Return-Path: <awaresystems.be+caf_=filtered=awaresystems.be@gmail.com>
Delivered-To: filtered@awaresystems.be
Received: from mx2.pub.mailpod2-cph3.one.com ([10.27.25.12])
	by mailstorage18.cst.mailpod2-cph3.one.com (Dovecot) with LMTP id 5vhKMB4EXVpqDAEAQS8vrQ
	for <filtered@awaresystems.be>; Mon, 15 Jan 2018 20:26:15 +0000
X-HalOne-ID: 54602f85-fa32-11e7-b104-ec0d9a6ed9fe
Received: from mail-ot0-f173.google.com (unknown [74.125.82.173])
	by mx2.pub.mailpod2-cph3.one.com (Halon) with ESMTPS
	id 54602f85-fa32-11e7-b104-ec0d9a6ed9fe;
	Mon, 15 Jan 2018 20:26:13 +0000 (UTC)
Received: by mail-ot0-f173.google.com with SMTP id v5so1354848oth.5
        for <filtered@awaresystems.be>; Mon, 15 Jan 2018 12:26:13 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-original-authentication-results:x-gm-message-state:delivered-to
         :date:from:to:in-reply-to:message-id:references:user-agent
         :mime-version:cc:subject:precedence:list-id:list-unsubscribe
         :list-archive:list-post:list-help:list-subscribe
         :content-transfer-encoding:sender:errors-to;
        bh=IDPy0tVKk7+bxDKitDueGqEOVg7jRe1P/p18K3JZXqM=;
        b=XgnTtGmeze1LMCISvzF/+nmUvdHb0SfmwsSaIjXjXcMG8o0Wy6pQPAow3gj41p/kbU
         NSa4Yk6dprEHlNS/4ZaBqWKzQudL/1mMnIdo2dtoS2FbsesbrSIVETZMlGNI5GN0xVj1
         4h9UbpNxYTe5oJeYUdYwdISgMurvZ/S5yO/1UsGxgbaAXL3kKuluLn0zJ4hyKr2sOPCv
         1kFm3CZdGEFmYCqSI66dXjT+l8dMNXM6Vby/cfS5cXnjpDh1u2J0PFfv6Gga1dweiToM
         9ij5gWbyoz3H+7u0TCwZVhTXcar2R+DVJ5LSI4nn33lExYubwv9nAtHyHyAKM/NgLIcy
         C92g==
X-Original-Authentication-Results: mx.google.com;       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org
X-Gm-Message-State: AKwxytfIV5z8AvivsXUVBoveds3kl7QBR+tDGvxNbdkB1S0QBVBQJO7i
	qXW2abzvvXWtyNHThAFF9lawVQs232ISHw64DNDx+r/h66V0lw8=
X-Received: by 10.157.49.44 with SMTP id e41mr11884709otc.68.1516047972464;
        Mon, 15 Jan 2018 12:26:12 -0800 (PST)
X-Forwarded-To: filtered@awaresystems.be
X-Forwarded-For: awaresystems.be@gmail.com filtered@awaresystems.be
Delivered-To: awaresystems.be@gmail.com
Received: by 10.202.235.11 with SMTP id j11csp3299932oih;
        Mon, 15 Jan 2018 12:26:11 -0800 (PST)
X-Google-Smtp-Source: ACJfBovj9HhCORp7E53iZrnYLI0m4/t/MtIVSYzW07BIwZZVHD0wMnNNwOttWjLJS4m9LDUr0L9e
X-Received: by 10.28.17.76 with SMTP id 73mr12140870wmr.104.1516047970888;
        Mon, 15 Jan 2018 12:26:10 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; t=1516047970; cv=none;
        d=google.com; s=arc-20160816;
        b=C69PfhL2/xZGBBE4HcqQOhU8n0Ru/vaIV+3Zlrn9qRJDYc2WKuAwqK5UtXybXjyL7x
         cCLmFpbSjtWERIjU2JqMfrvLbqJX9VRfekLug2cYizg/r66RegirX9rn+v00OABVt/U7
         oiSyD7ctcd+eGDJiL4zZbb3swqlBEL+M7wal9jqxS5pXm9ls2Hf+mH27PmwIvA0NMOIO
         5iJ99n91FDqkn5bE3ShwFl7/UUjZu96AVzldVaTlsNH4Cl10GJfZZwTjMcSjxfLrX+QZ
         TEpbKiyLSoh2ST/7G+TfwCVr1uQEDrt6zUot5V2niC75v45yraFop0yL9IeYRw9cXqaG
         PAkg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
        h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help
         :list-post:list-archive:list-unsubscribe:list-id:precedence:subject
         :cc:mime-version:user-agent:references:message-id:in-reply-to:to
         :from:date:arc-authentication-results;
        bh=IDPy0tVKk7+bxDKitDueGqEOVg7jRe1P/p18K3JZXqM=;
        b=WUmFIVr8MOUKTVw+Qi1LJmmhGKhbfdN0PLMEj/sAujhspdnUktf7qD6neBVq6zqAnS
         EZjnkyOY0do34lIYbLE8KGt1ZcTJQ20/KBEHIjGPS+gpTk6xlEHfQR33aGY0CTv8Ik+N
         zCQCoQuDlC7VBK/4LcEneUq7DZRHIoXaz8zYUCnsBHB9EcdtRnadh+wo4PQNgWDFLopB
         4USzoUEU2KO1kiQrqBNW3QljAu0PAYjqKO01gm18v4uuIRT7OfB3N7wXdKEkVP0ZwXPX
         puBa3uz+Ua/CKLy6pFCahSFfrRrTpt8cB0yUQ0vQpbICXPNb0qMNBHcXfcawG/A9NDpj
         90dA==
ARC-Authentication-Results: i=1; mx.google.com;
       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org
Return-Path: <tiff-bounces@lists.maptools.org>
Received: from lists.maptools.org (ks365428.kimsufi.com. [94.23.4.142])
        by mx.google.com with ESMTP id 3si357827wrv.436.2018.01.15.12.26.09;
        Mon, 15 Jan 2018 12:26:10 -0800 (PST)
Received-SPF: neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) client-ip=94.23.4.142;
Authentication-Results: mx.google.com;
       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org
Received: from [10.1.0.241] (localhost.localdomain [127.0.0.1])
	by lists.maptools.org (Postfix) with ESMTP id 5C1AF35C03B;
	Mon, 15 Jan 2018 15:25:11 -0500 (EST)
Received: from smtp.simplesystems.org (smtp.simplesystems.org [65.66.246.90])
	by lists.maptools.org (Postfix) with ESMTPS id AEDC535C035
	for <tiff@lists.maptools.org>; Mon, 15 Jan 2018 15:23:43 -0500 (EST)
Received: from scrappy.simplesystems.org (scrappy.simplesystems.org
	[65.66.246.73])
	by smtp.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id
	w0FKN7J8026003; Mon, 15 Jan 2018 14:23:07 -0600 (CST)
Date: Mon, 15 Jan 2018 14:23:07 -0600 (CST)
From: Bob Friesenhahn <bfriesen@simple.dallas.tx.us>
X-X-Sender: bfriesen@scrappy.simplesystems.org
To: Larry Gritz <lg@larrygritz.com>
In-Reply-To: <39EE49FE-23A5-43DE-8998-51526287D182@larrygritz.com>
Message-ID: <alpine.GSO.2.20.1801151400550.18206@scrappy.simplesystems.org>
References: <39EE49FE-23A5-43DE-8998-51526287D182@larrygritz.com>
User-Agent: Alpine 2.20 (GSO 67 2015-01-07)
MIME-Version: 1.0
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16
	(smtp.simplesystems.org [65.66.246.90]);
	Mon, 15 Jan 2018 14:23:07 -0600 (CST)
Cc: tiff@lists.maptools.org
Subject: Re: [Tiff] Strategies for multi-core speedups
X-BeenThere: tiff@lists.maptools.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: TIFF and libtiff discussion <tiff.lists.maptools.org>
List-Unsubscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=unsubscribe>
List-Archive: <http://lists.maptools.org/mailman/private/tiff>
List-Post: <mailto:tiff@lists.maptools.org>
List-Help: <mailto:tiff-request@lists.maptools.org?subject=help>
List-Subscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=subscribe>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Sender: tiff-bounces@lists.maptools.org
Errors-To: tiff-bounces@lists.maptools.org

On Mon, 15 Jan 2018, Larry Gritz wrote:
>
> I tried to do this myself using libtiff -- for the common case of 
> needing to read an entire image, my aim was read all the raw strips 
> (TIFFReadRawStrip, serially), then dole them out to different 
> threads to decompress in parallel (and when writing, compress in 
> parallel, then TIFFWriteRawStrip each one serially). If libIlmImf is 
> any indication, doing this on our typical 12 or 16 core machines 
> ought to speed up TIFF I/O by an order of magnitude, easily.

An order of magnitude is unlikely.  2-4x is possible.  The problem is 
that a computer has limited memory bandwidth.  With enough threading, 
the bottleneck moves from CPU to memory bandwidth and CPUs offer 2-4x 
memory-path parallelism (more on expensive server CPUs and less on 
desktop CPUs).  EXR uses floating point which requires a lot more CPU 
so there is more to be gained with many cores.

It is likely that even with sequential single-threaded I/O through 
libtiff that other parts of your app will benefit by threading.  I see 
up to 4x benefit when reading SMTPE DPX format sequentially (one 
thread doing I/O at a time), but then allowing each thread to continue 
processing the data in a multi-threaded fashion.

> But my plan was thwarted because it sure seems that the API for the 
> codecs is inherently stateful and non-reentrant.

This would be very useful to fix.

Bob
-- 
Bob Friesenhahn
bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
_______________________________________________
Tiff mailing list: Tiff@lists.maptools.org
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
