Return-Path: <tiff-bounces@lists.osgeo.org>
Received: from mailgate201.isp.belgacom.be ([195.238.22.33])
          by mailfep016.skynet.be
          (InterMail vM.8.04.03.28.4 201-2389-100-182-103-20180124)
          with ESMTP
          id <20201231014832.FNRX7525.mailfep016.skynet.be@mailgate201.isp.belgacom.be>
          for <chris.lindelaan@skynet.be>; Thu, 31 Dec 2020 02:48:32 +0100
Authentication-Results: mx201.skynet.be; dkim=none (message not signed) header.i=none; spf=None smtp.pra=tiff-bounces@lists.osgeo.org; spf=Pass smtp.mailfrom=tiff-bounces@lists.osgeo.org; spf=Pass smtp.helo=postmaster@lists.osgeo.org
Received-SPF: None (mx201.skynet.be: no sender authenticity
  information available from domain of
  tiff-bounces@lists.osgeo.org) identity=pra;
  client-ip=140.211.15.3; receiver=mx201.skynet.be;
  envelope-from="tiff-bounces@lists.osgeo.org";
  x-sender="tiff-bounces@lists.osgeo.org";
  x-conformance=sidf_compatible
Received-SPF: Pass (mx201.skynet.be: domain of
  tiff-bounces@lists.osgeo.org designates 140.211.15.3 as
  permitted sender) identity=mailfrom; client-ip=140.211.15.3;
  receiver=mx201.skynet.be;
  envelope-from="tiff-bounces@lists.osgeo.org";
  x-sender="tiff-bounces@lists.osgeo.org";
  x-conformance=sidf_compatible; x-record-type="v=spf1";
  x-record-text="v=spf1 mx a:mail.osgeo.org a:lists.osgeo.org
  a:mail.osgeo.osuosl.org a:tracsvn.osgeo.osuosl.org -all"
Received-SPF: Pass (mx201.skynet.be: domain of
  postmaster@lists.osgeo.org designates 140.211.15.3 as
  permitted sender) identity=helo; client-ip=140.211.15.3;
  receiver=mx201.skynet.be;
  envelope-from="tiff-bounces@lists.osgeo.org";
  x-sender="postmaster@lists.osgeo.org";
  x-conformance=sidf_compatible; x-record-type="v=spf1";
  x-record-text="v=spf1 mx a:mail.osgeo.org a:lists.osgeo.org
  a:mail.osgeo.osuosl.org a:tracsvn.osgeo.osuosl.org -all"
X-Ironport-Dmarc-Check-Result: validskip
IronPort-SDR: 619Fb7JgCn7d+D9FbST+VhFMF3yeQCv/hSQ6qG87+Fw2cJIx1o7OLgiPZFT7qgTPmHllB25akd
 WqOFIcXhUOwtUhnZqMmfTXasxbLJdSrGRE5ypO6AB1o3ifM51cMioNg/paOt6t8NiGzCIdqRfx
 ucMtonBaxT5QGsf5UBVAovJH2DEk08hpgH3Nf6E+4GrmHEg1LcnSli8qjrIZ7H0UuNnPAGqP3S
 XeJlQJbgvH8G4oKxWaavhPue7iMHdrKQ+UeMlrZkYQg40uZ/I3rkK59slqsSjO7cgcvNt59DPo
 G2cPTxEvU8PN2pBfdlL7GRx/
IronPort-PHdr: =?us-ascii?q?9a23=3AwNgKERLJXJbmf2W3oNmcpTZWNBhigK39O0sv0r?=
 =?us-ascii?q?FitYgXKPv+rarrMEGX3/hxlliBBdydt6sbzbCM7+u7CSQp2tWoiDg6aptCVh?=
 =?us-ascii?q?sI2409vjcLJ4q7M3D9N+PgdCcgHc5PBxdP9nC/NlVJSo6lPwWB6nK94iQPFR?=
 =?us-ascii?q?rhKAF7Ovr6GpLIj8Swyuu+54Dfbx9HiTagY75+Ngu6oRveu8UZj4ZvKbs6xw?=
 =?us-ascii?q?fUrHdPZ+lY335jK0iJnxb76Mew/Zpj/DpVtvk86cNOUrj0crohQ7BAAzsoL2?=
 =?us-ascii?q?465MvwtRneVgSP/WcTUn8XkhVTHQfI6gzxU4rrvSv7sup93zSaPdHzQLspVz?=
 =?us-ascii?q?mu87tnRRn1gyocKTU37H/YhdBxjKJDoRKuuRp/w5LPYIqIMPZyZ77Rcc8GSW?=
 =?us-ascii?q?ZEWMtaSi5PDZ6mb4YXAeQPPfhWoIv+qVsAoxSxChSiBPvtxzFKm3P707E23e?=
 =?us-ascii?q?s9Hg7YxwwtBM4OvXbOodnpKKsfS/65wa/VxjvDdfNW3jL95ZDVfxAmuv6HQL?=
 =?us-ascii?q?J+cdDQyUYxDAPKkFOQqYz/MDOV0OQGrm+W4PFuVeK1im4otRpxrSapxso3kY?=
 =?us-ascii?q?nJh4cVykjA9Spn2oo6OMe3SEp/YdG+C5tcrT2aNoRsTsM/WGFloSA3waAJtp?=
 =?us-ascii?q?CnZiYF0ognxwLBZPyddYiF+g7uWeeNLTp6inxpZrGyihe9/EW8zuDwStS53V?=
 =?us-ascii?q?dJoydFj9TBq24B2gLN5sWaRfZw4kes1CuT2g7T9+1JIE46mLfdJpU8zLAwkZ?=
 =?us-ascii?q?8Tvl7CHi/wgEj2l66WdkA8+uis9evnebXnpoSAN4BqjwH+Kb4imsulAeQ3Kg?=
 =?us-ascii?q?QOUHKb9v6k1L3j/E32XK5KjvgsnanWsZDVOccbqrS+Aw9P3YYv8xG/Dyyp0N?=
 =?us-ascii?q?sGmnkLNlVFeAiIj4XvJlrAI+j4AO+ij1mjjDtn3e7KMqD8DpnXM3TOkK3tcL?=
 =?us-ascii?q?hl50NdyAc/19FS6pNOBr0fL//+W1X9ud7WAxI/Lge62fzoCM9n2YMbQW+PAr?=
 =?us-ascii?q?GWMKfVsVKQ4uIvIvKMZJMOuDrnLPgl/PHugmE+mVMHeqmpx5QXZGi5HvRgPU?=
 =?us-ascii?q?qZfWLhjsoOHGsWpAYyUvHmhECYXTJOZ3u+Rawx6ik+CI28DIfDQo6tgKaG3C?=
 =?us-ascii?q?e+BpBWemFGCl6CEXvxbIqEXOoMaCSOLcB/iTwJTryhS4461RGyrw/20b1nLv?=
 =?us-ascii?q?Hb+iIGspLvzt915/fclR0q7zx7E9yd032RT2Fzhm4HWz823Lp+oUxm0FuD1a?=
 =?us-ascii?q?l4g/pFGtNN4/NGSB01NZjHwuxhCNDzWgTBccmVSFq8WdqpH2J5c9Qq3tVbY1?=
 =?us-ascii?q?phA87wyVfH3jG2GPkSjLOGA9oz6K2b2nHwI8N0zTHBzLUgiF89BddOO2P1u6?=
 =?us-ascii?q?hk6gKGAofIl1mexbqre6pZxiPU6WqY0UKKvVpES0h3SqrIWjYYfESBtsnz5E?=
 =?us-ascii?q?7JU+qzD686OBBK08+IJ/h3bMb0h3VLTfvuIoHef36pgDX3QhKJ3a+XKo3xdG?=
 =?us-ascii?q?gamiLHBw8BmgEX+H+Acg8mGiampXmZED1gGwHSZVjx+8lzoX+2VBoxxgCOc0?=
 =?us-ascii?q?ozy7u79FsPiOWBRugP9rYFozs67TJvE1u6mdXMBI2bugBjca5ANM47+0pNzm?=
 =?us-ascii?q?nDtgZwbaCneoRrnEQPOyttvl/8n0FzBa1EmNIwvXwwzRZgb6mf1QUSWSmf2M?=
 =?us-ascii?q?XaPbefBHPz8Quiba7f2leWhNyY4KYC8/k+g07iuRmtGwwu+nFj2sITz2va+4?=
 =?us-ascii?q?6cX1laao74Tktir0synLrdeCRootqMjSc2Y4Cprj/P3c4oD+I5yxGmOu1SK7?=
 =?us-ascii?q?6ACBSoS5xIAtj0cLVsgFu4aB8Yeupfp+Y4PMKjIv2B3qPjfONthymvgmkP5o?=
 =?us-ascii?q?dhmlmN+CxxRq+A35sMz/yCmArSUTD6gQTpqdj5zKZDYzxaBW+j0W78HocEeq?=
 =?us-ascii?q?x8c8MBDnqoKMCs7tF/nIL2HXlD+1OkQVQc15zhYgKcOmT0xhYYzkELuTqikC?=
 =?us-ascii?q?q8wSZzlmQ1r6SRmSbJ3uLpeQEvPm9QWHIkikzjLI/yiMoVDwCzdwZ8rBK+/g?=
 =?us-ascii?q?7hwrRD4qRyK26GWUBTYy3/NH1vSIO0t6GeeIhI85wltmNQSuvlOwKgR7Xwog?=
 =?us-ascii?q?UX32bYJ0UFnWlpUTast925mgwg0DvbN3ViqnfFP8t9l1/Z45THSPhd0yBOTT?=
 =?us-ascii?q?RkiTTRGlm3PsW4mLfc35bFvOekUmu9V5pVOSD1xIKEvSG/6CVkGxq61/y0n9?=
 =?us-ascii?q?TmF0A930qZn5F2VCvG6Rz7f4/g2r+SNeN8YlIuA0Tz5s08HZtx08MxiJwWxX?=
 =?us-ascii?q?kGl8CN53NU9AW7ed5f2K/4cD8MXWtSn4aTulG8nhQ+aCnaj5j0XXic3MZ7Mt?=
 =?us-ascii?q?yqODtPnDk+/sZHFOGe4e8h/2M9r1ymoAbWefU4kC0azK5k9H8UjaQDvxcrxS?=
 =?us-ascii?q?iGKrQTBlVDeyv3mBmPqdeko+8EAQTnOaj1z0d4kd27Wfueow9SHXz+Z5EpED?=
 =?us-ascii?q?RY6sRkLEmK12b854aif8PfJ4F21FXcg1LLiO5bL4g0n/wBiH98OG7zinYizv?=
 =?us-ascii?q?YykR1k2Zzj5tq9Jm5g/by0DltjDhOpOpNBwjj2lu4em9bMh9npBZZwHDgXGp?=
 =?us-ascii?q?HlH7qkFzcbtPKvMAjrcnV0tnaVF//VGhKZ40p9h3bOCIywcXaMK3Qei9J4S1?=
 =?us-ascii?q?GRKVdejwYdQDgh1sdlTkbwnZWnKRo/vXgY/Tua4lNUx/htNgXjX2uXvwquZj?=
 =?us-ascii?q?ouCdCeIBdQ8gBe9hLQOM2a4Ph0GnIQ9ZmgoQqRb22DMl0UVSdTAxXCXg2lZe?=
 =?us-ascii?q?rLh5GI6eWTC+ugIuGbZKXU8bUYTPSZw5+zlIFrrFPufo2COGdvC/oj1w9NR3?=
 =?us-ascii?q?d8To7FmjsBDiYQiiTGYtWzoBan5jYxqduw/fOtXxjgr9jqafMaIZB09ha6jL?=
 =?us-ascii?q?3WffKRiig8IzFF0JgB3lfMyaMDxxgVkC5jcH+qC7tK5kuvBOrA36RQCRAccS?=
 =?us-ascii?q?Z6MsBFurk90gd6Msnektrp17R8g619GxJfWFfmgM3sed0SLjT3KgbcHEjSfu?=
 =?us-ascii?q?fjR3WD04TtbKi7U7EVkOhEq0j6p2OAC0G6djGT3zC7UhmuNadLly6WO1pZoo?=
 =?us-ascii?q?71e0RpCHP/V5ThcRK2N5lwlzJTo/V8jyHPPG8Ydz9nfE5K6LqL8CRDhrN4AW?=
 =?us-ascii?q?MJ5yh5N+CPgDifqe7DJZsR9/xwHyNvk6Rd+ntyjrJR6GsdHq5PlSDfr8Bjrx?=
 =?us-ascii?q?SdqsfTl2I1aBNVsX4LgZLR5RwkIqbI8JRcH3zJrloB6mCZChJCoNUAaJWno6?=
 =?us-ascii?q?dL1t3Gj77+Mh9G8snI5o0bHMfZL4SAKnVpPRfyGTHSBRcIVnbyaTmZ3RUE1q?=
 =?us-ascii?q?jPsCbdp4NfyNCkgJcUT75HSFE5XugXDEhoBp1KIZt6WC8lja/OjMMM4in2px?=
 =?us-ascii?q?3QScNG+5HfA6vOWbO1cW/f1OQCPEhto/uwN4kYO4zl1lY3b0kgxN2MAEnBW9?=
 =?us-ascii?q?1R5CdsPWpW6A1A9mZzSmor1gfrcASotTUPFPC62xQxlw11bP8F8Db3/0xxIU?=
 =?us-ascii?q?HFqSx2l1M+04aA43jZYHvqIaG8UJsDQTLzrFQ0O4jnThxdaQypgVciMyzLSr?=
 =?us-ascii?q?UXiKFvPzMO6keUqd5EHvhSSrdBaRkbyKSMZvkm5l9brz2u2U5N4eaWQYsnjg?=
 =?us-ascii?q?YhdoSg6m5Rww82JsBgPrTefeAaqzoYzrLLpCKj0fo9hRMTN1pYun3HYzYG4Q?=
 =?us-ascii?q?QNJvEvdS+h/+gq7heNljwFf3IDH/N4pvNx6lh7PP+PwiWm1KRMTyL5f+XNKq?=
 =?us-ascii?q?qYvy7HjsGFRRU3zV4PjE0D+qJ5mc5xb1CdXFsxyv2WDBsEMozFMhxSdM4U82?=
 =?us-ascii?q?LcNWOHteGHmM4nD8CGDuntCNS2muMMmEv9Q1QtHp8Q9YIGBZah1wffN8i1dO?=
 =?us-ascii?q?dUmyVo3xziIRC+NNoMfRuKlDkdpMTmk81z3JJBPXcaGmpwNmOw/LmF/1Z30s?=
 =?us-ascii?q?rGZ88/ZzIhZqVBNn8yX5TlySpYpGhbSj6s3e8ejg+Y4G2lqw=3D=3D?=
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: =?us-ascii?q?A0GxAQD6LO1fhwMP04xiHgEBCxIMQIN?=
 =?us-ascii?q?zGAFkWQEeAwc2iAqFOYgHJQOKKZIDFgEBAgEBAQEBCRgLCgIEAQEBAYY7HwY?=
 =?us-ascii?q?BBDQTAgMBAQEDAgUBAQEEAQEBAgEBAgMCAgEBAhABAQEBDQsJCCksBlpmBGI?=
 =?us-ascii?q?GWwSBbAwzDII4BQIDHwEGbYEGAQEBAQEBAQEBAQEBAQEBAQEBAQEWAg1UaQE?=
 =?us-ascii?q?CAgEBAQE3DAoeCwMCAQECBgEBCg4oBAQIAwEjAS8GEwUEEoMLAYJhBSAFCq9?=
 =?us-ascii?q?XgTSEB4UJgUKBOI0pQYFBP4ERMwOCYz6CUgsBAwGHVASCcF8BKgFUByAVRwo?=
 =?us-ascii?q?hGyYtAo8yjCJXm1GCD3GJKoZ4hEKHFoMpj0o+jx+fG5FvhF2BbYF7gQIDO4J?=
 =?us-ascii?q?pCUQBAgECDQIBAgMBAgICAQECAgEBAo4qF4hihWMjATECNQIGCgEBAwlbiwm?=
 =?us-ascii?q?BbQEB?=
X-IPAS-Result: =?us-ascii?q?A0GxAQD6LO1fhwMP04xiHgEBCxIMQINzGAFkWQEeAwc2i?=
 =?us-ascii?q?AqFOYgHJQOKKZIDFgEBAgEBAQEBCRgLCgIEAQEBAYY7HwYBBDQTAgMBAQEDA?=
 =?us-ascii?q?gUBAQEEAQEBAgEBAgMCAgEBAhABAQEBDQsJCCksBlpmBGIGWwSBbAwzDII4B?=
 =?us-ascii?q?QIDHwEGbYEGAQEBAQEBAQEBAQEBAQEBAQEBAQEWAg1UaQECAgEBAQE3DAoeC?=
 =?us-ascii?q?wMCAQECBgEBCg4oBAQIAwEjAS8GEwUEEoMLAYJhBSAFCq9XgTSEB4UJgUKBO?=
 =?us-ascii?q?I0pQYFBP4ERMwOCYz6CUgsBAwGHVASCcF8BKgFUByAVRwohGyYtAo8yjCJXm?=
 =?us-ascii?q?1GCD3GJKoZ4hEKHFoMpj0o+jx+fG5FvhF2BbYF7gQIDO4JpCUQBAgECDQIBA?=
 =?us-ascii?q?gMBAgICAQECAgEBAo4qF4hihWMjATECNQIGCgEBAwlbiwmBbQEB?=
X-MGA-submission: =?us-ascii?q?MDGZfc0huMdFbD2hX81JNoYZBmtXRtodZVF5gr?=
 =?us-ascii?q?ceEz6WLevZiA+Of8eNHDJUoFJRyxabLd02cTnEotX7kRx81A70I/rMoX?=
 =?us-ascii?q?o5jZ1QY+W58Flik0x3HL0TtyI16i88/xjdwV8=3D?=
Received: from osgeo6.osgeo.osuosl.org (HELO lists.osgeo.org) ([140.211.15.3])
  by mx201.skynet.be with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 31 Dec 2020 02:48:29 +0100
Received: from osgeo6.osgeo.osuosl.org (localhost [127.0.0.1])
	by lists.osgeo.org (Postfix) with ESMTP id 5333C6143EAE;
	Wed, 30 Dec 2020 17:48:23 -0800 (PST)
Received: from smtp.simplesystems.org (smtp.simplesystems.org [65.66.246.90])
 by lists.osgeo.org (Postfix) with ESMTPS id C8FD46146626
 for <tiff@lists.osgeo.org>; Wed, 30 Dec 2020 17:48:21 -0800 (PST)
Received: from scrappy.simplesystems.org (scrappy.simplesystems.org
 [65.66.246.73])
 by smtp.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id 0BV1mHUv004453;
 Wed, 30 Dec 2020 19:48:17 -0600 (CST)
Date: Wed, 30 Dec 2020 19:48:17 -0600 (CST)
From: Bob Friesenhahn <bfriesen@simple.dallas.tx.us>
X-X-Sender: bfriesen@scrappy.simplesystems.org
To: Larry Gritz <lg@larrygritz.com>
cc: tiff@lists.osgeo.org
In-Reply-To: <20DAEBC0-D179-4C65-A807-B989425738F9@larrygritz.com>
Message-ID: <alpine.GSO.2.20.2012301932030.20407@scrappy.simplesystems.org>
References: <mailman.25.1609358402.3913.tiff@lists.osgeo.org>
 <20DAEBC0-D179-4C65-A807-B989425738F9@larrygritz.com>
User-Agent: Alpine 2.20 (GSO 67 2015-01-07)
MIME-Version: 1.0
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16
 (smtp.simplesystems.org [65.66.246.90]); Wed, 30 Dec 2020 19:48:17 -0600 (CST)
Subject: Re: [Tiff] Multithreading support?
X-BeenThere: tiff@lists.osgeo.org
X-Mailman-Version: 2.1.34
Precedence: list
List-Id: TIFF and libtiff discussion <tiff.lists.osgeo.org>
List-Unsubscribe: <https://lists.osgeo.org/mailman/options/tiff>,
 <mailto:tiff-request@lists.osgeo.org?subject=unsubscribe>
List-Post: <mailto:tiff@lists.osgeo.org>
List-Help: <mailto:tiff-request@lists.osgeo.org?subject=help>
List-Subscribe: <https://lists.osgeo.org/mailman/listinfo/tiff>,
 <mailto:tiff-request@lists.osgeo.org?subject=subscribe>
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="us-ascii"; Format="flowed"
Errors-To: tiff-bounces@lists.osgeo.org
Sender: "Tiff" <tiff-bounces@lists.osgeo.org>

On Wed, 30 Dec 2020, Larry Gritz wrote:

> But it would dramatically improve libtiff to have any or all of the following:
>
> * Add API calls for reading/writing a range of scanlines or tiles at a time.
> * Eschew any "statefulness" between calls -- i.e. a single call should be able to specify begin/end scanline, and the TIFF directory within the file to read from (so there is not hidden state between TIFFSetDirectory and TIFFReadScanline that currently requires a lock on the application side).
> * Reentrancy/thread safety of multiple read calls happening simultaneously on the same open TIFF handle.
> * Ideally, using an internal thread pool (or at least the ability to optionally use TBB or an external thread pool passed in, etc.) so that libtiff could parallelize everything but the raw I/O that has to be serialized. Or, alternately, ...
> * Exposing through the libtiff API a simple thread-safe "decompress this raw strip or tile that I already read" and "compress this pixel buffer to a raw strip buffer" that implements all of the codecs supported, thus allowing apps to split the raw read/write from the [de]compression, so they can more easily use their own threading to parallelize at least the [de]compression.
>
> Many applications could see an order of magnitude improvement in the I/O performance for TIFF files.

This sounds great to me, as long as it can be layered in without 
impacting existing usages.  Existing apps need to continue to compile 
and run and without any background cycles consumed that they did not 
request.  I expect that there would need to be a lot of internal 
changes.

Things like an internal thread pool would never be something that 
libtiff would itself decide to instantiate (given that the usage you 
describe is not the norm), but it could provide APIs which allow a 
using app to side-load a thread pool.

Others have said that support for async-I/O (offloading several I/O 
requests to the kernel and being notified when data is available) is 
the solution to perceived problems, but async-I/O is not as uniformly 
supported as threads are.  Async-I/O does not provide any assistance 
for compression/decompression requirements.

Libtiff is on GitLab now and the common way things work is that 
someone does the work, pushes a branch, and then submits a merge 
request.  Sometimes branches stay open/pending for a long time as 
development proceeds.  Are you (and others) ready and willing to do 
most of the work?

Bob
-- 
Bob Friesenhahn
bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
Public Key,     http://www.simplesystems.org/users/bfriesen/public-key.txt
_______________________________________________
Tiff mailing list
Tiff@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/tiff
