Return-Path: <awaresystems.be+caf_=filtered=awaresystems.be@gmail.com>
Delivered-To: filtered@awaresystems.be
Received: from mx1.pub.mailpod2-cph3.one.com ([10.27.25.11])
	by mailstorage18.cst.mailpod2-cph3.one.com (Dovecot) with LMTP id xLGzDyybXVqHqAAAQS8vrQ
	for <filtered@awaresystems.be>; Tue, 16 Jan 2018 07:34:07 +0000
X-HalOne-ID: a1949fad-fa8f-11e7-afe6-e0d848957041
Received: from mail-ot0-f179.google.com (unknown [74.125.82.179])
	by mx1.pub.mailpod2-cph3.one.com (Halon) with ESMTPS
	id a1949fad-fa8f-11e7-afe6-e0d848957041;
	Tue, 16 Jan 2018 07:34:06 +0000 (UTC)
Received: by mail-ot0-f179.google.com with SMTP id p36so4752217otd.10
        for <filtered@awaresystems.be>; Mon, 15 Jan 2018 23:34:06 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-original-authentication-results:x-gm-message-state:delivered-to
         :message-id:date:from:user-agent:mime-version:to:references
         :in-reply-to:subject:precedence:list-id:list-unsubscribe
         :list-archive:list-post:list-help:list-subscribe
         :content-transfer-encoding:sender:errors-to;
        bh=HDuWSR0Dw7KaoaCYGJ1POKyUvoUAPurd5ID7Wxu5PaA=;
        b=t59PQVKtCs04FDSvfFcqJC1xbaiD2g4JEymGJP0c2UhekrrP8k2WK/NPJVSH0rXaPr
         LGMut/ScBPxVkaa4qfCQXuV6o/50t4kwEzP5zU9r+qQVxFqJe6z34rbenYKPGSinE8zH
         w8oS4NodY0nM94FWZxaFsfx81kRATRgIEGLu2j5LBF1hTXPrAJj/TgMICui0n0jUXXMt
         5HRNdsIb/3xyET7UrkMQkqxY3DHkiIMJwFfnD5cO/pL3bv+kt/IXoU3a/IASHQVHkNcz
         0n69dkyO1IFC7KS5crJ8e27+2YKdfcmjx0AvY3zjktCAxHYRyuYPgHrQU7KfhJ/PAVEp
         k5tQ==
X-Original-Authentication-Results: mx.google.com;       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org
X-Gm-Message-State: AKwxytdwb8KLU73/20QhABiJKdAFbrJ1XLgfEcFLbBrKUPB8MOLnmAcz
	VF6ncKAS8fCiVHTBkwPbeqKyvB8QfzIr89qXLWKTcuvLyAqa8DI=
X-Received: by 10.157.25.197 with SMTP id k63mr20660900otk.204.1516088045236;
        Mon, 15 Jan 2018 23:34:05 -0800 (PST)
X-Forwarded-To: filtered@awaresystems.be
X-Forwarded-For: awaresystems.be@gmail.com filtered@awaresystems.be
Delivered-To: awaresystems.be@gmail.com
Received: by 10.202.235.11 with SMTP id j11csp3835915oih;
        Mon, 15 Jan 2018 23:34:03 -0800 (PST)
X-Google-Smtp-Source: ACJfBosqBJVrWyGfn/U969/npJ4Nu+NthbUMKqtZMZA9jHW2T6bI9QY85lnKADiTMGzxrq9NCAv/
X-Received: by 10.223.188.80 with SMTP id a16mr17948335wrh.20.1516088043859;
        Mon, 15 Jan 2018 23:34:03 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; t=1516088043; cv=none;
        d=google.com; s=arc-20160816;
        b=fkGg6XhTXdBNSbA7CumVye7V7CoFp8FvTA14qMKqj8I6Yv5fdCyDfpyKKVk5aPQdVp
         MUz9rtL878v6PhLKCuNtaBy6/zLgriwEliBFW4xmrlq8OEzcBHytj29RKJ7QR1oIFVQj
         u9YMRRPDOeKalRcGQhkQ9KxJLTKH7pOHwtoRv8mc7JsICsCh9qr/xJMRtgiYWvfmlmAU
         UjKv++u+vQOORNbzCR23zT4TSyOtzobVrVaopb6Tkb7L6jQKcLLhe4WifFRkNvm41/fm
         riDp1J5G9o5oAU4xfKZrAtK/Z3PYrVBSmKQtGbwxcvP5Aj1K1CYOAli2Z9Uj95qUkpJ7
         rKHQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
        h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help
         :list-post:list-archive:list-unsubscribe:list-id:precedence:subject
         :in-reply-to:references:to:mime-version:user-agent:from:date
         :message-id:arc-authentication-results;
        bh=HDuWSR0Dw7KaoaCYGJ1POKyUvoUAPurd5ID7Wxu5PaA=;
        b=XIERKL8eyWc4Y0F/QweoE5DRbAtVXQ4OqKMH5jVWGGqK4hxgPDbCIfWm/6q1OxmDoU
         fF+El86kTqATBhsgzdzvIg1WNFT07QICLWQBCAQeSnIcYgy/prJ+MXyWGYw4hj4Mq75k
         AUlX3RNy4dirGO7WtwEEVKnCR4pM5P2Ic6mGSYUd2PsVVvt+1j5Ry3hEAQSwUC/ajO34
         TnPZU7Ja2W3AxMfGpIK5LDxJklTFR5MXaY7v9Hs9u7XTMB5Zb+x/3fluMaxgBHSdf9Ud
         BzgHXFYoryxf3btnedV4vZgldflrdrXYz7D337nf5/wycJ13P8p2W58WNgYWo7W/5Oem
         k1vw==
ARC-Authentication-Results: i=1; mx.google.com;
       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org
Return-Path: <tiff-bounces@lists.maptools.org>
Received: from lists.maptools.org (ks365428.kimsufi.com. [94.23.4.142])
        by mx.google.com with ESMTP id k14si1199786wrf.484.2018.01.15.23.34.03;
        Mon, 15 Jan 2018 23:34:03 -0800 (PST)
Received-SPF: neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) client-ip=94.23.4.142;
Authentication-Results: mx.google.com;
       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org
Received: from [10.1.0.241] (localhost.localdomain [127.0.0.1])
	by lists.maptools.org (Postfix) with ESMTP id 3B34035C094;
	Tue, 16 Jan 2018 02:33:01 -0500 (EST)
Received: from polaris.osa.pri.ee (osa.pri.ee [213.219.84.188])
	by lists.maptools.org (Postfix) with ESMTP id 53F3635C092
	for <tiff@lists.maptools.org>; Tue, 16 Jan 2018 02:32:48 -0500 (EST)
Received: from localhost (localhost [127.0.0.1])
	by polaris.osa.pri.ee (Postfix) with ESMTP id 4BF766E69D
	for <tiff@lists.maptools.org>; Tue, 16 Jan 2018 09:32:48 +0200 (EET)
X-Virus-Scanned: amavisd-new at osa.pri.ee
Received: from polaris.osa.pri.ee ([127.0.0.1])
	by localhost (osa.pri.ee [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id hK++WGzk+eV8 for <tiff@lists.maptools.org>;
	Tue, 16 Jan 2018 09:31:56 +0200 (EET)
Received: from [192.168.1.208] (134-171-190-90.dyn.estpak.ee [90.190.171.134])
	(using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(Client did not present a certificate)
	by polaris.osa.pri.ee (Postfix) with ESMTPSA id 6A87D6E669
	for <tiff@lists.maptools.org>; Tue, 16 Jan 2018 09:31:56 +0200 (EET)
Message-ID: <5A5DAA6A.6090402@osa.pri.ee>
Date: Tue, 16 Jan 2018 09:31:54 +0200
From: Paavo Helde <paavo@osa.pri.ee>
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64;
	rv:25.8) Gecko/20151117 FossaMail/25.1.9
MIME-Version: 1.0
To: tiff@lists.maptools.org
References: <39EE49FE-23A5-43DE-8998-51526287D182@larrygritz.com>	<alpine.GSO.2.20.1801151400550.18206@scrappy.simplesystems.org>	<4252FC16-C07D-4FA1-8B7B-25D626633236@larrygritz.com>	<F78FB533EE93A044B560CD25E17A4DAF18388904@mbx031-e1-va-4.exch031.domain.local>	<D4C4B1FE-4C28-4E7C-964F-750BBCF87DAC@larrygritz.com>	<alpine.GSO.2.20.1801151816490.18206@scrappy.simplesystems.org>
	<E3AA6EDE-85B1-4398-98BF-B91781818EBF@killerbytes.com>
In-Reply-To: <E3AA6EDE-85B1-4398-98BF-B91781818EBF@killerbytes.com>
Subject: Re: [Tiff] Strategies for multi-core speedups
X-BeenThere: tiff@lists.maptools.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: TIFF and libtiff discussion <tiff.lists.maptools.org>
List-Unsubscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=unsubscribe>
List-Archive: <http://lists.maptools.org/mailman/private/tiff>
List-Post: <mailto:tiff@lists.maptools.org>
List-Help: <mailto:tiff-request@lists.maptools.org?subject=help>
List-Subscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=subscribe>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Sender: tiff-bounces@lists.maptools.org
Errors-To: tiff-bounces@lists.maptools.org


On 16.01.2018 2:58, scott ribe wrote:
> On Jan 15, 2018, at 5:24 PM, Bob Friesenhahn <bfriesen@simple.dallas.tx.us> wrote:
>> The I/O is not necessarily serialized.  When using mmap, pread/pwrite,
>> or asynchronous I/O, the I/O can also benefit from multi-threading
>> given that the underlying storage is a multi-disk array or SSDs using
>> PCIe NVMe (capable of up to 1024 simultaneous I/Os).
> Also, I/O calls are likely only to RAM, not to disk, thanks to caching...
>
With TIFFClientOpen() it is pretty easy to set up one's own I/O so that 
everything happens in RAM definitely, not only likely. All our libtiff 
reading happens via memory, either mmap-ped or obtained otherwise.

That does not mean RAM is always super fast. If the files are large 
enough and there are many frames then traversing them takes a lot of 
time because they do not fit in L3 caches. The main culprit in libtiff 
is TIFFSetDirectory() which always traverses the whole file. This 
becomes pretty slow when you have something like 500 frames. In our app 
we are trying to cache the known IFD offsets externally and (ab)using 
TIFFSetSubDirectory() for frame hopping instead.

Cheers
Paavo

_______________________________________________
Tiff mailing list: Tiff@lists.maptools.org
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
