Return-Path: <awaresystems.be+caf_=filtered=awaresystems.be@gmail.com>
Delivered-To: filtered@awaresystems.be
Received: from mx2.pub.mailpod2-cph3.one.com ([10.27.25.12])
	by mailstorage18.cst.mailpod2-cph3.one.com (Dovecot) with LMTP id dkqRDVhVXlo2mQAAQS8vrQ
	for <filtered@awaresystems.be>; Tue, 16 Jan 2018 20:07:24 +0000
X-HalOne-ID: dd708a9d-faf8-11e7-b104-ec0d9a6ed9fe
Received: from mail-ot0-f174.google.com (unknown [74.125.82.174])
	by mx2.pub.mailpod2-cph3.one.com (Halon) with ESMTPS
	id dd708a9d-faf8-11e7-b104-ec0d9a6ed9fe;
	Tue, 16 Jan 2018 20:07:24 +0000 (UTC)
Received: by mail-ot0-f174.google.com with SMTP id 53so14749041otj.2
        for <filtered@awaresystems.be>; Tue, 16 Jan 2018 12:07:24 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-original-authentication-results:x-gm-message-state:delivered-to
         :in-reply-to:references:to:mime-version:from:message-id:date:cc
         :subject:precedence:list-id:list-unsubscribe:list-archive:list-post
         :list-help:list-subscribe:mime-version:sender:errors-to;
        bh=UBjVggFOelN8q8QidaujgxjZK0hXVkpvf4zL+Tj5MVA=;
        b=LcKQDl0ZpxMfMmsGlv7ID/vFDyT/oxwjOnxy8+LUDyp39e9yVHV+BODsLCXsC4eP4+
         m47NsPo592njTMM1a10Et38VdX+t18bcAcV5MW9MEUymVSzzdG1JLKyGOv+9FetVrzZ5
         +sgiDLqZQe0948WkdV/so1ble9NCSfHhICMzLDOMBZzP1pxfv4ndI+xJKg0uvRbMPt3m
         SVWoWUCidJEp40mHXypA0+3JLdOGvYLwUdOWfxGXaorazDnGvg7Zrk2IxuSdwMgmEf2A
         y1OqRnsUpOXWgK2mzn414JCKDgCY4TDRKFzA5JZS0ecURf4+2j5CVVS8T/SaXFpg/N4a
         NBFw==
X-Original-Authentication-Results: mx.google.com;       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org
X-Gm-Message-State: AKwxytdxjRl3nEBrhDcbqoLuTPVQsI1WVGZlVICOIzOXrHfpzhqtnbMS
	aGtllYj2zAAhN/tgWcIGzXVoenT8QEHBZz0jgvijS0NmXesLGlM=
X-Received: by 10.157.43.28 with SMTP id o28mr22033366otb.236.1516133242858;
        Tue, 16 Jan 2018 12:07:22 -0800 (PST)
X-Forwarded-To: filtered@awaresystems.be
X-Forwarded-For: awaresystems.be@gmail.com filtered@awaresystems.be
Delivered-To: awaresystems.be@gmail.com
Received: by 10.202.235.11 with SMTP id j11csp4601865oih;
        Tue, 16 Jan 2018 12:07:22 -0800 (PST)
X-Google-Smtp-Source: ACJfBovr3FBm9faNS+EPp1VScDYtGHVBWe55bJ9CkRHl8U7UgGxO7shZrPQ62jiIdr2CcYhJYG4m
X-Received: by 10.28.96.86 with SMTP id u83mr197748wmb.63.1516133241980;
        Tue, 16 Jan 2018 12:07:21 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; t=1516133241; cv=none;
        d=google.com; s=arc-20160816;
        b=M+ghYxln7na8HGYT5bzufB6waROji54OnlHZ4HNsl0PxUouUNpXkI+PDyP8kWB5Rz9
         mErRhJZamiD6zRwgPCOgxS1LbJS40F3qnWIOkHcxG9keVkqxdth8eizxnXdfb+hrSXHW
         JyewBgIfCtJLNcyEXCr/cwoeNE+/rgX2P6BkKg9ADIYrVFPYJQ6V1aMcom/4e6g1Ch29
         gNt4+3TGoL8AgsI3wT6n66I72m6UJvLu0WyF9tvQxVQLh51kt+JnNtNqO0tdIZM6+UKp
         0ZBiY/tlhGPNQOG/oKrtf2+xt7r5kp0uLAkCIGx38EDQ/DiVjHmz/qUoKYlW5bF3TTE3
         V5qg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
        h=errors-to:sender:mime-version:list-subscribe:list-help:list-post
         :list-archive:list-unsubscribe:list-id:precedence:subject:cc:date
         :message-id:from:mime-version:to:references:in-reply-to
         :arc-authentication-results;
        bh=UBjVggFOelN8q8QidaujgxjZK0hXVkpvf4zL+Tj5MVA=;
        b=zcMKsnKsNg/MVaAypgTd9gWLhGUJiyS5BGZ5MLKJ4WybR9eCAXjlA6bjnE562OZqDT
         lRqXHjbDUeKZjlp8W3xeUz+WqLmkhhnYYZQOfuifxybey3I5Fuzrlk+l+nDEF+/xH2Wo
         c/MKNlY2xeW4PuUF99mI+26b8ISxW+JdHns+Hc97xXiMh88OPaum6g9WwWJ08OjrZqjb
         b6W78+lZJnhuyfuyoOYRFdqG5X70Qj9l6VHHhGN20ltgitjrpSmBNbDqMhg9sK2bGSeH
         0MyPrsOe2ZtzkKCClWWEbFYcGuaTkjQbrwDuOhzCoMuYtjBmpmDs4oyFDn1vDf7vmXfK
         mVzA==
ARC-Authentication-Results: i=1; mx.google.com;
       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org
Return-Path: <tiff-bounces@lists.maptools.org>
Received: from lists.maptools.org (ks365428.kimsufi.com. [94.23.4.142])
        by mx.google.com with ESMTP id i64si2334601wmc.112.2018.01.16.12.07.21;
        Tue, 16 Jan 2018 12:07:21 -0800 (PST)
Received-SPF: neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) client-ip=94.23.4.142;
Authentication-Results: mx.google.com;
       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org
Received: from [10.1.0.241] (localhost.localdomain [127.0.0.1])
	by lists.maptools.org (Postfix) with ESMTP id E89A535C0BB;
	Tue, 16 Jan 2018 15:03:39 -0500 (EST)
Received: from mx0a-00145802.pphosted.com (mx0a-00145802.pphosted.com
	[67.231.148.190])
	by lists.maptools.org (Postfix) with ESMTPS id 170FC35C0AA;
	Tue, 16 Jan 2018 14:57:49 -0500 (EST)
Received: from pps.filterd (m0054486.ppops.net [127.0.0.1])
	by mx0a-00145802.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id
	w0GJseDo022720; Tue, 16 Jan 2018 14:57:47 -0500
Received: from mgamac05.us.ricoh.ds (mcnhost1.ikon.com [205.145.16.4])
	by mx0a-00145802.pphosted.com with ESMTP id 2ffcbq9x58-2;
	Tue, 16 Jan 2018 14:57:47 -0500
In-Reply-To: <5A5E53A9.4020707@osa.pri.ee>
References: <39EE49FE-23A5-43DE-8998-51526287D182@larrygritz.com>	<alpine.GSO.2.20.1801151400550.18206@scrappy.simplesystems.org>
	<4252FC16-C07D-4FA1-8B7B-25D626633236@larrygritz.com>	<F78FB533EE93A044B560CD25E17A4DAF18388904@mbx031-e1-va-4.exch031.domain.local>
	<alpine.GSO.2.20.1801151728170.18206@scrappy.simplesystems.org>	<f279e2e7-a91a-0cd8-a513-026479f571ff@cubewerx.com>
	<OF239FD1E6.9CEBEE62-ON87258217.0064336B-87258217.00663BB4@ricoh-usa.com>
	<5A5E53A9.4020707@osa.pri.ee>
To: Paavo Helde <paavo@osa.pri.ee>
MIME-Version: 1.0
X-KeepSent: 54AE7D2A:761E261D-87258217:006CE6C5;
 type=4; name=$KeepSent
X-Mailer: IBM Notes Release 9.0 March 08, 2013
From: Joe.Maniaci@ricoh-usa.com
Message-ID: <OF54AE7D2A.761E261D-ON87258217.006CE6C5-87258217.006DA8CF@ricoh-usa.com>
Date: Tue, 16 Jan 2018 12:57:45 -0700
X-MIMETrack: Serialize by Router on MGAMAC05/Servers/RCUS(Release
	9.0.1FP7|August 17, 2016) at 01/16/2018 14:57:46,
	Serialize complete at 01/16/2018 14:57:46
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, ,
	definitions=2018-01-16_10:, , signatures=0
Cc: tiff-bounces@lists.maptools.org, tiff@lists.maptools.org
Subject: Re: [Tiff] Strategies for multi-core speedups
X-BeenThere: tiff@lists.maptools.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: TIFF and libtiff discussion <tiff.lists.maptools.org>
List-Unsubscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=unsubscribe>
List-Archive: <http://lists.maptools.org/mailman/private/tiff>
List-Post: <mailto:tiff@lists.maptools.org>
List-Help: <mailto:tiff-request@lists.maptools.org?subject=help>
List-Subscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=subscribe>
Content-Type: multipart/mixed; boundary="===============0135623820=="
Mime-version: 1.0
Sender: tiff-bounces@lists.maptools.org
Errors-To: tiff-bounces@lists.maptools.org

This is a multipart message in MIME format.
--===============0135623820==
Content-Type: multipart/alternative; boundary="=_alternative 006DA85A87258217_="

This is a multipart message in MIME format.
--=_alternative 006DA85A87258217_=
Content-Type: text/plain; charset="US-ASCII"

These laptops have SSDs but they're some of the worst I have ever seen, 
less than 100Mb read or write, it comes close single threaded but there 
was always a hitch where you could tell it was loading new tiles into 
memory. I basically have tiles in view being drawn, then i have a 
perimeter of tiles around those either leaving/entering memory based on 
their proximity to the viewing area. I was actually designing a pyramid 
template(I'm building this tool out for various other filetypes) without 
knowing it was a thing, but instead I opted to just generate a few 
downsampled examples of the image when zoomed out, and then move into 
tiled mode once the user zoomed in to where you can see actual pixels 
because we validate literally on a pixel by pixel to basis. 

Joseph Maniaci
Staff Software Engineer
RPPS



From:   Paavo Helde <paavo@osa.pri.ee>
To:     tiff@lists.maptools.org
Date:   01/16/2018 12:46 PM
Subject:        Re: [Tiff] Strategies for multi-core speedups [EXTERNAL]
Sent by:        tiff-bounces@lists.maptools.org




On 16.01.2018 20:36, Joe.Maniaci@ricoh-usa.com wrote:
I am a complete newbie to digital imagery and TIFFs in general, so please 
forgive any ignorance. 

I'm an automated test tool developer and I am actually going to start 
working on a viewer for the TIFFs that my company generates, to 
potentially include relying on the bigtiff format and this is the thing 
I've spent most of my time thinking about. I've come to the conclusion 
that for the images we generate, up to 10GB uncompressed(soon to be 40GB), 
that I will need to preprocess a single TIFF into multiple TIFFs 
containing intelligently placed tiles just to get around the I/O issue. By 
intelligently placed, I mean that if a user were to scroll their view 
window to the right, and five tiles(vertically aligned) had to be loaded 
into memory, I could theoretically have those files intelligently located 
in different TIFF files(subTIFFs?) that would allow multiple threads to 
operate on. So each thread only had to find one tile. So for my tool it 
would be a staggering amount of pre-processing, but real time manipulation 
is extremely fluid for the user which is basically my number one 
requirement. 

So for a case like mine, thread overhead is nothing to worry about. That 
and the more I read about the C++11 thread library, the less I worry. I 
can just leave it to the library to determine how many threads should 
operate in my 4-core environments, or my 16-core environments. At least 
that's the impression I've garnered. 

If all you want is a viewer then you are limited by the screen sizes which 
are typically not more than ca 2000x2000 pixels. And this is pretty small 
image nowadays, there is no need to multithread anything, you could just 
read the needed tiles from the TIFF file in real time and maybe add a 
small cache of the read tiles so you don't need to read them again if the 
user pans left 1 pixel.

For zooming out the TIFF file should have different frames of the same 
image stored in decreasing resolutions so that a suitable resolution can 
be chosen for display - these are called pyramidal tiffs. The TIFF 
standard suggests that such varying resolution images should be stored as 
sub-IFD-s but as many viewers do not support sub-IFD-s they are typically 
stored just in standard IFD-s, and some XML piece in the first IFD 
provides the details. I am sure there are multiple variants of such 
pyramidal tiff formats, one from our company as well, and of course there 
are multiple readers supporting such tiff files.

hth
Paavo


_______________________________________________
Tiff mailing list: Tiff@lists.maptools.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.maptools.org_mailman_listinfo_tiff&d=DwICAg&c=5hYF0Zu0Yz-C6S-kaHDItw&r=HnVw9YJBS4ST0tOBucT3NvWUJRRGhDxXu4zHnS5AkgY&m=RcsXrpPDw-KXo-5NeLU-puXNVyvQfx7dLHdgzA4wxTw&s=4-nAkbKaJaJMlatde8ffh_QxF23k8hdM9UgRBbWwIsM&e=

https://urldefense.proofpoint.com/v2/url?u=http-3A__www.remotesensing.org_libtiff_&d=DwICAg&c=5hYF0Zu0Yz-C6S-kaHDItw&r=HnVw9YJBS4ST0tOBucT3NvWUJRRGhDxXu4zHnS5AkgY&m=RcsXrpPDw-KXo-5NeLU-puXNVyvQfx7dLHdgzA4wxTw&s=2t33zDcteNgQ1Ihq6G5nrh669iJFqB535PFOMpJ1VAE&e=

--=_alternative 006DA85A87258217_=
Content-Type: text/html; charset="US-ASCII"

<font size=2 face="sans-serif">These laptops have SSDs but they're some
of the worst I have ever seen, less than 100Mb read or write, it comes
close single threaded but there was always a hitch where you could tell
it was loading new tiles into memory. I basically have tiles in view being
drawn, then i have a perimeter of tiles around those either leaving/entering
memory based on their proximity to the viewing area. I was actually designing
a pyramid template(I'm building this tool out for various other filetypes)
without knowing it was a thing, but instead I opted to just generate a
few downsampled examples of the image when zoomed out, and then move into
tiled mode once the user zoomed in to where you can see actual pixels because
we validate literally on a pixel by pixel to basis. </font>
<br>
<br><font size=2 face="sans-serif">Joseph Maniaci</font>
<br><font size=2 face="sans-serif">Staff Software Engineer</font>
<br><font size=2 face="sans-serif">RPPS</font>
<br>
<br>
<br>
<br><font size=1 color=#5f5f5f face="sans-serif">From: &nbsp; &nbsp; &nbsp;
&nbsp;</font><font size=1 face="sans-serif">Paavo Helde &lt;paavo@osa.pri.ee&gt;</font>
<br><font size=1 color=#5f5f5f face="sans-serif">To: &nbsp; &nbsp; &nbsp;
&nbsp;</font><font size=1 face="sans-serif">tiff@lists.maptools.org</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Date: &nbsp; &nbsp; &nbsp;
&nbsp;</font><font size=1 face="sans-serif">01/16/2018 12:46 PM</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Subject: &nbsp; &nbsp;
&nbsp; &nbsp;</font><font size=1 face="sans-serif">Re: [Tiff] Strategies
for multi-core speedups [EXTERNAL]</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Sent by: &nbsp; &nbsp;
&nbsp; &nbsp;</font><font size=1 face="sans-serif">tiff-bounces@lists.maptools.org</font>
<br>
<hr noshade>
<br>
<br>
<br>
<br><font size=3>On 16.01.2018 20:36, </font><a href="mailto:Joe.Maniaci@ricoh-usa.com"><font size=3 color=blue><u>Joe.Maniaci@ricoh-usa.com</u></font></a><font size=3>
wrote:</font>
<br><font size=2 face="sans-serif">I am a complete newbie to digital imagery
and TIFFs in general, so please forgive any ignorance.</font><font size=3>
<br>
</font><font size=2 face="sans-serif"><br>
I'm an automated test tool developer and I am actually going to start working
on a viewer for the TIFFs that my company generates, to potentially include
relying on the bigtiff format and this is the thing I've spent most of
my time thinking about. I've come to the conclusion that for the images
we generate, up to 10GB uncompressed(soon to be 40GB), that I will need
to preprocess a single TIFF into multiple TIFFs containing intelligently
placed tiles just to get around the I/O issue. By intelligently placed,
I mean that if a user were to scroll their view window to the right, and
five tiles(vertically aligned) had to be loaded into memory, I could theoretically
have those files intelligently located in different TIFF files(subTIFFs?)
that would allow multiple threads to operate on. So each thread only had
to find one tile. So for my tool it would be a staggering amount of pre-processing,
but real time manipulation is extremely fluid for the user which is basically
my number one requirement.</font><font size=3> <br>
</font><font size=2 face="sans-serif"><br>
So for a case like mine, thread overhead is nothing to worry about. That
and the more I read about the C++11 thread library, the less I worry. I
can just leave it to the library to determine how many threads should operate
in my 4-core environments, or my 16-core environments. At least that's
the impression I've garnered.</font><font size=3> </font>
<br><font size=3><br>
If all you want is a viewer then you are limited by the screen sizes which
are typically not more than ca 2000x2000 pixels. And this is pretty small
image nowadays, there is no need to multithread anything, you could just
read the needed tiles from the TIFF file in real time and maybe add a small
cache of the read tiles so you don't need to read them again if the user
pans left 1 pixel.<br>
<br>
For zooming out the TIFF file should have different frames of the same
image stored in decreasing resolutions so that a suitable resolution can
be chosen for display - these are called pyramidal tiffs. The TIFF standard
suggests that such varying resolution images should be stored as sub-IFD-s
but as many viewers do not support sub-IFD-s they are typically stored
just in standard IFD-s, and some XML piece in the first IFD provides the
details. I am sure there are multiple variants of such pyramidal tiff formats,
one from our company as well, and of course there are multiple readers
supporting such tiff files.<br>
<br>
hth<br>
Paavo<br>
<br>
<br>
</font><tt><font size=2>_______________________________________________<br>
Tiff mailing list: Tiff@lists.maptools.org<br>
</font></tt><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.maptools.org_mailman_listinfo_tiff&amp;d=DwICAg&amp;c=5hYF0Zu0Yz-C6S-kaHDItw&amp;r=HnVw9YJBS4ST0tOBucT3NvWUJRRGhDxXu4zHnS5AkgY&amp;m=RcsXrpPDw-KXo-5NeLU-puXNVyvQfx7dLHdgzA4wxTw&amp;s=4-nAkbKaJaJMlatde8ffh_QxF23k8hdM9UgRBbWwIsM&amp;e="><tt><font size=2>https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.maptools.org_mailman_listinfo_tiff&amp;d=DwICAg&amp;c=5hYF0Zu0Yz-C6S-kaHDItw&amp;r=HnVw9YJBS4ST0tOBucT3NvWUJRRGhDxXu4zHnS5AkgY&amp;m=RcsXrpPDw-KXo-5NeLU-puXNVyvQfx7dLHdgzA4wxTw&amp;s=4-nAkbKaJaJMlatde8ffh_QxF23k8hdM9UgRBbWwIsM&amp;e=</font></tt></a><tt><font size=2><br>
</font></tt><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.remotesensing.org_libtiff_&amp;d=DwICAg&amp;c=5hYF0Zu0Yz-C6S-kaHDItw&amp;r=HnVw9YJBS4ST0tOBucT3NvWUJRRGhDxXu4zHnS5AkgY&amp;m=RcsXrpPDw-KXo-5NeLU-puXNVyvQfx7dLHdgzA4wxTw&amp;s=2t33zDcteNgQ1Ihq6G5nrh669iJFqB535PFOMpJ1VAE&amp;e="><tt><font size=2>https://urldefense.proofpoint.com/v2/url?u=http-3A__www.remotesensing.org_libtiff_&amp;d=DwICAg&amp;c=5hYF0Zu0Yz-C6S-kaHDItw&amp;r=HnVw9YJBS4ST0tOBucT3NvWUJRRGhDxXu4zHnS5AkgY&amp;m=RcsXrpPDw-KXo-5NeLU-puXNVyvQfx7dLHdgzA4wxTw&amp;s=2t33zDcteNgQ1Ihq6G5nrh669iJFqB535PFOMpJ1VAE&amp;e=</font></tt></a>
<br>
--=_alternative 006DA85A87258217_=--

--===============0135623820==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Tiff mailing list: Tiff@lists.maptools.org
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
--===============0135623820==--
