Return-Path: <awaresystems.be+caf_=filtered=awaresystems.be@gmail.com>
Delivered-To: filtered@awaresystems.be
Received: from mx1.pub.mailpod2-cph3.one.com ([10.27.25.11])
	by mailstorage18.cst.mailpod2-cph3.one.com (Dovecot) with LMTP id fVUgJXZZXlrJ8gAAQS8vrQ
	for <filtered@awaresystems.be>; Tue, 16 Jan 2018 20:02:43 +0000
X-HalOne-ID: 36069723-faf8-11e7-afe7-e0d848957041
Received: from mail-ot0-f179.google.com (unknown [74.125.82.179])
	by mx1.pub.mailpod2-cph3.one.com (Halon) with ESMTPS
	id 36069723-faf8-11e7-afe7-e0d848957041;
	Tue, 16 Jan 2018 20:02:43 +0000 (UTC)
Received: by mail-ot0-f179.google.com with SMTP id x15so14734328ote.0
        for <filtered@awaresystems.be>; Tue, 16 Jan 2018 12:02:43 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-original-authentication-results:x-gm-message-state:delivered-to
         :mime-version:in-reply-to:references:from:date:message-id:to:cc
         :subject:precedence:list-id:list-unsubscribe:list-archive:list-post
         :list-help:list-subscribe:mime-version:sender:errors-to;
        bh=0/BIfrASDNyF/dKik4qKUqQwDo7msErkLH1J4ZqDD1A=;
        b=jxQOc1NlTjC3WIFtW8/VEpR85EEBQSYwRPDkYerAG/I09Kt3SG6geS0Og2wRVhHZPX
         0/W4iFMYKZRBCDoGdLI245HuoWYK4s9wxTxb6zlvKoxNkJzc/ZcaW9WoOVrhBvZgqt4s
         RDrc9/XYxE7/FOL+d6i1LBBVAYqjCgzNk496Cw8ywKw2UuhYOLWhHjD76fSZQGMe0hJb
         xpdyrj5ztK+m8PbwMgxSSYgA+HytI4Y7pSAtz3VX6R8bo1FmH5QkKMdvxAZ/+t7KpoVh
         oybYO+mnmsXqNjzCprOhy9rv/SqYRMTHhRKNNPB1xkbsUrPSrnDNtNNV+N/82S4ht4iC
         BaDg==
X-Original-Authentication-Results: mx.google.com;       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org
X-Gm-Message-State: AKwxytej+5xmXNYWx+z2WEN7bMeIidLeSe5j1lq2DweCw3t4kRASDxCA
	lrxWm8FDCBAJO0JxGN2cxi4c2oh7Kx9d5HZCSP53VHLq2Vic0qA=
X-Received: by 10.157.5.248 with SMTP id 111mr9123132otd.303.1516132961954;
        Tue, 16 Jan 2018 12:02:41 -0800 (PST)
X-Forwarded-To: filtered@awaresystems.be
X-Forwarded-For: awaresystems.be@gmail.com filtered@awaresystems.be
Delivered-To: awaresystems.be@gmail.com
Received: by 10.202.235.11 with SMTP id j11csp4596585oih;
        Tue, 16 Jan 2018 12:02:41 -0800 (PST)
X-Google-Smtp-Source: ACJfBos1242WLXVQ6xZVjbmL8Mw0sxYAOnO4U/qW6iD59vnR1EtbAHAsRUm9Xm4G2yuf8LVd1ubF
X-Received: by 10.223.150.213 with SMTP id u79mr348346wrb.137.1516132961110;
        Tue, 16 Jan 2018 12:02:41 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; t=1516132961; cv=none;
        d=google.com; s=arc-20160816;
        b=y5lIYjfemmBzUX82rUs/+yZaAYjfDRWpiNrBYNf1iedIelvI5lDpd+Ijqp5QWFeeOq
         4fAn/2Yd8KMVMUjbS7+nnmcjZ09OOf0UdPfdogZrPCn2Qiru4RekQrIXfQybH/kTkgkU
         I5MhQiNhZoLyhntNJUp0tCymaFImJauVTiHEnhMl3QggXxcD+iFqgBx8MNp6Nmyxvmb0
         AYzFYVtb5oBES+G3tisxybj5Jr6mXfAVNGQFiLXOTT+tcU+tMjTtVBGCj+yp2TTbSCQV
         kL5g9jsVb6l7K9ZtZ//YzVNEFXf3qURhWPnk+r9a3SVUgoz2FeZreNA2cxvlHgsaviYi
         8t9w==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
        h=errors-to:sender:mime-version:list-subscribe:list-help:list-post
         :list-archive:list-unsubscribe:list-id:precedence:subject:cc:to
         :message-id:date:from:references:in-reply-to:mime-version
         :arc-authentication-results;
        bh=0/BIfrASDNyF/dKik4qKUqQwDo7msErkLH1J4ZqDD1A=;
        b=sYhmmijA/KtDVXWEA+zIC7x+4X4vGftBf5djEWu0rTArSQtnsNUSOCnhrxtT30Of9A
         NTVb2RV59KV7k3FwcC7MyPqwGC/+pZXfj3w2GvOLJjkG5UDC8lTs36FFyjR7/hIy6ITI
         16doHjheKk49Oi8YAJe2ufVUmz41gY6ez/etESjDF/VLsKHttDirQUnUg9TaVi6FX47R
         ExGGHjQnpXdM0cpNgZEkmUQWx5QvKyZRvpHa1nz8CFZuGMV3tPr/gKMfb29WqpB5Cibl
         WhUwu13pzN24vtgovLEqzejus6I+dkv8/Onn+CCkwkyGegpI8psHIBOc49ITEiclKJ9e
         k4uQ==
ARC-Authentication-Results: i=1; mx.google.com;
       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org
Return-Path: <tiff-bounces@lists.maptools.org>
Received: from lists.maptools.org (ks365428.kimsufi.com. [94.23.4.142])
        by mx.google.com with ESMTP id j6si2283037wmj.151.2018.01.16.12.02.40;
        Tue, 16 Jan 2018 12:02:41 -0800 (PST)
Received-SPF: neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) client-ip=94.23.4.142;
Authentication-Results: mx.google.com;
       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org
Received: from [10.1.0.241] (localhost.localdomain [127.0.0.1])
	by lists.maptools.org (Postfix) with ESMTP id 585B135C0AB;
	Tue, 16 Jan 2018 15:01:46 -0500 (EST)
Received: from mail-wr0-f193.google.com (mail-wr0-f193.google.com
	[209.85.128.193])
	by lists.maptools.org (Postfix) with ESMTPS id EE98135C0AA
	for <tiff@lists.maptools.org>; Tue, 16 Jan 2018 14:52:46 -0500 (EST)
Received: by mail-wr0-f193.google.com with SMTP id o7so16507157wro.8
	for <tiff@lists.maptools.org>; Tue, 16 Jan 2018 11:52:46 -0800 (PST)
X-Received: by 10.223.173.165 with SMTP id w34mr315080wrc.75.1516132366268;
	Tue, 16 Jan 2018 11:52:46 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.187.17 with HTTP; Tue, 16 Jan 2018 11:52:45 -0800 (PST)
In-Reply-To: <5A5E53A9.4020707@osa.pri.ee>
References: <39EE49FE-23A5-43DE-8998-51526287D182@larrygritz.com>
	<alpine.GSO.2.20.1801151400550.18206@scrappy.simplesystems.org>
	<4252FC16-C07D-4FA1-8B7B-25D626633236@larrygritz.com>
	<F78FB533EE93A044B560CD25E17A4DAF18388904@mbx031-e1-va-4.exch031.domain.local>
	<alpine.GSO.2.20.1801151728170.18206@scrappy.simplesystems.org>
	<f279e2e7-a91a-0cd8-a513-026479f571ff@cubewerx.com>
	<OF239FD1E6.9CEBEE62-ON87258217.0064336B-87258217.00663BB4@ricoh-usa.com>
	<5A5E53A9.4020707@osa.pri.ee>
From: Kemp Watson <kemp@objectivepathology.com>
Date: Tue, 16 Jan 2018 14:52:45 -0500
Message-ID: <CACp925S20fu_Mtu7iRbfEkeTEYsvGx1ms5yHrYGhzq0su-cSPw@mail.gmail.com>
To: Paavo Helde <paavo@osa.pri.ee>
Cc: tiff@lists.maptools.org
Subject: Re: [Tiff] Strategies for multi-core speedups
X-BeenThere: tiff@lists.maptools.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: TIFF and libtiff discussion <tiff.lists.maptools.org>
List-Unsubscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=unsubscribe>
List-Archive: <http://lists.maptools.org/mailman/private/tiff>
List-Post: <mailto:tiff@lists.maptools.org>
List-Help: <mailto:tiff-request@lists.maptools.org?subject=help>
List-Subscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=subscribe>
Content-Type: multipart/mixed; boundary="===============1608999783=="
Mime-version: 1.0
Sender: tiff-bounces@lists.maptools.org
Errors-To: tiff-bounces@lists.maptools.org

--===============1608999783==
Content-Type: multipart/alternative; boundary="f403045cf3c61fced30562ea1367"

--f403045cf3c61fced30562ea1367
Content-Type: text/plain; charset="UTF-8"

Also Joseph, you might find yourself limited by network bandwidth or BDP
long before you are limited by application threading, good idea to think
about network simulation (if relevant) before detailed design.

On Tue, Jan 16, 2018 at 2:34 PM, Paavo Helde <paavo@osa.pri.ee> wrote:

>
> On 16.01.2018 20:36, Joe.Maniaci@ricoh-usa.com wrote:
>
> I am a complete newbie to digital imagery and TIFFs in general, so please
> forgive any ignorance.
>
> I'm an automated test tool developer and I am actually going to start
> working on a viewer for the TIFFs that my company generates, to potentially
> include relying on the bigtiff format and this is the thing I've spent most
> of my time thinking about. I've come to the conclusion that for the images
> we generate, up to 10GB uncompressed(soon to be 40GB), that I will need to
> preprocess a single TIFF into multiple TIFFs containing intelligently
> placed tiles just to get around the I/O issue. By intelligently placed, I
> mean that if a user were to scroll their view window to the right, and five
> tiles(vertically aligned) had to be loaded into memory, I could
> theoretically have those files intelligently located in different TIFF
> files(subTIFFs?) that would allow multiple threads to operate on. So each
> thread only had to find one tile. So for my tool it would be a staggering
> amount of pre-processing, but real time manipulation is extremely fluid for
> the user which is basically my number one requirement.
>
> So for a case like mine, thread overhead is nothing to worry about. That
> and the more I read about the C++11 thread library, the less I worry. I can
> just leave it to the library to determine how many threads should operate
> in my 4-core environments, or my 16-core environments. At least that's the
> impression I've garnered.
>
>
> If all you want is a viewer then you are limited by the screen sizes which
> are typically not more than ca 2000x2000 pixels. And this is pretty small
> image nowadays, there is no need to multithread anything, you could just
> read the needed tiles from the TIFF file in real time and maybe add a small
> cache of the read tiles so you don't need to read them again if the user
> pans left 1 pixel.
>
> For zooming out the TIFF file should have different frames of the same
> image stored in decreasing resolutions so that a suitable resolution can be
> chosen for display - these are called pyramidal tiffs. The TIFF standard
> suggests that such varying resolution images should be stored as sub-IFD-s
> but as many viewers do not support sub-IFD-s they are typically stored just
> in standard IFD-s, and some XML piece in the first IFD provides the
> details. I am sure there are multiple variants of such pyramidal tiff
> formats, one from our company as well, and of course there are multiple
> readers supporting such tiff files.
>
> hth
> Paavo
>
>
>
>
> _______________________________________________
> Tiff mailing list: Tiff@lists.maptools.org
> http://lists.maptools.org/mailman/listinfo/tiff
> http://www.remotesensing.org/libtiff/
>

--f403045cf3c61fced30562ea1367
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Also Joseph, you might find yourself limited by network ba=
ndwidth or BDP long before you are limited by application threading, good i=
dea to think about network simulation (if relevant) before detailed design.=
<br></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue,=
 Jan 16, 2018 at 2:34 PM, Paavo Helde <span dir=3D"ltr">&lt;<a href=3D"mail=
to:paavo@osa.pri.ee" target=3D"_blank">paavo@osa.pri.ee</a>&gt;</span> wrot=
e:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-l=
eft:1px #ccc solid;padding-left:1ex">
 =20
   =20
 =20
  <div bgcolor=3D"#FFFFFF" text=3D"#000000"><span class=3D"">
    <br>
    <div class=3D"m_-6608426305772452080moz-cite-prefix">On 16.01.2018 20:3=
6,
      <a class=3D"m_-6608426305772452080moz-txt-link-abbreviated" href=3D"m=
ailto:Joe.Maniaci@ricoh-usa.com" target=3D"_blank">Joe.Maniaci@ricoh-usa.co=
m</a> wrote:<br>
    </div>
    <blockquote type=3D"cite"><font face=3D"sans-serif" size=3D"2">I am a c=
omplete
        newbie to digital imagery
        and TIFFs in general, so please forgive any ignorance.</font>
      <br>
      <br>
      <font face=3D"sans-serif" size=3D"2">I&#39;m an automated test tool
        developer
        and I am actually going to start working on a viewer for the
        TIFFs that
        my company generates, to potentially include relying on the
        bigtiff format
        and this is the thing I&#39;ve spent most of my time thinking about=
.
        I&#39;ve come
        to the conclusion that for the images we generate, up to 10GB
        uncompressed(soon
        to be 40GB), that I will need to preprocess a single TIFF into
        multiple
        TIFFs containing intelligently placed tiles just to get around
        the I/O
        issue. By intelligently placed, I mean that if a user were to
        scroll their
        view window to the right, and five tiles(vertically aligned) had
        to be
        loaded into memory, I could theoretically have those files
        intelligently
        located in different TIFF files(subTIFFs?) that would allow
        multiple threads
        to operate on. So each thread only had to find one tile. So for
        my tool
        it would be a staggering amount of pre-processing, but real time
        manipulation
        is extremely fluid for the user which is basically my number one
        requirement.</font>
      <br>
      <br>
      <font face=3D"sans-serif" size=3D"2">So for a case like mine, thread
        overhead
        is nothing to worry about. That and the more I read about the
        C++11 thread
        library, the less I worry. I can just leave it to the library to
        determine
        how many threads should operate in my 4-core environments, or my
        16-core
        environments. At least that&#39;s the impression I&#39;ve garnered.=
</font>
      <font face=3D"sans-serif" size=3D"2"></font><br>
    </blockquote>
    <br></span>
    If all you want is a viewer then you are limited by the screen sizes
    which are typically not more than ca 2000x2000 pixels. And this is
    pretty small image nowadays, there is no need to multithread
    anything, you could just read the needed tiles from the TIFF file in
    real time and maybe add a small cache of the read tiles so you don&#39;=
t
    need to read them again if the user pans left 1 pixel.<br>
    <br>
    For zooming out the TIFF file should have different frames of the
    same image stored in decreasing resolutions so that a suitable
    resolution can be chosen for display - these are called pyramidal
    tiffs. The TIFF standard suggests that such varying resolution
    images should be stored as sub-IFD-s but as many viewers do not
    support sub-IFD-s they are typically stored just in standard IFD-s,
    and some XML piece in the first IFD provides the details. I am sure
    there are multiple variants of such pyramidal tiff formats, one from
    our company as well, and of course there are multiple readers
    supporting such tiff files.<br>
    <br>
    hth<span class=3D"HOEnZb"><font color=3D"#888888"><br>
    Paavo<br>
    <br>
    <br>
    <br>
  </font></span></div>

<br>______________________________<wbr>_________________<br>
Tiff mailing list: <a href=3D"mailto:Tiff@lists.maptools.org">Tiff@lists.ma=
ptools.org</a><br>
<a href=3D"http://lists.maptools.org/mailman/listinfo/tiff" rel=3D"noreferr=
er" target=3D"_blank">http://lists.maptools.org/<wbr>mailman/listinfo/tiff<=
/a><br>
<a href=3D"http://www.remotesensing.org/libtiff/" rel=3D"noreferrer" target=
=3D"_blank">http://www.remotesensing.org/<wbr>libtiff/</a><br></blockquote>=
</div><br></div>

--f403045cf3c61fced30562ea1367--

--===============1608999783==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Tiff mailing list: Tiff@lists.maptools.org
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
--===============1608999783==--
