Return-Path: <awaresystems.be+caf_=filtered=awaresystems.be@gmail.com>
Delivered-To: filtered@awaresystems.be
Received: from mx10.public.one.com ([10.246.18.30])
	by mail6.local.one.com (Dovecot) with LMTP id SmwvJEb5U1lWYQAAsyOMew
	for <filtered@awaresystems.be>; Wed, 28 Jun 2017 19:28:11 +0000
X-HalOne-ID: eb76fe07-5c37-11e7-84ad-246e9615723c
Received: from mail-oi0-f48.google.com (unknown [209.85.218.48])
	by mx10.public.one.com (Halon) with ESMTPS
	id eb76fe07-5c37-11e7-84ad-246e9615723c;
	Wed, 28 Jun 2017 19:28:11 +0000 (UTC)
Received: by mail-oi0-f48.google.com with SMTP id p66so49697433oia.0
        for <filtered@awaresystems.be>; Wed, 28 Jun 2017 12:28:10 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-original-authentication-results:x-gm-message-state:delivered-to
         :mime-version:in-reply-to:references:from:date:message-id:to:cc
         :subject:precedence:list-id:list-unsubscribe:list-archive:list-post
         :list-help:list-subscribe:mime-version:sender:errors-to;
        bh=IOTD8LUgYi76KxezBZp6g+r/wMyNdSoJ1tHC3yx9Ckw=;
        b=CSV//Gy+8KDQPdnTNzjnGO+oeyBNSL/srJi0KkDAxrfFoSIPGzk5LIWNz/AbQkuuO5
         r0fRDZbUTsPAM+ZgfqzhQDmHOs9ISoCYlcVklCfYTOt8jnHt6GUE3q84X+vqyqNewQwJ
         hyqkgeiUrt/ydODkqQzshHtA29ZJiezRh/4We5VEz6PNIlsqTZ2G/ebnAkHUwdfa4NM/
         ui7PqTi63L6mGuPCadUDi6S6yX4dSO8ATKGazkbg28BqLNFnBrpUCbvUxNs5jseqaE6o
         xjO8MJ2BrhX00B+ZW/noauxuFKDyh8b+EC7OlFDr51LnkvlmL0FRWKYuS8Ygu4nyeEv5
         m1pA==
X-Original-Authentication-Results: mx.google.com;       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org;       dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=gmail.com
X-Gm-Message-State: AKS2vOzljjlLAT46LOXoHRmPPFlyk+UPiqGppwKAIH8wCvSFR5vNiDNC
	kTFrgH0Kz9+284OEuI+wMIHc5383XeQ7BvV7+1BU/g==
X-Received: by 10.202.208.85 with SMTP id h82mr7779290oig.80.1498678089694;
        Wed, 28 Jun 2017 12:28:09 -0700 (PDT)
X-Forwarded-To: filtered@awaresystems.be
X-Forwarded-For: awaresystems.be@gmail.com filtered@awaresystems.be
Delivered-To: awaresystems.be@gmail.com
Received: by 10.202.64.84 with SMTP id n81csp3884778oia;
        Wed, 28 Jun 2017 12:28:08 -0700 (PDT)
X-Received: by 10.28.134.11 with SMTP id i11mr7603615wmd.77.1498678088087;
        Wed, 28 Jun 2017 12:28:08 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; t=1498678088; cv=none;
        d=google.com; s=arc-20160816;
        b=PsGhux5AOMSxJPifPtryf9dIgUxprPqGzUWuuK0FXyxFE/QDvaCeNJbt+sGfyuvFwC
         ALFHigeEXVnG4i6voRD8FBGzgG7DnvGMhVbfDGKYclAV7PHwE9HAkFUyue2EHvPga746
         vYR1DoH+5HeJ0PKPXgvLhrJfSTHhYMuGd49wCoGSHJBqxrPzNDf0XBtaNOH+SBBtLRMd
         XBi8UXPzDLmz3YvTMAL6Q/ZDFtNi7W6d5+D6ub4mduR+lDIZGYGWrDjwNQM4nE9yRtn6
         PGx2ePg2CrVoO56SlOUfB2avvTkLmWJg1mYKYrEYdv+BTV1d6eNmlHJtnIyQ7sAS22iA
         w8CQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
        h=errors-to:sender:mime-version:list-subscribe:list-help:list-post
         :list-archive:list-unsubscribe:list-id:precedence:subject:cc:to
         :message-id:date:from:references:in-reply-to:mime-version
         :arc-authentication-results;
        bh=IOTD8LUgYi76KxezBZp6g+r/wMyNdSoJ1tHC3yx9Ckw=;
        b=t4hUhZ6GIfLwYnsAHcr2G+jpg27QVTkhUkOY/523ODNUNMO1LMNnzh1Zcgb7S/5JLS
         SNJwJroWO2C0DIPiqsbHcF9scEtBkmRJvi9lW9W66AQzVSZ74ni1X9VgZuzOoeZH/6Gl
         naM7EMxkx1PnJYzj8T+B41mPf3LgJQArN0xTG2z6TD/bV1PXVoCiErqttUzCK7y1cAiP
         bEChvEsBJ3k8d9H6Cjug2g99d7dzxFVVtbRLIPNw2sVJCnY1Yhpqo2pKAvRrIV9ZHnfj
         puHz3+yBQPnDDDm3tBWOhMZTCQj/+q5plped/waG6kI5cbB2YPavclExWYbwGo2+648+
         WB8g==
ARC-Authentication-Results: i=1; mx.google.com;
       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org;
       dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=gmail.com
Return-Path: <tiff-bounces@lists.maptools.org>
Received: from lists.maptools.org (ks365428.kimsufi.com. [94.23.4.142])
        by mx.google.com with ESMTP id l7si6162510wml.156.2017.06.28.12.28.07;
        Wed, 28 Jun 2017 12:28:08 -0700 (PDT)
Received-SPF: neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) client-ip=94.23.4.142;
Authentication-Results: mx.google.com;
       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org;
       dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=gmail.com
Received: from [10.1.0.241] (localhost.localdomain [127.0.0.1])
	by lists.maptools.org (Postfix) with ESMTP id 5E7F035C06C;
	Wed, 28 Jun 2017 14:26:49 -0500 (EST)
Received: from mail-pf0-f176.google.com (mail-pf0-f176.google.com
	[209.85.192.176])
	by lists.maptools.org (Postfix) with ESMTPS id 074C835C06A
	for <tiff@lists.maptools.org>; Wed, 28 Jun 2017 14:01:12 -0500 (EST)
Received: by mail-pf0-f176.google.com with SMTP id s66so37867303pfs.1
	for <tiff@lists.maptools.org>; Wed, 28 Jun 2017 12:01:12 -0700 (PDT)
X-Received: by 10.99.6.145 with SMTP id 139mr11881175pgg.157.1498676470623;
	Wed, 28 Jun 2017 12:01:10 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.100.185.5 with HTTP; Wed, 28 Jun 2017 12:01:09 -0700 (PDT)
In-Reply-To: <alpine.GSO.2.20.1706281033400.6797@scrappy.simplesystems.org>
References: <CABoQFUADrHiYLs7=2gVoRGkpex3P2jHWRPCZhehS8+OQRxTvYQ@mail.gmail.com>
	<1588415.R2acFO4bS3@even-i700>
	<CAAcuQeYOCExBMHb15EsYpUttGKyeJ0y=Xmqtf-nkUgM_o61GXQ@mail.gmail.com>
	<1879895.yTvgkPFqHA@even-i700>
	<CAAcuQeZgiBE9GvFyncYdHbqrW=sB8yPS0roicPKXBjvJvzbnWg@mail.gmail.com>
	<alpine.GSO.2.20.1706281033400.6797@scrappy.simplesystems.org>
From: Rob Tillaart <rob.tillaart@gmail.com>
Date: Wed, 28 Jun 2017 21:01:09 +0200
Message-ID: <CANoujwGk7-eagBqfrUTAaaGr1U8wTR3jHcgDRsvUvV1h7tz7cA@mail.gmail.com>
To: Bob Friesenhahn <bfriesen@simple.dallas.tx.us>
Cc: TIFF mailing list <tiff@lists.maptools.org>
Subject: Re: [Tiff] Excessive memory allocation while chopping strips
X-BeenThere: tiff@lists.maptools.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: TIFF and libtiff discussion <tiff.lists.maptools.org>
List-Unsubscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=unsubscribe>
List-Archive: <http://lists.maptools.org/mailman/private/tiff>
List-Post: <mailto:tiff@lists.maptools.org>
List-Help: <mailto:tiff-request@lists.maptools.org?subject=help>
List-Subscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=subscribe>
Content-Type: multipart/mixed; boundary="===============0227066561=="
Mime-version: 1.0
Sender: tiff-bounces@lists.maptools.org
Errors-To: tiff-bounces@lists.maptools.org

--===============0227066561==
Content-Type: multipart/alternative; boundary="001a114f74e8aa31bb055309ce0b"

--001a114f74e8aa31bb055309ce0b
Content-Type: text/plain; charset="UTF-8"

(long time no see :)

It makes sense to have hard limits in place where exceeding those would
cause undefined behaviour. (e.g. due to 32 bit overflow etc)

Furthermore some soft limits that would make systems e.g. extreme slow
(zillion allocs whatever) makes sense too but these
should be overrulable if people really want to. Possibly use #define for
the soft limits in a config file ?
In the end these limits will depend on the target system where the
application runs..


wrt the question:
Very large strips and or tiles are used by people who expect that it will
increase the performance of their application in some sense,
and yes there might be some gain.Most computers can handle MB easily (even
a .$35 RaspberryPi 3 can do serious processing).
64 GB of memory (on a multicore 64bit cpu) is not rare anymore in an age of
 "BIG" data (recall BIG Tiff came first ;)

For some types of devices row based processing is the natural way of
working. If I look e.g. at a wide format (around 60" or 1.5 mtr)
scanner at 2400dpi results in 144K pixels per row @ 4 bytes per pixel is
about 0.6 MB per row. It could have 300K rows (120" or 3mtr),
resulting in 200GB image files. Increasing the bitdepth to 16 bit per
channel brings us to 0.4 TB per image.

Stitching applications are also quite common and these can make extreme
large images in one or both dimensions. Large rows would
not be strange.

So in short, there is a need to support large images, which implies larges
tiles/stripes....

<thinking out loud modus started>
I dont think that a strip should be as big as the imagewidth for every
possible value. From some distance (abstract level) there are
only tiles (in extreme heigth == 1  && width == imagewidth  ==> single row).
If imagewidth is "too" large one could split that row in e.g. 4 tiles
(heigth == 1 && width == imagewidth/4 ) to be processable.

maybe all the lib algorithms should become tile based and should we
deprecate stripe based ones...
</>

my 2 cents,
Rob Tillaart



On Wed, Jun 28, 2017 at 5:42 PM, Bob Friesenhahn <
bfriesen@simple.dallas.tx.us> wrote:

> On Wed, 28 Jun 2017, Olivier Paquet wrote:
> >
> > I think you misunderstood me. I meant the problem that uncompressed
> > strip size can be quite large and, as far as I know, can't be
> > validated without decompressing the data. A hard limit on strip size
> > would cover that neatly and sane applications will keep it reasonable
> > anyway.
>
> I agree that implementing a hard limit on strip/tile size (including
> when writing) is beneficial.
>
> The question is what the justification for using very large
> strips/tiles and what is the largest justifiable large strip/tile
> size?
>
> For a very large image, if a strip is just one row, then it needs to
> be allowed to be large enough to support the maximum image width.
>
> Bob
> --
> Bob Friesenhahn
> bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
> GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
> _______________________________________________
> Tiff mailing list: Tiff@lists.maptools.org
> http://lists.maptools.org/mailman/listinfo/tiff
> http://www.remotesensing.org/libtiff/
>

--001a114f74e8aa31bb055309ce0b
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">(long time no see :)<div><br></div><div>It makes sense to =
have hard limits in place where exceeding those would cause undefined behav=
iour. (e.g. due to 32 bit overflow etc)</div><div><br></div><div>Furthermor=
e some soft limits that would make systems e.g. extreme slow (zillion alloc=
s whatever) makes sense too but these=C2=A0</div><div>should be overrulable=
 if people really want to. Possibly use #define for the soft limits in a co=
nfig file ?</div><div>In the end these limits will depend on the target sys=
tem where the application runs..</div><div><br></div><div><br></div><div>wr=
t the question:</div><div>Very large strips and or tiles are used by people=
 who expect that it will increase the performance of their application in s=
ome sense,</div><div>and yes there might be some gain.Most computers can ha=
ndle MB easily (even a .$35 RaspberryPi 3 can do serious processing).=C2=A0=
</div><div>64 GB of memory (on a multicore 64bit cpu) is not rare anymore i=
n an age of =C2=A0&quot;BIG&quot; data (recall BIG Tiff came first ;)</div>=
<div><br></div><div>For some types of devices row based processing is the n=
atural way of working. If I look e.g. at a wide format (around 60&quot; or =
1.5 mtr)=C2=A0</div><div>scanner at 2400dpi results in 144K pixels per row =
@ 4 bytes per pixel is about 0.6 MB per row. It could have 300K rows (120&q=
uot; or 3mtr),</div><div>resulting in 200GB image files. Increasing the bit=
depth to 16 bit per channel brings us to 0.4 TB per image.</div><div><br></=
div><div>Stitching applications are also quite common and these can make ex=
treme large images in one or both dimensions. Large rows would</div><div>no=
t be strange.=C2=A0</div><div><br></div><div>So in short, there is a need t=
o support large images, which implies larges tiles/stripes....</div><div><b=
r></div><div>&lt;thinking out loud modus started&gt;</div><div>I dont think=
 that a strip should be as big as the imagewidth for every possible value. =
From some distance (abstract level) there are=C2=A0</div><div>only tiles (i=
n extreme heigth =3D=3D 1 =C2=A0&amp;&amp; width =3D=3D imagewidth =C2=A0=
=3D=3D&gt; single row).</div><div>If imagewidth is &quot;too&quot; large on=
e could split that row in e.g. 4 tiles (heigth =3D=3D 1 &amp;&amp; width =
=3D=3D imagewidth/4 ) to be processable.</div><div><br></div><div>maybe all=
 the lib algorithms should become tile based and should we deprecate stripe=
 based ones...</div><div>&lt;/&gt;</div><div><br></div><div>my 2 cents,</di=
v><div>Rob Tillaart</div><div><br></div><div><br></div></div><div class=3D"=
gmail_extra"><br><div class=3D"gmail_quote">On Wed, Jun 28, 2017 at 5:42 PM=
, Bob Friesenhahn <span dir=3D"ltr">&lt;<a href=3D"mailto:bfriesen@simple.d=
allas.tx.us" target=3D"_blank">bfriesen@simple.dallas.tx.us</a>&gt;</span> =
wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bord=
er-left:1px #ccc solid;padding-left:1ex"><span class=3D"">On Wed, 28 Jun 20=
17, Olivier Paquet wrote:<br>
&gt;<br>
&gt; I think you misunderstood me. I meant the problem that uncompressed<br=
>
&gt; strip size can be quite large and, as far as I know, can&#39;t be<br>
&gt; validated without decompressing the data. A hard limit on strip size<b=
r>
&gt; would cover that neatly and sane applications will keep it reasonable<=
br>
&gt; anyway.<br>
<br>
</span>I agree that implementing a hard limit on strip/tile size (including=
<br>
when writing) is beneficial.<br>
<br>
The question is what the justification for using very large<br>
strips/tiles and what is the largest justifiable large strip/tile<br>
size?<br>
<br>
For a very large image, if a strip is just one row, then it needs to<br>
be allowed to be large enough to support the maximum image width.<br>
<span class=3D"HOEnZb"><font color=3D"#888888"><br>
Bob<br>
--<br>
Bob Friesenhahn<br>
<a href=3D"mailto:bfriesen@simple.dallas.tx.us">bfriesen@simple.dallas.tx.u=
s</a>, <a href=3D"http://www.simplesystems.org/users/bfriesen/" rel=3D"nore=
ferrer" target=3D"_blank">http://www.simplesystems.org/<wbr>users/bfriesen/=
</a><br>
GraphicsMagick Maintainer,=C2=A0 =C2=A0 <a href=3D"http://www.GraphicsMagic=
k.org/" rel=3D"noreferrer" target=3D"_blank">http://www.GraphicsMagick.org/=
</a><br>
</font></span><div class=3D"HOEnZb"><div class=3D"h5">_____________________=
_________<wbr>_________________<br>
Tiff mailing list: <a href=3D"mailto:Tiff@lists.maptools.org">Tiff@lists.ma=
ptools.org</a><br>
<a href=3D"http://lists.maptools.org/mailman/listinfo/tiff" rel=3D"noreferr=
er" target=3D"_blank">http://lists.maptools.org/<wbr>mailman/listinfo/tiff<=
/a><br>
<a href=3D"http://www.remotesensing.org/libtiff/" rel=3D"noreferrer" target=
=3D"_blank">http://www.remotesensing.org/<wbr>libtiff/</a><br>
</div></div></blockquote></div><br></div>

--001a114f74e8aa31bb055309ce0b--

--===============0227066561==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Tiff mailing list: Tiff@lists.maptools.org
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
--===============0227066561==--
