Return-Path: <awaresystems.be+caf_=filtered=awaresystems.be@gmail.com>
Delivered-To: filtered@awaresystems.be
Received: from mx3.public.one.com ([10.246.18.23])
	by mail6.local.one.com (Dovecot) with LMTP id m39DD+WOVFkwLQEAsyOMew
	for <filtered@awaresystems.be>; Thu, 29 Jun 2017 06:14:34 +0000
X-HalOne-ID: 376c00f0-5c92-11e7-abe3-b82a72d83ab6
Received: from mail-oi0-f44.google.com (unknown [209.85.218.44])
	by mx3.public.one.com (Halon) with ESMTPS
	id 376c00f0-5c92-11e7-abe3-b82a72d83ab6;
	Thu, 29 Jun 2017 06:14:33 +0000 (UTC)
Received: by mail-oi0-f44.google.com with SMTP id p188so2959087oia.0
        for <filtered@awaresystems.be>; Wed, 28 Jun 2017 23:14:33 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-original-authentication-results:x-gm-message-state:delivered-to
         :message-id:date:from:user-agent:mime-version:to:references
         :in-reply-to:subject:precedence:list-id:list-unsubscribe
         :list-archive:list-post:list-help:list-subscribe
         :content-transfer-encoding:sender:errors-to;
        bh=KCZZ0TpRxgpffrOwD2+950lz46xc1M8iyl0JMLsvx+Y=;
        b=O0QeT0IMCWgpSYcsbt4tVfTrA+dUdM4ojj6HyEXFeBlaPezZZoAiNqUyBPWNkxOgVJ
         S6LQ60UP0MNyaTl1hscHHFRNV2JyuP0bhO4O155vVfK/nVuIBv5Ic7RJiMoI4e+QUKSr
         2f+8V6JsZMao7uBQ702e/r7QpFTwgqGrRPRyjOe/xrrZ6uhQWlbUJ2CahqPcYwTY0wDP
         +LDJRkflqJS90yMUvkU6/1dIQF5ThFyjLG/LmgfG0MxRTg64SozwTlfL6iS6+poGt628
         jBcXZ0XBRjiG6WBE0tpVMtgiPXqWF9Fqbi8nietnlNzT2dcolOIz0xwsKzS/Ahw4Ojxw
         idhw==
X-Original-Authentication-Results: mx.google.com;       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org
X-Gm-Message-State: AKS2vOz08G0vvxt1aI6JKHztwvftu6sHVmAqiloQ6jjewT4OiV+aGC6s
	NN6aPWscWORw/VWru3G54PEg3yO8+A98G9fnIGZKGA==
X-Received: by 10.202.0.65 with SMTP id 62mr8896768oia.86.1498716871768;
        Wed, 28 Jun 2017 23:14:31 -0700 (PDT)
X-Forwarded-To: filtered@awaresystems.be
X-Forwarded-For: awaresystems.be@gmail.com filtered@awaresystems.be
Delivered-To: awaresystems.be@gmail.com
Received: by 10.202.64.84 with SMTP id n81csp4312790oia;
        Wed, 28 Jun 2017 23:14:30 -0700 (PDT)
X-Received: by 10.28.172.4 with SMTP id v4mr488605wme.72.1498716870491;
        Wed, 28 Jun 2017 23:14:30 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; t=1498716870; cv=none;
        d=google.com; s=arc-20160816;
        b=YbtTpCu9EbuYW536lyR2egevl2YkX3xx3BHssuygt37fV1CfLiIKyK7P+4Ak82WJpF
         GgcNTMh6P5ej7P5YlywWb43l5XMThtZYyAl5EDb5Hcx89BjOvW7zCeqeqfsIfac+Uzbv
         hf+Pn+Xjc8EEXdFz364s8erWK0wyOfjYhbQARquyo1Er+FWBnoc5ghNeCbc5CHBvMX6Z
         AyirkAdIzRpehm+dvl1qO12XwM0uqGL4+eEHlOisAC/2jQdog7inALBA8DPCbIbzoQ9H
         ArGorPuYTpnjRJq9eGu2HeSrV2B2TEuIOos8ZeP6mQLmKUQOMovFbTXv4Iby6HCBKN8I
         4rEQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
        h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help
         :list-post:list-archive:list-unsubscribe:list-id:precedence:subject
         :in-reply-to:references:to:mime-version:user-agent:from:date
         :message-id:arc-authentication-results;
        bh=KCZZ0TpRxgpffrOwD2+950lz46xc1M8iyl0JMLsvx+Y=;
        b=QQyEECciiwQk0ziLY3o8QK/uL/m6ebltK8qSxiavY0Bp2rez0NahGIIyJE+TZyyXsQ
         r21+nmgs8lT4G2nEZ9umU3gLMopeVMjrwlKsbdcnxHDrbQ+8JzZPxuHN6PyLTfTRPQ36
         7uEXV1a2hgTUMvVMmC8gDgwittR0w/ct8tVa1QE1g6BYdMuPpDQEohm75RtP2w0D0ttg
         sBwq4RsXR6i+IbZeYdlicCX1W5EdzdqPHV9Qvq81sGJMERMVYTsot/cDhQjvvs5Tioov
         XybwJfwiPA809Cs2eRq1BUlQHJj67l+nBVmpdSpXbwwzPuVtTRrhgQIVt0YJKxG8zCBf
         dBBw==
ARC-Authentication-Results: i=1; mx.google.com;
       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org
Return-Path: <tiff-bounces@lists.maptools.org>
Received: from lists.maptools.org (ks365428.kimsufi.com. [94.23.4.142])
        by mx.google.com with ESMTP id p73si317876wmf.62.2017.06.28.23.14.28;
        Wed, 28 Jun 2017 23:14:30 -0700 (PDT)
Received-SPF: neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) client-ip=94.23.4.142;
Authentication-Results: mx.google.com;
       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org
Received: from [10.1.0.241] (localhost.localdomain [127.0.0.1])
	by lists.maptools.org (Postfix) with ESMTP id BBE1235C077;
	Thu, 29 Jun 2017 01:12:25 -0500 (EST)
Received: from polaris.osa.pri.ee (osa.pri.ee [213.219.84.188])
	by lists.maptools.org (Postfix) with ESMTP id CCE2B35C023
	for <tiff@lists.maptools.org>; Thu, 29 Jun 2017 00:53:13 -0500 (EST)
Received: from localhost (localhost [127.0.0.1])
	by polaris.osa.pri.ee (Postfix) with ESMTP id 46A7A6E69A
	for <tiff@lists.maptools.org>; Thu, 29 Jun 2017 08:53:12 +0300 (EEST)
X-Virus-Scanned: amavisd-new at osa.pri.ee
Received: from polaris.osa.pri.ee ([127.0.0.1])
	by localhost (osa.pri.ee [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id LqP315KKwg8T for <tiff@lists.maptools.org>;
	Thu, 29 Jun 2017 08:52:20 +0300 (EEST)
Received: from [192.168.1.208] (10-170-190-90.dyn.estpak.ee [90.190.170.10])
	(using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(Client did not present a certificate)
	by polaris.osa.pri.ee (Postfix) with ESMTPSA id 82D686E652
	for <tiff@lists.maptools.org>; Thu, 29 Jun 2017 08:52:20 +0300 (EEST)
Message-ID: <59549593.1020900@osa.pri.ee>
Date: Thu, 29 Jun 2017 08:52:19 +0300
From: Paavo Helde <paavo@osa.pri.ee>
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64;
	rv:25.8) Gecko/20151117 FossaMail/25.1.9
MIME-Version: 1.0
To: tiff@lists.maptools.org
References: <CABoQFUADrHiYLs7=2gVoRGkpex3P2jHWRPCZhehS8+OQRxTvYQ@mail.gmail.com>	<1588415.R2acFO4bS3@even-i700>	<CAAcuQeYOCExBMHb15EsYpUttGKyeJ0y=Xmqtf-nkUgM_o61GXQ@mail.gmail.com>	<1879895.yTvgkPFqHA@even-i700>	<CAAcuQeZgiBE9GvFyncYdHbqrW=sB8yPS0roicPKXBjvJvzbnWg@mail.gmail.com>	<alpine.GSO.2.20.1706281033400.6797@scrappy.simplesystems.org>	<CANoujwGk7-eagBqfrUTAaaGr1U8wTR3jHcgDRsvUvV1h7tz7cA@mail.gmail.com>
	<alpine.GSO.2.20.1706281529130.6797@scrappy.simplesystems.org>
In-Reply-To: <alpine.GSO.2.20.1706281529130.6797@scrappy.simplesystems.org>
Subject: Re: [Tiff] Excessive memory allocation while chopping strips
X-BeenThere: tiff@lists.maptools.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: TIFF and libtiff discussion <tiff.lists.maptools.org>
List-Unsubscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=unsubscribe>
List-Archive: <http://lists.maptools.org/mailman/private/tiff>
List-Post: <mailto:tiff@lists.maptools.org>
List-Help: <mailto:tiff-request@lists.maptools.org?subject=help>
List-Subscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=subscribe>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Sender: tiff-bounces@lists.maptools.org
Errors-To: tiff-bounces@lists.maptools.org


On 28.06.2017 23:37, Bob Friesenhahn wrote:
> On Wed, 28 Jun 2017, Rob Tillaart wrote:
>> wrt the question:
>> Very large strips and or tiles are used by people who expect that it will
>> increase the performance of their application in some sense,
>> and yes there might be some gain.Most computers can handle MB easily (even
> Even now, most (Intel) CPUs have L2 caches optimized for looping
> through no more than 256k data at a time.  Even if more data is read
> into RAM, the processing loops should usually not consume more than
> about 128k and performance definitely drops off once the working set
> exceeds 256k.

Yes, limiting the strip or tile size to some pretty large limit would be 
reasonable. However, people are not reasonable. Also, lots of processing 
software read the whole image into memory before any processing so the 
potential performance benefits from piecewise processing would be lost 
anyway.

Writing the whole image always in a single strip is also simpler then 
chopping it up to strips or tiles, so I am sure there are lots of 
software pieces out there which are writing single-strip TIFF files 
without using libtiff or any other library (this is actually pretty 
simple) and these strips are growing larger all the time because of 
hardware advancements. A fully reasonable 2000x2000x16bpp image would 
already be 8 MB and that doesn't stop there. The reading part (libtiff) 
should be lenient and allow for reading such files as long as they fit 
in memory.

Just my 2 euro cents
Paavo

_______________________________________________
Tiff mailing list: Tiff@lists.maptools.org
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
