Return-Path: <awaresystems.be+caf_=filtered=awaresystems.be@gmail.com>
Delivered-To: filtered@awaresystems.be
Received: from mx9.public.one.com ([195.47.247.105])
	by mail6.local.one.com (Dovecot) with LMTP id A24YD24OmVhRCgEAsyOMew
	for <filtered@awaresystems.be>; Tue, 07 Feb 2017 00:05:01 +0000
X-HalOne-ID: 0fec319d-ecc9-11e6-855f-246e961730ec
Received: from mail-ot0-f170.google.com (unknown [74.125.82.170])
	by mx9.public.one.com (Halon) with ESMTPS
	id 0fec319d-ecc9-11e6-855f-246e961730ec;
	Tue, 07 Feb 2017 00:04:59 +0000 (UTC)
Received: by mail-ot0-f170.google.com with SMTP id f9so74481013otd.1
        for <filtered@awaresystems.be>; Mon, 06 Feb 2017 16:04:58 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-original-authentication-results:x-gm-message-state:delivered-to
         :date:from:to:in-reply-to:message-id:references:user-agent
         :mime-version:cc:subject:precedence:list-id:list-unsubscribe
         :list-archive:list-post:list-help:list-subscribe
         :content-transfer-encoding:sender:errors-to;
        bh=mh/Kq3bbnAicPUpQBfgHwreFTjyDDVwIUB3Z2QHPkUo=;
        b=ZxZ4BPal08z4B0CieTxKIxgd1az8x4x23Ap8osSE9EAU+VadMHQSl8bJg4meJgeJfm
         F9fykoj3uo0zVUQqmGF0+otuYCQNFUwz8EXAXkHsEvN0JZP5dfwEIzfnmSWX3SyJWcKi
         WZXAzo7Bd7QmZlVDj4n6xTKMz22554B//tccF9CcAC+Y/BJJTQdI5C37Q/T14EJKZgnD
         hxYWndzLbgbsyWMuk4UZoiF0xz/6olQ+hjuvQHMUf0rf4vD0+J20CUQguyhnEr0Avbqc
         dmgyzzKJODfx+ketHvUuIjoIoJLvACtzhWhGJ3agiBn7cPVP3WtUTcgmmmmLdza4R5In
         0bXg==
X-Original-Authentication-Results: mx.google.com;       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org
X-Gm-Message-State: AMke39lJVoDtjsZ5rLCU3y4J1d4cKEteXmksvG5U4zW7+6mbHR/tG3sOwDj2HDFZR7ekXIT5Hx4pb1wt5QrUockdT5Of0lA=
X-Received: by 10.157.56.70 with SMTP id r6mr5912285otd.154.1486425897617;
        Mon, 06 Feb 2017 16:04:57 -0800 (PST)
X-Forwarded-To: filtered@awaresystems.be
X-Forwarded-For: awaresystems.be@gmail.com filtered@awaresystems.be
Delivered-To: awaresystems.be@gmail.com
Received: by 10.202.175.141 with SMTP id y135csp1920021oie;
        Mon, 6 Feb 2017 16:04:56 -0800 (PST)
X-Received: by 10.28.111.78 with SMTP id k75mr11115024wmc.71.1486425896595;
        Mon, 06 Feb 2017 16:04:56 -0800 (PST)
Return-Path: <tiff-bounces@lists.maptools.org>
Received: from lists.maptools.org (ks365428.kimsufi.com. [94.23.4.142])
        by mx.google.com with ESMTP id 107si2815127wra.205.2017.02.06.16.04.56;
        Mon, 06 Feb 2017 16:04:56 -0800 (PST)
Received-SPF: neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) client-ip=94.23.4.142;
Authentication-Results: mx.google.com;
       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org
Received: from [10.1.0.241] (localhost.localdomain [127.0.0.1])
	by lists.maptools.org (Postfix) with ESMTP id 7BE0D35C02E;
	Mon,  6 Feb 2017 19:04:13 -0500 (EST)
Received: from smtp.simplesystems.org (smtp.simplesystems.org [65.66.246.90])
	by lists.maptools.org (Postfix) with ESMTPS id 9BBCC35C02C
	for <tiff@lists.maptools.org>; Mon,  6 Feb 2017 19:04:06 -0500 (EST)
Received: from freddy.simplesystems.org (freddy.simplesystems.org
	[65.66.246.65])
	by smtp.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id
	v17042M4027494; Mon, 6 Feb 2017 18:04:02 -0600 (CST)
Date: Mon, 6 Feb 2017 18:04:02 -0600 (CST)
From: Bob Friesenhahn <bfriesen@simple.dallas.tx.us>
X-X-Sender: bfriesen@freddy.simplesystems.org
To: Dinesh Iyer <dsi.maillist@gmail.com>
In-Reply-To: <CAHcSuBoRvQ9eA1OTRJ=Cy+aTJbUDCp+kutsQ72RuKNYVuF9Rdg@mail.gmail.com>
Message-ID: <alpine.GSO.2.20.1702061749190.10840@freddy.simplesystems.org>
References: <CAHcSuBoRvQ9eA1OTRJ=Cy+aTJbUDCp+kutsQ72RuKNYVuF9Rdg@mail.gmail.com>
User-Agent: Alpine 2.20 (GSO 67 2015-01-07)
MIME-Version: 1.0
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16
	(smtp.simplesystems.org [65.66.246.90]);
	Mon, 06 Feb 2017 18:04:02 -0600 (CST)
Cc: Tiff List <tiff@lists.maptools.org>
Subject: Re: [Tiff] Qs about support for more than 2^16 IFDs and writing
 performance
X-BeenThere: tiff@lists.maptools.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: TIFF and libtiff discussion <tiff.lists.maptools.org>
List-Unsubscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=unsubscribe>
List-Archive: <http://lists.maptools.org/mailman/private/tiff>
List-Post: <mailto:tiff@lists.maptools.org>
List-Help: <mailto:tiff-request@lists.maptools.org?subject=help>
List-Subscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=subscribe>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Sender: tiff-bounces@lists.maptools.org
Errors-To: tiff-bounces@lists.maptools.org

On Mon, 6 Feb 2017, Dinesh Iyer wrote:

> Hi everyone,
> I am developing an application that uses libTIFF for reading/writing TIFF
> files and I have the following questions about libTIFF. I apologize in
> advance if these questions have been asked but I could not find answers to
> them in the archive.
>
> 1. I noticed that libTIFF does not support reading images from TIFF files
> as IFD's greater than 65535. The TIFF specification does not appear to
> impose such a limit but libTIFF does. I have a few clients who have 100,000
> image TIFF stacks that they would like to be able to read.
>
> 2. I noticed that libTIFF does appear to support writing > 65536 images
> into a TIFF file. Is this the expected behaviour or an over-sight?

It is likely that the arbitrary limit is intended to foil looping IDFs 
(denial of service opportunity) while also limiting the performance 
and memory impact of the check. The linear scan is perhaps not the 
fastest algorithm but it should not take terribly long (much less than 
a second) to scan 65535 entries.

> 3. Lastly, when i tested writing a large number of images into a TIFF file,
> I found that the performance was very slow. It took ~2.5 hrs to write 66200
> images, each of size just 64x64. I came across this github post:
> https://github.com/escabe/libtiff/commit/58b4c3ba4478987ecfe1e793b9d925e59eecfa36
>
> about why the performance is poor and a possible fix for this. Is this
> patch recommended?

It looks like this patch also caches the IFD offset lists in the 
writer but also improves lookup time for the case where the offset 
searched for is the one just stored.

Perhaps an algorithm like a Bloom filter 
(https://en.wikipedia.org/wiki/Bloom_filter) could be used to improve 
efficiency, but I am not sure of how much code is required for an 
implementation.  The Bloom filter can have false positives so an 
exhaustive scan would sometimes be needed to eliminate false 
positives.

> Are there plans to limit the 65536 IFD restriction for TIFF files and also
> improve the writing performance anytime soon? Or is the recommendation to
> patch my local copy of the TIFF library?

The 65536 IFD restriction is likely arbitrary.

The most important thing is that have you profiled the code for your 
slow test case to see why it is so slow to write?  Are you able to 
post simple portable code for your test case so that others can 
observe the same issue that you are?

It is quite possible that the writing bottleneck is something other 
than this part of the code.

Bob
-- 
Bob Friesenhahn
bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
_______________________________________________
Tiff mailing list: Tiff@lists.maptools.org
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
