Return-Path: <tiff-bounces@lists.maptools.org>
X-Original-To: info@awaresystems.be
Delivered-To: info@awaresystems.be
Received: from mx-s.b-one.net (mx-s.one.com [91.198.169.250])
	by mail9.b-one.net (Postfix) with ESMTP id 78B78400343
	for <info@awaresystems.be>; Fri, 18 Feb 2011 23:10:01 +0100 (CET)
Received: by mx-s.b-one.net (Postfix, from userid 106)
	id 68EB7177C530; Fri, 18 Feb 2011 23:09:25 +0100 (CET)
X-Virus-Scanned: by one.com
X-Spam-Flag: NO
X-Spam-Score: 0
X-Spam-Level: 
X-Spam-Status: No, score=0 tagged_above=0 required=4 tests=[none]
X-Greylist: domain auto-whitelisted by SQLgrey-1.7.4
Received: from lists.maptools.org (lists.maptools.org [65.39.174.144]) by
 mx-s.b-one.net (Postfix) with ESMTP id 8B200177CE51 for
 <info@awaresystems.be>; Fri, 18 Feb 2011 23:09:24 +0100 (CET)
Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by
 lists.maptools.org (Postfix) with ESMTP id 8A9EA86BC9; Fri, 18 Feb 2011
 17:07:52 -0500 (EST)
Received: from sasl.smtp.pobox.com (a-pb-sasl-sd.pobox.com [64.74.157.62]) by
 lists.maptools.org (Postfix) with ESMTP id 73C6D86BDE for
 <tiff@lists.maptools.org>; Fri, 18 Feb 2011 17:06:59 -0500 (EST)
Received: from sasl.smtp.pobox.com (unknown [127.0.0.1]) by
 a-pb-sasl-sd.pobox.com (Postfix) with ESMTP id 5DD784785 for
 <tiff@lists.maptools.org>; Fri, 18 Feb 2011 17:08:07 -0500 (EST)
Received: from a-pb-sasl-sd.pobox.com (unknown [127.0.0.1]) by
 a-pb-sasl-sd.pobox.com (Postfix) with ESMTP id 5ADF94784 for
 <tiff@lists.maptools.org>; Fri, 18 Feb 2011 17:08:07 -0500 (EST)
Received: from [192.168.2.24] (unknown [69.156.78.30]) (using TLSv1 with
 cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested)
 by a-pb-sasl-sd.pobox.com (Postfix) with ESMTPSA id 2084C4783 for
 <tiff@lists.maptools.org>; Fri, 18 Feb 2011 17:08:06 -0500 (EST)
Message-ID: <4D5EED81.2090405@pobox.com>
Date: Fri, 18 Feb 2011 17:06:57 -0500
From: Frank Warmerdam <warmerdam@pobox.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13)
 Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: TIFF mailing list <tiff@lists.maptools.org>
X-Pobox-Relay-ID:
 90A3D78A-3BAB-11E0-8519-AF401E47CF6F-06493634!a-pb-sasl-sd.pobox.com
Subject: [Tiff] Deferring loading of strip/tile offsets and sizes
X-BeenThere: tiff@lists.maptools.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: TIFF and libtiff discussion <tiff.lists.maptools.org>
List-Unsubscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
 <mailto:tiff-request@lists.maptools.org?subject=unsubscribe>
List-Archive: <http://lists.maptools.org/mailman/private/tiff>
List-Post: <mailto:tiff@lists.maptools.org>
List-Help: <mailto:tiff-request@lists.maptools.org?subject=help>
List-Subscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
 <mailto:tiff-request@lists.maptools.org?subject=subscribe>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Sender: tiff-bounces@lists.maptools.org
Errors-To: tiff-bounces@lists.maptools.org

Folks,

In a recent performance analysis of the MapServer application with TIFF
files with overviews (each as a distinct directory) it was determinated that
a disproportionate amount of time was being spent loading the arrays of
offsets and sizes of tiles while reading directories.

In order to establish what overviews are available the applications reads
through all the directories in the file to get the sizes.  The file in
question is (at base) 131000 x 150000 and has nearly 900000 tiles.  With
64bit offsets and sizes that is around 14MB of reading just to get the
offsets and sizes for the tiles.

In order to optimize my application, I have incorporated a new feature into
libtiff (CVS head / aka libtiff 4.0).  This is a compile-time option to
defer loading the strip/tile offsets and sizes until such time as the are
actually needed to access imagery data.

This feature is enabled with the --enable-defer-strile-load configure
option which enables the DEFER_STRILE_LOAD macro.  Appropriate code is
surrounded by #if defined(DEFER_STRILE_LOAD) and it is my hope that there
is no risk when the feature is disabled.

When enabled, any part of libtiff that might need to use the offset/size
arrays from the directory call _TIFFFillStriles(TIFF*) in advance to ensure
the values are loaded.

This has dramatically improved performance for my particular application
in one use context.  I also think it could be helpful to others - particularly
when dealing with files that have:

a) a lot of tiles or strips - ie. large files.
b) multiple image directories
c) only needing to grab a bit of image data from one of the directories.

The change is now in libtiff CVS head.  I consider it experimental and
a bit risky so it is off by default, though if we get comfortable with it
at some point in the future it could be changed to always-on.

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent

_______________________________________________
Tiff mailing list: Tiff@lists.maptools.org
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
