Return-Path: <tiff-bounces@remotesensing.org>
Received: from inmx009.isp.belgacom.be (inmx009.isp.belgacom.be [195.238.3.146])
        by mailst035.isp.belgacom.be (8.12.11/8.12.11/Skynet-MAILSTORE-2.14) with ESMTP id j0L9XLt2027407
        for <joris.at.lebbeke@skynet.be>; Fri, 21 Jan 2005 10:33:21 +0100
        (envelope-from <tiff-bounces@remotesensing.org>)
Received: from xserve.flids.com (adsl-065-083-169-202.sip.mco.bellsouth.net [65.83.169.202])
        by inmx009.isp.belgacom.be (8.12.11/8.12.11/Skynet-IN-2.32) with ESMTP id j0L9WwhX032146
        for <joris.at.lebbeke@skynet.be>; Fri, 21 Jan 2005 10:33:01 +0100
        (envelope-from <tiff-bounces@remotesensing.org>)
Received: from adsl-065-083-169-202.sip.mco.bellsouth.net (localhost [127.0.0.1])
	by xserve.flids.com (Postfix) with ESMTP
	id 7FEB65D7819; Fri, 21 Jan 2005 04:32:26 -0500 (EST)
X-Original-To: tiff@remotesensing.org
Delivered-To: tiff@remotesensing.org
Received: from mailout06.sul.t-online.com (mailout06.sul.t-online.com
	[194.25.134.19])
	by xserve.flids.com (Postfix) with ESMTP id 30B525D780A
	for <tiff@remotesensing.org>; Fri, 21 Jan 2005 04:32:24 -0500 (EST)
Received: from fwd03.aul.t-online.de by mailout06.sul.t-online.com with smtp 
	id 1Crv9S-0001V1-04; Fri, 21 Jan 2005 10:32:22 +0100
Received: from MAGNUM
	(Tn7M1+ZXweIt+N5O1d2adqk1FzOez9sRJNp9kZ5ULvrTROxZQfOCso@[217.245.160.54])
	by fmrl03.sul.t-online.com
	with smtp id 1Crv9K-0VzAie0; Fri, 21 Jan 2005 10:32:14 +0100
From: Sherlog <sherlog@t-online.de>
To: tiff@remotesensing.org
Subject: Re: [Tiff] tiff question - performance on large files
Date: Fri, 21 Jan 2005 10:32:19 +0100
X-Meow: Zrbj!
Message-ID: <41f0c6d8$cee8c0f@meow-labs.de>
References: <OF73E7A20B.7273FEE2-ONC2256F7D.002FFE01@amat.com>
	<931f8ea905010306256bf66e5b@mail.gmail.com>
In-Reply-To: <931f8ea905010306256bf66e5b@mail.gmail.com>
Organization: SNAFU Labs in Radis, Germany
X-No-Archive: yes
X-Mailer: Forte Agent 1.92/32.572:085  /\/\eo\/\/
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
X-ID: Tn7M1+ZXweIt+N5O1d2adqk1FzOez9sRJNp9kZ5ULvrTROxZQfOCso@t-dialin.net
X-TOI-MSGID: 668e27d6-767d-43d3-a6e3-2f5cd7d43b30
X-BeenThere: tiff@remotesensing.org
X-Mailman-Version: 2.1.2
Precedence: list
List-Id: TIFF and libtiff discussion  <tiff.remotesensing.org>
List-Unsubscribe: <http://xserve.flids.com/mailman/listinfo/tiff>,
	<mailto:tiff-request@remotesensing.org?subject=unsubscribe>
List-Archive: <http://xserve.flids.com/pipermail/tiff>
List-Post: <mailto:tiff@remotesensing.org>
List-Help: <mailto:tiff-request@remotesensing.org?subject=help>
List-Subscribe: <http://xserve.flids.com/mailman/listinfo/tiff>,
	<mailto:tiff-request@remotesensing.org?subject=subscribe>
Sender: tiff-bounces@remotesensing.org
Errors-To: tiff-bounces@remotesensing.org
X-UIDL: 7ec7dd78b2301f2d36dac28485791c7d

Frank Warmerdam in <931f8ea905010306256bf66e5b@mail.gmail.com>:

> On Sun, 2 Jan 2005 10:55:23 +0200, Natalya_Katz@amat.com
>
> > problem is the poor performance in case of very large files
...
> I suspect the problem is due to the way libtiff just "walks" the list
> of images when doing various kinds of operations rather than 
> maintaining a master in-memory list of all availalbe image directories.

I think it is the same problem with quite a bunch of image format packages.
Many of them seem to have been designed with single-image files in mind and
hacked for multi-page TIFF support afterwards (not libtiff, of course <g>).
Some - like LEAD tools - even re-open the file and close it again for reading
field values of image pages that have just been retrieved (that is, one
open-walk-close to retrieve a certain image page, one open-walk-close for
each field value requested). This can add up to considerable amounts of time
for files containing hundreds or even thousands of pages. 

Add the brand-new network file handling b0rkenness in WinXP SP2 - which
basically adds a semi-random delay of about 0.5 s to closing *large* network
files - and you are looking at something like half an hour for what should
take less than 5 seconds if you consider only network throughput and latency
(and which does take less than 5 seconds if you load the file into memory and
process it from there). Consider LEAD tools: reading image pages and two
fields per page -> 1000 * (1 + 1 + 1) == 3000 file closes -> ~1500 seconds. I
sh*t you not, this happened to our production system when we added a couple
of WinXP SP2 boxen to help with the workload.

Of course, libtiff does not have the SP2 problem but it still suffers the
consequences of TIFF files being simple linked lists (or, lately, trees)
without global directory information. Accessing IFD <n> requires you to parse
the <n-1> preceding IFDs and this can cause a lot of data to be pulled into
OS FS caches, apart from the latency if network access is involved. For the
past couple of years we avoided the problem by the simple expedient of
storing single-page TIFFs in memo fields of data base tables, but for various
reasons this will no longer do.

I have seen some workarounds, which basically boil down to 'add fake
out-of-line TIFF field at the end of the file that points to the last IFD'.
This ensures constant-time append performance regardless of the number of
pages already in the file, but of course this violates the rule that TIFF
fields except for strip/tile offset and SubIFD (and old JPEG <g>) must not
contain file offsets.

I think a better solution is to keep the trailer out of the TIFF structure
entirely. From the POV of a TIFF reader it would not exist at all and from
the POV of a thorough TIFF structure checker (does such a beast exist?) it
would appear as some unaccounted-for space at the end of the file. So,
structurally the file would be sound regardless of what's written at the end
of the file since slack space content is not part of the specification. 

The only drawbacks I can see are these:
(1) the space is wasted if an unaware tool appends a new page to the file
(2) the data will become stale or even corrupted if an unaware tool modifies
    pages within the file in a manner that leaves the file size unchanged

Regarding (2), if the trailer says that page <k> is located at file offset
<o> and we find something looking like a valid IFD at this file offset this
does not mean that we do indeed have a valid IFD. It may be a stale copy that
is no longer part of the IFD chain because a tool rewrote the IFD for that
page at another place within the file or simply deleted (unlinked) it. To a
certain degree this can be worked around by including the file's last
modified time in the trailer and careful choice of file open sharing modes
(to be sure that no other process is currently modifying the file because the
time stamp may only get updated at file close).

As far as I can see there are two compatible motivations for a trailer:
- locating the last IFD quickly in order to get constant append performance
- locating any IFD quickly (by page index) in order to get constant retrieval
  performance

I think if people are going to write such trailers - they are doing so
already - it might be desirable to find a consensus as to how exactly go
about it. As far as I can see the trailer should contain
- a magic
- the total page (IFD) count of the 'main' chain
- an array of offsets for the main IFDs
- perhaps the file's last modified time when the trailer was generated

The last item is a bit thorny (low filetime resolution on UN*X, even lower on
FAT, time warping twice a year under Win32, local time vs. UTC and so on) but
in a business process this can work, even without the time thingy.

Is there any interest in such schemes here on this list? I mean schemes that
work when opening a multi-page TIFF 'cold'. If the chain of IFDs has been
walked already then the software *should* have remembered the offset of the
last IFD so that the next IFD offset can be patched in, which gives constant
peformance for repeated appends without any structural tricks like guerilla
trailers.

P.S.: apologies to Frank Warmerdam - I accidentally flung this post into his
inbox instead of posting it to the list. Mea culpa. In the meantime I have
found that newer versions of the LEAD tools allow the user to pass the file
offset of a 'base' IFD when accessing pages in a multipage file. This means
processing multiple pages in a big multi-page TIFF is O(n), instead of
O(n**2) as with the unhinted approach. I think such an approach would be
reasonably easy to integrate into libtiff, and it would remove a class of
performance problems without requiring things like trailer records.

Of course, the same speedup is possible even if the TIFF-reading library is
uncooperative: extract the data for the page under consideration from the
multi-page TIFF, repackage as a single-page TIFF, and then hand that to the
uncooperative reader (e.g. libtiff). This is reasonably easy (we have been
using such a scheme successfully for more than seven years) but it requires
knowledge of TIFF fields where in addition to the usual TIFF field logic the
*values* also need to be recalculated, e.g. StripOffsets, TileOffsets and so
on. 

Regards,
Sherlog
_______________________________________________
Tiff mailing list
Tiff@remotesensing.org
http://xserve.flids.com/mailman/listinfo/tiff
