Return-Path: <tiff-bounces@lists.maptools.org>
Received: from inas013.isp.belgacom.be (localhost [127.0.0.1])
        by inav005.isp.belgacom.be (8.12.11/8.12.11/Skynet-IN-AV-3.00) with ESMTP id jAN4BaVH026965
        for <joris.at.lebbeke@skynet.be>; Wed, 23 Nov 2005 05:11:36 +0100
        (envelope-from <tiff-bounces@lists.maptools.org>)
Received: from inmx009.isp.belgacom.be (inmx009.isp.belgacom.be [195.238.5.146])
        by inas013.isp.belgacom.be (8.12.11/8.12.11/Skynet-IN-AS-2.03) with ESMTP id jAN4BPQ0027465
        for <joris.at.lebbeke@skynet.be>; Wed, 23 Nov 2005 05:11:27 +0100
        (envelope-from <tiff-bounces@lists.maptools.org>)
Received: from duke.maptools.org (r153.dmsolutions.ca [209.217.116.153] (may be forged))
        by inmx009.isp.belgacom.be (8.12.11/8.12.11/Skynet-IN-2.32) with ESMTP id jAN4BN7i031826
        for <joris.at.lebbeke@skynet.be>; Wed, 23 Nov 2005 05:11:23 +0100
        (envelope-from <tiff-bounces@lists.maptools.org>)
Received: from duke.maptools.org (duke.maptools.org [127.0.0.1])
	by duke.maptools.org (8.12.11/8.12.10) with ESMTP id jAN46otI003453;
	Tue, 22 Nov 2005 23:06:55 -0500
Received: from wproxy.gmail.com (wproxy.gmail.com [64.233.184.200])
	by duke.maptools.org (8.12.11/8.12.10) with ESMTP id jAN46l4O003450
	for <tiff@lists.maptools.org>; Tue, 22 Nov 2005 23:06:47 -0500
Received: by wproxy.gmail.com with SMTP id i34so807202wra
	for <tiff@lists.maptools.org>; Tue, 22 Nov 2005 20:06:44 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com;
	h=received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
	b=N0VSp+vIyLw09pPE8ZAPTYbzedyfbP2MgdgE19TPfHSFzxyaBkeAuZxN5gWTdo/7r+IwfsuGoW3RH8d/slRca0fERZHw4cdcCmu3DMBFJQOERJsxAHKDjTrn9UT8TOowlJ+VmlYyDfuzvlb31ubMC0WAGu0tcMRLXBqCrAkzcgw=
Received: by 10.64.27.6 with SMTP id a6mr5488018qba;
	Tue, 22 Nov 2005 20:06:44 -0800 (PST)
Received: by 10.65.150.12 with HTTP; Tue, 22 Nov 2005 20:06:44 -0800 (PST)
Message-ID: <931f8ea90511222006n2a139c0bva296a18a40c2ff24@mail.gmail.com>
Date: Tue, 22 Nov 2005 23:06:44 -0500
From: Frank Warmerdam <warmerdam@pobox.com>
To: Joris <joris.at.lebbeke@skynet.be>
Subject: Re: [Tiff] libtiff and over 4GB files
In-Reply-To: <000a01c5efdd$5c737ad0$b9bcf551@dfgrrxsknr4e>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Disposition: inline
References: <931f8ea90511221832p1117fa67n66fcb210199c60b4@mail.gmail.com>
	<000a01c5efdd$5c737ad0$b9bcf551@dfgrrxsknr4e>
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by duke.maptools.org id
	jAN46l4O003450
Cc: TIFF <tiff@lists.maptools.org>
X-BeenThere: tiff@lists.maptools.org
X-Mailman-Version: 2.1.4
Precedence: list
List-Id: TIFF and libtiff discussion <tiff.lists.maptools.org>
List-Unsubscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=unsubscribe>
List-Archive: <http://lists.maptools.org/mailman/private/tiff>
List-Post: <mailto:tiff@lists.maptools.org>
List-Help: <mailto:tiff-request@lists.maptools.org?subject=help>
List-Subscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=subscribe>
Sender: tiff-bounces@lists.maptools.org
Errors-To: tiff-bounces@lists.maptools.org
X-UIDL: 000000000023ceae.00000e67
X-MSP-STORE: Wed, 23 Nov 2005 05:11:40 +0100
Status:   

On 11/22/05, Joris <joris.at.lebbeke@skynet.be> wrote:
> There's writing beyond 4 gig, and there's writing beyond 4 gig. Let's
> ignore current wrapping back behaviour, and discuss what is actual
> desired behaviour. Is a classic TIFF truelly limited to 4 gig to begin
> with?
>
> In theory, the stuff that makes the 4 gig boundary, is the fact that
> offsets are limited to 32 bits. But, from a file format point of view,
> one could argue that not every byte needs a offset pointing to it. For
> example, if a TIFF has a tile of 1 gig compressed space, and this tile
> is written as the very last datablock, then the highest offset used
> inside the TIFF is 1 gig less then the size of TIFF. Thus,
> theoretically, such a TIFF could grow to 5 gig in size.

Joris,

As you note, there are various potentially legal TIFF files
somewhat larger than 4GB.  I personally would be in
favor of preventing libtiff from writing such files, though I
wouldn't necessarily want to preclude it from reading them.
I could also live with us just ensuring that we don't write
files with any invalid offsets.

I would add that since libtiff normally writes the directory
*after* writing the imagery, it is somewhat hard to construct
a TIFF file with libtiff that has imagery going past the 4GB
boundary since when the directory is written past the end of
that it will be all screwed up.  It is, in theory, possible, if you
flush the directory to disk before writing imagery (and then
refresh the offsets).

> > Does anyone depend on libtiff being able to write a bit
> > past the 4GB mark?
>
> If I remember correctly, there was a user reporting on the list writing
> files > 4 gig and not being able to read them back not too long ago. I
> seem to also remember this was caused by current wrapping back
> behaviour, thus causing LibTiff to override previously written file
> header and all...

Right, and this is what I see happening frequently in the GDAL
user community.  The current lack of checking is resulting in the
generation of lots of corrupt files, without users realizing why.

Several years ago when I was at PCI I tried to make the application
level code calling libtiff recognise in advance if the file was going to
be too large, but it is very hard to do this reliably, especially when
compression is being used.  That is why I am now promoting the
idea of fixing this right in libtiff.

> Are there situations where you think you can safely recommend users to
> depend on libtiff currently being able to write 'a bit' past the 4 gig
> mark? Can you define 'a bit' such that it makes operational sense to the
> LibTiff user, and not just from a file format point of view? (These
> questions are not rethorical, I do not know the answer.)

Well, we could allow writing a bit past the 4GB mark by just
checking that offsets are valid, but I don't see this as a particularly
valuable behavior.  I was hoping that no one would speak up in
favor of this need so I could just strictly limit libtiff to 4GB files.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent

_______________________________________________
Tiff mailing list: Tiff@lists.maptools.org
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
