Replied: Fri, 04 Jun 1999 09:32:36 +0200
Replied: rvdt
Replied: Frank Warmerdam <warmerda@home.com>
Replied: "Forsberg, Bruce" <Bruce.Forsberg@marconi-is.com>
Replied: TIFF Mailing List <tiff@olympiakos.com>
Received: by smtp02.oce.nl (4.1/SMI-4.1)
	id AA28923; Fri, 4 Jun 99 06:10:05 +0200
Received: from gate7.olympiakos.com ([207.113.13.68])
	by oce.nl (8.9.0/8.9.0) with ESMTP id FAA26882;
	Fri, 4 Jun 1999 05:46:33 +0200 (MET DST)
Received: from localhost (bin@localhost)
	by gate7.olympiakos.com (8.8.7/8.8.7) with SMTP id XAA08314;
	Thu, 3 Jun 1999 23:45:42 -0400
Received: by gate7.olympiakos.com (bulk_mailer v1.9); Thu, 3 Jun 1999 23:45:20 -0400
Received: (from majordom@localhost)
	by gate7.olympiakos.com (8.8.7/8.8.7) id XAA08301
	for tiff-outgoing; Thu, 3 Jun 1999 23:45:19 -0400
X-Authentication-Warning: gate7.olympiakos.com: majordom set sender to owner-tiff@olympiakos.com using -f
Received: from mail.rdc1.on.home.com (imail@ha1.rdc1.on.wave.home.com [24.2.9.66])
	by gate7.olympiakos.com (8.8.7/8.8.7) with ESMTP id XAA08298
	for <tiff@olympiakos.com>; Thu, 3 Jun 1999 23:45:15 -0400
Received: from home.com ([24.65.34.185]) by mail.rdc1.on.home.com
          (InterMail v4.01.01.00 201-229-111) with ESMTP
          id <19990604034511.NJND23601.mail.rdc1.on.home.com@home.com>;
          Thu, 3 Jun 1999 20:45:11 -0700
Message-Id: <37574BDB.6367B7B5@home.com>
Date: Thu, 03 Jun 1999 23:45:31 -0400
From: Frank Warmerdam <warmerda@home.com>
X-Mailer: Mozilla 4.51 [en] (X11; U; Linux 2.0.27 i586)
X-Accept-Language: en
Mime-Version: 1.0
To: "Forsberg, Bruce" <Bruce.Forsberg@marconi-is.com>,
        TIFF Mailing List <tiff@olympiakos.com>
Subject: Re: Large File Support
References: <E130F8876BE8D211A2B90008C7CBFB48A776E9@usexch01.sd.gd.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Sender: owner-tiff@olympiakos.com

"Forsberg, Bruce" wrote:
> 
> I am investigating for our project what file formats support large
> images (ie > 2GB). I am not a TIFF expert but noticed that the TIFF
> 6.0 spec has a field in the Directory Entry structure called "count".
> Which is 4 bytes unsigned. This would seem to limit TIFF to an image
> size to something less than 4 GB. Is this correct?
> 
> For our UNIX machines (HP, SGI, SUN) we are compiling 32 bit applications
> and using the transitional API for accessing files > 2GB defined by
> the Large File Summit. Most of these are just the same calls with 64
> added on (ie open is open64). I noticed that the tiff library does not
> use these. Will tiff be modified in the future for this or is 2GB the
> current limit of tiff for 32 bit compiles?
> 
> I am new to this mail list so if this has been discussed before I am sorry.
> I tried to retrieve an archive from the mail list but it never responded.

Bruce, and the list,

My understanding is that the 4GB limit is primarily imposed by the use
of 32bit offsets for the beginning of tile/strip data chunks within the
file itself.  As I see it, the fix for this is 64 bit offsets and this will
be a major compatibility break.

Is there even an Adobe (or defacto) approved approach to this? 

In my opinion:

 o Agreement should be reached on a way of handling 64bit offsets in TIFF
   files.

 o libtiff should be modified to support reading of 64bit files. 

 o libtiff should be modified to be able to generate 64bit files.  Of course
   it should not do that by default.

Even with this approach it will be ages between 64bit files are widely
supported, but at least if we can get a 64bit toolkit out there the process
would begin. 

Bruce, I presume you are with the photogrammetry group at Marconi?  Within the
remote sensing, and photogrammetry community the 4GB limit is becoming
significant already.  I might be able to get funding from a government
agency, or software vendor to extend libtiff if there was general agreement
on an approach. 

I can think of a few approachs.  One is to add a TIFF_LONGLONG type
to the list of data types, and this would be a 64bit integer.  The 
TIFFTAG_STRIPOFFSETS and TIFFTAG_TILEOFFSETS could optionally be of this type.

A full 64 bit solution would also allow any other file offset or data block
size to be 64bit, but I think that is overkill.  On the other hand, it would
avoid problems with treating some offset specially, and if we are taking the
64bit plunge maybe it should be done comprehensively. 

A third approach would be to add new TIFFTAG_TILEOFFSET32 and 
TIFFTAG_STRIPOFFSET32.  If found, the additional 32bit number would be a 
4GB page on which the chunk starts.  Put together with the normal offset
it gives a 64bit address.  Files smaller than 4GB, but generated with 
64bit offset handling would still be compatible with older applications
since they would just ignore the extra tags.

Any opinion on the approaches?  Is this a solved problem, but just not 
incorporated into libtiff?

Best regards,

---------------------------------------+--------------------------------------
I set the clouds in motion - turned up | Frank Warmerdam, warmerda@home.com
light and sound - activate the windows | http://members.home.com/warmerda
and watch the world go round - Rush    | Geospatial Programmer for Rent
