Return-Path: <tiff-admin@remotesensing.org>
Received: from inav008.isp.belgacom.be (inav008.isp.belgacom.be [195.238.2.78])
        by six.skynet.be (8.12.9/8.12.9/Skynet-MAILSTORE-2.14) with ESMTP id i2F2vU5M025726
        for <joris.at.lebbeke@skynet.be>; Mon, 15 Mar 2004 03:57:30 +0100
        (envelope-from <tiff-admin@remotesensing.org>)
Received: from inmx008.isp.belgacom.be (inmx008.isp.belgacom.be [195.238.3.88])
        by inav008.isp.belgacom.be (8.12.9/8.12.9/Skynet-IN-AV-2.02) with ESMTP id i2F2vMxo025291
        for <joris.at.lebbeke@skynet.be>; Mon, 15 Mar 2004 03:57:22 +0100
        (envelope-from <tiff-admin@remotesensing.org>)
Received: from yoda.imagelinks.com ([208.24.120.42])
        by inmx008.isp.belgacom.be (8.12.9/8.12.9/Skynet-IN-2.32) with ESMTP id i2F2vJaj032085
        for <joris.at.lebbeke@skynet.be>; Mon, 15 Mar 2004 03:57:20 +0100
        (envelope-from <tiff-admin@remotesensing.org>)
Received: from yoda.imagelinks.com (localhost.localdomain [127.0.0.1])
	by yoda.imagelinks.com (8.11.6/8.11.2) with ESMTP id i2F2uLl32338;
	Sun, 14 Mar 2004 21:56:21 -0500
Received: from outmx008.isp.belgacom.be (outmx008.isp.belgacom.be [195.238.3.235])
	by yoda.imagelinks.com (8.11.6/8.11.2) with ESMTP id i2F2tgl32318
	for <tiff@remotesensing.org>; Sun, 14 Mar 2004 21:55:43 -0500
Received: from dfgrrxsknr4e (153.123-136-217.adsl.skynet.be [217.136.123.153])
        by outmx008.isp.belgacom.be (8.12.9/8.12.9/Skynet-OUT-2.21) with SMTP id i2F2teS1027700
        for <tiff@remotesensing.org>; Mon, 15 Mar 2004 03:55:40 +0100
        (envelope-from <joris.at.lebbeke@skynet.be>)
Message-ID: <001001c40a39$19f05150$997b88d9@dfgrrxsknr4e>
From: "Joris" <joris.at.lebbeke@skynet.be>
To: "Tiff mailing list" <tiff@remotesensing.org>
References: <BAY1-DAV38i4u6LbosM00026558@hotmail.com>
Subject: Re: [Tiff] Tiff question
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2800.1158
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165
X-RAVMilter-Version: 8.4.3(snapshot 20030212) (inav008.isp.belgacom.be)
X-RAVMilter-Version: 8.4.3(snapshot 20030212) (outmx008.isp.belgacom.be)
Sender: tiff-admin@remotesensing.org
Errors-To: tiff-admin@remotesensing.org
X-BeenThere: tiff@remotesensing.org
X-Mailman-Version: 2.0.11
Precedence: bulk
List-Help: <mailto:tiff-request@remotesensing.org?subject=help>
List-Post: <mailto:tiff@remotesensing.org>
List-Subscribe: <http://remotesensing.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@remotesensing.org?subject=subscribe>
List-Id: TIFF and libtiff discussion <tiff.remotesensing.org>
List-Unsubscribe: <http://remotesensing.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@remotesensing.org?subject=unsubscribe>
List-Archive: <http://remotesensing.org/pipermail/tiff/>
Date: Mon, 15 Mar 2004 03:56:20 +0100
X-UIDL: 7cce84d0eec0ae6b7366512eff30c177

Arthur Schuchter wrote:
> I have a question . I have huge TIFF files and with my software
> I cant handle them, they are simply too large. I am a
> programmer as well, is it possible to cretae something to
> handle those files.

There's two distinct parts to this problem: the file, and the internal decoded
raster (object) representation of your imaging library. First part doesn't
really matter that much, though, of course it's simply sensible to store such
huge TIFF pages tiled. It's mainly the second part that is actually difficult,
and that really matters.

The problem with allocating/managing a single internal raster representation
that big is that one single continuous chunk of memory is needed that is way too
big for comfort. Such a chunck is a serious headache to the operating system, if
allocating it is at all possible, and causes constant swapping activity. That is
why this problem is best solved by splitting up this raster into more managable
parts. You could split it up in bands or in tiles. Bands are pieces of same
width as total raster, but limited height. Tiles are square or almost square
regions. Of these two techniques, tiling comes most recommended. Here's a
comparison of both techniques, that may also be helpfull in giving you a feel of
why exactly big images are a problem and why exactly splitting the raster is a
solution:

http://www.awaresystems.be/techtalk/techtalks/002_tiling_vs_banding.html

A good tiling solution can handle rasters of the size you've mentioned. It's
best implemented in a raster object, managing an array of tiles, that are in
turn arrays of pixel color data, solving your need from big images to the huge
ones from that point on. Such a raster object is best designed to hide this
complication to the world of code outside of the object. Thing is, of course,
this raster does not to be read or exposed at some stage. That need in
particular, I find, is best solved by a dedicated 'scanline stream object', that
kinda 'serves' the raster, line by line. Such a scanline stream object should be
viewed as a chain of individual steps. Next, it's only a matter of designing the
actual TIFF encoder/decoder to work as a step in such a chain, for instance.
This way, you can easily and independently solve stuff like putting together
TIFF strips/tiles of any size or decoding a single-strip TIFF on the one hand,
or writing out the result, line by line, into your raster tile array object, or
resampling, buffering only the minimum amount of lines needed (eg 2 in a linear
interpolation resampler), or filtering, or... etc. As all of these steps become
implemented as single independent steps you can link together in a 'scanline
stream' chain, reading and writing from and to tile array rasters, you've build
yourself an imaging library that withstands the big image problem.

This whole thing really is quite a bit of work. I can tell you that it's not a
dead end, but a definite solution, but I cannot guarantee that it's actually
worth the effort if you percieve this as an isolated need. In that case, hacking
together something that reads such a TIFF tile by tile, writing out each tile to
a destination TIFF file, as soon it is processed, and forgetting about it before
reading in the next tile, for example, may be more worth your while. Of course,
such a quick hack is no longer 'universal', depends on particular properties of
the particular files and the particular task and such, which may perhaps not be
a problem.



Joris Van Damme
info@awaresystems.be
http://www.awaresystems.be
Download your free TIFF tag viewer for windows here:
http://www.awaresystems.be/free_downloads/astifftools.html

_______________________________________________
Tiff mailing list
Tiff@remotesensing.org
http://remotesensing.org/mailman/listinfo/tiff
