From: warmerdam@pobox.com (Frank Warmerdam)
Date: Mon, 01 Sep 2003 16:02:30 -0400
Subject: [Tiff] Reading and writing to memory.
In-Reply-To: <20030901114922.GA4270@vital>
References: <20030901114922.GA4270@vital>
Message-ID: <3F53A5D6.2020202@pobox.com>

Väinö Järvelä wrote:
> Hello,
> 
> 
> My software has to manipulate some very large TIFF images. The images
> are so big that they won't fit in the memory uncompressed, so i need to
> have the images compressed in the memory and uncompress it strip at a
> time , manipulate and compress back to the memory. When the
> manipulations are complete, then write it to a file.
> 
> I know i could pump up the swap space, but it might be faster to do it
> all in memory even with all this uncompressing and compressing than
> writing and reading gigabytes of data from and to swap(done
> automatically by kernel ofcourse) and work from there.
> 
> I have tried using TIFFClientOpen by creating my own
> read,write,seek,size procs which pass a struct (through thandle_t) to
> the functions and manipulate the data(image buffer) in the struct. But
> when i try to "open" the memory with
> 
> Tiff = TIFFClientOpen( "dummy", "w", (thandle_t) &mt, tiffMemReadProc,
> tiffMemWriteProc, tiffMemSeekProc, tiffMemCloseProc, tiffMemSizeProc,
> NULL, NULL );
> 
> It fails by saying: "dummy: Not a TIFF file, bad magic number 0 (0x0)."
> 
> I don't undestand this error message.. i'm not trying to read or append
> (to) an old TIFF file, but it apparently still tries to read one?
> 
> 
> Is there anything specific i have to do in those procs? Return a
> specific value on the first run or something?

Vaino,

There are indeed fairly detailed semantics required of the IO functions.
I don't know that these assumptions are well documented anywhere.  You
might be best off reviewing existing low level implementations carefully
if you want to make your own.

There are already at least two memory based TIFF implementations for
libtiff.  in contrib/mfs there is one.  I also have one I use as part
of GDAL, based loosely on mfs.

I am not absolutely certain why you are getting the "dummy: Not a
TIFF file, bad magic number" message.  It may be that as currently
setup even a writable data stream needs to support read, and that you
are not allowing the tiff header to be read back by the code.

> Is mmap and unmap procs mandatory?

They must be supplied, but a simple "unsupported" implementation
is fine.

I can't help but wonder though if your overall approach is appropriate.
Why is it that you need to hold your whole image in memory?  Why not operate
on the image in chunks as Pushkar suggests?

Best regards,

-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent



