Return-Path: <bfriesen@simple.dallas.tx.us>
Received: from inmx015.isp.belgacom.be (inmx015.isp.belgacom.be [195.238.2.218])
        by mailst034.isp.belgacom.be (8.12.11/8.12.11/Skynet-MAILSTORE-2.14) with ESMTP id i8H20fw3000740
        for <joris.at.lebbeke@skynet.be>; Fri, 17 Sep 2004 04:00:41 +0200
        (envelope-from <bfriesen@simple.dallas.tx.us>)
Received: from mta208-rme.xtra.co.nz (mta208-rme.xtra.co.nz [210.86.15.119])
        by inmx015.isp.belgacom.be (8.12.11/8.12.11/Skynet-IN-2.32) with ESMTP id i8H20YLk004228
        for <joris.at.lebbeke@skynet.be>; Fri, 17 Sep 2004 04:00:35 +0200
        (envelope-from <bfriesen@simple.dallas.tx.us>)
Received: from mta1-rme.xtra.co.nz ([210.86.15.157])
          by mta208-rme.xtra.co.nz with ESMTP
          id <20040917020033.DZTL13497.mta208-rme.xtra.co.nz@mta1-rme.xtra.co.nz>;
          Fri, 17 Sep 2004 14:00:33 +1200
Received: from zeus.concept.co.nz ([219.88.64.156]) by mta1-rme.xtra.co.nz
          with ESMTP
          id <20040917020033.LUH56.mta1-rme.xtra.co.nz@zeus.concept.co.nz>;
          Fri, 17 Sep 2004 14:00:33 +1200
Received: from zeus.concept.co.nz (zeus [10.0.0.250]) by zeus.concept.co.nz with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13)
	id SLFF13W3; Fri, 17 Sep 2004 14:00:22 +1200
Received: by zeus.concept.co.nz (Microsoft Exchange Connector for POP3 Mailboxes 4.50.2113) with SMTP (Global POP3 Download)
	 id MSG09172004-140017-42207.MMD@concept.co.nz; Fri, 17 Sep 2004 14:00:17 +1200
Delivered-To: computerconcepts-co-nz-martin@computerconcepts.co.nz
X-Envelope-To: <martin@computerconcepts.co.nz>
Received: (qmail 24578 invoked by uid 408); 17 Sep 2004 13:55:01 +1200
Received: from tiff-bounces@remotesensing.org by seraph.iserve.net.nz by uid 401 with qmail-scanner-1.22 
 (clamdscan: 0.71.  Clear:RC:0(65.83.169.202):. 
 Processed in 0.852446 secs); 17 Sep 2004 01:55:01 -0000
Received: from unknown (HELO xserve.flids.com) (65.83.169.202)
  by mx1-6.iserve.net.nz with SMTP; 17 Sep 2004 13:55:00 +1200
Received: from adsl-065-083-169-202.sip.mco.bellsouth.net (localhost [127.0.0.1])
	by xserve.flids.com (Postfix) with ESMTP
	id 192202F211A; Thu, 16 Sep 2004 21:54:33 -0400 (EDT)
X-Original-To: tiff@remotesensing.org
Delivered-To: tiff@remotesensing.org
Received: from scooby.simplesystems.org (scooby.simplesystems.org
	[65.66.246.67])
	by xserve.flids.com (Postfix) with ESMTP id 6C5012F20F6
	for <tiff@remotesensing.org>; Thu, 16 Sep 2004 21:54:30 -0400 (EDT)
Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74])
	by scooby.simplesystems.org (8.12.10+Sun/8.12.10) with ESMTP id
	i8H1sT9p026914; Thu, 16 Sep 2004 20:54:29 -0500 (CDT)
Date: Thu, 16 Sep 2004 20:54:29 -0500 (CDT)
From: Bob Friesenhahn <bfriesen@simple.dallas.tx.us>
X-X-Sender: bfriesen@blade.simplesystems.org
To: Joris <joris.at.lebbeke@skynet.be>
Subject: Re: [Tiff] BigTIFF Tag Value Count issue
In-Reply-To: <001401c49c4d$bff536f0$379688d9@dfgrrxsknr4e>
Message-ID: <Pine.SOC.4.60.0409162021080.15634@blade.simplesystems.org>
References: <001c01c49c21$2bf33730$c325c850@dfgrrxsknr4e>
	<Pine.SOC.4.60.0409161905220.15634@blade.simplesystems.org>
	<001401c49c4d$bff536f0$379688d9@dfgrrxsknr4e>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
Cc: Tiff mailing list <tiff@remotesensing.org>
X-BeenThere: tiff@remotesensing.org
X-Mailman-Version: 2.1.2
Precedence: list
List-Id: TIFF and libtiff discussion  <tiff.remotesensing.org>
List-Unsubscribe: <http://xserve.flids.com/mailman/listinfo/tiff>,
	<mailto:tiff-request@remotesensing.org?subject=unsubscribe>
List-Archive: <http://xserve.flids.com/pipermail/tiff>
List-Post: <mailto:tiff@remotesensing.org>
List-Help: <mailto:tiff-request@remotesensing.org?subject=help>
List-Subscribe: <http://xserve.flids.com/mailman/listinfo/tiff>,
	<mailto:tiff-request@remotesensing.org?subject=subscribe>
Sender: tiff-bounces@remotesensing.org
Errors-To: tiff-bounces@remotesensing.org
X-Spam-Virus-Prob: 0.0001
X-Spam-Classification: good
X-Spam-Probability: 0.1291
X-Spam-Server: d2s.iserve.net.nz
X-Spam-File: conmail@pop3.iserve.net.nz/good/8671438012d414e
X-UIDL: 6762a3a4a155330b19f5b88cb1762610

On Fri, 17 Sep 2004, Joris wrote:
>
> Partially true. Well, completely true, I guess, if memory mapping is involved.
> However, memory mapping of files of a format that is specifically designed to be
>> 4 gig, might not be a good idea. In windows, you can have a 'window' of a file

Stop that!  We are discussing BigTIFF here!!! :-)

There was a time when 640K seemed like a hard limit but that time is 
long past.  Yesterday 4GB seemed like a hard limit but now 64-bit PCs 
can be purchased for under $1000.  In a couple of years, most PCs will 
come with 4GB of RAM.  Years later people may be cursing the 
inventors of BigTIFF because they only thought about problems based on 
past experiences and didn't think ahead.

> memory mapped, called 'view' IIRC, limiting stress on address space, but this
> too, will not be a good idea in the BigTIFF case, since TIFF is inherently
> completely random access and so with this scheme you might have to swap to
> another 'window' for just about every new read.

TIFF may be random access but it can hardly be described as 
"completely random access".  Raster image pixel data is usually 
ordered in a contiguous fashion.

64 bit CPUs are readily able to memory map huge files, limited more by 
the design of the MMU than anything else.

> As a sidenote, slightly of topic as to this particular BigTIFF issue, but on
> topic as to the memory mapping... I've never quite understood why anyone would
> want to memory map a TIFF file for use in eg LibTiff. I believe that there is no
> speed or system advantage. The main advantage of memory mapping, I believe, is
> merely the convenience of accessing memory instead of having to do IO all the
> time, which is a bit less troublesome code logic... but, if your codec is build
> to accommodate all this IO, why would one next essentially wire IO routines to
> accommodate memory mapping?

With most modern operating systems, memory-mapped accesses are 
significantly faster than stdio-based or read/write access, 
particularly if the data is accessed more than once.  The limitation 
on this is how effective the system is at forgetting mapped pages 
which are no longer needed anymore.  If unneeded pages are not purged 
when they should be the system tends to choke and "thrash" once all 
physical memory has been consumed.  Most OSs (even Windows) provide a 
call to tell the system that certain pages are no longer required.  I 
have done quite a bit of investigation and benchmarking in this area 
and have learned that some OSs do tremendously better than others. 
Linux was among those that fared the worst.

> Or am I wrong in thinking there is no actual speed/system/cache/management
> advantage?

There is always an advantage when the natural filesystem caching is 
used directly and data copies and function calls are minimized.

> Option 0: stick with 4 byte tag count members (alignment, tag data <= 4 gig,
> Frank's choice)
> Option 1: make it an 8 byte tag count (no alignment, tag data > 4 gig allowed)
> Option 2, courtesy of Bob: make it 8 byte tag count, add 4 padding bytes
> (alignment, tag data > 4 gig allowed)
>
> Can I put you down as an option 2 vote, or were you merely signalling the
> option?

Sure, there should always be a second option so put me down for option 
2. :-)

Bob
======================================
Bob Friesenhahn
bfriesen@simple.dallas.tx.us
http://www.simplesystems.org/users/bfriesen
_______________________________________________
Tiff mailing list
Tiff@remotesensing.org
http://xserve.flids.com/mailman/listinfo/tiff



