Received: from mailgate001.isp.belgacom.be (mailgate001.isp.belgacom.be [195.238.6.86])
	by maildelivery001.isp.belgacom.be (Postfix) with ESMTP id EBA7F5A800D
	for <joris.at.lebbeke@skynet.be>; Thu,  5 Jul 2007 18:18:35 +0200 (CEST)
Received: from r153.dmsolutions.ca (HELO duke.maptools.org) ([209.217.116.153])
  by in.mx.skynet.be with ESMTP; 05 Jul 2007 18:18:35 +0200
Received: from duke.maptools.org (duke.maptools.org [127.0.0.1])
	by duke.maptools.org (8.12.11/8.12.10) with ESMTP id l65G2mRe012327;
	Thu, 5 Jul 2007 12:02:52 -0400
Received: from scooby.simplesystems.org (scooby.simplesystems.org
	[65.66.246.67])
	by duke.maptools.org (8.12.11/8.12.10) with ESMTP id l65G29gT012291
	for <tiff@lists.maptools.org>; Thu, 5 Jul 2007 12:02:10 -0400
Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74])
	by scooby.simplesystems.org (8.12.10+Sun/8.12.10) with ESMTP id
	l65G28W7027833; Thu, 5 Jul 2007 11:02:08 -0500 (CDT)
Date: Thu, 5 Jul 2007 11:02:08 -0500 (CDT)
From: Bob Friesenhahn <bfriesen@simple.dallas.tx.us>
X-X-Sender: bfriesen@blade.simplesystems.org
To: Joris <joris.at.lebbeke@skynet.be>
Subject: Re: [Tiff] 16-Bit-Per-Channel Lossless Compression
In-Reply-To: <002401c7bf0a$cc4fd2b0$0201a8c0@AWareCore2Duo>
Message-ID: <Pine.SOC.4.60.0707051041100.922@blade.simplesystems.org>
References: <20070705105619.GA13773@ak4719.spb.edu><01f201c7befc$746a2d60$da60fea9@EKAT>
	<20070705125702.GA15896@ak4719.spb.edu>
	<002401c7bf0a$cc4fd2b0$0201a8c0@AWareCore2Duo>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
Cc: Tiff List <tiff@lists.maptools.org>
X-BeenThere: tiff@lists.maptools.org
X-Mailman-Version: 2.1.4
Precedence: list
List-Id: TIFF and libtiff discussion <tiff.lists.maptools.org>
List-Unsubscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=unsubscribe>
List-Archive: <http://lists.maptools.org/mailman/private/tiff>
List-Post: <mailto:tiff@lists.maptools.org>
List-Help: <mailto:tiff-request@lists.maptools.org?subject=help>
List-Subscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=subscribe>
Sender: tiff-bounces@lists.maptools.org
Errors-To: tiff-bounces@lists.maptools.org
X-UIDL: 000000000023ceae.00002566
X-MSP-STORE: Thu, 05 Jul 2007 18:18:36 +0200
Status:   

On Thu, 5 Jul 2007, Joris wrote:

> As I don't speak gdalese, what size do tiles have in your test? It's my 
> opinion ideally efficient tiles over a broad range of compression modes, that 
> fit workflow on today's machines best, are in the order of about 1 to 3 
> megabyte, not the legacy 8 kilobyte (uncompressed, of course). I believe that 
> for some compression modes, tile size does have a considerable impact on 
> efficienty.

Joris,

You make many assumptions in the above statement, and particularly 
about how applications store/access their pixels.  I would not 
encourage anyone to write such huge tiles.

There are many major factors which influence performance:

   o Application pixel storage organization
   o CPU cache
   o Disk cache
   o Main memory bandwidth
   o Disk filesystem implementation
   o Operating system I/O implementation
   o Network filesystem implementation

I have done considerable testing and benchmarking in various scenarios 
and results usually show that data access sizes of 8K or 16K are best, 
with performance diminishing past 32K.

Your huge tiles have these problems:

   o They are larger than the cache on anything but the most exotic
     RISC systems.  Regardless, the first level cache is usually fairly
     small.
   o They are larger than many disk drive caches.
   o They don't mesh well with the typical 8K or 16K filesystem block
     sizes.
   o They don't mesh well with network filesystem buffering ("block")
     sizes.
   o The application is blocked on large I/O when it could be
     simultaneously processing data (parsing/storing pixels).
   o They require the application to allocate more memory.
   o They defeat the operating system's built in "read ahead"
     logic for sequential file access.  The operating system is only
     going to be willing to buffer a limited amount of unrequested
     data in anticipation that it might be used.  Only a rather
     foolish OS will gamble on reading 3MB of data in advance with
     the hope that it will be used.  If the OS is only willing to
     do read ahead by 8K or 16K, then performance is best if the
     application does not need more than that in order to proceed.

Bob
======================================
Bob Friesenhahn
bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

_______________________________________________
Tiff mailing list: Tiff@lists.maptools.org
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
