Return-Path: <tiff-bounces@lists.maptools.org>
Delivered-To: info@awaresystems.be
Received: (qmail 21480 invoked from network); 22 Aug 2008 14:48:14 -0000
Received: from floyd.telenet-ops.be ([195.130.136.4])
          (envelope-sender <tiff-bounces@lists.maptools.org>)
          by angelia.telenet-ops.be (qmail-ldap-1.03) with SMTP
          for <info@awaresystems.be>; 22 Aug 2008 14:48:14 -0000
Received: from waldo.telenet-ops.be (waldo.telenet-ops.be [195.130.136.10])
	by floyd.telenet-ops.be (8.13.1/8.13.1) with ESMTP id m7MEmECC023864
	for <info@awaresystems.be>; Fri, 22 Aug 2008 16:48:14 +0200
Received: from localhost (localhost.localdomain [127.0.0.1])
	by waldo.telenet-ops.be (Postfix) with SMTP id 5E02C1C009
	for <info@awaresystems.be>; Fri, 22 Aug 2008 16:48:14 +0200 (CEST)
Received: from apate.telenet-ops.be (apate.telenet-ops.be [195.130.132.43])
	by waldo.telenet-ops.be (Postfix) with ESMTP id 587821C006
	for <info@awaresystems.be>; Fri, 22 Aug 2008 16:48:14 +0200 (CEST)
Received: from duke.maptools.org (r153.dmsolutions.ca [209.217.116.153])
	by apate.telenet-ops.be (Postfix) with ESMTP id 0311127EC07
	for <info@awaresystems.be>; Fri, 22 Aug 2008 16:48:13 +0200 (CEST)
Received: from duke.maptools.org (duke.maptools.org [127.0.0.1])
	by duke.maptools.org (8.12.11/8.12.10) with ESMTP id m7MEj69M022253;
	Fri, 22 Aug 2008 10:47:10 -0400
Received: from scooby.simplesystems.org (scooby.simplesystems.org
	[65.66.246.67])
	by duke.maptools.org (8.12.11/8.12.10) with ESMTP id m7MEj1j7022240
	for <tiff@lists.maptools.org>; Fri, 22 Aug 2008 10:45:01 -0400
Received: from freddy.simplesystems.org (freddy.simplesystems.org
	[65.66.246.65])
	by scooby.simplesystems.org (8.12.10+Sun/8.12.10) with ESMTP id
	m7MEirCI000876; Fri, 22 Aug 2008 09:44:53 -0500 (CDT)
Date: Fri, 22 Aug 2008 09:44:53 -0500 (CDT)
From: Bob Friesenhahn <bfriesen@simple.dallas.tx.us>
X-X-Sender: bfriesen@freddy.simplesystems.org
To: Rogier Wolff <R.E.Wolff@bitwizard.nl>
Subject: Re: [Tiff] creating sparse files......
In-Reply-To: <20080822085634.GA21294@bitwizard.nl>
Message-ID: <Pine.SOC.4.64.0808220936280.1110@freddy.simplesystems.org>
References: <20080822085634.GA21294@bitwizard.nl>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
Cc: tiff@lists.maptools.org
X-BeenThere: tiff@lists.maptools.org
X-Mailman-Version: 2.1.4
Precedence: list
List-Id: TIFF and libtiff discussion <tiff.lists.maptools.org>
List-Unsubscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=unsubscribe>
List-Archive: <http://lists.maptools.org/mailman/private/tiff>
List-Post: <mailto:tiff@lists.maptools.org>
List-Help: <mailto:tiff-request@lists.maptools.org?subject=help>
List-Subscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=subscribe>
Sender: tiff-bounces@lists.maptools.org
Errors-To: tiff-bounces@lists.maptools.org
X-SpamCatcher-Score: 10 [X]

On Fri, 22 Aug 2008, Rogier Wolff wrote:
>
> I'm stitching kind of large panoramas. This results in big
> intermediate files. On my last run, which took overnight to stitch,
> I thought 42 Gb of free disk space would be enough. Wrong!
>
> I got over thrity files of over 1.2Gbytes, filling up the disk.
>
> It turns out that most of the files contain lots of zeroes. On Unix
> this can be stored effciently by not issueing a "write" with a buffer
> full of zeroes, but by seeking over the area. The operating system
> will act as if the area was filled with zeroes.

This is an interesting issue.  While holey files seem like a panacea, 
there can be some drawbacks.  They are best for files which are 
written just once (like core files) and not so good for files which 
are expected to be updated in place.  For files which are updated in 
place, the updated hole is quite likely to increase disk fragmentation 
since now it takes more space and the space will need to be from some 
other place on disk.  Fragmentation behavior is quite filesystem 
dependent.

It is worth considering enabling filesystem compression, or using 
whole-file compression.  Perhaps even just enabling normal TIFF 
compression (e.g. LZW) is sufficient to eliminate the long spans of 
zeros.

Bob
======================================
Bob Friesenhahn
bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

_______________________________________________
Tiff mailing list: Tiff@lists.maptools.org
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
