Return-Path: <tiff-bounces@lists.maptools.org>
Delivered-To: info@awaresystems.be
Received: (qmail 3642 invoked from network); 17 Nov 2008 21:48:11 -0000
Received: from cliffy.telenet-ops.be ([195.130.136.2])
          (envelope-sender <tiff-bounces@lists.maptools.org>)
          by angelia.telenet-ops.be (qmail-ldap-1.03) with SMTP
          for <info@awaresystems.be>; 17 Nov 2008 21:48:11 -0000
Received: from mindy.telenet-ops.be (mindy.telenet-ops.be [195.130.136.8])
	by cliffy.telenet-ops.be (8.13.1/8.13.1) with ESMTP id mAHLmB2Z021820
	for <info@awaresystems.be>; Mon, 17 Nov 2008 22:48:11 +0100
Received: from localhost (localhost.localdomain [127.0.0.1])
	by mindy.telenet-ops.be (Postfix) with SMTP id 7F1F860008
	for <info@awaresystems.be>; Mon, 17 Nov 2008 22:48:11 +0100 (CET)
Received: from ladon.telenet-ops.be (ladon.telenet-ops.be [195.130.132.50])
	by mindy.telenet-ops.be (Postfix) with ESMTP id 76C2560005
	for <info@awaresystems.be>; Mon, 17 Nov 2008 22:48:11 +0100 (CET)
Received: from duke.maptools.org (r153.dmsolutions.ca [209.217.116.153])
	by ladon.telenet-ops.be (Postfix) with ESMTP id 14D4C312773
	for <info@awaresystems.be>; Mon, 17 Nov 2008 22:48:10 +0100 (CET)
Received: from duke.maptools.org (duke.maptools.org [127.0.0.1])
	by duke.maptools.org (8.12.11/8.12.10) with ESMTP id mAHLiZkJ026214;
	Mon, 17 Nov 2008 16:46:05 -0500
Received: from yx-out-2324.google.com (yx-out-2324.google.com [74.125.44.30])
	by duke.maptools.org (8.12.11/8.12.10) with ESMTP id mAHLiWG7026211
	for <tiff@lists.maptools.org>; Mon, 17 Nov 2008 16:44:32 -0500
Received: by yx-out-2324.google.com with SMTP id 8so1136517yxg.9
	for <tiff@lists.maptools.org>; Mon, 17 Nov 2008 13:44:31 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;
	h=domainkey-signature:received:received:message-id:date:from:to
	:subject:cc:in-reply-to:mime-version:content-type
	:content-transfer-encoding:content-disposition:references;
	bh=EU4jSuAHFEwYHaXJOshhFuzFfnmSeCz/Ie1twts8vcY=;
	b=e+bUDUfdWe1G/7URMrCeIIvznRlkUeJRl5cE5l+HjZF1CBA2AMEIq+33J/Az9RZr/o
	Q7u70Mtsk0t3XIkcwbihRxi5RLfABOH2wGyKji+uKj+8iVswfjWXMcAPkIlZzBaaYt/v
	UZHJG9uM7CgIDeLruBGDHXhN5b1GfPcmUN1r4=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=message-id:date:from:to:subject:cc:in-reply-to:mime-version
	:content-type:content-transfer-encoding:content-disposition
	:references;
	b=TQiG0zGJ4uAr4KGYBTXt+N04a1w0Kq8kLChJM6E1o7y3qSLfWvOnuiPVgICu4JEfAz
	kMkTCFI+gEjGW1iPyWyTTwEy8b4os/5v3p2wCzZjEuKjNK4SySzip9jmBW5JaeW1jZwU
	e2KiKT494DIDNI9lhzWwD9HiILE/afbaFG9NM=
Received: by 10.142.214.5 with SMTP id m5mr2190277wfg.110.1226958270725;
	Mon, 17 Nov 2008 13:44:30 -0800 (PST)
Received: by 10.143.157.17 with HTTP; Mon, 17 Nov 2008 13:44:30 -0800 (PST)
Message-ID: <522c6460811171344r46c06cf8od2e9a1a5ee32d516@mail.gmail.com>
Date: Mon, 17 Nov 2008 21:44:30 +0000
From: jcupitt@gmail.com
To: "Frank Warmerdam" <warmerdam@pobox.com>
Subject: Re: [Tiff] Memory leak (TIFFOpen, TIFFReadTile)?
In-Reply-To: <4921A7DF.9000202@pobox.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <200811142137.mAELb1jC000522@tux.cubewerx.com>
	<4921A7DF.9000202@pobox.com>
Cc: tiff@lists.maptools.org
X-BeenThere: tiff@lists.maptools.org
X-Mailman-Version: 2.1.4
Precedence: list
List-Id: TIFF and libtiff discussion <tiff.lists.maptools.org>
List-Unsubscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=unsubscribe>
List-Archive: <http://lists.maptools.org/mailman/private/tiff>
List-Post: <mailto:tiff@lists.maptools.org>
List-Help: <mailto:tiff-request@lists.maptools.org?subject=help>
List-Subscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=subscribe>
Sender: tiff-bounces@lists.maptools.org
Errors-To: tiff-bounces@lists.maptools.org
X-SpamCatcher-Score: 50 [XX]

2008/11/17 Frank Warmerdam <warmerdam@pobox.com>:
>> I have found that memory-mapped I/O is completely FUBARed, even on Linux.
>> Even if the kernel is ultimately going to release the memory, the system
>> still thrashes like crazy.  It really shouldn't be the default mode
>> for TIFFOpen().
>
> I am interested in advice on whether to change TIFFOpen() to *not* use
> memory mapped IO by default.

I tried a couple of benchmarks:

1) Sequential read of a large file

I made a 1.8 gb RGB 8-bit tiff in strips and read it in order. I got
about 60s real time, 12s user, 5s sys for both read() and mmap().
There was quite a lot of variability between runs, but no clear
winner.

Watching progress in "top", you could see that the read() version had
a steady and low RSS, whereas the mmap() one slowly crept up to 1.8 gb
as the operation ran. However,  resident shared pages are counted in
RSS, so this wasn't a "real" 1.8 gb process, it was just showing (in
effect) the system disc cache as part of the process size. The read()
version will have just as dramatic effect on the state of memory, as
the system tries to cache all the pages it passes to read(). I think
the two versions are equivalent in terms of memory use.

2) Random read of a large file

I made a 250 mb tiled RGB 8-bit tiff and rotated it 90 degrees. I had
thought that the heavy seeking would favour mmap(), but again they
seem to have almost identical performance. Both versions ran in about
10s real, 3s user, 1s sys.

Again, although they look very different to "top", I think they are
equivalent in terms of mem use as well.

To summarise, I'm a bit surprised as I thought that mmap did have some
performance advantages. It certainly did 15 years ago, or whenever it
was that we were all so excited about it. I guess read() has become a
lot better. But I'm not sure that mmap() has any real disadvantages
either, at least in this case (local disc). I didn't try a networked
device.

John
_______________________________________________
Tiff mailing list: Tiff@lists.maptools.org
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
