Return-Path: <awaresystems.be+caf_=filtered=awaresystems.be@gmail.com>
Delivered-To: filtered@awaresystems.be
Received: from mx2.pub.mailpod2-cph3.one.com ([10.27.25.12])
	by mailstorage18.cst.mailpod2-cph3.one.com with LMTP id KEZ/F2IYNlvB6wEAQS8vrQ
	for <filtered@awaresystems.be>; Fri, 29 Jun 2018 11:30:42 +0000
X-HalOne-ID: da8bb8a2-7b8f-11e8-a68c-ec0d9a6ed9fe
Received: from mail-yb0-f182.google.com (unknown [209.85.213.182])
	by mx2.pub.mailpod2-cph3.one.com (Halon) with ESMTPS
	id da8bb8a2-7b8f-11e8-a68c-ec0d9a6ed9fe;
	Fri, 29 Jun 2018 11:30:42 +0000 (UTC)
Received: by mail-yb0-f182.google.com with SMTP id h127-v6so3220857ybg.12
        for <filtered@awaresystems.be>; Fri, 29 Jun 2018 04:30:41 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-original-authentication-results:x-gm-message-state:delivered-to
         :to:references:from:message-id:date:user-agent:mime-version
         :in-reply-to:content-language:subject:precedence:list-id
         :list-unsubscribe:list-archive:list-post:list-help:list-subscribe
         :content-transfer-encoding:sender:errors-to;
        bh=t0mnT1VW3SeuqBsjebUOncvK9MYEWutSfSvdRJ+U5S4=;
        b=LJJBXyjNvG/TspX20m4soMQHXcFaNtab6A18dnzHZ/D5wgl3hHJHGezY9iy2HgSQ6i
         6OfMNrG0Sgtsea8M/1HVdJzZZaEkFM5bTWakClETwtxxRtjE5Gbq0kyc2HlMJZYOa8t4
         n/iLVo9SeeBVmbGi5yE0cyVhbty+OtjZ78f2X9YgpEgvH5dwa19NruAD2xC/siiKrpkd
         zk/cpvaCkcCoZV6U6wLAG2gZMDvjRZGh13wacAsHHshIQTM0wk34nt7vMyi8q/8LIs+6
         n2HjyrewiXCvyGbvxCDJPuz5RsG06dIGizVahXDRMkwuTJMw2F167WPPFZBgGsV3XQwK
         0Ifw==
X-Original-Authentication-Results: mx.google.com;       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org
X-Gm-Message-State: APt69E0wzf92lZE846V2qUDHkrvtLWqVstZMcrQ0EeUboNDGSIdkvDTA
	IgKZORPO+JJ8jc3rFmOjQyWLPKT0v3CFNNPgml9aVWaodg/VuVA=
X-Received: by 2002:a25:688a:: with SMTP id d132-v6mr1656779ybc.160.1530271840660;
        Fri, 29 Jun 2018 04:30:40 -0700 (PDT)
X-Forwarded-To: filtered@awaresystems.be
X-Forwarded-For: awaresystems.be@gmail.com filtered@awaresystems.be
Delivered-To: awaresystems.be@gmail.com
Received: by 2002:a25:2d05:0:0:0:0:0 with SMTP id t5-v6csp639263ybt;
        Fri, 29 Jun 2018 04:30:39 -0700 (PDT)
X-Google-Smtp-Source: AAOMgpcOF99Fzu5AJApoVdS88F28AkocQpKbycO8J0ivX7waWtQ11W/oFcW1ogB7hKLATP1hJKsy
X-Received: by 2002:adf:8610:: with SMTP id 16-v6mr11585806wrv.255.1530271839279;
        Fri, 29 Jun 2018 04:30:39 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; t=1530271839; cv=none;
        d=google.com; s=arc-20160816;
        b=Q2VKQ64yRJ72k4bUmR1F/AXoEkLsL6bUdbjeObIO8JMp9YTs0lnmJVlh2fJiwEGtRd
         ErxCORr2B0bLpO2oxjnyC3may0P2McwUt+1Y056/9UZXzNAy4pE0PgLpQLSM6EKPM68F
         55Av54IKFa2xPey98wC86oc/iArKet123eZ8KY9MXts/8KHloQzKR+cslmHBM4SxO6wA
         XcZlDXyAJPHzRCZyg4wAne01FwKGSDdK1RVBUWO7xfx62nMDmlt337uYLXrwJixKKPAO
         spC3L1fArdDy0QGLIimI5v6H1iRMNJn9O693KZ7XA1zO0vGS+i9ByJ+ztoOcb6l4UQ54
         4/nw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
        h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help
         :list-post:list-archive:list-unsubscribe:list-id:precedence:subject
         :content-language:in-reply-to:mime-version:user-agent:date
         :message-id:from:references:to:arc-authentication-results;
        bh=t0mnT1VW3SeuqBsjebUOncvK9MYEWutSfSvdRJ+U5S4=;
        b=CgTtR+grZX+wYMh/JSlV0T4QsRxpYtl/rRYtWHH1NxT7vQQDWL7eoYbdqwwhOGpQYW
         AfW2zkeyF5SF0AMfR/zsLojUUylitLWptd92pkRclMO3j19sd/FyVeThqhvekPI3Qpc9
         rI1AKpUc8A0qCcuqWTblk03CWuVpKaB7OHwuA1fbnapcKLUbsKyOqSSJ9HQj4Mx9tvkK
         C8qDM1k1QQFR/v9Ag90wh6fsQ8mrV+SP4WZRr2pLTa7KaZr+FYlf+RU9/QkMHp6aHQiJ
         Y4/AaFmCK1ZA9fKgLeUjEA7jLSBNqcW4lnGqsgtFEAREcVJ+YL9tXnq8MGhW3NoZaSPc
         S1bg==
ARC-Authentication-Results: i=1; mx.google.com;
       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org
Return-Path: <tiff-bounces@lists.maptools.org>
Received: from lists.maptools.org (ks365428.kimsufi.com. [94.23.4.142])
        by mx.google.com with ESMTP id z5-v6si2649647wrr.252.2018.06.29.04.30.38;
        Fri, 29 Jun 2018 04:30:39 -0700 (PDT)
Received-SPF: neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) client-ip=94.23.4.142;
Authentication-Results: mx.google.com;
       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org
Received: from [10.1.0.241] (localhost.localdomain [127.0.0.1])
	by lists.maptools.org (Postfix) with ESMTP id 21B8835C0CA;
	Fri, 29 Jun 2018 06:28:10 -0500 (EST)
Received: from b-painless.mh.aa.net.uk (b-painless.mh.aa.net.uk [81.187.30.52])
	by lists.maptools.org (Postfix) with ESMTPS id C10F835C027
	for <tiff@lists.maptools.org>; Fri, 29 Jun 2018 06:27:50 -0500 (EST)
Received: from
	e.7.e.f.a.b.e.7.b.9.f.3.7.2.9.e.d.b.d.d.0.6.8.0.0.b.8.0.1.0.0.2.ip6.arpa
	([2001:8b0:860:ddbd:e927:3f9b:7eba:fe7e])
	by b-painless.mh.aa.net.uk with esmtpsa
	(TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.89)
	(envelope-from <rleigh@codelibre.net>) id 1fYrZ8-0002mS-9J
	for tiff@lists.maptools.org; Fri, 29 Jun 2018 12:27:50 +0100
To: tiff@lists.maptools.org
References: <d5029186-bbdd-27c4-9f8d-8a00a5156a84@codelibre.net>
	<7222068.FQGs7OQbg5@even-i700>
From: Roger Leigh <rleigh@codelibre.net>
Message-ID: <3b20a5c2-4634-3789-372c-193aeea14c28@codelibre.net>
Date: Fri, 29 Jun 2018 12:27:38 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
	Thunderbird/52.8.0
MIME-Version: 1.0
In-Reply-To: <7222068.FQGs7OQbg5@even-i700>
Content-Language: en-US
Subject: Re: [Tiff] Writing tiles out of order
X-BeenThere: tiff@lists.maptools.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: TIFF and libtiff discussion <tiff.lists.maptools.org>
List-Unsubscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=unsubscribe>
List-Archive: <http://lists.maptools.org/mailman/private/tiff>
List-Post: <mailto:tiff@lists.maptools.org>
List-Help: <mailto:tiff-request@lists.maptools.org?subject=help>
List-Subscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=subscribe>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Sender: tiff-bounces@lists.maptools.org
Errors-To: tiff-bounces@lists.maptools.org

On 29/06/18 11:27, Even Rouault wrote:
> On vendredi 29 juin 2018 11:08:00 CEST Roger Leigh wrote:
>> Hi folks,
>>
>> Right now I'm using libtiff to write tiled images completely in order:
>> each tile of each plane in ascending order, adding new IFDs as needed.
>> One of my users has asked if it's possible to stripe writes across
>> directories.  That is, for example, to write Tile 0 of IFDs 0-12, then
>> Tile 1 of IFDs 0-12 etc.
>>
>> Looking at the API, it's possible to visit previously written IFDs, but
>> you have to call TIFFRewriteIFD, which gets expensive to duplicate when
>> you have 10k-100k tiles per image and will be rewriting that many times
>> over.  If I reopen an old IFD and then call TIFFWrite*Tile, should I
>> expect this to work, and update the existing IFD tile offsets+sizes in
>> place, or is this strictly forbidden?
> 
> Update in place, and switching between IFD, works. GDAL does this in some
> scenarios. But as you pointed, if your IFD have many tiles/strips, loading and
> rewriting the offset and bytecount arrays each time you switch from IFD can be
> time expensive.

Just to clarify, this is only time [I/O] expensive when reading and 
updating the tile offsets+sizes?

Or is it also wasteful of space by duplicating the whole IFD each time I 
do an update?

Looking at GDAL, I see you calling TIFFRewriteDirectory in 
frmts/gtiff/geotiff.cpp.  If you're only writing additional tiles, would 
it be sufficient to call TIFFCheckpointDirectory to update the already 
written directory in place?  Or will this only work before the first 
call to TIFFWriteDirectory?  Or would TIFFWriteDirectory be possible to 
use here?  (Apologies for the confusion, this seems a bit of a 
non-standard thing to do and it's not covered by the documentation.)

> If performance is terrible, what you could potentially do is create the
> structure of the TIFF (ie all IFDs but without writing any tile/strip), close
> it, and then re-open it in update mode as many times as you have IFDs, and use
> a given TIF handle for a given IFD. Multiple handles in update on the same
> file is probably to be considered undefined behaviour, but I believe (untested
> though !) that given the current implementation of libtiff, that should work,
> since it uses TIFFSeekFile(tif, 0, SEEK_END) to get the offset of a new tile/
> strip (assuming the implementation of seek() you use actually seeks to the
> file, and doesn't cache the file size). Of course, do not write in a multi-
> treaded way...

Thanks, I'll investigate this too.  I've already wrapped libtiff in a 
muxtex so should be relatively safe...  On the other hand, for large 
images with multiple planes for Z/T/C, I might run out of file descriptors!


Thanks,
Roger
_______________________________________________
Tiff mailing list: Tiff@lists.maptools.org
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
