Return-Path: <awaresystems.be+caf_=filtered=awaresystems.be@gmail.com>
Delivered-To: filtered@awaresystems.be
Received: from mx1.pub.mailpod2-cph3.one.com ([10.27.25.11])
	by mailstorage18.cst.mailpod2-cph3.one.com (Dovecot) with LMTP id KTkPBPCxXlpGWwAAQS8vrQ
	for <filtered@awaresystems.be>; Wed, 17 Jan 2018 02:24:19 +0000
X-HalOne-ID: 84aaab58-fb2d-11e7-afe7-e0d848957041
Received: from mail-ot0-f174.google.com (unknown [74.125.82.174])
	by mx1.pub.mailpod2-cph3.one.com (Halon) with ESMTPS
	id 84aaab58-fb2d-11e7-afe7-e0d848957041;
	Wed, 17 Jan 2018 02:24:18 +0000 (UTC)
Received: by mail-ot0-f174.google.com with SMTP id 44so14596088otk.8
        for <filtered@awaresystems.be>; Tue, 16 Jan 2018 18:24:18 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-original-authentication-results:x-gm-message-state:delivered-to
         :from:mime-version:date:references:to:in-reply-to:message-id:subject
         :precedence:list-id:list-unsubscribe:list-archive:list-post
         :list-help:list-subscribe:mime-version:sender:errors-to;
        bh=fb4DVDQ8WHxG8Ul86cQbfOfP6xhAAw3TAkDR8yWk3Zc=;
        b=R8pCPqRfxwvy0nemFxmT6ldcQPBaFqHnEdMzA0vVbTyuIO+f1yv1wRMKFfgDMPgW30
         FuRoMhda1mfAuDPBmtxfH2SE6Cve/e15kyhqi7B6wQtZj2mPhLI+nhf+Jve04xGcQtzQ
         ah1QO+OspqiWQGhFALvL++LLFisz2kjKQRQyEdiyZKiCVRAZ7YBuGZxHP+Y9mAfYxQHh
         qbCdmxuJvuqYY8WpvYIN1x7rSSM8WB0vt+t6trdk0okWFt0tG2ujs5dwNGksPGwuss62
         o64t2bJEgrUo4QsMKj83EmxMm6WBLGmlPxGVR+9BAtXquluA5eFHdlR90Ftp+8Y1RACZ
         vnAA==
X-Original-Authentication-Results: mx.google.com;       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org
X-Gm-Message-State: AKwxytfL7HNksYdgpjGk8Yud45G0v7ZEX9dfvYsExPTY4lR7O73g8mYn
	Qbv84T+69Ni+9MdwudQZI7HEMqvIVkVO5KwEMZVJySg1uKbeD/E=
X-Received: by 10.157.50.5 with SMTP id t5mr247688otc.172.1516155857130;
        Tue, 16 Jan 2018 18:24:17 -0800 (PST)
X-Forwarded-To: filtered@awaresystems.be
X-Forwarded-For: awaresystems.be@gmail.com filtered@awaresystems.be
Delivered-To: awaresystems.be@gmail.com
Received: by 10.202.235.11 with SMTP id j11csp4915230oih;
        Tue, 16 Jan 2018 18:24:16 -0800 (PST)
X-Google-Smtp-Source: ACJfBot0CNZzAP6BZxm+VeUDBbSEefdPUIfOm0QAkmENFkfRrUVwVOXZ74a2tyn7/WkMN1bR32wx
X-Received: by 10.223.176.39 with SMTP id f36mr938979wra.113.1516155855997;
        Tue, 16 Jan 2018 18:24:15 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; t=1516155855; cv=none;
        d=google.com; s=arc-20160816;
        b=sxnbO+Pym7WP0gVGo1iC0EjU2BPqdZPC5ToOBj8fpp4J93Ib1b9Yh3Id7RzTUe+HUr
         Xtt0u6+SYNvYvU/FTA/HzmpITMv5zip1cn2YZwWeW5J76QWfS2cafkptETO6eGMOkEkk
         R8Y1Ls4rd2mmVeZinZOFlULSHnELucQEfKqaUNKZUbc/2k3Dv0CC68FDWk6+oLZqVXT9
         5CVBeDK8xTepL/2qCDxa0zJYVZnj6dBm2B7CuvV9hTW5N68pwGYuNzDITWBOaaRFjzRT
         Ga94blW32JqhAMeZita075Hbi4RelVSk6mzgPZCGXK7KDmmrTT2kgiFj6Y99WlNsMigZ
         Yuaw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
        h=errors-to:sender:mime-version:list-subscribe:list-help:list-post
         :list-archive:list-unsubscribe:list-id:precedence:subject:message-id
         :in-reply-to:to:references:date:mime-version:from
         :arc-authentication-results;
        bh=fb4DVDQ8WHxG8Ul86cQbfOfP6xhAAw3TAkDR8yWk3Zc=;
        b=dnhW8il04xv0UEszbRug1v92noqf6SxJpB2ntBS4B9vADgYlY1P4J3SoWojuvyYsC6
         ZcrC+NBgbFUl7JRpVbIzfqLySC9xGNQT7GR6kMkLT9vxAnLxP4Kb3S3gFOQVmdwMaSfj
         qa9ST1zIG+M+KMEWhPPP72Dt/M4isuTvElYMH2qlv97rqCETGc8BCq5Cw/TePbjSk/uU
         9Uj2RjR3Y209pjEOReNkzy/Lf9PqvKX1onFG1HHq8gSnvg2GD7S9QcU+sTVabOHOZJeo
         oum/JkCsXr9LJBLIa6ING+unYCB618/YyCgTbynwQH1gVHH4bzjUuf2Maa9mAWMdhusX
         ZsuA==
ARC-Authentication-Results: i=1; mx.google.com;
       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org
Return-Path: <tiff-bounces@lists.maptools.org>
Received: from lists.maptools.org (ks365428.kimsufi.com. [94.23.4.142])
        by mx.google.com with ESMTP id s4si3216411wrf.176.2018.01.16.18.24.15;
        Tue, 16 Jan 2018 18:24:15 -0800 (PST)
Received-SPF: neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) client-ip=94.23.4.142;
Authentication-Results: mx.google.com;
       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org
Received: from [10.1.0.241] (localhost.localdomain [127.0.0.1])
	by lists.maptools.org (Postfix) with ESMTP id 183DD35C0D1;
	Tue, 16 Jan 2018 21:22:18 -0500 (EST)
Received: from homiemail-a7.g.dreamhost.com (homie.mail.dreamhost.com
	[208.97.132.208])
	by lists.maptools.org (Postfix) with ESMTPS id 3C56A35C0D1
	for <tiff@lists.maptools.org>; Tue, 16 Jan 2018 21:00:58 -0500 (EST)
Received: from homiemail-a7.g.dreamhost.com (localhost [127.0.0.1])
	by homiemail-a7.g.dreamhost.com (Postfix) with ESMTP id 7234125C064
	for <tiff@lists.maptools.org>; Tue, 16 Jan 2018 18:00:56 -0800 (PST)
Received: from [172.30.12.190] (unknown [160.33.4.9])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	(Authenticated sender: lg@larrygritz.com)
	by homiemail-a7.g.dreamhost.com (Postfix) with ESMTPSA id 4C28C25C062
	for <tiff@lists.maptools.org>; Tue, 16 Jan 2018 18:00:56 -0800 (PST)
From: Larry Gritz <lg@larrygritz.com>
Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\))
Date: Tue, 16 Jan 2018 18:00:27 -0800
References: <39EE49FE-23A5-43DE-8998-51526287D182@larrygritz.com>
	<alpine.GSO.2.20.1801151400550.18206@scrappy.simplesystems.org>
	<4252FC16-C07D-4FA1-8B7B-25D626633236@larrygritz.com>
	<F78FB533EE93A044B560CD25E17A4DAF18388904@mbx031-e1-va-4.exch031.domain.local>
	<D4C4B1FE-4C28-4E7C-964F-750BBCF87DAC@larrygritz.com>
To: "tiff@lists.maptools.org" <tiff@lists.maptools.org>
In-Reply-To: <D4C4B1FE-4C28-4E7C-964F-750BBCF87DAC@larrygritz.com>
Message-Id: <89E6CAE9-699D-42A5-8DCF-90EA1FEDB553@larrygritz.com>
X-Mailer: Apple Mail (2.3273)
Subject: Re: [Tiff] Strategies for multi-core speedups
X-BeenThere: tiff@lists.maptools.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: TIFF and libtiff discussion <tiff.lists.maptools.org>
List-Unsubscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=unsubscribe>
List-Archive: <http://lists.maptools.org/mailman/private/tiff>
List-Post: <mailto:tiff@lists.maptools.org>
List-Help: <mailto:tiff-request@lists.maptools.org?subject=help>
List-Subscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=subscribe>
Content-Type: multipart/mixed; boundary="===============1559157808=="
Mime-version: 1.0
Sender: tiff-bounces@lists.maptools.org
Errors-To: tiff-bounces@lists.maptools.org


--===============1559157808==
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_8A76D94E-6104-40FF-98D1-FA7708693339"


--Apple-Mail=_8A76D94E-6104-40FF-98D1-FA7708693339
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Status report: I've implemented this, thanks to Scott Ribe's suggestion =
of not trying to struggle with libtiff's codec internals, but just =
calling zlib directly, then TIFFWriteRawStrip (this strategy only works =
for deflate compression, but that's my common case anyway). The strips' =
data are compressed in parallel.

I'm getting a 6x improvement in total time for writing large TIFF files.

I've only done scanline (i.e. strip) output so far. I still need to =
implement it for the case of having lots of tiles ready to write at =
once. And also need to do similar for the tile and scanline input side =
of things. But now that I have the blueprint, that should be =
straightforward.

This is all implemented in OpenImageIO, so as soon as I'm done with this =
work, anybody who is doing whole-image reads and writes of TIFF files =
using OIIO rather than libtiff directly will get this speedup without =
having to do anything extra.

--
Larry Gritz
lg@larrygritz.com





--Apple-Mail=_8A76D94E-6104-40FF-98D1-FA7708693339
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=us-ascii

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D"">Status report: I've implemented this, thanks to Scott Ribe's =
suggestion of not trying to struggle with libtiff's codec internals, but =
just calling zlib directly, then TIFFWriteRawStrip (this strategy only =
works for deflate compression, but that's my common case anyway). The =
strips' data are compressed in parallel.<div class=3D""><br =
class=3D""></div><div class=3D"">I'm getting a 6x improvement in total =
time for writing large TIFF files.</div><div class=3D""><br =
class=3D""></div><div class=3D"">I've only done scanline (i.e. strip) =
output so far. I still need to implement it for the case of having lots =
of tiles ready to write at once. And also need to do similar for the =
tile and scanline input side of things. But now that I have the =
blueprint, that should be straightforward.</div><div class=3D""><div =
class=3D""><br class=3D""></div><div class=3D"">This is all implemented =
in OpenImageIO, so as soon as I'm done with this work, anybody who is =
doing whole-image reads and writes of TIFF files using OIIO rather than =
libtiff directly will get this speedup without having to do anything =
extra.</div><div class=3D""><br class=3D""></div><div class=3D""><div =
class=3D"">
<div style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; =
-webkit-line-break: after-white-space;" class=3D""><div style=3D"color: =
rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: =
normal; font-variant-caps: normal; font-weight: normal; letter-spacing: =
normal; text-align: start; text-indent: 0px; text-transform: none; =
white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: =
0px;">--</div><div style=3D"color: rgb(0, 0, 0); font-family: Helvetica; =
font-size: 14px; font-style: normal; font-variant-caps: normal; =
font-weight: normal; letter-spacing: normal; text-align: start; =
text-indent: 0px; text-transform: none; white-space: normal; =
word-spacing: 0px; -webkit-text-stroke-width: 0px;">Larry =
Gritz</div><div style=3D"color: rgb(0, 0, 0); font-family: Helvetica; =
font-size: 14px; font-style: normal; font-variant-caps: normal; =
font-weight: normal; letter-spacing: normal; text-align: start; =
text-indent: 0px; text-transform: none; white-space: normal; =
word-spacing: 0px; -webkit-text-stroke-width: 0px;"><a =
href=3D"mailto:lg@larrygritz.com" =
class=3D"">lg@larrygritz.com</a></div><div style=3D"color: rgb(0, 0, 0); =
font-family: Helvetica; font-size: 14px; font-style: normal; =
font-variant-caps: normal; font-weight: normal; letter-spacing: normal; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><br=
 class=3D""></div><br class=3D"Apple-interchange-newline"></div><br =
class=3D"Apple-interchange-newline">
</div>
<br class=3D""></div></div></body></html>=

--Apple-Mail=_8A76D94E-6104-40FF-98D1-FA7708693339--

--===============1559157808==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Tiff mailing list: Tiff@lists.maptools.org
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
--===============1559157808==--
