X-Account-Key: account1
X-UIDL: 54e3457b1138b755fce583399a940be5
X-Mozilla-Status: 0011
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:                                                                                 
Received: from lists.maptools.org (lothlorien.howardsilvan.com [127.0.0.1])
	by mail.howardsilvan.com (Postfix) with ESMTPS id 08B596721A7
	for <faxguy@howardsilvan.com>; Tue, 26 Jan 2016 05:46:27 -0800 (PST)
Received: from lists.maptools.org ([94.23.4.142] helo=lists.maptools.org)
	by lothlorien.howardsilvan.com
	with SMTP (2.3.3); 26 Jan 2016	05:46:27 -0800
Received: from [10.1.0.241] (localhost.localdomain [127.0.0.1])
	by lists.maptools.org (Postfix) with ESMTP id 5B98235C0DD;
	Tue, 26 Jan 2016 08:45:31 -0500 (EST)
Received: from mail-qk0-f178.google.com (mail-qk0-f178.google.com	[209.85.220.178])
	by lists.maptools.org (Postfix) with ESMTPS id 813F435C0BB
	for <tiff@lists.maptools.org>; Tue, 26 Jan 2016 08:45:18 -0500 (EST)
Received: by mail-qk0-f178.google.com with SMTP id s68so62551779qkh.3
	for <tiff@lists.maptools.org>; Tue, 26 Jan 2016 05:45:18 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;	d=1e100.net;
 s=20130820;	h=x-gm-message-state:mime-version:in-reply-to:references:date
	:message-id:subject:from:to:cc:content-type;
	bh=Ud4Wi+kzUES3Q6Vowr2J+K6urny+5bv6r+zzaknlN6Q=;
	b=LR6jtv5bsQ0YEBCp1OiLnOvJSu4zBRremb9Q5d/wi5Hibm7VNIVMGbZYLIBe3CA/SR
	W3Rb+h1gK9+CjQT3rO3ipJJvhXQitVZaUqmz46Mo2aQLVDcJxOxQWh6RcEA9aheXXrht
	KnUS9H+OUAjpBfJyK5/30dGcND9WMsfnd+YeHIoINAzASOhNQXytqSMPpUs+pslEl1Vo
	NYivuoGEt2Jjh8GVi3BFdbRDBOUf9px8dR8AnBdWEBYUi2LBbu5AfeyEDq8tQUm36q8q
	ZuSO7b98WNHQST3j+KGmgV+QPabryvMvG+g22ammGn+n3vV3ZOfaQmXEaAVxuXyHCKtz	0xEQ==
X-Gm-Message-State: 
 AG10YOSN8+kFVHTJYccEJxg5F9SjoyvSaclOYqVtn99zuua4Y+lNtKWMCoxGj+xtyY/Asd6/hjD
	L53c/THR7iA==
MIME-Version: 1.0
X-Received: by 10.55.72.87 with SMTP id v84mr28852390qka.9.1453815917212;
 Tue,	26 Jan 2016 05:45:17 -0800 (PST)
Received: by 10.140.85.69 with HTTP; Tue, 26 Jan 2016 05:45:17 -0800 (PST)
In-Reply-To: 
 <CAGNS0RtXiz0qPgEuXpzVwJGvbv7TACX2JD-quMi_urwo7P05HA@mail.gmail.com>
References:
	<CAK8GPyAzKfkxRzoG9BSLMT6WS1+pOuD5-cn08huFevXBAM9Y3g@mail.gmail.com>
	<CAGNS0RtXiz0qPgEuXpzVwJGvbv7TACX2JD-quMi_urwo7P05HA@mail.gmail.com>
Date: Tue, 26 Jan 2016 08:45:17 -0500
Message-ID: <CAK8GPyCHEWgbMgj23+f0c_xrv=cptBLVeeChC=Z=WBDq9Rvgiw@mail.gmail.com>
From: Aaron Boxer <boxerab@gmail.com>
To: John Cupitt <jcupitt@gmail.com>
Cc: TIFF mailing list <tiff@lists.maptools.org>
Subject: Re: [Tiff] OpenMP enabled libtiff
X-BeenThere: tiff@lists.maptools.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: TIFF and libtiff discussion <tiff.lists.maptools.org>
List-Unsubscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=unsubscribe>
List-Archive: <http://lists.maptools.org/mailman/private/tiff>
List-Post: <mailto:tiff@lists.maptools.org>
List-Help: <mailto:tiff-request@lists.maptools.org?subject=help>
List-Subscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=subscribe>
Mime-version: 1.0
Sender: tiff-bounces@lists.maptools.org
Errors-To: tiff-bounces@lists.maptools.org
X-Assp-ID: lothlorien.howardsilvan.com m1-15987-12751
X-Assp-Session: 7FE09B908770 (mail 1)
X-Assp-Detected-RIP: 209.85.220.178
X-Assp-Source-IP: 209.85.220.178
X-Assp-Envelope-From: tiff-bounces@lists.maptools.org
X-Assp-Intended-For: faxguy@howardsilvan.com
X-Assp-Version: 2.3.3(14029) on lothlorien.howardsilvan.com
X-Assp-Server-TLS: yes
X-Assp-Delay: not delayed (whitelisted); 26 Jan 2016 05:46:28 -0800
X-Assp-Message-Score: -40 (KnownGoodHelo)
X-Assp-IP-Score: -40 (KnownGoodHelo)
X-Assp-Whitelisted: Yes (whitelistdb)
X-Assp-DKIM: not verified
Return-Path: tiff-bounces@lists.maptools.org
Content-Type: multipart/mixed;
	boundary="===============0403851679=="


--===============0403851679==
Content-Type: multipart/alternative; boundary=001a11487efe507ba6052a3ce5a4


--001a11487efe507ba6052a3ce5a4
Content-Type: text/plain; charset=UTF-8

Thanks, everyone. It will take me a little time to fully grok all of this
information.

Sounds like the simplest approach is to allow the user to set a lock
callback method,
which, if not null, will protect the global/static variables with a lock of
the user's choice.

I don't think the user could use OpenMP for this, since OpenMP is a
fork/join architecture. So,
it would also be nice to provide a light-weight cross platform lock
library, wrapping pthreads / win32 API.

This idea of "blasting" the entire file into RAM, and then reading with
libtiff; is this currently possible? Can
libtiff read from a memory buffer ?

Cheers,
Aaron







On Tue, Jan 26, 2016 at 4:12 AM, <jcupitt@gmail.com> wrote:

> On 25 January 2016 at 18:25, Aaron Boxer <boxerab@gmail.com> wrote:
> > Is anyone interested in having OpenMP support in libtiff ?
> > Is this feasible?  I am planning on adding this feature for another
> project
> > I am working on, so I am trying to gauge interest.
>
> I think I would not add openmp support directly. Scattered reads and
> writes are too difficult to support efficiently with the current
> structure.
>
> Instead, as Bob said, I think I would split the library in two. Have a
> very low level interface which just reads or writes compressed strips
> or tiles, essentially just a thin wrapper around the read() and
> write() system calls, and then have a higher layer to do compression,
> decompression, packing and unpacking. The low-level read/write layer
> should be single-threaded, the higher layer should be threadsafe.
>
> Now you can serialize read/write to match the disc structure, but the
> application can use SMP in whatever framework it prefers to accelerate
> the CPU-intensive parts.
>
> Uncompressed tiffs already work rather like this: if you call
> TIFFReadEncodedStrip() on an uncompressed image, it's just a few
> function calls around read() --- it's very easy to saturate any disc
> system. We just need to move the decompress stage outside the serial
> code path.
>
> John
> _______________________________________________
> Tiff mailing list: Tiff@lists.maptools.org
> http://lists.maptools.org/mailman/listinfo/tiff
> http://www.remotesensing.org/libtiff/
>

--001a11487efe507ba6052a3ce5a4
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div><div><div><div><div>Thanks, everyone. It wi=
ll take me a little time to fully grok all of this information.<br><br></di=
v>Sounds like the simplest approach is to allow the user to set a lock call=
back method,<br></div>which, if not null, will protect the global/static va=
riables with a lock of the user&#39;s choice.<br><br></div><div>I don&#39;t=
 think the user could use OpenMP for this, since OpenMP is a fork/join arch=
itecture. So,<br></div>it would also be nice to provide a light-weight cros=
s platform lock library, wrapping pthreads / win32 API.<br><br></div>This i=
dea of &quot;blasting&quot; the entire file into RAM, and then reading with=
 libtiff; is this currently possible? Can <br></div>libtiff read from a mem=
ory buffer ?=C2=A0 <br><br></div>Cheers,<br></div>Aaron<br><div><div><div><=
div><br><br><div><div>=C2=A0 <br><br><br><br><div><div><div><div class=3D"g=
mail_extra"><br><div class=3D"gmail_quote">On Tue, Jan 26, 2016 at 4:12 AM,=
  <span dir=3D"ltr">&lt;<a href=3D"mailto:jcupitt@gmail.com" target=3D"_bla=
nk">jcupitt@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_q=
uote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1e=
x"><span>On 25 January 2016 at 18:25, Aaron Boxer &lt;<a href=3D"mailto:box=
erab@gmail.com" target=3D"_blank">boxerab@gmail.com</a>&gt; wrote:<br>
&gt; Is anyone interested in having OpenMP support in libtiff ?<br>
&gt; Is this feasible?=C2=A0 I am planning on adding this feature for anoth=
er project<br>
&gt; I am working on, so I am trying to gauge interest.<br>
<br>
</span>I think I would not add openmp support directly. Scattered reads and=
<br>
writes are too difficult to support efficiently with the current<br>
structure.<br>
<br>
Instead, as Bob said, I think I would split the library in two. Have a<br>
very low level interface which just reads or writes compressed strips<br>
or tiles, essentially just a thin wrapper around the read() and<br>
write() system calls, and then have a higher layer to do compression,<br>
decompression, packing and unpacking. The low-level read/write layer<br>
should be single-threaded, the higher layer should be threadsafe.<br>
<br>
Now you can serialize read/write to match the disc structure, but the<br>
application can use SMP in whatever framework it prefers to accelerate<br>
the CPU-intensive parts.<br>
<br>
Uncompressed tiffs already work rather like this: if you call<br>
TIFFReadEncodedStrip() on an uncompressed image, it&#39;s just a few<br>
function calls around read() --- it&#39;s very easy to saturate any disc<br=
>
system. We just need to move the decompress stage outside the serial<br>
code path.<br>
<span><font color=3D"#888888"><br>
John<br>
</font></span><div><div>_______________________________________________<br>
Tiff mailing list: <a href=3D"mailto:Tiff@lists.maptools.org" target=3D"_bl=
ank">Tiff@lists.maptools.org</a><br>
<a href=3D"http://lists.maptools.org/mailman/listinfo/tiff" rel=3D"noreferr=
er" target=3D"_blank">http://lists.maptools.org/mailman/listinfo/tiff</a><b=
r>
<a href=3D"http://www.remotesensing.org/libtiff/" rel=3D"noreferrer" target=
=3D"_blank">http://www.remotesensing.org/libtiff/</a><br>
</div></div></blockquote></div><br></div></div></div></div></div></div></di=
v></div></div></div></div>

--001a11487efe507ba6052a3ce5a4--

--===============0403851679==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Tiff mailing list: Tiff@lists.maptools.org
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
--===============0403851679==--


