Return-Path: <awaresystems.be+caf_=filtered=awaresystems.be@gmail.com>
Delivered-To: filtered@awaresystems.be
Received: from mx-m.one.com (mx-m.one.com [91.198.169.60])
	by mail6.local.one.com (Postfix) with ESMTP id B3429300306283
	for <filtered@awaresystems.be>; Mon, 15 Dec 2014 22:52:42 +0000 (UTC)
X-Virus-Scanned: by one.com
Received-SPF: pass (mx-m: domain of awaresystems.be+caf_=filtered=awaresystems.be@gmail.com designates 209.85.217.182 as permitted sender)
X-Greylist: whitelisted by SQLgrey-1.8.0-rc2
Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182])
	by mx-m.one.com (Postfix) with ESMTP id 7D58C2081E590
	for <filtered@awaresystems.be>; Mon, 15 Dec 2014 22:52:42 +0000 (UTC)
Received: by mail-lb0-f182.google.com with SMTP id f15so10924970lbj.27
        for <filtered@awaresystems.be>; Mon, 15 Dec 2014 14:52:42 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20130820;
        h=x-original-authentication-results:delivered-to:delivered-to:from:to
         :subject:date:user-agent:cc:references:in-reply-to:mime-version
         :content-type:content-transfer-encoding:message-id;
        bh=TNbaMQtK+jQgKpuU+q56E2O27aoFJnygckPeXD25sWI=;
        b=b8bWtAMC2AaGedRmPZheP2PA38s1ZG5cu+ittkfZe8THjjOrgkU8eT/sB+Wp4ddKQL
         4A27g2lDr4gcqzRe6VqRCBb5ObcDwFXigUO2eAm74/+c0U51gtHhM9MVOSavTOnowylq
         o2ERK3mSKgcUWR68/et4URTA2LEtbWBgevjsciSubi8FYbYCgjjhQlwxJVJDUKy84Npz
         fxQhnZb7pEehZ3b19Pfg6hE8YmcG1DUCIV1wxgErg3ItqlGdS0zdJZzSrOyjiSQawua2
         bEJiWDDuk0xxKuXzeYV30Kfdh1yyBmIlQ+Bt6laYnJUAAN33mE8hiyIJRdhCPbfzWSx+
         m9lA==
X-Original-Authentication-Results: mx.google.com;       spf=none (google.com: even.rouault@spatialys.com does not designate permitted sender hosts) smtp.mail=even.rouault@spatialys.com
X-Received: by 10.152.1.201 with SMTP id 9mr8542273lao.36.1418683962250;
        Mon, 15 Dec 2014 14:52:42 -0800 (PST)
X-Forwarded-To: filtered@awaresystems.be
X-Forwarded-For: awaresystems.be@gmail.com filtered@awaresystems.be
Delivered-To: awaresystems.be@gmail.com
Received: by 10.114.181.134 with SMTP id dw6csp316184ldc;
        Mon, 15 Dec 2014 14:52:41 -0800 (PST)
X-Received: by 10.152.88.44 with SMTP id bd12mr32157875lab.88.1418683957939;
        Mon, 15 Dec 2014 14:52:37 -0800 (PST)
Authentication-Results: mx.google.com;
       spf=none (google.com: even.rouault@spatialys.com does not designate permitted sender hosts) smtp.mail=even.rouault@spatialys.com
Received-SPF: none (google.com: even.rouault@spatialys.com does not designate permitted sender hosts) client-ip=91.198.169.55;
Received: by 10.152.45.225 with POP3 id q1mf4813070lam.37;
        Mon, 15 Dec 2014 14:52:37 -0800 (PST)
X-Gmail-Fetch-Info: info@awaresystems.be 1 pop.awaresystems.be 110 info@awaresystems.be
Delivered-To: info@awaresystems.be
Received: from mx-h.one.com (mx-h.one.com [91.198.169.55])
	by mail6.local.one.com (Postfix) with ESMTP id A6E03300306283
	for <info@awaresystems.be>; Mon, 15 Dec 2014 22:49:11 +0000 (UTC)
X-Greylist: whitelisted by SQLgrey-1.8.0-rc2
Received: from smtp5-g21.free.fr (smtp5-g21.free.fr [212.27.42.5])
	by mx-h.one.com (Postfix) with ESMTP id 7A2A0C6AED9
	for <info@awaresystems.be>; Mon, 15 Dec 2014 22:49:11 +0000 (UTC)
Received: from even-desktop.localnet (unknown [88.162.84.61])
	by smtp5-g21.free.fr (Postfix) with ESMTPS id D602DD48002;
	Mon, 15 Dec 2014 23:47:30 +0100 (CET)
From: Even Rouault <even.rouault@spatialys.com>
To: "Joris Van Damme (AWare Systems)" <info@awaresystems.be>
Subject: Re: [Tiff] Unnecessary bigger JPEG-compressed TIFF
Date: Mon, 15 Dec 2014 23:49:08 +0100
User-Agent: KMail/1.13.5 (Linux/2.6.32-67-generic; KDE/4.4.5; x86_64; ; )
Cc: TIFF <tiff@lists.maptools.org>,
 gdal-dev@lists.osgeo.org
References: <201412151849.41073.even.rouault@spatialys.com> <201412152117.36105.even.rouault@spatialys.com> <CA+NZCsS8Om-hNsSTH6h_n3GEATD3t+qcVgJDa0UcLw--4ewSAQ@mail.gmail.com>
In-Reply-To: <CA+NZCsS8Om-hNsSTH6h_n3GEATD3t+qcVgJDa0UcLw--4ewSAQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: Text/Plain;
  charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Message-Id: <201412152349.08894.even.rouault@spatialys.com>

Le lundi 15 d=C3=A9cembre 2014 23:09:17, Joris Van Damme (AWare Systems) a =
=C3=A9crit :
> Even,
>=20
> > Hum http://www.remotesensing.org/libtiff/TIFFTechNote2.html mentions "An
> > image segment may not redefine any table defined in JPEGTables.", so my
> > understanding is that a strip/tiff that would want to have custom tables
> > should use different table numbers.
>=20
> Yes, that is correct. My choice of words was unfortunate. From your
> next comments however, I can see you understood my bad choice of words
> hardly matters, using different tables number rather then overriding
> the same number, doesn't change anything.

Agreed

>=20
> >> for whatever strips or tiles it wishes to update, but still leave the
> >> tag as is to apply to any other strip or tile that may not override
> >> it.
> >=20
> > Yes I agree the JPEG-in-TIFF specification allows this, but my
> > understanding libtiff has never really supported this. It would require
> > comparing the tables generated with the current quality setting with the
> > ones in JpegTables tag, and if they do not match, install new tables and
> > reference them in the libjpeg component definition.
>=20
> I'm not sure I understand correctly. Are you talking about what is
> required in the encoder changing a separate strip or tile this way, or
> the decoder handling the resulting file? Either case, I don't think I
> agree, but I may be missing something.

Was only about the encoder. I believe the decoder should work OK already. =
=20
Obviously it did since for 4 years, it has produced files with redefined=20
quantization tables in strip/tiles, and actually with the same table number=
 as=20
in the JpegTable tag...
I'm not sure however if it works in conformant files where the redefined ta=
bles=20
would have different numbers (perhaps it does, just haven't a test example=
=20
handy)

>=20
> In the encoder, it should not be required to compare any tables at
> all, in order to use this updating scheme. The only reason why an
> encoder that proceeds this way, would even read the JpegTables data,
> would be so as to make sure it does indeed use different table
> numbers.
>=20
> This is exactly the reason why I regard this "overriding with
> different table numbers" the most intuitive way to update single
> strips or tiles in a JPEG-in-TIFF file.
>=20
> In the decoder; strictly speaking, what is required is to load the
> JpegTables data into the JPEG subcodec before decoding the first strip
> or tile data with the same said subcodec. At least, that's how it
> would work with LibJpeg, there's some level of implementation detail
> involved in the fact that fully decoding a tables-only stream is the
> standard way to "infuse" LibJpeg with those tables.
>=20
> In a more robust implementation of the decoder, one may want to
> protect against a strip/tile redefining tables and mistakenly using
> the same table numbers. That would be reasonably simple, the way to do
> this would be to load the JpegTables data into the JPEG subcodec
> before decoding *any* strip or tile, instead of merely doing it before
> decoding *the first* strip or tile. That's how my codec does it. This
> more robust implementation is actually a requirement anyway for a
> decoder that wishes to support decoding different strips or tiles in
> different thread contexts, for better multicore usage, like mine.
>=20
> If full decoding of a tables-only JPEG stream is the only way to
> infuse the JPEG subcodec with tables, I can see there may be a need to
> quantify the possible overhead involved in this before deciding upon
> this "more robust" implementation. In my own JPEG codec, I decided to
> make the tables a separate class for this exact reason, with the
> caller level being able to simply "transplant and reuse" decoded
> tables into other JPEG decoder objects. Maybe this feature got added
> to LibJpeg, I don't know anything about LibJpeg beyond version 6b.
>=20
> So I'm unsure why any table comparison would be required with the
> scheme of strip/tile updating, anywhere.
>=20
> Unless you're  specifically talking about space-saving in strip/tile
> updating, by comparing the planned "local overriding" tables with the
> default tables in the JpegTables tag, and not overriding at all if
> equal? I wasn't thinking along those lines in my implementation. With
> tables being implementation specific, possibly image-specific,
> possibly specific to other factors like LibJpeg quality inside the
> same single implementation, etc, I chose to regard matching tables as
> astronomically improbable. Besides, my reasoning is, if table size
> does contribute even remotely significantly to the file size, then
> RowsPerStrip or TileWidth+TileHeight is very badly chosen.=20

Well, I think the intent of TIFFTAG_JPEGTABLESMODE =3D 3 (the default mode)=
 is=20
to avoid writing quantization&huffman tables in tiles/strips when it is not=
=20
necessary. And more generally achieving the smallest binary size seems to b=
e a=20
reasonable objective.

> I can see
> how it may be justified to regard matching tables a legit and way more
> probable concept, though, in your part of the world. Working with
> LibTiff and LibJpeg you sort-off have the luxury you can assume 98% of
> the planet sharing your implementation so the words
> "implementation-specific" almost loose their meaning. If next you're
> able to work around the "libjpeg-implementation-specific
> quality-specific" issue as well... Still, there's the "image-specific"
> issue, at least for huffman tables, when huffman optimization is
> applied, isn't there? Or does LibTiff+LibJpeg not support huffman
> optimization in this context?

AFAICS, in JPEGTABLESMODE_HUFF mode, huffman optimization is disabled.

>=20
> > libtiff + libjpeg... tricky
>=20
> Absolutely. There's a humongous amount of detail to keep in your head
> wherever LibTiff and LibJpeg mix. I'm sorry if I added to it.

One of the issue that was encountered in the past and lead to the regressio=
n I=20
fixed (tried to fix...) was when switching back and forth between directori=
es=20
that might have different current JPEG tables. Each time you do this, the J=
PEG=20
state is reset, so you have to reload the tables in case the quality settin=
gs=20
were different between the directories, and in the current state of tif_jpe=
g.c=20
the easiest solution is to call jpeg_set_quality().

I'm not saying doing something better is impossible, it is just that I don'=
t=20
feel pursuing that right now ;-) Trivial changes in that code have a high=20
chance of causing issues in some cases, so non-trivial changes, and especia=
lly=20
since I don't think the test suite actually tests that a lot.

Even

=2D-=20
Spatialys - Geospatial professional services
http://www.spatialys.com
