Return-Path: <awaresystems.be+caf_=filtered=awaresystems.be@gmail.com>
Delivered-To: filtered@awaresystems.be
Received: from mx5.public.one.com ([10.246.18.25])
	by mail6.local.one.com (Dovecot) with LMTP id IU19C0aoU1lHEwAAsyOMew
	for <filtered@awaresystems.be>; Wed, 28 Jun 2017 13:54:12 +0000
X-HalOne-ID: 42f836cd-5c09-11e7-bb21-b82a72d82f61
Received: from mail-oi0-f52.google.com (unknown [209.85.218.52])
	by mx5.public.one.com (Halon) with ESMTPS
	id 42f836cd-5c09-11e7-bb21-b82a72d82f61;
	Wed, 28 Jun 2017 13:54:11 +0000 (UTC)
Received: by mail-oi0-f52.google.com with SMTP id p187so41866743oif.3
        for <filtered@awaresystems.be>; Wed, 28 Jun 2017 06:54:11 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-original-authentication-results:x-gm-message-state:delivered-to
         :mime-version:in-reply-to:references:from:date:message-id:to:cc
         :subject:precedence:list-id:list-unsubscribe:list-archive:list-post
         :list-help:list-subscribe:content-transfer-encoding:sender:errors-to;
        bh=cskS4ITw2xEN840v8rqRJg+Dre6BKqauUbR+esvnYNM=;
        b=pHAzaKT81+HAijUdwSZ/51aReBFz64WjUyWpxx9isSYSj+IEqv5RsYv2IF1cENebJL
         on17fF0xSm5OBlLm7tdIQtTCgurjaZw+RvGhVipJsMdX8w9gL0l8L+S7yXCU6tGnhR4g
         OV03LpXPEr97eeIY7gvV4DPxB9iQ3Jzp37j9CtfcC9sbHKeMs8gCFH3DPE5p9WbZ7V6Z
         Xj0hjJNqnl+Sa7BeHgoqBkKB4GQPtYSEc2EqzSs3pCSMfOEB5C3pB4528rJZALGf5XXs
         3n8ZNXt6xhVGtZQupHnm2XQ1kyNNx2Kv5wTGBaTPZ+yDulksq2+XNGrvTyNuKGZWrnva
         I9Bw==
X-Original-Authentication-Results: mx.google.com;       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org;       dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=gmail.com
X-Gm-Message-State: AKS2vOzQf82QOFjGhHXiYbbWxOS0mNxG6+Y6RKPnye4CzHrKKK7qGOQH
	emTaprT/VR4SZcnQ9VByFwYEry2mUJ/EvvqZlIoCHg==
X-Received: by 10.202.65.9 with SMTP id o9mr5707950oia.134.1498658050150;
        Wed, 28 Jun 2017 06:54:10 -0700 (PDT)
X-Forwarded-To: filtered@awaresystems.be
X-Forwarded-For: awaresystems.be@gmail.com filtered@awaresystems.be
Delivered-To: awaresystems.be@gmail.com
Received: by 10.202.64.84 with SMTP id n81csp3571475oia;
        Wed, 28 Jun 2017 06:54:08 -0700 (PDT)
X-Received: by 10.28.140.194 with SMTP id o185mr8020989wmd.87.1498658048576;
        Wed, 28 Jun 2017 06:54:08 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; t=1498658048; cv=none;
        d=google.com; s=arc-20160816;
        b=KZAxm35izVE3yMYKr98N4n5VGCG3zZEqGx8uxX3UZfUgUwqLHYcBWGSfmFOyfLmHXD
         mpnAj+9B2m2IHHBeqv20cRCnSdKNdjJ6JwHOc6epVI6NJy5kxvx1tHj91TdufiDyCcTw
         sS53JsTR1oQDepcknL8cjDWnfsrkGXNJR7T7YmkctCDHEVdTkQtdbE3K0PKy+8qng0lI
         zg0qFMLjDRWvYK65nybVjwCbaUSVUjWTgPSw37K1cpv+PtkyGG9L0I3lJHmXqKeoaQfG
         aGyDQcCDFaq36+dm+dkt8yVnWmhgWhQ+XZjQvmGD5glNYyxDT2tsdtGYcelNfHMi0F8d
         3wHQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
        h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help
         :list-post:list-archive:list-unsubscribe:list-id:precedence:subject
         :cc:to:message-id:date:from:references:in-reply-to:mime-version
         :arc-authentication-results;
        bh=cskS4ITw2xEN840v8rqRJg+Dre6BKqauUbR+esvnYNM=;
        b=kOTiEESyYwr1MtjQvD1cDsB6Hbqym3TDG5yQvtTt5dMx8VJFLZZlx2xmZR2gkd0qdk
         IBRS1YneduCbHhcqX2FOMmkTZ5GKvkR/wJZgccrOUcbE6Tj2iG72tL2d7CN8I59Cr4NJ
         fb2qCg8f+agc2cUN9vn1YyFvX/Z/h+czmktwVPPJZCMZL7AV/MYyQja2QQY4k4ywZg7A
         trxEQBvfbzN+mvCF0bzuOGNUz/VnyKceIY7RdSii55EjjdgjZCNP5BpTvt8DuR8SYv5S
         yxatzVFHS0Go8OITo5m18B4w/SxV2Nbb7pPqCa1loNzhQ3VHec2MPbmrOh1eu9GN0CGX
         ErNA==
ARC-Authentication-Results: i=1; mx.google.com;
       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org;
       dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=gmail.com
Return-Path: <tiff-bounces@lists.maptools.org>
Received: from lists.maptools.org (ks365428.kimsufi.com. [94.23.4.142])
        by mx.google.com with ESMTP id y93si1858238wrb.0.2017.06.28.06.54.07;
        Wed, 28 Jun 2017 06:54:08 -0700 (PDT)
Received-SPF: neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) client-ip=94.23.4.142;
Authentication-Results: mx.google.com;
       spf=neutral (google.com: 94.23.4.142 is neither permitted nor denied by best guess record for domain of tiff-bounces@lists.maptools.org) smtp.mailfrom=tiff-bounces@lists.maptools.org;
       dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=gmail.com
Received: from [10.1.0.241] (localhost.localdomain [127.0.0.1])
	by lists.maptools.org (Postfix) with ESMTP id 8B34F35C051;
	Wed, 28 Jun 2017 08:52:53 -0500 (EST)
Received: from mail-yb0-f170.google.com (mail-yb0-f170.google.com
	[209.85.213.170])
	by lists.maptools.org (Postfix) with ESMTPS id 4FD4935C04F
	for <tiff@lists.maptools.org>; Wed, 28 Jun 2017 08:49:58 -0500 (EST)
Received: by mail-yb0-f170.google.com with SMTP id e201so19219560ybb.1
	for <tiff@lists.maptools.org>; Wed, 28 Jun 2017 06:49:58 -0700 (PDT)
X-Received: by 10.37.54.31 with SMTP id d31mr7829157yba.165.1498657797632;
	Wed, 28 Jun 2017 06:49:57 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.37.171.147 with HTTP; Wed, 28 Jun 2017 06:49:56 -0700 (PDT)
In-Reply-To: <1588415.R2acFO4bS3@even-i700>
References: <CABoQFUADrHiYLs7=2gVoRGkpex3P2jHWRPCZhehS8+OQRxTvYQ@mail.gmail.com>
	<17844748.quhNjf2p3V@even-i700>
	<CAAcuQeYE_e+5AAMTqqosSWDYvj142bUh1FMyKSZQhgLiKU_zrQ@mail.gmail.com>
	<1588415.R2acFO4bS3@even-i700>
From: Olivier Paquet <olivier.paquet@gmail.com>
Date: Wed, 28 Jun 2017 09:49:56 -0400
Message-ID: <CAAcuQeYOCExBMHb15EsYpUttGKyeJ0y=Xmqtf-nkUgM_o61GXQ@mail.gmail.com>
To: Even Rouault <even.rouault@spatialys.com>
Cc: TIFF mailing list <tiff@lists.maptools.org>
Subject: Re: [Tiff] Excessive memory allocation while chopping strips
X-BeenThere: tiff@lists.maptools.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: TIFF and libtiff discussion <tiff.lists.maptools.org>
List-Unsubscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=unsubscribe>
List-Archive: <http://lists.maptools.org/mailman/private/tiff>
List-Post: <mailto:tiff@lists.maptools.org>
List-Help: <mailto:tiff-request@lists.maptools.org?subject=help>
List-Subscribe: <http://lists.maptools.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@lists.maptools.org?subject=subscribe>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Sender: tiff-bounces@lists.maptools.org
Errors-To: tiff-bounces@lists.maptools.org

2017-06-28 6:09 GMT-04:00 Even Rouault <even.rouault@spatialys.com>:
> I wouldn't call the tool "broken". It is a reasonable behaviour in most
> cases

I would. It's reporting problems for code which runs fine. It's
causing "fixes" to be written which may introduce new security issues,
the very thing the tool is meant to detect in the first place.

That does not mean we might not still be better off with the tool than
without. But we have to be careful about what we do based on its
reports. It reminds me a lot of the early days of valgrind, where I'd
"fix" some things which we not real bugs to be able to keep using the
tool. They were simpler fixes, mostly uninitialized memory, but even
those could have unwanted consequences (eg. calloc() is a whole lot
slower than malloc()).

> For strip reading, there's the mechanism I already put in place and
> described in my previous email. Small files that would declare large strip
> byte count will no longer cause the full allocation to be done before
> beginning reading the strip content.

Looks like a reasonable fix but:
- It's not that simple (should ideally have a few pair of eyes go over it).
- It made reading the large strips a little slower.
- It may have increased the peak amount of memory required (in case
realloc needs to copy).
- It does not help with compressed strips, as I understand it.

>> But otherwise, it looks like something which might be
>
>> useful to someone required to deal with unsafe input.
>
>
> Given the number of teams that fuzz libtiff / tiff tools and report issues
> in Bugzilla, it seems there's a lot of interest for such use cases.

I don't know about that. Just because people like to use their shiny
new fuzzer on everything they can get their hands on does not mean
code will be written to use the new API. I might be wrong but I get
the feeling that in many cases libtiff is being dragged along
indirectly with a bunch of other dependencies, typically through some
flavor of an abstract image handling library. Do you know better from
reading the bug reports?

If we're going to have this, I think it should be the default behavior
across the board. We should first fix the cases needed by well formed
large files (ie. progressive allocation for large tags, as you did for
the strips). Then simply refuse strips/tiles larger than X (say, 1 GB
or even 100 MB) and add a flag to TIFFOpen() to override that for the
rare few projects which really need to deal with such files and don't
care about the potential consequences. Making it the default requires
that we make it work well and ensures it will be used. It's also more
work, unfortunately.

Olivier
_______________________________________________
Tiff mailing list: Tiff@lists.maptools.org
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
