Return-Path: <tiff-admin@remotesensing.org>
Received: from tee.skynet.be (tee.skynet.be [195.238.3.4])
        by twentyeight.skynet.be (8.12.9/8.12.9/Skynet-MAILSTORE-2.14) with ESMTP id hBGFZIn5015123
        for <joris.at.lebbeke@skynet.be>; Tue, 16 Dec 2003 16:35:18 +0100
        (envelope-from <tiff-admin@remotesensing.org>)
Received: from yoda.imagelinks.com ([208.24.120.42])
        by tee.skynet.be (8.12.9/8.12.9/Skynet-IN-2.32) with ESMTP id hBGFYrnC031175
        for <joris.at.lebbeke@skynet.be>; Tue, 16 Dec 2003 16:34:53 +0100
        (envelope-from <tiff-admin@remotesensing.org>)
Received: from yoda.imagelinks.com (localhost.localdomain [127.0.0.1])
	by yoda.imagelinks.com (8.11.6/8.11.2) with ESMTP id hBGFVXb24322;
	Tue, 16 Dec 2003 10:31:33 -0500
Received: from december.skynet.be (december.skynet.be [195.238.3.62])
	by yoda.imagelinks.com (8.11.6/8.11.2) with ESMTP id hBGFURb24282
	for <tiff@remotesensing.org>; Tue, 16 Dec 2003 10:30:29 -0500
Received: from dfgrrxsknr4e (162.162-201-80.adsl.skynet.be [80.201.162.162])
        by december.skynet.be (8.12.9/8.12.9/Skynet-OUT-2.21) with SMTP id hBGFUEPg018908;
	Tue, 16 Dec 2003 16:30:14 +0100
        (envelope-from <joris.at.lebbeke@skynet.be>)
Message-ID: <000b01c3c3ea$14badce0$a2a2c950@dfgrrxsknr4e>
From: "Joris" <joris.at.lebbeke@skynet.be>
To: "Frank Warmerdam" <warmerdam@pobox.com>,
   "Tiff mailing list" <tiff@remotesensing.org>
References: <000501c3c189$d2804d90$2022c850@dfgrrxsknr4e> <3FDB3D9C.1030308@pobox.com> <4.2.0.58.20031213162904.00bc9c60@mail.cerious.com> <4.2.0.58.20031216075535.017f13f0@mail.cerious.com> <000d01c3c3db$26d56800$fb2cc850@dfgrrxsknr4e> <3FDF1668.9020909@pobox.com>
Subject: Re: [Tiff] Stupid question
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2800.1158
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165
X-RAVMilter-Version: 8.4.3(snapshot 20030212) (december.skynet.be)
Sender: tiff-admin@remotesensing.org
Errors-To: tiff-admin@remotesensing.org
X-BeenThere: tiff@remotesensing.org
X-Mailman-Version: 2.0.11
Precedence: bulk
List-Help: <mailto:tiff-request@remotesensing.org?subject=help>
List-Post: <mailto:tiff@remotesensing.org>
List-Subscribe: <http://remotesensing.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@remotesensing.org?subject=subscribe>
List-Id: TIFF and libtiff discussion <tiff.remotesensing.org>
List-Unsubscribe: <http://remotesensing.org/mailman/listinfo/tiff>,
	<mailto:tiff-request@remotesensing.org?subject=unsubscribe>
List-Archive: <http://remotesensing.org/pipermail/tiff/>
Date: Tue, 16 Dec 2003 16:34:17 +0100
X-UIDL: 67d7e3bd9b69221d37b7551b0653575a

> > I think my only hope in completing this project is by not going about
things
> > manually anyway. For example, the 'old' on-line archive that goes back
to
> > aug 99 is 6217 messages long, and accessable only through 6217 seperate
HTML
> > pages. I didn't quite feel up to hitting 'Save as' 6217 times, so I brew
a
> > few lines of code that downloaded the 6217 messages to my hard disc with
a
> > single click. I figure that's the only feasable strategy for all
> > post-processing and most of the grouping and indexing too. So I don't
mind
> > CR/LF, in fact, that's going to be quite a relief after brewing code to
> > extract the true messages from the HTML pages and filtering out all
viagra
> > related stuff.
>
> Joris,
>
> Yikes.

You can say that again. ;-)

No, really, it's not that bad. The HTML is very predictable and uniform
accross pages, as is typical for generated HTML, of course, so I coded up a
little thing to extract the first message (your 'test' message, quite a
symbol), and that is already enough to handle most pages.

> Note, Andrey and I are happy to provide more direct access to the
> archives ... such as the mbox format recent archive if you need it.

The extraction from those HTML pages is nearly finished. But I figure I need
to handle this mbox format anyhow if I understand you correctly that this is
the format of the 2003 archive. And I'll most defenetly end up with code
eliminating duplicates, so anything is helpfull.

Nevertheless, the folder with the auto-downloaded HTML pages, each to a
seperate text file, is about 30 meg. This mbox format is a lot more
efficient, but still, I think it'll be 10 to 15 meg, right? It's not going
to be easy to mail that to me, especially seeing my provider's mail servers
are having a bad week again (or is it a bad year?). So, bottom line is: Yes,
I appreciate receive 'em, unless it's too much trouble sending them.

> Also,
> we are happy to host what you come up with as a cleaned up archive.

Let's first wait and see if I get this job done, I'm not yet quite 100% sure
that it's going to be feasable, even with maximum non-manual handling. I've
got a few ideas for usefull regrouping as a kind of context-sensitive help.
Except that it'll be more like 'TIFF-sensitive' instead of
context-sensitive, and more like 'knowledge base' instead of help. That is
not quite the complete archive but is almost guaranteed to be feasable I
think... So I guess something usefull is going to come out of this, anyhow,
even though it may perhaps not be a completely restored archive for the last
15 years or so. Anyway, let's just wait and see first.

As to the hosting... That may be the only part of the job that might be
somewhat beneficial to me. I may want to take you up on the hosting offer,
but I'm also wondering if you would mind me hosting the results myself. Even
if I'm allowed to host it myself, it will - of course - be a completely free
download, no strings or even advertising attached, except for the hosting
domain name and maybe a single pointer to my site or something. If this is
not acceptable, just say so, I'm not going to do anything on this project
that either you or Andrey object to. Your approval is important to me.



Joris

_______________________________________________
Tiff mailing list
Tiff@remotesensing.org
http://remotesensing.org/mailman/listinfo/tiff
