Back at the dawn of time when I wrote my first email program I was on dial up. I was also a member of several Yahoo groups (Delphi related naturally) and Yahoo had a habit of embedding ads into the emails from the groups which were meant to change each time you looked at them requiring you to go on-line. Before I wrote my program I used Outlook Express

To stop the Yahoo groups emails wanting to connect as part of the design for my email program I downloaded web graphics and stored locally.

This worked well for a loooong time then came the Financial Times new way of things. I don't know just what it is they are doing but I suspect it has to do with advertising. The problem is that a web graphic for one of their newsfeed emails may require up to 25 redirects around the web before it finally finds the graphic or gives up. My application will then try to download the missing ones in the foreground which may again take a lot of redirects before giving up. Needless to say this is very annoying and I've finally had enough and I'm going to download missing web graphics in the background.

I use HTMLViewer for email display and the downloaded graphics are stored in a BLOB using a homebrew image store component. My thoughts are to replace the BLOB with an additional table (emImageStore) which would be:

EMailID (autoinc from emails)
ID (integer CRC32 has of URL)
URL (CLOB cos its wildly variable length)
Image (BLOB)

The process would be when HTMLViewer.OnImageRequest is triggered generate a CRC32 has of the image url from the email, use that and the email ID to see if the image is there, if so just use it. If not post a request to a background thread to fetch it from the web, when found post a message back to tell the email program so it can shove it onto the email display. Part of that would be to see if the current email id was different from the one owning the image just found and if so just ignore.

Another minor problem is that users have the option of downloading the web graphics every time the email is viewed (some people are masochists) which is easy currently - just don't bother to file the image store and I should be able to take care of by using either a disk or memory based table depending on they choice.

In many ways I'd like to keep the image store and file with the email but I can't see a way of doing so reliably. Firstly by using a table I can easily discard images for emails as a user scrolls up and down.

I'm sort of rambling in the hope a better scheme will occur to me (or even to one of you)

Roy Lambert

Images are always interesting, due mainly to the fact they are usually
used for tracking. Thus a spammer can tell that you got his email and
know to send more. Thus by default, I don't have any images from the
web shown, except from specific known good senders.

I'd certainly not want the images to be fetched from the web repeatedly
- once is enough, unless perhaps I explicitly request it, and why would
I?

I think your idea is ideal - but perhaps create a work list for the
current email and have them ticked off as the downloads happen and when
all done, refresh the page to get the [now] cached images. That way 23
images don't cause 23 refreshes. You could optimise it with a refresh
after 2 seconds with what you have perhaps, and then a final one.

The images could be stored in a cache database where the time stamp is
used to delete the image after N days or something. Stop junk from
filling it up.

--

Matthew Jones

Matthew

>Images are always interesting, due mainly to the fact they are usually
>used for tracking. Thus a spammer can tell that you got his email and
>know to send more. Thus by default, I don't have any images from the
>web shown, except from specific known good senders.

Unless they build some sort of id in how do they do that? Or to put it another way - can I detect it and disable it?

>I'd certainly not want the images to be fetched from the web repeatedly
>- once is enough, unless perhaps I explicitly request it, and why would
>I?

This is why I started the approach -back when it was a single user (me) that was affected. Then I included it in my business application - if 10 of us were going to read the FT newsfeed did I want to download things 10 times?

>I think your idea is ideal - but perhaps create a work list for the
>current email and have them ticked off as the downloads happen and when
>all done, refresh the page to get the [now] cached images. That way 23
>images don't cause 23 refreshes. You could optimise it with a refresh
>after 2 seconds with what you have perhaps, and then a final one.

Interesting thought I'm going to have to investigate that. 23 refreshes may be preferable to waiting for the images - the FT has a slew of them and it takes (what feels like) ages. I'm also not sure there is a 100% reliable way of determining if HTMLViewer has asked for all the images.

>The images could be stored in a cache database where the time stamp is
>used to delete the image after N days or something. Stop junk from
>filling it up.

Nah - as I've been told so often disk is cheap - more to the point they'll be deleted when the email is.


Roy Lambert

Roy Lambert wrote:

> Unless they build some sort of id in how do they do that? Or to put
> it another way - can I detect it and disable it?

You're new to email aren't you? 8-)

<img src="http://click.email.argos.co.uk/open.aspx?ffcb10-fe8
31175716d0c7a73-fdf911727065057d701d7372-fe998292907a76-ff61167871-fe311
370766d0779771472-ff021571766503&d=40043" width="1" height="1">

That's the first I found, and it is a one pixel (probably transparent)
tracker image. (I altered the GUID rather to break it)

Key is, you can't change it otherwise it won't work to show the image.
All you can do is choose to either show images, or not. Google, in
Gmail, are now doing some fancy thing to stop tracking, but I suspect
that it starts with them cacheing to stop repeat lookups. And they too
don't show images in spam emails.

With regard to the repeated loads, I'd look at some basic timer that
may be on a fast time for good response, but heuristically adjusts to
update according to the speed of input. So if 20 come back in 10ms,
refresh, but if only one in 3 seconds, refresh. Or whatever - tune to
fit our craven need for speed of response while not being horrible on
flicker. (Browsers do this automatically of course, leaving space if
they have the image size to fill later - if you can make your calls to
get the image wait for the cache, that would solve the problem, but I
suspect that HTMLViewer is not designed that way.)

--

Matthew Jones

Matthew

>> Unless they build some sort of id in how do they do that? Or to put
>> it another way - can I detect it and disable it?
>
>You're new to email aren't you? 8-)

Oh yes - it was as recently as D6 I email enabled my first app. Started by using Indy - saved my sanity by switching to Mail2000 and now use Synapse.

However, I have only very recently (ie today) become interested in the subject of tracking. Up til now my focus has been on being able to decode and display the rubbish that some think is correct and bombing spam using header information before I download the body.

This probably explains the massive number of redirects for the images served up by the FT.

>
><img src="http://click.email.argos.co.uk/open.aspx?ffcb10-fe8
>31175716d0c7a73-fdf911727065057d701d7372-fe998292907a76-ff61167871-fe311
>370766d0779771472-ff021571766503&d=40043" width="1" height="1">
>
>That's the first I found, and it is a one pixel (probably transparent)
>tracker image. (I altered the GUID rather to break it)
>
>Key is, you can't change it otherwise it won't work to show the image.
>All you can do is choose to either show images, or not. Google, in
>Gmail, are now doing some fancy thing to stop tracking, but I suspect
>that it starts with them cacheing to stop repeat lookups. And they too
>don't show images in spam emails.

What I can do is decide not to download any image below a certain size (at least I can if the size information is included in the url), or one where there isn't a recognisable image type in it.

My business application already has global switches to turn off image download so it would be easy to prevent download. There's also a switch to say don't bother displaying images in HTMLViewer even if you've got them so I could combine them to give images where wanted. Unfortunately with some sites the email is essentially one large image.

>With regard to the repeated loads, I'd look at some basic timer that
>may be on a fast time for good response, but heuristically adjusts to
>update according to the speed of input. So if 20 come back in 10ms,
>refresh, but if only one in 3 seconds, refresh. Or whatever - tune to
>fit our craven need for speed of response while not being horrible on
>flicker. (Browsers do this automatically of course, leaving space if
>they have the image size to fill later - if you can make your calls to
>get the image wait for the cache, that would solve the problem, but I
>suspect that HTMLViewer is not designed that way.)

HTMLViewer has an event - OnImageRequest - which is called when it needs an image. If I've already got it then its just a matter of stuffing it into a stream and its displayed. There's also a method - OnImageRequested - which can be used to supply an image fetched from elsewhere when its ready.


Roy

Roy Lambert wrote:

> There's also a method - OnImageRequested - which can be used to
> supply an image fetched from elsewhere when its ready.

That sounds ideal then.

--

Matthew Jones