For the email part of an app I'm considering several options for the db structure and I'm torn as to the best choice. Its essentially a split between "good practice" and speed of display. I thought I'd ask a savvy group for their opinions, or at least start a good argument

Options:

1. store email as receive and decode if necessary each time its viewed
This will definitely be the slowest for viewing. Most emails (that I currently have on file) will decode in <150ms, but some (eg a 15mb attachment) take up to 2.5 seconds. It does give me the original data and is second best in terms of storage.

2. store only the decoded format
Fast for viewing since its just a matter of throw text at htmlviewer, best for storage since it doesn't have the mime encoding overhead but I loose the original email

3. store email as received and in decoded format
As good as 2 for viewing, gives me the original email but needs about double the storage.

I'm leaning towards 3 but I hate the data duplication.

All input will be read with interest.



Roy Lambert

Roy Lambert wrote:
....
> 3. store email as received and in decoded format
> As good as 2 for viewing, gives me the original email but needs about
> double the storage.

That would be my choice. And you should consider to store the data in a
compressed form, at least the viewing part.

Ralf

I'd go with #1, but separate decoding into separate thread and
cache decoded messages within current session. I believe this is
how it's done by most mail clients. I think you must store messages
as is. Optionally you can store decoded version. However, you don't
have to store them all permanently - just cache 100-200 last accessed
messages, that's it.

Regards,
Michael

Roy Lambert wrote:
> For the email part of an app I'm considering several options for the db structure and I'm torn as to the best choice. Its essentially a split between "good practice" and speed of display. I thought I'd ask a savvy group for their opinions, or at least start a good argument
>
> Options:
>
> 1. store email as receive and decode if necessary each time its viewed
> This will definitely be the slowest for viewing. Most emails (that I currently have on file) will decode in <150ms, but some (eg a 15mb attachment) take up to 2.5 seconds. It does give me the original data and is second best in terms of storage.
>
> 2. store only the decoded format
> Fast for viewing since its just a matter of throw text at htmlviewer, best for storage since it doesn't have the mime encoding overhead but I loose the original email
>
> 3. store email as received and in decoded format
> As good as 2 for viewing, gives me the original email but needs about double the storage.
>
> I'm leaning towards 3 but I hate the data duplication.
>
> All input will be read with interest.
>
>
>
> Roy Lambert

Michael


>I'd go with #1, but separate decoding into separate thread and
>cache decoded messages within current session. I believe this is
>how it's done by most mail clients. I think you must store messages
>as is. Optionally you can store decoded version. However, you don't
>have to store them all permanently - just cache 100-200 last accessed
>messages, that's it.

Hmm. That's sort of what I'm doing in TMaN, just storing the decoded details as well. Unless you can decode a wadge in advance I'm not sure of the benefit of decoding in a thread. You can't display anything until its decoded, at least the message part, and in my tests, there's little difference between just decoding the message and decoding the lot.

I don't know about you, but most of the time with emails, I just go in one direction eg next one unread, or open a mailbox and scroll in one direction to find the one I want. Only rarely do I go up and down so I'm not sure how I could decide what to cache.

Mind you I'm always willing to learn.

Roy Lambert

Hi Roy,

Agree with Mike B

> I'm not sure of the benefit of decoding in a thread

Your front-end will appear to hang during the decode otherwise.

> Only rarely do I go up and down so I'm not sure how I could decide what to
> cache.

Just create a size-managed object list, return an existing entry (if already
in the list), or add a new entry (if not in the list), then delete the first
entry (i.e the oldest) if appropriate. You could allow you users to set the
cache size. Revistiing an entry would be blindingly fast then (much faster
than db persistence of decoded message)
--
Best regards

Steve

"Roy Lambert" <roy.lambert@skynet.co.uk> wrote in message
news:19990744-BD52-484F-9A22-FA4C48A97F59@news.elevatesoft.com...

Steve

>> I'm not sure of the benefit of decoding in a thread

>Your front-end will appear to hang during the decode otherwise.

It may just be me being thick but consider the case where I'm visiting a record for the first time. If I don't store the decoded email in a database then before I can display anything it has to be decoded. So I decode in foreground and then display, the app "hangs" whilst its being decoded or I decode in a thread and the app waits for the thread to complete so it can display something again appearing to "hang". From your answer there's something wrong with my thinking, but my poor old brain can't see what.

One approach I have thought of is start decoding the entire mailbox in a thread, use a memory table to hold the decoded messages, if the decode keeps ahead of the user great, if not I have to decode in the foreground for the one just hit, and then store it in the memory table. Any comments.

>> Only rarely do I go up and down so I'm not sure how I could decide what to
>> cache.
>
>Just create a size-managed object list, return an existing entry (if already
>in the list), or add a new entry (if not in the list), then delete the first
>entry (i.e the oldest) if appropriate. You could allow you users to set the
>cache size. Revistiing an entry would be blindingly fast then (much faster
>than db persistence of decoded message)

Not worked with object lists, and I can see that working with memory would be faster than disk. I'll have to move up and down the table anyway so I'll have a play and see what sort of speed difference I can get. The vast majority of emails will decode in less than 150ms (in fact most are probably sub 50ms) so for display purposes unless I have superman pressing the up and down arrow key its not a problem its just those with large attachments where it can take up to 3 seconds.

Any suggestions about your approach, code snippets for a moron etc would be well appreciated, especially as to how multiple attachments, in-line images would be handled. At present, in TMaN, these are stored in a memory table which is streamed into a memo field.

Roy Lambert

Hi Roy,

> It may just be me being thick but consider the case where I'm visiting a
> record for the first time.

I understand what you mean, but without a thread, the application will
actually hang until the message is decoded. If you use a thread you can show
progress of the decode, allow the user to select other messages for decoding
... i.e. continue using the application until the thread process finishes.

> Any suggestions about your approach, code snippets for a moron etc would
> be well appreciated

Possibly the quickest and easiest to get you started would be to use a
string list, the strings would hold your message references and the related
objects would contain an instance of your container class with the decoded
message (and any other stuff about the message)

e.g.

type
 TTMANMessage = class(TObject)
   private
     FMessageID: Integer;
     FMessage: String;
     etc.
   public
     MessageID: Integer read FMessageID write FMessageID;
     Message: String read FMessage write FMessage;
     etc.
 end;

 private
   MessageList: TStringList;


procedure AddMessage(Message: TTMANMessage);
begin
 if (MessageList.IndexOf(IntToStr(Message.MessageID)) = -1) then
 begin
   if MessageList.Count = 10 then
   begin
     TTMANMessage(MessageList.Objects[0]).Free;
     MessageList.Delete(0);
   end;
   MessageList.AddObject(IntToStr(Message.MessageID), Message)
 end;
end;

function GetMessage(const MessageId: Integer; out Message: TTMANMessage):
Boolean;
var
 Index: Integer;
begin
 Result := False;
 Index := MessageList.IndexOf(IntToStr(MessageID));
 if (Index <> -1) then
 begin
   Message := MessageList.Objects[Index];
   Result := True;
 end;
end;

Very rough, but should give you the idea.
--
Best regards

Steve

"Roy Lambert" <roy.lambert@skynet.co.uk> wrote in message
news:8AF17B82-4E3A-4D05-8195-25B82475981F@news.elevatesoft.com...
> Steve
>
>>> I'm not sure of the benefit of decoding in a thread
>
>>Your front-end will appear to hang during the decode otherwise.
>
> It may just be me being thick but consider the case where I'm visiting a
> record for the first time. If I don't store the decoded email in a
> database then before I can display anything it has to be decoded. So I
> decode in foreground and then display, the app "hangs" whilst its being
> decoded or I decode in a thread and the app waits for the thread to
> complete so it can display something again appearing to "hang". From your
> answer there's something wrong with my thinking, but my poor old brain
> can't see what.
>
> One approach I have thought of is start decoding the entire mailbox in a
> thread, use a memory table to hold the decoded messages, if the decode
> keeps ahead of the user great, if not I have to decode in the foreground
> for the one just hit, and then store it in the memory table. Any comments.
>
>>> Only rarely do I go up and down so I'm not sure how I could decide what
>>> to
>>> cache.
>>
>>Just create a size-managed object list, return an existing entry (if
>>already
>>in the list), or add a new entry (if not in the list), then delete the
>>first
>>entry (i.e the oldest) if appropriate. You could allow you users to set
>>the
>>cache size. Revistiing an entry would be blindingly fast then (much faster
>>than db persistence of decoded message)
>
> Not worked with object lists, and I can see that working with memory would
> be faster than disk. I'll have to move up and down the table anyway so
> I'll have a play and see what sort of speed difference I can get. The vast
> majority of emails will decode in less than 150ms (in fact most are
> probably sub 50ms) so for display purposes unless I have superman pressing
> the up and down arrow key its not a problem its just those with large
> attachments where it can take up to 3 seconds.
>
> Any suggestions about your approach, code snippets for a moron etc would
> be well appreciated, especially as to how multiple attachments, in-line
> images would be handled. At present, in TMaN, these are stored in a memory
> table which is streamed into a memo field.
>
> Roy Lambert

Hi Roy,

Obviously, multi-threading can only work when you're able to split the
process into things that can be done in parallel. In your case, you only
have problem decoding really long messages with attachments. This means
that you need to split message decoding process into decoding main message
body (which is instantaneous) and decoding attachments (which may take long).
You create a thread and give that thread your message Blob stream and
resulting TMessage object (empty at this time).
Right after that you create your message viewing form connected to that
same TMessage object.
The thread will read message stream and fill out the TMessage object with
portions of information, like body, attachment 1, 2, etc. After each
portion it will invoke some method of TMessage (synchronized), so that
TMessage can notify UI that it has changed. This way, you open up
the form instantly, then once the body is decoded you show the body portion,
then when the attachments are decoded you show that portion and so on.
The UI will not freeze at all in this case and your user will be able
to read the message while decoding thread is processing huge attachments.
Once you've decoded a message and the user closes the form, you don't drop the
TMessage object, but save it (cache) someplace (i.e. TStringList as Steve
suggested). The next time user needs to view a message you check if
your TStringList already contains that message and if yes - display
it instantly, otherwise, start the decoding thread. Ideally, you allocate
some space to store message in and keep them decoded between user
sessions (something like IE cache). This way your most often read
messages (in inbox) will be always decoded and available.

The most tricky part in this approach is to implement TMessage in a
thread safe manner, cause it needs to share its data between UI form
and decoding thread. All the data access properties/methods must therefore
be wrapped into critical sections and UI calls made from decoding thread
must be synchronized.

Regards,
Michael

Roy Lambert wrote:
> Steve
>
>>> I'm not sure of the benefit of decoding in a thread
>
>> Your front-end will appear to hang during the decode otherwise.
>
> It may just be me being thick but consider the case where I'm visiting a record for the first time. If I don't store the decoded email in a database then before I can display anything it has to be decoded. So I decode in foreground and then display, the app "hangs" whilst its being decoded or I decode in a thread and the app waits for the thread to complete so it can display something again appearing to "hang". From your answer there's something wrong with my thinking, but my poor old brain can't see what.
>
> One approach I have thought of is start decoding the entire mailbox in a thread, use a memory table to hold the decoded messages, if the decode keeps ahead of the user great, if not I have to decode in the foreground for the one just hit, and then store it in the memory table. Any comments.
>
>>> Only rarely do I go up and down so I'm not sure how I could decide what to
>>> cache.
>> Just create a size-managed object list, return an existing entry (if already
>> in the list), or add a new entry (if not in the list), then delete the first
>> entry (i.e the oldest) if appropriate. You could allow you users to set the
>> cache size. Revistiing an entry would be blindingly fast then (much faster
>> than db persistence of decoded message)
>
> Not worked with object lists, and I can see that working with memory would be faster than disk. I'll have to move up and down the table anyway so I'll have a play and see what sort of speed difference I can get. The vast majority of emails will decode in less than 150ms (in fact most are probably sub 50ms) so for display purposes unless I have superman pressing the up and down arrow key its not a problem its just those with large attachments where it can take up to 3 seconds.
>
> Any suggestions about your approach, code snippets for a moron etc would be well appreciated, especially as to how multiple attachments, in-line images would be handled. At present, in TMaN, these are stored in a memory table which is streamed into a memo field.
>
> Roy Lambert

Michael


Thanks for the pointers. I've only done a bit of work with threads and I know how they bite if you don't get them isolated properly

I think I'm going to go for a compound approach. I'll store the encoded email, the decoded message body, subject etc, decoded in-line and web graphics and a list of attachments. Scrolling though the table will then be fast, and if someone wants to open an attachment it may take a couple of seconds longer which should be fine.

Roy Lambert

Get BMThread components from Torry (free with src).  Make threading a safe, fast pleasure
to use.  If I can use it, anyone can!
Yet to find their equal and they have nifty design-time editors to set up what happens
where.  What have you got to lose?

Cheers,
Graham W.

Roy Lambert <roy.lambert@skynet.co.uk> wrote:

Michael


Thanks for the pointers. I've only done a bit of work with threads and I know how they
bite if you don't get them isolated properly

I think I'm going to go for a compound approach. I'll store the encoded email, the decoded
message body, subject etc, decoded in-line and web graphics and a list of attachments.
Scrolling though the table will then be fast, and if someone wants to open an attachment
it may take a couple of seconds longer which should be fine.

Roy Lambert