Tim,

I am still occupied in a Java  project, but  have began to think about some planned Delphi
projects/udates lately I had to delay. So I am looking at EDB once again as the db engine
to migrate to.

In  one of the projects when all devices are connected it requires storing around
10000-15000 integer values in every half a second.  At the same time the recorded data is
graphicaly displayed for he user who may be jumping back and forth, so some reading may
occur too. Depending the screen resolution the control displaying the data uses 2000-4000
integer values for a redraw, which happens at least in every half a second.   

I figured that  the only way to do this is to use compressed blob fields. (Or write every
data channel directly to a separate file, if this amount of data is too much to handle for
a DB engine).

The data is  sort of periodic (large part of it is ECG --> 0.5-4 cycles in a sec). In my
experience compression ratio is around 0.3 -0.4  when files with whole recordings are
compressed with standard zip.    


So the main numbers again:

- Max 30000 integers/sec to store - This speed must be maintened for several hours.
- The minimum amount of data  needs to be stored at once is ~16 integer values. (Happens
often.)
- The maximum amount of data needs to be stored at once is ~10000 integer values. (Can be
sliced)
- A block of   10 - 4000  integer values must be read with the best possible speed. - The
speed of reading weighs most.

Every measurement has it's own database, so it stores only some hours of data the most.

Based on these numbers can you make suggestions how to optimize the setting for:

1. Blob block size
2. Blob compression
3. How much data would be optimal to store in a single field?


Leslie

Forgot one settings which may be important:

4. Blob Buffer size.

Leslie,

<< I figured that  the only way to do this is to use compressed blob fields.
(Or write every data channel directly to a separate file, if this amount of
data is too much to handle for a DB engine).  >>

It's not the amount of data that is the issue, rather it's the real-time
requirements.  EDB isn't a real-time database, so the response time could
vary enough that you wouldn't have the response time required.

You would have to try it with BLOBs in order to see if it works.  If you
want to implement a test application in order to test the idea, I can look
at it for you and see if there are any optimizations that can be done.

A question, though: are the integer values accumulated in one shot, and then
written out to the BLOB column ?  Or will you need to constantly be
re-reading the BLOB and updating it ?

--
Tim Young
Elevate Software
www.elevatesoft.com

"Tim Young [Elevate Software]" wrote:

Tim,


<<It's not the amount of data that is the issue, rather it's the real-time
requirements.>>

Yes, the title of the question was not finished, and realised it too late, so it  is
missleading.  

<<A question, though: are the integer values accumulated in one shot, and then
written out to the BLOB column ?  Or will you need to constantly be
re-reading the BLOB and updating it ?>>

At this point it is more of the first case, the data meant to be stored in blobs do not
change.

<<EDB isn't a real-time database, so the response time could
vary enough that you wouldn't have the response time required.>>

The response time matters most when reading the data the graphical display requires, and
it is not more then ~4000 integers. If writing is suspended while these requests are
processed, in worst case scenario it would mean te single thing for the engine should do
is reading 1x16k or 2x8k  or 4x4k or ...  blob data from the disk. (It is a local
database, with one user.)  

The data can be cached in the application as long as there is a confirmation from the db
engine that it has made it to the disk. So some  fluctuation in speed should not be a
problem as long the everage throughput is fast enough. What I am thinking about is feeding
the data to a memory database first and having a thread (or maybe a Job?) transfering the
data to a disk db from there. It seems, an easily controlable more sophisticated cache
would be an advantage for the calculations made on the data later, and I figoured that
managing the cache through SQL would be an easy and hopefully fast enough way to do it.


Anyhow, you are right, i cannot escape from making a test app for performance testing with
different settings.

Thank you for your answer!

Leslie