Login ProductsSalesSupportDownloadsAbout |
Home » Technical Support » ElevateDB Technical Support » Support Forums » ElevateDB General » View Thread |
Messages 1 to 4 of 4 total |
Optimal settings for lots of data |
Tue, Jul 28 2009 3:20 PM | Permanent Link |
Leslie | Tim,
I am still occupied in a Java project, but have began to think about some planned Delphi projects/udates lately I had to delay. So I am looking at EDB once again as the db engine to migrate to. In one of the projects when all devices are connected it requires storing around 10000-15000 integer values in every half a second. At the same time the recorded data is graphicaly displayed for he user who may be jumping back and forth, so some reading may occur too. Depending the screen resolution the control displaying the data uses 2000-4000 integer values for a redraw, which happens at least in every half a second. I figured that the only way to do this is to use compressed blob fields. (Or write every data channel directly to a separate file, if this amount of data is too much to handle for a DB engine). The data is sort of periodic (large part of it is ECG --> 0.5-4 cycles in a sec). In my experience compression ratio is around 0.3 -0.4 when files with whole recordings are compressed with standard zip. So the main numbers again: - Max 30000 integers/sec to store - This speed must be maintened for several hours. - The minimum amount of data needs to be stored at once is ~16 integer values. (Happens often.) - The maximum amount of data needs to be stored at once is ~10000 integer values. (Can be sliced) - A block of 10 - 4000 integer values must be read with the best possible speed. - The speed of reading weighs most. Every measurement has it's own database, so it stores only some hours of data the most. Based on these numbers can you make suggestions how to optimize the setting for: 1. Blob block size 2. Blob compression 3. How much data would be optimal to store in a single field? Leslie |
Tue, Jul 28 2009 3:24 PM | Permanent Link |
Leslie | Forgot one settings which may be important:
4. Blob Buffer size. |
Wed, Jul 29 2009 12:27 PM | Permanent Link |
Tim Young [Elevate Software] Elevate Software, Inc. timyoung@elevatesoft.com | Leslie,
<< I figured that the only way to do this is to use compressed blob fields. (Or write every data channel directly to a separate file, if this amount of data is too much to handle for a DB engine). >> It's not the amount of data that is the issue, rather it's the real-time requirements. EDB isn't a real-time database, so the response time could vary enough that you wouldn't have the response time required. You would have to try it with BLOBs in order to see if it works. If you want to implement a test application in order to test the idea, I can look at it for you and see if there are any optimizations that can be done. A question, though: are the integer values accumulated in one shot, and then written out to the BLOB column ? Or will you need to constantly be re-reading the BLOB and updating it ? -- Tim Young Elevate Software www.elevatesoft.com |
Wed, Jul 29 2009 1:36 PM | Permanent Link |
Leslie | "Tim Young [Elevate Software]" wrote:
Tim, <<It's not the amount of data that is the issue, rather it's the real-time requirements.>> Yes, the title of the question was not finished, and realised it too late, so it is missleading. <<A question, though: are the integer values accumulated in one shot, and then written out to the BLOB column ? Or will you need to constantly be re-reading the BLOB and updating it ?>> At this point it is more of the first case, the data meant to be stored in blobs do not change. <<EDB isn't a real-time database, so the response time could vary enough that you wouldn't have the response time required.>> The response time matters most when reading the data the graphical display requires, and it is not more then ~4000 integers. If writing is suspended while these requests are processed, in worst case scenario it would mean te single thing for the engine should do is reading 1x16k or 2x8k or 4x4k or ... blob data from the disk. (It is a local database, with one user.) The data can be cached in the application as long as there is a confirmation from the db engine that it has made it to the disk. So some fluctuation in speed should not be a problem as long the everage throughput is fast enough. What I am thinking about is feeding the data to a memory database first and having a thread (or maybe a Job?) transfering the data to a disk db from there. It seems, an easily controlable more sophisticated cache would be an advantage for the calculations made on the data later, and I figoured that managing the cache through SQL would be an easy and hopefully fast enough way to do it. Anyhow, you are right, i cannot escape from making a test app for performance testing with different settings. Thank you for your answer! Leslie |
This web page was last updated on Saturday, May 4, 2024 at 09:18 PM | Privacy PolicySite Map © 2024 Elevate Software, Inc. All Rights Reserved Questions or comments ? E-mail us at info@elevatesoft.com |