Hey everyone, I just wanted to see if anyone wanted to do a very quick beta test on a new feature coming up in EDB 2.28, as well as let everyone know what's coming up with EDB 2.28.

I've developed a new buffered I/O layer for the EDB engine that should resolve any issues with the performance of very large files (> 15-20GB) under Windows.  This new mode typically provides about a ~50-60% improvement in the I/O performance in general.

The way it works is this:

- The TEDBEngine.ExclusiveFileAccess property is now gone. In it's place is a Boolean property called BufferedFileIO, a new FileIOBufferSettings (TStrings) property that controls the buffering settings, and a new FileIOBufferFlushInterval (Integer) property that controls how many seconds can go by before the engine automatically flushes any dirty buffers to disk.

- When the BufferedFileIO property is turned on, all locking is performed locally within the EDB engine and only the EDB engine can open any database files (not including non-database files in stores, etc.).  In addition, all I/O is buffered using a buffer manager per database file.  This means that you can control how the configuration, catalogs, and table files are all buffered.

- There *is* a downside with the buffering, and that is the possibility of lost writes if the server process goes down during operation, but before EDB gets to flush any dirty buffers to the Windows file system.  The buffering manager *does* flush any writes to the Windows file system after the last file handle is closed, and when a transaction commits with a flush, but singleton updates and transaction commits without flushes, absent any
concurrent transactions (with flushed commits) on the same file, are buffered until the periodic flush occurs.

In addition:

- I've also modified the way that we do read-write locks in the engine so that they aren't using spin locks anymore, which has reduced any ill effects from polling/yielding threads and maximizes the throughput under heavy concurrency levels (20-50 threads pounding away on the same table).  Combined with the buffered I/O, the effects are impressive.  More importantly, the performance tends to *improve* as the database continues to be used and buffered until it plateaus when the max buffer size is hit.

- The TEDBEngine.LargeFileSupport property is now gone in 2.28, and large file support is the default and only mode.  I've been meaning to do this for some time.

This is an example of how you would use the new buffering:

  Engine.BufferedFileIO:=False;
  with Engine.FileIOBufferSettings do
     begin
     Add('"*EDBConfig.EDBLck",1024,1024');
     Add('"*EDBDatabase.EDBLck",1024,1024');
     Add('"*Customer.EDBTbl",65536,50331648');
     Add('"*Customer.EDBIdx",65536,50331648');
     Add('"*Orders.EDBTbl",65536,268435456');
     Add('"*Orders.EDBIdx",65536,536870912');
     Add('"*Orders.EDBBlb",65536,536870912');
     end;

The first parameter is the wildcard match on the file to buffer, the second is the block size to use, and the third is the max buffer size to use.  The wildcards are case-insensitive and evaluated from back-to-front, so you can perform a type of cascading configuration of files from the most-specific to the least-specific (*.edbtbl).

So, I'm looking for just a handful of beta testers to try the new buffering out and make sure that it is behaving.  I've stress-tested the new code thoroughly, as well as run it through our test framework, but I'd like to have it tried out in other applications to make sure that I didn't miss anything.

Thanks !

Tim Young
Elevate Software
www.elevatesoft.com

Please ignore the "it's" vs "its" typos, it's been a long day.

Tim Young
Elevate Software
www.elevatesoft.com

Tim


Both forms are accepted usage so no need to apologise.


I don't have any real files that sort of size but I'm happy to create lots of junk if you don't get enough other volunteers.

Roy Lambert

Roy,

<< Both forms are accepted usage so no need to apologise. >>

Not in the US.

<< I don't have any real files that sort of size but I'm happy to create lots of junk if you don't get enough other volunteers. >>

Sure, anything that you can chuck at it is good.  I'm just covering bases - as I said, I've been running stress tests on it for about a week, and it's pretty bullet-proof at this point.

Tim Young
Elevate Software
www.elevatesoft.com

Tim

><< Both forms are accepted usage so no need to apologise. >>
>
>Not in the US.

I had forgotten for a moment that you lot over there can't spell for toffee


><< I don't have any real files that sort of size but I'm happy to create lots of junk if you don't get enough other volunteers. >>
>
>Sure, anything that you can chuck at it is good. I'm just covering bases - as I said, I've been running stress tests on it for about a week, and it's pretty bullet-proof at this point.

OK let me know when I should download and where from

Roy

Tim Young [Elevate Software] wrote:

> Roy,
>
> << Both forms are accepted usage so no need to apologise. >>
>
> Not in the US.
>

I'm with you Tim .. and I'm a Brit.
Its and it's have very different meanings.
However, in this age of social media, speed and context rule - leaving
grammar and accuracy trailing far behind.

Malcolm

Malcolm


>I'm with you Tim .. and I'm a Brit.
>Its and it's have very different meanings.
>However, in this age of social media, speed and context rule - leaving
>grammar and accuracy trailing far behind.

Whilst I can be a bit of a grammar nazi (I love those threads on El Reg) and I accept that dictionaries can differentiate between the two it has been common usage for much longer than the timespan of social media.

I used to be punctilious about the use of its and it's now I only use an apostrophe only when the second word isn't is.

Roy

Hello Tim,
 I'm here for testing ...

Michael,

<<  I'm here for testing ... >>

Thanks, I've added you, so you should get a notification shortly.

Tim Young
Elevate Software
www.elevatesoft.com

Hi Tim,

If you need any other testers I'm up for it.

= Steve