Hi Tim,

I've written a .NET migration utility to convert from DBISAM4 (UNICODE) ->
ElevateDB.

On a small test database with (77 tables in total) the migration seems to
progressively slow down as it progresses through the tables, until towards
the end, it's taking many times longer for each table to migrate compared to
the start.

Again, when I run the same migration in ElevateDB - no problem - it
consistently runs through at the same pace and the whole migration is done
in less than a minute. In .NET it's nearer 5 minutes.

The problem is that as migrating this small database takes an unacceptably
long time in .NET, it's going to be completely unworkable to migrate larger
'real' databases for my customers.

Here's the sample DBISAM4 database I've been working with (this is much
smaller than the previous ones I've uploaded) :-
http://files.pioneersoftware.co.uk/temp/SampleDB.7z

And here's the complete C# project source of my migrator app :-
http://files.pioneersoftware.co.uk/temp/EDBMigrator.7z

Look forward to any suggestions!

Hedley,

<< I've written a .NET migration utility to convert from DBISAM4
(UNICODE) -> ElevateDB.

On a small test database with (77 tables in total) the migration seems to
progressively slow down as it progresses through the tables, until towards
the end, it's taking many times longer for each table to migrate compared to
the start.

Again, when I run the same migration in ElevateDB - no problem - it
consistently runs through at the same pace and the whole migration is done
in less than a minute. In .NET it's nearer 5 minutes. >>

I'm getting "cannot open file as archive" errors on the two linked .7z
archives.

--
Tim Young
Elevate Software
www.elevatesoft.com

Oops! Managed to upload in ASCII format! I've re-uploaded the files.

The main routine is 'void MigrateDatabase()' - you can probably just comment
out the rest.

Please note that I have also had to comment out the "DROP MIGRATOR" command
as it throws the following exception :-
----------------
LoaderLock was detected
Attempting managed execution inside OS Loader lock. Do not attempt to run
managed code inside a DllMain or image initialization function since doing
so can cause the application to hang.
----------------

Any idea what might be causing that? Also, does it matter if the migrator is
not dropped? i.e. is anything persistent about a migrator or does it just
local to that session?

Thanks!

Hedley,

<< Please note that I have also had to comment out the "DROP MIGRATOR"
command as it throws the following exception :- >>

If this is only in the VS debugger, then it's probably this issue:

http://vivekthangaswamy.blogspot.com/2006/11/loaderlock-was-detected-error-when.html
http://harriyott.com/2006/08/loaderlock-was-detected.aspx

--
Tim Young
Elevate Software
www.elevatesoft.com

Hedley,

That migrator project indicates that I'm missing the PSLib assembly.  Do you
have a compiled version of it that I can use with the EDBMigrator test
project ?

--
Tim Young
Elevate Software
www.elevatesoft.com

PSLib.dll is in the "EDBMigrator\bin" folder in the original archive :-
http://files.pioneersoftware.co.uk/temp/EDBMigrator.7z

The DLL reference might be set to "Copy Local" in the solution which means
it will get deleted when you re-compile the project.

You can comment out the calls to PSLib as they're just there to get the
program arguments and the registry settings. In fact, it might be best to
comment out the following functions :-

GetRegistrySettings
GetParameters

and simply do the following in the 'frmMain_Shown' routine :-

STR_PATH_SETTINGS = "<your EDB ConfigPath>";
SrcFolder = "<dbisam database folder>";
DBName = "<new edb databse name to use>";

The ActivateDLL() routine can be removed too - it's just there to
enable/disable the "edbmigratedbisam4.dll" so that x64 machines don't crash
when trying to use the ConfigPath.

Hope this helps!

Hedley,

<< That migrator project indicates that I'm missing the PSLib assembly. >>

Never mind, cancel that - I don't need it.

As for the slowdown, this is pretty much normal for .NET.  Writing out the
database catalog takes a bit longer under .NET than under Win32 due to the
fact that .NET requires things like strings to be converted to byte arrays
before being saved.

I'll do some more looking into this to see if I can squeeze some more
performance out.  No promises, though.

--
Tim Young
Elevate Software
www.elevatesoft.com

> As for the slowdown, this is pretty much normal for .NET.  Writing out the
> database catalog takes a bit longer under .NET than under Win32 due to the
> fact that .NET requires things like strings to be converted to byte arrays
> before being saved.

On my machine, at the start of the conversion it's whizzing through the
tables and the progressbar motors up to about 30% then it starts to slow
down and by the time it's at 80%+ it's painfully slow.

Is that what you found too and is that expected behaviour?

There's no appreciable slow down on my machine when doing the same thing in
EDB Manager - the speed seems to be constant.

So... if I understand you correctly, this is being caused by the fact that
the catalog file is growing with each table conversion and editing the
catalog therefore gets slower each time?

Hmmmm... whos idea was it to move the table catalogs out of the table
headers and into a single file? <vbg>

Hedley

>There's no appreciable slow down on my machine when doing the same thing in
>EDB Manager - the speed seems to be constant.

Unless Tim has built a version of EDBManager for each access type (I know there's one for Unicode and non-unicode) EDBManager will be compiled VCL not .NET which would explain it.

I remember from a previous post that the migrators are basically writing the structure and then running a bloody great wadge of INSERTs for the table. On that basis you could try modding the migrator to write all the structures out and then write all the data. Don't know if it would be any faster but it would give you something to do

Roy Lambert [Team Elevate]

Hedley,

BTW, I found the issue and it will be fixed in 2.02 B9 or 2.03, whichever
comes first.  However, if you want a copy of the new assembly to test with,
just send me an email and I'll respond with the new assembly.

The issue was with the memory allocation used for writing out the database
catalog when running under .NET.  The Win32 memory manager didn't have an
issue with it, but the .NET memory manager is very sensitive to intensive
memory reallocation, and therefore experienced a major slowdown after a
while.  It's a lot easier to tax the .NET memory manager, which is why stuff
like this crops up occasionally.

--
Tim Young
Elevate Software
www.elevatesoft.com