Tim

For some reason I took it into my head to try migrating the data files for TMaN into ElevateDB. Have you done something to the migrator? I seem to remember it being rather nippy the last time I tried this. OK the data files have grown but I left ElevateDB for c90mins and it had only chomped its way a quarter through the MandN table - its a biggy but I don't think it should be this slow.

MandN

Records c220k
.blb 2.2Gb.
.dat 97Mb.
.idx 217Mb.

I initially thought it was the full text indexing so I copied the tables and zapped that but still deadly slow.

I have large file support turned on, and progress interval set to 100 (I tried 1 but you sneakily changed it).


Roy Lambert

Roy,

<< For some reason I took it into my head to try migrating the data files
for TMaN into ElevateDB. Have you done something to the migrator? I seem to
remember it being rather nippy the last time I tried this. OK the data files
have grown but I left ElevateDB for c90mins and it had only chomped its way
a quarter through the MandN table - its a biggy but I don't think it should
be this slow. >>

It spits out the data as INSERT statements that are executed against the EDB
database, so it's a slow process when you've got that much data.

--
Tim Young
Elevate Software
www.elevatesoft.com

Tim

>It spits out the data as INSERT statements that are executed against the EDB
>database, so it's a slow process when you've got that much data.

If I'd looked at the code I would have know that

Do you have any gut feel for when its worth "rolling your own", and is there anything that could be rolled that would be significantly faster? It doesn't really matter for me since I can happily run it overnight but for people who have a large app with wadges of data that's running 24x7 it may cause a few problems.

The other thing that occurred to me as I knocked out the full text index to see if that did any good is the customised text filter/word generator issue. With the major app I'm rewriting its not a problem since I'm doing so much restructuring of the tables BEFORE migrating adding in a column to control it was no big deal. I'm only aware of one other of your DBISAM customers having written a custom filter (the guy who was preparing the app for a show and bought some serious hardware to get performance) so I don't know how many people it could impact. Any suggestions as to go about it apart from the obvious - drop full text indexing, restructure the DBISAM table and add a column, do the migration, add full text indexing?

Roy Lambert

Roy,

<< Do you have any gut feel for when its worth "rolling your own", and is
there anything that could be rolled that would be significantly faster? It
doesn't really matter for me since I can happily run it overnight but for
people who have a large app with wadges of data that's running 24x7 it may
cause a few problems. >>

I've got plans to make a binary equivalent for faster performance.  The big
problem that the INSERT statement method runs into is with large binary
BLOBs.  I'm going to see if I can work it into 2.02.

<< The other thing that occurred to me as I knocked out the full text index
to see if that did any good is the customised text filter/word generator
issue. With the major app I'm rewriting its not a problem since I'm doing so
much restructuring of the tables BEFORE migrating adding in a column to
control it was no big deal. I'm only aware of one other of your DBISAM
customers having written a custom filter (the guy who was preparing the app
for a show and bought some serious hardware to get performance) so I don't
know how many people it could impact. Any suggestions as to go about it
apart from the obvious - drop full text indexing, restructure the DBISAM
table and add a column, do the migration, add full text indexing? >>

Unfortunately I can't think of any way to automatically add a filter column
during the migration, as it exists.  Perhaps I can add a detail layer that
specifies this type of information in order to avoid extra steps.

--
Tim Young
Elevate Software
www.elevatesoft.com

Tim


>I've got plans to make a binary equivalent for faster performance. The big
>problem that the INSERT statement method runs into is with large binary
>BLOBs. I'm going to see if I can work it into 2.02.

I tried the transfer 3 different ways

1. As is
2. remove FTI
3. remove FTI and adjust BlobBlockSize from 512 to 4096

Didn't seem to be any difference between them. Mind you I didn't wait around for any but the last version to complete.

Roy Lambert

Roy,

Did you get a chance to try this migration again with the latest EDB build ?
The performance should have improved quite a bit.

--
Tim Young
Elevate Software
www.elevatesoft.com

Tim

>Did you get a chance to try this migration again with the latest EDB build ?
>The performance should have improved quite a bit.

Not yet I'm in the process of recovering from a bout of food poisoning a week ago followed by a tooth abscess leading to a face like a football or balloon the antibiotics for which seem to be acting as a superior quality sleeping tablet.

Roy Lambert

Roy,

<< Not yet I'm in the process of recovering from a bout of food poisoning a
week ago followed by a tooth abscess leading to a face like a football or
balloon the antibiotics for which seem to be acting as a superior quality
sleeping tablet. >>

I'm sorry to hear that.  I hope you're feeling better soon.

--
Tim Young
Elevate Software
www.elevatesoft.com

Tim

>I'm sorry to hear that. I hope you're feeling better soon.

Thanks - at least my legs only feel slightly rubbery now.

Roy Lambert