Login ProductsSalesSupportDownloadsAbout |
Home » Technical Support » DBISAM Technical Support » Support Forums » DBISAM General » View Thread |
Messages 1 to 4 of 4 total |
Is there a way to block copy a TDBISAM memory table to disk? |
Mon, Jul 16 2007 10:49 AM | Permanent Link |
Dave Harrison | I have a process that has to complete in 30 minutes. Right now it takes
about 90 minutes if I write it directly out to disk. I can build the table without requiring an index. 1) I thought if I write the rows to a memory table and then find a high speed way of writing it to disk. Of course I can copy it from the memory table record by record to a disk based table, but it has 30 million rows and is around 1gb in size. Is there a faster way I can get it from the memory table to disk (like a block copy)? 2) Also has anyone had any recommendations for disk caches? Do they really work? I tried SpeedCacheII over the weekend and it didn't seem to make a difference even though I gave it 512MB of ram to work with. 3) The other alternative I thought of is to build the TDBISAM table on a 1gb RAM disk and when the process is finished close the table and just do a DOS copy of the file to disk. Does anyone have any recommendations? TIA Dave (DBISAM 4.25) |
Mon, Jul 16 2007 11:08 AM | Permanent Link |
"Robert" | "Dave Harrison" <daveh_18824@spammore.com> wrote in message news:8A6923F2-D4E5-4EB2-91B9-19E8173A80F7@news.elevatesoft.com... >I have a process that has to complete in 30 minutes. Right now it takes >about 90 minutes if I write it directly out to disk. I can build the table >without requiring an index. > > 1) I thought if I write the rows to a memory table and then find a high > speed way of writing it to disk. Of course I can copy it from the memory > table record by record to a disk based table, but it has 30 million rows > and is around 1gb in size. Is there a faster way I can get it from the > memory table to disk (like a block copy)? > > 2) Also has anyone had any recommendations for disk caches? Do they really > work? I tried SpeedCacheII over the weekend and it didn't seem to make a > difference even though I gave it 512MB of ram to work with. > > 3) The other alternative I thought of is to build the TDBISAM table on a > 1gb RAM disk and when the process is finished close the table and just do > a DOS copy of the file to disk. > > Does anyone have any recommendations? TIA > If you are SURE that memory processing will be significantly faster, look into table SaveToStream (memory) and LoadFromStream (disk) methods. But I suspect creating that large a table in memory is not a good option. If feasible, you could try saving the records to a stringlist or a text file, and use the importtable method to get the data into a table. Robert |
Mon, Jul 16 2007 11:10 AM | Permanent Link |
Roy Lambert NLH Associates Team Elevate | Dave
>1) I thought if I write the rows to a memory table and then find a high >speed way of writing it to disk. Of course I can copy it from the memory >table record by record to a disk based table, but it has 30 million rows >and is around 1gb in size. Is there a faster way I can get it from the >memory table to disk (like a block copy)? Try table.CopyTable - might be faster > >2) Also has anyone had any recommendations for disk caches? Do they >really work? I tried SpeedCacheII over the weekend and it didn't seem to >make a difference even though I gave it 512MB of ram to work with. With the newer versions of Windows I haven't found one worth bothering with. They generally seem to be fighting M$'s caching Roy Lambert |
Mon, Jul 16 2007 4:31 PM | Permanent Link |
Tim Young [Elevate Software] Elevate Software, Inc. timyoung@elevatesoft.com | Dave,
<< I have a process that has to complete in 30 minutes. Right now it takes about 90 minutes if I write it directly out to disk. I can build the table without requiring an index. >> Are you currently using a transaction for every X number of rows ? That's the fastest way to speed up any batch write operation with a lot of rows. Psuedo-code while Adding Rows do begin Start Transaction; Add Row Increment Counter if Counter mod Block Size = 0 then Commit Transaction end if In Transaction then Commit Transaction -- Tim Young Elevate Software www.elevatesoft.com |
This web page was last updated on Tuesday, April 23, 2024 at 08:10 AM | Privacy PolicySite Map © 2024 Elevate Software, Inc. All Rights Reserved Questions or comments ? E-mail us at info@elevatesoft.com |