I have a process that has to complete in 30 minutes. Right now it takes
about 90 minutes if I write it directly out to disk. I can build the
table without requiring an index.

1) I thought if I write the rows to a memory table and then find a high
speed way of writing it to disk. Of course I can copy it from the memory
table record by record to a disk based table, but it has 30 million rows
and is around 1gb in size. Is there a faster way I can get it from the
memory table to disk (like a block copy)?

2) Also has anyone had any recommendations for disk caches? Do they
really work? I tried SpeedCacheII over the weekend and it didn't seem to
make a difference even though I gave it 512MB of ram to work with.

3) The other alternative I thought of is to build the TDBISAM table on a
1gb RAM disk and when the process is finished close the table and just
do a DOS copy of the file to disk.

Does anyone have any recommendations? TIA

Dave
(DBISAM 4.25)

"Dave Harrison" <daveh_18824@spammore.com> wrote in message
news:8A6923F2-D4E5-4EB2-91B9-19E8173A80F7@news.elevatesoft.com...
>I have a process that has to complete in 30 minutes. Right now it takes
>about 90 minutes if I write it directly out to disk. I can build the table
>without requiring an index.
>
> 1) I thought if I write the rows to a memory table and then find a high
> speed way of writing it to disk. Of course I can copy it from the memory
> table record by record to a disk based table, but it has 30 million rows
> and is around 1gb in size. Is there a faster way I can get it from the
> memory table to disk (like a block copy)?
>
> 2) Also has anyone had any recommendations for disk caches? Do they really
> work? I tried SpeedCacheII over the weekend and it didn't seem to make a
> difference even though I gave it 512MB of ram to work with.
>
> 3) The other alternative I thought of is to build the TDBISAM table on a
> 1gb RAM disk and when the process is finished close the table and just do
> a DOS copy of the file to disk.
>
> Does anyone have any recommendations? TIA
>

If you are SURE that memory processing will be significantly faster, look
into table SaveToStream (memory) and LoadFromStream (disk) methods. But I
suspect creating that large a table in memory is not a good option. If
feasible, you could try saving the records to a stringlist or a text file,
and use the importtable method to get the data into a table.

Robert

Dave

>1) I thought if I write the rows to a memory table and then find a high
>speed way of writing it to disk. Of course I can copy it from the memory
>table record by record to a disk based table, but it has 30 million rows
>and is around 1gb in size. Is there a faster way I can get it from the
>memory table to disk (like a block copy)?

Try table.CopyTable - might be faster

>
>2) Also has anyone had any recommendations for disk caches? Do they
>really work? I tried SpeedCacheII over the weekend and it didn't seem to
>make a difference even though I gave it 512MB of ram to work with.

With the newer versions of Windows I haven't found one worth bothering with. They generally seem to be fighting M$'s caching



Roy Lambert

Dave,

<< I have a process that has to complete in 30 minutes. Right now it takes
about 90 minutes if I write it directly out to disk. I can build the table
without requiring an index. >>

Are you currently using a transaction for every X number of rows ?  That's
the fastest way to speed up any batch write operation with a lot of rows.

Psuedo-code

while Adding Rows do

  begin

  Start Transaction;

  Add Row

  Increment Counter

  if Counter mod Block Size = 0 then
      Commit Transaction

  end

if In Transaction then
  Commit Transaction

--
Tim Young
Elevate Software
www.elevatesoft.com