Ralf


Congratulations. Now your next task is ZZZZzzzzzz

Roy Lambert

"Ralf Mimoun" <nospam@rad-on.de> wrote in message
news:5FBDADB9-3697-4161-ABA4-D30CEFA1F197@news.elevatesoft.com...
> Roy Lambert wrote:
>> Ralf
>>
>>
>> When the dust has all settled don't forget to let us all know what
>> you managed to do and how.
>
> Wow. Done. I can't believe it.
>
....
> The result is astonishing, the system is flying. I can only recommend that
> tool if you have huge imports that must be done asap.
>
> Ralf
Ralf,

Thank you very much for this info. Having some large files myself I really
appreciate your feedback here!

Kind regards,

Frans van Daalen

Roy Lambert wrote:
> Ralf
>
>
> Congratulations. Now your next task is ZZZZzzzzzz

Ah, that would be great. I have to test the application, set up the disk for
shipment, pack everything together etc. Holidays... Roy, I'll be on my way
tomorrow, visiting you for a week or two

Ralf

Ralf,

<< Tim, is there _anything_ I can do? I can't deliver something that needs a
whole day to index a gigabyte. >>

Sorry for not getting on this earlier, but I've been trying to use the
weekends to get ahead on the coding and docs.

The issue that you're running into is that DBISAM only caches a small amount
of index data at one time, even during index builds.  This basically means
that it will perform a lot of OS reads/writes when building the text
indexes.  Normally this isn't too much of an issue, but with large files it
can quickly force Windows to use of most of the file system cache space and
become disk-bound.  Once this takes place with Windows, the whole thing
slows to a crawl and Windows never seems to be able to recover.

Basically, what you found out to be the case is the best solution - you'll
need to throw a lot of memory at the issue in order to keep the process from
becoming disk-bound.

--
Tim Young
Elevate Software
www.elevatesoft.com

Tim Young [Elevate Software] wrote:
....
> The issue that you're running into is that DBISAM only caches a small
> amount of index data at one time, even during index builds.  This
> basically means that it will perform a lot of OS reads/writes when
> building the text indexes.  Normally this isn't too much of an issue,
> but with large files it can quickly force Windows to use of most of
> the file system cache space and become disk-bound.  Once this takes
> place with Windows, the whole thing slows to a crawl and Windows
> never seems to be able to recover.

Is there any parameter in TDBISAMEngine I can change to accelerate the
indexing?

Ralf

Ralf,

<< Is there any parameter in TDBISAMEngine I can change to accelerate the
indexing? >>

You could try bumping up the MaxTableIndexBufferSize/Count properties a bit
and see if that helps.  It may not help much, however, if the issue is with
the process becoming disk-bound.  It will just delay things a bit.

--
Tim Young
Elevate Software
www.elevatesoft.com

Ralf,
   Something else to consider. Have you tried Rubicon from Tamarack? It is
a more heavy duty fulltext search than DBISAM and will work with DBISAM.
There are no royalties and may eliminate the need to install disk
caching software on all machines using your software. They have a demo
you can download and have the ability to assign more ram to increase the
index building speed. Download it from www.tamaracka.com

Dave

Dave,

Dave Harrison wrote:
> Ralf,
> Something else to consider. Have you tried Rubicon from Tamarack? It
> is a more heavy duty fulltext search than DBISAM and will work with
> DBISAM. There are no royalties and may eliminate the need to install
> disk caching software on all machines using your software. They have
> a demo you can download and have the ability to assign more ram to
> increase the index building speed. Download it from www.tamaracka.com

Thank you for the hint. That huge import will run every 90 days, on a
central machine. The result is a 3.5 GB table (.dat+.idx+.blb), which comes
down to about 650 MB after compression. These data are only needed for new
installations, and I can tell my customer what machine he needs for that
run. Better that than trying to write a transgene query that looks at the
SQL WHERE clause and the text seach parameters.

The search itself with DBISAM is blazijng fast. The first search might need
up to 15 seconds if the machine has not that much RAM, but then it flies.

Ralf

Ralf Mimoun wrote:
> Dave,
>
> Dave Harrison wrote:
>
>> Ralf,
>> Something else to consider. Have you tried Rubicon from Tamarack? It
>> is a more heavy duty fulltext search than DBISAM and will work with
>> DBISAM. There are no royalties and may eliminate the need to install
>> disk caching software on all machines using your software. They have
>> a demo you can download and have the ability to assign more ram to
>> increase the index building speed. Download it from www.tamaracka.com
>
>
> Thank you for the hint. That huge import will run every 90 days, on a
> central machine. The result is a 3.5 GB table (.dat+.idx+.blb), which
> comes down to about 650 MB after compression. These data are only needed
> for new installations, and I can tell my customer what machine he needs
> for that run. Better that than trying to write a transgene query that
> looks at the SQL WHERE clause and the text seach parameters.
>
> The search itself with DBISAM is blazijng fast. The first search might
> need up to 15 seconds if the machine has not that much RAM, but then it
> flies.
>
> Ralf

Ralf,
    Your compression is just 20% of the original, pretty darn good.
Just out of curiosity, have you tried putting it all on a CD-ROM or DVD
to see how fast it is? DBISAM will automatically open the table as
readonly so it will buffer the contents pretty good. A while back I was
able to compress my .exe file and put it on a floppy disk with a DBISAM
database containing images. Worked great.

Dave

Dave Harrison wrote:
....
>     Your compression is just 20% of the original, pretty darn good.

More exactly, it's 13% or less But the honor goes to Ray and his
suggestions about preparing the text, and to SuperCache II.

> Just out of curiosity, have you tried putting it all on a CD-ROM or
> DVD to see how fast it is? DBISAM will automatically open the table as
> readonly so it will buffer the contents pretty good. A while back I
> was able to compress my .exe file and put it on a floppy disk with a
> DBISAM database containing images. Worked great.

Sorry, but the application needs write access to other tables. But I am sure
that it will be fast enough.

Ralf