Icon View Thread

The following is the text of the current message along with any replies.
Messages 1 to 10 of 21 total
Thread db to be delivered on dvd
Wed, Jun 14 2006 11:09 AMPermanent Link

Marc Pelletier
Hello,

I am a fairly beginner level db programmer, and am examining dbisam for use
in a new application with LOTS of data. The data are primarily images,
compressed in jpeg4-2-1 with timestamps for filenames. Right now I just
store them in directories of ~150 images each (ie 1 minute), and if I can't
find the required directory I prompt the user to put in the proper disk.
This involves building some rather large filelists on the fly, and is just
generally awkward. A medium sized datasets will include 50 dvd's.

The natural organization of the data is by flight and line. I want to
create a table containing flight, line, start and end times, and use this
to generate queries that will return the images by line ( or part thereof).
Ideally the linelist information would be on the user's machine and the
user would be prompted to insert the appropriate disk when necessary. This
seems to break the "everything in one directory" dbIsam model. Is it doable
with dbisam?

Maybe the solution is that its not all one database? I could treat each
disk as a database somehow?

Ideas? Suggestions?

Thanks,

Marc Pelletier
Goldak Airborne Surveys
Wed, Jun 14 2006 11:43 AMPermanent Link

"Ralf Mimoun"
Marc Pelletier wrote:
....
> Maybe the solution is that its not all one database? I could treat
> each disk as a database somehow?
>
> Ideas? Suggestions?

If I understand you correctly, then the database itself, without the images,
will be relatively small. You could store the whole db on each disk and copy
it at startup into in-memory tables. Each record has a CD number, and that'
what you check for.

Ralf
Wed, Jun 14 2006 11:46 AMPermanent Link

Roy Lambert

NLH Associates

Team Elevate Team Elevate

Marc

Why not a mix and match approach. Since (IIRC) your users are unlikely to ever load the whole thing onto one hard disk have a table which stores the basic information (from what you say flight, line, start and end times), plus a disk number and a filename (probably keep the timestamp as the filename). Keep the jpegs as individual files and you can then use the data in the master table to prompt for a disk change.

That keeps it fairly simple, probably means the least effort for you and leaves things as near as they are for the users.

It would also be possible to have a master table and subsidiary tables on dvd. It would just be a matter of closing one table and opening its replacement when the dvd is changed. A bit more difficult to develop but still not to bad.

Roy Lambert
Wed, Jun 14 2006 12:38 PMPermanent Link

Marc Pelletier
"Ralf Mimoun" <nospam@rad-on.de> wrote in
news:1B4D996A-D213-4C2F-868D-2E4497C2F1EE@news.elevatesoft.com:

> If I understand you correctly, then the database itself, without the
> images, will be relatively small. You could store the whole db on each
> disk and copy it at startup into in-memory tables. Each record has a
> CD number, and that' what you check for.
>

Hmm.. that sounds like a good answer, but in the end I think its
impractical because of temporary storage. Data acquisition can take months.
Every day or two the operator will be creating a new dvd with a set of
images, but the final index won't be ready until the project is finished.
To go back and add the index onto the disks would require reburning dozens
of dvds, which is just too painful to contemplate.

I think the indexes have to stand alone.

I didn't make it clear in my original post but I am also going to be
putting the images into tables. I expect there will be a bit of a
performance hit that way, but it will allow me to simplify the image
retrieval somewhat by using queries.

cheers

Marc
Wed, Jun 14 2006 1:09 PMPermanent Link

Tim Young [Elevate Software]

Elevate Software, Inc.

Avatar

Email timyoung@elevatesoft.com

Marc,

<< The natural organization of the data is by flight and line. I want to
create a table containing flight, line, start and end times, and use this
to generate queries that will return the images by line ( or part thereof).
Ideally the linelist information would be on the user's machine and the user
would be prompted to insert the appropriate disk when necessary. This seems
to break the "everything in one directory" dbIsam model. Is it doable with
dbisam? >>

I'm not sure how I understand that the setup that you have doesn't store the
data in one directory.  Isn't the database table(s) on the local hard drive,
with the jpeg images on the DVDs, or are the database tables also on the DVD
?  If the case is the latter, then I would simply merge the DVD table into a
master table on the user's machine when a new DVD is inserted.  It won't
take long at all, and it only has to be done once for each DVD.  Kind of
like a CD database that Windows Media Player, ITunes, etc. use locally after
grabbing the information from the Internet when a new audio CD is inserted
into the CD drive.

--
Tim Young
Elevate Software
www.elevatesoft.com

Wed, Jun 14 2006 1:38 PMPermanent Link

Marc Pelletier
"Tim Young [Elevate Software]" <timyoung@elevatesoft.com> wrote in
news:B1EFDF4A-979C-4918-81AA-94327E91EC23@news.elevatesoft.com:

> I'm not sure how I understand that the setup that you have doesn't
> store the data in one directory.  Isn't the database table(s) on the
> local hard drive, with the jpeg images on the DVDs, or are the
> database tables also on the DVD
>

Tim,

As I mentioned in my reply above, I also want to put the images into a
table. I'm now planning on creating a series of tables, each 1 dvd in size,
and breaking them up into multiple databases at distribution time. Maybe
that just isn't necessary, but it seems cleaner somehow, and its a good
learning exercise for me. If it turns out to be a dog, I can revert to the
existing method.

But this brings up another issue, how do I determine the size of the
current table? The images themselves are already compressed so I should be
able to predict the table size based on the fields and number of images,
right?

Thanks, everyone.

Marc
Wed, Jun 14 2006 2:30 PMPermanent Link

Roy Lambert

NLH Associates

Team Elevate Team Elevate

Tim


Interesting approach - but 50 DVD's worth of data - 250Gb say.......

Roy Lambert
Thu, Jun 15 2006 1:27 AMPermanent Link

"Clive"
Hi,
I have been down a similar track before.. Firstly storing blobs in a
database was not good, what I did was to just store a thumbnail in the
database and store a reference to the actual disk file name.

For multiple disks this process works fine, the user can view the thumbnail
without the disk in, but gets prompted for correct disk when not found.


"Marc Pelletier" <marc@stopspam.goldak.ca> wrote in message
news:Xns97E25C2D1AEE8mmpp1234.dd@64.65.248.118...
> Hello,
>
> I am a fairly beginner level db programmer, and am examining dbisam for
> use
> in a new application with LOTS of data. The data are primarily images,
> compressed in jpeg4-2-1 with timestamps for filenames. Right now I just
> store them in directories of ~150 images each (ie 1 minute), and if I
> can't
> find the required directory I prompt the user to put in the proper disk.
> This involves building some rather large filelists on the fly, and is just
> generally awkward. A medium sized datasets will include 50 dvd's.
>
> The natural organization of the data is by flight and line. I want to
> create a table containing flight, line, start and end times, and use this
> to generate queries that will return the images by line ( or part
> thereof).
> Ideally the linelist information would be on the user's machine and the
> user would be prompted to insert the appropriate disk when necessary. This
> seems to break the "everything in one directory" dbIsam model. Is it
> doable
> with dbisam?
>
> Maybe the solution is that its not all one database? I could treat each
> disk as a database somehow?
>
> Ideas? Suggestions?
>
> Thanks,
>
> Marc Pelletier
> Goldak Airborne Surveys

Thu, Jun 15 2006 3:43 AMPermanent Link

"Uffe Kousgaard"
"Marc Pelletier" <marc@stopspam.goldak.ca> wrote in message
news:Xns97E25C2D1AEE8mmpp1234.dd@64.65.248.118...
> A medium sized datasets will include 50 dvd's.

Have you considered to use large USB harddisks for deployment? 50 DVD's is
appr. 250 GB, which costs 200 USD or so.

Thu, Jun 15 2006 10:12 AMPermanent Link

Sean McCall
Clive,

Interesting. I've haven't played much with images. Is there
a procedure or method anywhere Delphi or windows that can
take any image and convert it into a thumbnail or do you
have some code you could share?

Thanks,

Sean

Clive wrote:

> Hi,
> I have been down a similar track before.. Firstly storing blobs in a
> database was not good, what I did was to just store a thumbnail in the
> database and store a reference to the actual disk file name.
>
> For multiple disks this process works fine, the user can view the thumbnail
> without the disk in, but gets prompted for correct disk when not found.
>
>
> "Marc Pelletier" <marc@stopspam.goldak.ca> wrote in message
> news:Xns97E25C2D1AEE8mmpp1234.dd@64.65.248.118...
>
>>Hello,
>>
>>I am a fairly beginner level db programmer, and am examining dbisam for
>>use
>>in a new application with LOTS of data. The data are primarily images,
>>compressed in jpeg4-2-1 with timestamps for filenames. Right now I just
>>store them in directories of ~150 images each (ie 1 minute), and if I
>>can't
>>find the required directory I prompt the user to put in the proper disk.
>>This involves building some rather large filelists on the fly, and is just
>>generally awkward. A medium sized datasets will include 50 dvd's.
>>
>>The natural organization of the data is by flight and line. I want to
>>create a table containing flight, line, start and end times, and use this
>>to generate queries that will return the images by line ( or part
>>thereof).
>>Ideally the linelist information would be on the user's machine and the
>>user would be prompted to insert the appropriate disk when necessary. This
>>seems to break the "everything in one directory" dbIsam model. Is it
>>doable
>>with dbisam?
>>
>>Maybe the solution is that its not all one database? I could treat each
>>disk as a database somehow?
>>
>>Ideas? Suggestions?
>>
>>Thanks,
>>
>>Marc Pelletier
>>Goldak Airborne Surveys
>
>
>
Page 1 of 3Next Page »
Jump to Page:  1 2 3
Image