Icon View Thread

The following is the text of the current message along with any replies.
Messages 1 to 7 of 7 total
Thread Tools for the Paranoid
Fri, Apr 17 2009 10:52 AMPermanent Link

BAMaustin
I was wondering what tools others have found mandatory in their diagnostic toolboxes for DBISAM tables?  

We've used the Elevate database system utility (DBsys) tool for a long time.  But it wasn't until using the DBISAM Viewer  (DATview from
http://www.scalabium.com/dbisam/ ) tool today that we could actually SEE the garbage accumulating in edited records.  The garbage was not
shown after using DBsys to “Optimize” … but is it REALLY gone?  

DatView's “Recover deleted records” action is a good reminder that deleting isn't enough if you're dealing with sensitive data.  Should I be
shredding records before I delete?  Or maybe shredding deleted records when Optimizing?

When that file size suddenly dropped, maybe that chunk of data is sitting intact on the drive waiting  for a hard core hacker.

I'm paranoid enough to think that it is a good idea to periodically file shred, defrag the drive and then shred those unused blocks too.

I like DATview and the fact that it doesn't use the DBISAM engine.  (By using both Dbsys and DATview can help isolate more variables in the
diagnostic process.)   But considering that tables are parsed and displayed on multiple GUI pages, I am not confident that I'm seeing
everything that's stored in the .dat and .idx files.  I have a lot of confidence if I use Hex Workshop... but a lot of confusion too. Wink Has anyone
built a lower level parser to that shows parsed table/index data mapped to the raw file?

Brian
--------------
"As I was going up the stair; I saw a man who wasn’t there; He wasn’t there again today; Oh, how I wish he’d go away..."  from Antigonish by
Wm. Hughes Mearns.
Fri, Apr 17 2009 10:58 AMPermanent Link

BAMaustin
When testing a Windows workstation DBISAM application, I like using Windows Task Manager  with the CPU, CPU Time, Mem Usage, Peak
Memory & VM Size columns on the Processes tab.

I also have a window showing the folder(s) with any temporary or work tables.  (Although I have to F5 force refreshes to get updated file
sizes.)

Watching how the resource usage changes can help point you down a better path.
Fri, Apr 17 2009 1:28 PMPermanent Link

"Raul"
Sorry - can't help you with any tools but if your main concern is doing a
proper data erase and understanding binary data files on a byte-level then
just license the DBISAM source-code version. You can then customize it to
your needs and use it to build any tool you'd ever want.

Raul



"BAMaustin" <brianm@districtupport.com> wrote in message
news:FEA2761A-F000-4ED5-8C71-EF78AA0D731C@news.elevatesoft.com...
>I was wondering what tools others have found mandatory in their diagnostic
>toolboxes for DBISAM tables?
>
> We've used the Elevate database system utility (DBsys) tool for a long
> time.  But it wasn't until using the DBISAM Viewer  (DATview from
> http://www.scalabium.com/dbisam/ ) tool today that we could actually SEE
> the garbage accumulating in edited records.  The garbage was not
> shown after using DBsys to "Optimize" . but is it REALLY gone?
>
> DatView's "Recover deleted records" action is a good reminder that
> deleting isn't enough if you're dealing with sensitive data.  Should I be
> shredding records before I delete?  Or maybe shredding deleted records
> when Optimizing?
>
> When that file size suddenly dropped, maybe that chunk of data is sitting
> intact on the drive waiting  for a hard core hacker.
>
> I'm paranoid enough to think that it is a good idea to periodically file
> shred, defrag the drive and then shred those unused blocks too.
>
> I like DATview and the fact that it doesn't use the DBISAM engine.  (By
> using both Dbsys and DATview can help isolate more variables in the
> diagnostic process.)   But considering that tables are parsed and
> displayed on multiple GUI pages, I am not confident that I'm seeing
> everything that's stored in the .dat and .idx files.  I have a lot of
> confidence if I use Hex Workshop... but a lot of confusion too. Wink Has
> anyone
> built a lower level parser to that shows parsed table/index data mapped to
> the raw file?
>
> Brian
> --------------
> "As I was going up the stair; I saw a man who wasn't there; He wasn't
> there again today; Oh, how I wish he'd go away..."  from Antigonish by
> Wm. Hughes Mearns.
>

Sun, Apr 19 2009 8:08 PMPermanent Link

"Adam H."

"BAMaustin" <brianm@districtupport.com> wrote in message
news:FEA2761A-F000-4ED5-8C71-EF78AA0D731C@news.elevatesoft.com...
>I was wondering what tools others have found mandatory in their diagnostic
>toolboxes for DBISAM tables?
>
> We've used the Elevate database system utility (DBsys) tool for a long
> time.  But it wasn't until using the DBISAM Viewer  (DATview from
> http://www.scalabium.com/dbisam/ ) tool today that we could actually SEE
> the garbage accumulating in edited records.  The garbage was not
> shown after using DBsys to "Optimize" . but is it REALLY gone?

Not if edisoft's made a backup of the tables before doing the optimize. Wink

You may want to check for .dbk, idk and blk files and delete them too. From
what I'm aware of (and someone will jump in and correct me if I'm wrong very
soon), DBISam does not fully delete a record when .delete is called, but
marks the record as being deleted.

The table needs to be rebuilt (ie, optimised) in order for the records to be
truly deleted, and the space recovered.

Cheers

Adam.
Mon, Apr 20 2009 12:55 PMPermanent Link

Joel Schaubert

test1 ------------------------------------- simple undelete -----------------

File with 10 student records.
Using dbsys ran sql  "delete from student where suffix='JR'"
this deleted 3 records
7 records remained

Use scalabium to undelete.
Reopen with dbsys.
10 records are present, BUT scalabium did not seem to get the indices correct.
Running sql delete again gives
 ERROR 8965  index page buffers corrupt
Running 'repair table' fixes up the index problems


test2 ---------- delete + RepairTable() call
File with 10 student records.
Using dbsys ran sql  "delete from student where suffix='JR'"
this deleted 3 records
run RepairTable()        // in our app our user icon calls this reindexing the tables
7 records remained


run scalabian and undelete
still have 7 records present, the 3 deleted are NOT recovered.
And the indices are corrupt again



So delete and RepairTable() are making the records inaccessible to scalabium undelete

Joel
Tue, Apr 21 2009 1:35 PMPermanent Link

Tim Young [Elevate Software]

Elevate Software, Inc.

Avatar

Email timyoung@elevatesoft.com

Brian,

<< We've used the Elevate database system utility (DBsys) tool for a long
time.  But it wasn't until using the DBISAM Viewer  (DATview from
http://www.scalabium.com/dbisam/ ) tool today that we could actually SEE the
garbage accumulating in edited records.  The garbage was not shown after
using DBsys to “Optimize” … but is it REALLY gone? >>

It's gone as far as any applications are concerned.  As far as the actual
hard drive and the Windows file system, no, those clusters are still hanging
around until the file system re-uses them.  You can find utilities for
wiping this data also, if you really need the data gone completely.
SysInternals has one:

http://technet.microsoft.com/en-us/sysinternals/bb897443.aspx

<< DatView's “Recover deleted records” action is a good reminder that
deleting isn't enough if you're dealing with sensitive data.  Should I be
shredding records before I delete?  Or maybe shredding deleted records when
Optimizing? >>

If you need information to be really "erased" then yes, you should wipe the
fields with an Edit before actually deleting them with a Delete.

--
Tim Young
Elevate Software
www.elevatesoft.com

Sun, Apr 26 2009 12:42 PMPermanent Link

Jon Lloyd Duerdoth
Not sure whether running the SDelete (SysInternals) utility
from the same drive you're trying to clean-up will take care
of all issues, such as cluster tips (especially if they're
locked in some way by the operating system).

I've been experimenting with UBCD4Win to boot off a CD and treat the
C: drive as a data drive.  Eraser v5 is part of that package and
(I think) handles the above issue.

Caveat:  I'm no expert on this topic!

JonD

Tim Young [Elevate Software] wrote:
> Brian,
>
> << We've used the Elevate database system utility (DBsys) tool for a long
> time.  But it wasn't until using the DBISAM Viewer  (DATview from
> http://www.scalabium.com/dbisam/ ) tool today that we could actually SEE the
> garbage accumulating in edited records.  The garbage was not shown after
> using DBsys to “Optimize” … but is it REALLY gone? >>
>
> It's gone as far as any applications are concerned.  As far as the actual
> hard drive and the Windows file system, no, those clusters are still hanging
> around until the file system re-uses them.  You can find utilities for
> wiping this data also, if you really need the data gone completely.
> SysInternals has one:
>
> http://technet.microsoft.com/en-us/sysinternals/bb897443.aspx
>
> << DatView's “Recover deleted records” action is a good reminder that
> deleting isn't enough if you're dealing with sensitive data.  Should I be
> shredding records before I delete?  Or maybe shredding deleted records when
> Optimizing? >>
>
> If you need information to be really "erased" then yes, you should wipe the
> fields with an Edit before actually deleting them with a Delete.
>
Image