André,

<< But for me it doesn't look like a feature.

It's not a bug? It's a feature? >>

It's neither.  It's a problem caused by another piece of software.  For
example, if someone wrote a piece of software that simply monitored the hard
drive and deleted or moved any .dat file that was created in the temporary
files directory, would you consider that a DBISAM bug ?  That's essentially
what we're dealing with here, and the evidence that it isn't a bug is in the
changes that we made in 4.x - we simply allow the developer to change the
file extension that we're using for temporary tables.

--
Tim Young
Elevate Software
www.elevatesoft.com

Tim,

From the posts before, I understood the problem as if it's caused by the creation-flag of the files .. (delete on close) couldn't this be left away, in this case
it should be unimportant if any other software is monitoring and opening tempfiles. Because even if the temporaryfileextension is changed to *.tmp or
whatever another monitoring software could harm those too.

Even if you allow the developer in the 4.xx versions to change the extension wasn't it possible that it will happen with any extension if a monitoring
software is hitting just all instead only .dat-files?

What about changing the handling of tempfile creation and deletion? So that a tempfile is created without any winAPIs that causes flags to delete a
temporary created file again after beeing opened by another application or service.
And on a point where the tempfile is surely not nessessary anymore, it could get deleted explicitly.

greetings André

"Tim Young [Elevate Software]" <timyoung@elevatesoft.com> wrote:

André,

<< But for me it doesn't look like a feature.

It's not a bug? It's a feature? >>

It's neither.  It's a problem caused by another piece of software.  For
example, if someone wrote a piece of software that simply monitored the hard
drive and deleted or moved any .dat file that was created in the temporary
files directory, would you consider that a DBISAM bug ?  That's essentially
what we're dealing with here, and the evidence that it isn't a bug is in the
changes that we made in 4.x - we simply allow the developer to change the
file extension that we're using for temporary tables.

--
Tim Young
Elevate Software
www.elevatesoft.com

> Temporary tables are opened exclusively.  There should be no way that an
> external process could interfere with the files once they are open.  

Indeed.  I was not implying that.  Here are the scenarios I had in mind...

The AV software has requested Windows to notify it when new files are
created.  It has a thread sitting there idle just waiting to act on that
notification, so it will get in on the act very early on in the lifetime of
the file.

Scenario 1.
The AV opens the file, scans it, finds no virus, and closes it.  If this
happens at a time when the file is not currently open in DBISAM, but when it
does have a delete on close flag associated with it, then Windows will delete
the file.  

Scenario 2.
The AV opens the file, and begins to scan it.  Before the AV closes it,
DBISAM tries to open it in exclusive mode but fails because it is still open.
This failure could look like the file has been deleted - especially if it
does disappear shortly afterwards as a result of the delete on close flag.

Both of the above scenarios could be "faked" by running DBISAM under a
debugger, stepping it through the initial OS calls to create/close/open the
file, and simultaneously opening and closing the file in another process
(scenario 1), or opening and leaving it open (scenario 2).

It strikes me that the AV software is probably not doing anything different
with files in the temporary directory compared to files in the application
directory.  The difference between the two types of file is more likely down
to the difference in how they are created/opened.  The software doing the
deleting is probably Windows itself.

--Bill Sparrow--

André,

<< From the posts before, I understood the problem as if it's caused by the
creation-flag of the files .. (delete on close) >>

That is not correct.

<< Because even if the temporaryfileextension is changed to *.tmp or
whatever another monitoring software could harm those too. >>

Sure.  But now you're getting into the territory where there's nothing you
can do short of disabling the AV software on the desired directory.

<< Even if you allow the developer in the 4.xx versions to change the
extension wasn't it possible that it will happen with any extension if a
monitoring software is hitting just all instead only .dat-files? >>

Yes.

<< What about changing the handling of tempfile creation and deletion? So
that a tempfile is created without any winAPIs that causes flags to delete a
temporary created file again after beeing opened by another application or
service. >>

As stated before, we don't use the delete-on-close flag for the creation of
the temporary table, only for the open after the creation (for obvious
reasons - you can't use the flag if you intend to close the file right after
creating it).  The problem is that the AV software is deleting/moving the
file before the open ever occurs.

--
Tim Young
Elevate Software
www.elevatesoft.com

Bill,

<< The AV opens the file, scans it, finds no virus, and closes it.  If this
happens at a time when the file is not currently open in DBISAM, but when it
does have a delete on close flag associated with it, then Windows will
delete the file. >>

I understand, but DBISAM does not use the delete-on-close flag during file
creation.  Only during the opens.  In addition, the delete-on-close flag is
a Win32 API call parameter, not a file attribute.  It does not persist
beyond the current API call, therefore there is no way for Windows to know
that DBISAM wants this parameter until it gets the CreateFile call with the
parameter.  Finally, you cannot logically have a file that is not open with
the delete-on-close flag.

<< The AV opens the file, and begins to scan it.  Before the AV closes it,
DBISAM tries to open it in exclusive mode but fails because it is still
open.  This failure could look like the file has been deleted - especially
if it does disappear shortly afterwards as a result of the delete on close
flag. >>

I've never encountered a case where Windows reports a "file not found" issue
as a "access error" issue, or vice-versa.  If DBISAM says that the file is
not there, then it means that Windows told it that the file isn't there.  In
this particular case, it means that the AV software moved it or deleted it.
If DBISAM says that it can't access the file, then it's most likely due to
the AV software having the file open after DBISAM created it and interfering
with the exclusive open (again, DBISAM is informed by Windows that there is
an access error).

<< The difference between the two types of file is more likely down to the
difference in how they are created/opened. >>

DBISAM doesn't create temporary tables any differently than it does normal
tables.  The fact that normal tables work is due to the fact that the AV
software scans particular directories, like the local hard drive directories
or the temporary files directory.

<< The software doing the deleting is probably Windows itself. >>

Absolutely not.  What you're describing completely fails to account for the
fact that it works fine in situations where the AV software is not present.
DBISAM only uses the delete-on-close flag for the exclusive opens, hence it
should never fail or have an issue unless something messes with the file
after the create/close but before the exclusive open.

--
Tim Young
Elevate Software
www.elevatesoft.com

Ah.  It looks like I have misunderstood some Windows fundamentals.  Thanks
for putting me right, Tim, and apologies for casting aspersions on your
coding!

I thought the delete on close was an attribute of the file on disk, now I see
it is an attribute stored in the singleton associated with the open file.

Looks like I was getting mixed up with FILE_ATTRIBUTE_TEMPORARY which can be
specified as a parameter to SetFileAttributes.

Bill,

<< Ah.  It looks like I have misunderstood some Windows fundamentals.
Thanks for putting me right, Tim, and apologies for casting aspersions on
your coding! >>

No problem.  And I'm not perfect, so it certainly pays to ask questions when
things look like they might be incorrect.  Sometimes that jogs my noggin in
a way that reveals a solution.

--
Tim Young
Elevate Software
www.elevatesoft.com