Login ProductsSalesSupportDownloadsAbout |
Home » Technical Support » DBISAM Technical Support » Support Forums » DBISAM General » View Thread |
Messages 11 to 20 of 31 total |
#8963 BLOB block buffers corrupt |
Wed, May 18 2011 6:08 PM | Permanent Link |
Tim Young [Elevate Software] Elevate Software, Inc. timyoung@elevatesoft.com | Dave,
<< The application that populates the tables typically accesses the database files across the LAN and I'm beginning to think that this is perhaps at the root of our problem. >> Yes, that could very well be an issue. One thing that can verify this for sure is by putting a simple logging utility in your application that logs when the application starts up, and when it closes. Be sure to have the logging work like this: 1) Open log file using FileOpen, if it doesn't exist create it using FileCreate. Make sure that the log file is on a local drive. 2) Write out the log entry using FileWrite. 3) Flush the file to disk using this function: function FileFlush(Handle: Integer): Boolean; begin Result:=Windows.FlushFileBuffers(THandle(Handle)); end; 4) Close the log file using FileClose. This will allow you to very easily see if workstations are killing your application process while it is running, thus causing possible problems in the database. Once you have verified whether this is taking place or not, you can have a better idea of how to proceed. In general, with that much data, I would seriously recommend going to the DBISAM Database Server instead of direct access over the LAN. It will eliminate the possibility of an improper application shutdown hurting the database, and will give you much more control over the sessions that are connecting to the database(s). -- Tim Young Elevate Software www.elevatesoft.com |
Thu, May 19 2011 6:16 AM | Permanent Link |
Dave Sellers | Hi Tim
I'm not sure if we're quite on the same wavelength here. If I understand you correctly, the logging would prove that the application had closed normally rather than by somebody 'killing' it via taskmanager? That is not the issue. What generally happens is that the application will work through a long list of files loading their contents into fields in a DBISAM table. Sometimes it will get so far and then, for no apparent reason, each insert will ellicit a #8963 BLOB block buffers corrupt exception. The user will come back to the machine some hours later and my application will report something like "Process completed: 24,976 errors, see log for details" The log will list the file it was trying to load and the exception that was raised. If I can believe what I'm told, that AV and backup are not touching the DBISAM files, then my hope is that it is the act of doing all this work across a LAN is causing the corruption and that working on a local copy will fix it. I appreciate that a change to a C/S would be a good idea, it simply would not be practical at the moment for various reasons - not least, off the top of my head I couldn't tell you how many individual databases we have but it is several thousand!. We are storing OCR data and currently we OCR two million pages a month resulting in updating many hundreds of DBISAM databases each month. Those very large files are the exception rather than the rule and we get no more corruption issues with those than we do with ones of just a few hundred kB - it's just the big ones are the real nightmare because they take so long to repopulate. Regards Dave |
Thu, May 19 2011 7:04 AM | Permanent Link |
Roy Lambert NLH Associates Team Elevate | Dave
This is the first time you mention OCR. My previous experience (way out of date now) is that you can get some interesting non-alphanumeric characters embedded in a document. So to develop Tim's logging suggestion a bit more why not log which of the documents caused the problem then investigate it and the previous 3 or 5 just to make sure you've caught it. My bet is you'll find something nasty lurking in there. Roy Lambert [Team Elevate] |
Thu, May 19 2011 10:27 AM | Permanent Link |
Dave Sellers | Hi Roy
That is a thought but... The OCR engine is spitting out (allegedly) ANSI encoded files which I then load into DBISAM. I don't think it is the file content that is the issue though because if I delete and recreate the database and reload it from scratch it will load exactly the same files without a hitch. Surely if it were 'illegal' characters causing the problem they'd cause it every time wouldn't they? Regards Dave |
Thu, May 19 2011 11:32 AM | Permanent Link |
Roy Lambert NLH Associates Team Elevate | Dave
>Surely if it were >'illegal' characters causing the problem they'd cause it every time >wouldn't they? That would be my assumption, but when I get desperate I look under every stone! Can you post the actual code you're using just in case anyone can spot something? Not the data obviously both from a viewpoint of confidentiality and also there's a lot of it Roy Lambert [Team Elevate] |
Thu, May 19 2011 12:09 PM | Permanent Link |
Dave Sellers | Roy Lambert wrote:
> Dave > > > Surely if it were > > 'illegal' characters causing the problem they'd cause it every time > > wouldn't they? > > That would be my assumption, but when I get desperate I look under > every stone! > > Can you post the actual code you're using just in case anyone can > spot something? Not the data obviously both from a viewpoint of > confidentiality and also there's a lot of it > > Roy Lambert [Team Elevate] Hi Roy Fair comment. I'll try and extract the relevant sections hopefully tomorrow - I'm right in the middle of some remedial work on the application that handles the OCR itself... Cheers Dave |
Thu, May 19 2011 8:16 PM | Permanent Link |
Gregory Sebastian | <<then my hope is that it is the act of doing all this work across a LAN is
causing the corruption and that working on a local copy will fix it. >> Have you tried moving the DB to the local hard disk and test again ? That would probably the quickest way to verify this. Regards Greg |
Fri, May 20 2011 4:51 AM | Permanent Link |
Dave Sellers | Gregory Sebastian wrote:
> <<then my hope is that it is the act of doing all this work across a > LAN is causing the corruption and that working on a local copy will > fix it. >> Have you tried moving the DB to the local hard disk and > test again ? That would probably the quickest way to verify this. > > Regards > Greg Hi Greg Yes but the intermittency makes it hard to know whether you've fixed it or just got lucky. The only thing I can do is make the changes and then run it for a month and see if we get any new corruption. Thanks Dave |
Mon, May 23 2011 8:49 AM | Permanent Link |
Tim Young [Elevate Software] Elevate Software, Inc. timyoung@elevatesoft.com | Dave,
<< If I understand you correctly, the logging would prove that the application had closed normally rather than by somebody 'killing' it via taskmanager? That is not the issue. >> No, the logging would ensure that somebody didn't close down the application improperly - there would be a "login" entry, but no corresponding "logout" entry. When dealing with direct connections to the database over a LAN, you have to eliminate this as a possibility *first*. Not doing so will just leave you guessing as to the cause. << What generally happens is that the application will work through a long list of files loading their contents into fields in a DBISAM table. Sometimes it will get so far and then, for no apparent reason, each insert will ellicit a #8963 BLOB block buffers corrupt exception. >> Do records get recycled in this table (deleted and then new records re-added) ? Or is this table always appended to ? If the former, then there is a good chance that an improper shutdown could cause this type of behavior. An improper shutdown could cause this behavior in both cases, but the first is the more likely. -- Tim Young Elevate Software www.elevatesoft.com |
Mon, May 23 2011 11:23 AM | Permanent Link |
Dave Sellers | Tim Young [Elevate Software] wrote:
> Dave, > > << If I understand you correctly, the logging would prove that the > application had closed normally rather than by somebody 'killing' it > via taskmanager? That is not the issue. >> > > No, the logging would ensure that somebody didn't close down the > application improperly - there would be a "login" entry, but no > corresponding "logout" entry. When dealing with direct connections > to the database over a LAN, you have to eliminate this as a > possibility first. Not doing so will just leave you guessing as to > the cause. Hi Tim I take your point and if there was any doubt about the application being improperly closed I would certainly implement it The application is not being closed improperly, it is still running announcing the details of the error. Here's an example: 21/05/2011 15:39:12.456 - Page 12 of file xyz added to table 21/05/2011 15:39:12.789 - Page 13 of file xyz added to table 21/05/2011 15:39:13.456 - Page 14 of file xyz added to table 21/05/2011 15:39:13.789 - Page 15 of file xyz added to table 21/05/2011 15:39:14.456 - Problem adding Page 16 of file xyz added to table: Error message #8963 BLOB block buffers corrupt 21/05/2011 15:39:14.789 - Problem adding Page 17 of file xyz added to table: Error message #8963 BLOB block buffers corrupt ...... This error will now repeat for every page of every file in the folder structure being processed until: 21/05/2011 18:16:14.345 - Update complete, 12,345 errors - see above for details. This happens repeatedly. The users calls me, I remote into their machine and there is the evidence in front of me. In my mind this is evidence enough that the application was not closed improperly because it has not been closed at all. > Do records get recycled in this table (deleted and then new records > re-added) ? Or is this table always appended to ? If the former, > then there is a good chance that an improper shutdown could cause > this type of behavior. An improper shutdown could cause this > behavior in both cases, but the first is the more likely. Records for a given file/page are deleted if they exist and then the new record inserted. I have never established whether the errors always occur only when deletions have actually taken place, this could be logged but again, improper shutdown is not taking place. Dave |
« Previous Page | Page 2 of 4 | Next Page » |
Jump to Page: 1 2 3 4 |
This web page was last updated on Tuesday, May 7, 2024 at 06:25 PM | Privacy PolicySite Map © 2024 Elevate Software, Inc. All Rights Reserved Questions or comments ? E-mail us at info@elevatesoft.com |