Icon View Thread

The following is the text of the current message along with any replies.
Messages 1 to 10 of 18 total
Thread Client App freezes
Thu, Oct 2 2014 12:27 PMPermanent Link

KyleL

Hi everyone,

I've been perusing the forum to try and get some information to help troubleshoot an issue our software is having in a customer's office, however I don't believe I've found any existing threads that deals with the specific kind of issue we are encountering. To start let me explain that I am the software QA person for the company that is using DBISam for our software, not the developer, however I do have a basic understanding of programming and should have all the resources needed to answer questions. Now to my question.

We are using DBISAM V 4.26 with our application coded with Delphi 7. Our client/server version of our software works fine with the "out-of-the-box" build of the server application in most offices, but we do encounter issues with one office in particular. This office uses our software on 30 computers, which when connecting to the server can open up to about 60 connection sessions depending on what they are doing in our application (we have a main session thread and a secondary one used for speeding up calculations when they are on the relevant segment of our application). The server machine in this office is a Dell server running Windows Server 2003 (32bit) with 4 gigs of ram and all it does is host the server application service and the data.

The daily order of events at this office is as follows. Office opens up and runs using our application. They typically go through the day without issues for about 6 - 8 hours. Soon after the office begins to experience slow down when posts to the database are being made (I've noticed that navigation through the software is fine, but a point is reached when any attempt to amend the database tables results in the application hanging). The nature of the application freeze doesn't trigger the OS task manager (the OS on the client machines is a hodge podge of windows xp and windows 7 machines) to say our application is not responding, its simply frozen. I've checked for window popup issues disappearing behind our main app and that is not happening. it seems that when this issues first happens on one computer then over the course of the next 15 minutes or so all will experience the issue. Checking memory usage on the server it ranges from 400MB to over 800 MB. The customer then goes to the server and start/stops the service, and the clients ctrl-alt-delete out of our application and reopen. Rinse repeat.

I checked with our developer and we are using FastMM with the compiled server app the customer is using, and other than that change the base server app is the standard distribution from Elevate. I would think perhaps our client software might have been having problems, but if that were the case then why would it work for most of the day then freeze on all clients? I've tried several adjustments to session time out, dead session cleanup, etc from the admin utility. After increasing the dead session cleanup interval to three hours (we originally had it at too low a value) this seemed to improve how long they last before failure and memory usage would grow higher, but the critical freezing event still occurs.

My understanding of FastMM is that it's an alternate memory manager that's been proven to reduce fragmentation with large amounts of memory tables, so I wonder if this issue might be a more sever extension of that problem where memory tables are becoming fragmented due to the sheer volume created from the 35 - 50 remote connections being up at any given time. When the applications freeze the server application is apparently not frozen as we are able to login to the admin utility and dump the server logs before the customer needs to reset the service. When the freezing occurs you can still open our application and navigate through the software up until you attempt to generate a report or save new data to the database. We have reindexed the database with dbsys as well for what it's worth.

Any help that can be offered for this would be greatly appreciated as the customer is quite obviously becoming tired of having to deal with this problem and the things we've tried to address this do not seem to work.

Thanks
KyleL
Thu, Oct 2 2014 3:08 PMPermanent Link

Jose Eduardo Helminsky

HPro Informatica

KyleL

This is very hard to find troubles like that but first of all you have to check some things:

- Set the "Max # of dead sessions" to 0 (zero).
- Set the "Dead session expiration" to 15
- Set the "Dead session cleanup interval" to 30

I have similar issues with one of my instalations with around 200 users

What username the session is considering to connect to the server ? If the hang time is around the same hour of the day, have you already checked the disk usage ? Someone making a huge backup (some Gbytes) can slow down a lot the disk access.

So far, this is the questions

Eduardo (HPro)
Fri, Oct 3 2014 4:51 AMPermanent Link

Roy Lambert

NLH Associates

Team Elevate Team Elevate

KyleL


If following Eduardo's ideas doesn't improve things then its very difficult.

You might want to try recompiling the server with MadExcept or EurekaLog to see if they can spot the problem.

The fact that you can log in and dump server logs indicates that the server itself isn't frozen so I wonder if there is some sort of race condition eg two queries both trying to set transaction locks and both waiting on the other to finish.

Finally, if there are other offices about the same size, about the same work pattern, then suspect the site rather than your software

Roy Lambert

Fri, Oct 3 2014 10:21 AMPermanent Link

KyleL

Jose Eduardo Helminsky wrote:

KyleL

This is very hard to find troubles like that but first of all you have to check some things:

- Set the "Max # of dead sessions" to 0 (zero).
- Set the "Dead session expiration" to 15
- Set the "Dead session cleanup interval" to 30

I have similar issues with one of my instalations with around 200 users

What username the session is considering to connect to the server ? If the hang time is around the same hour of the day, have you already checked the disk usage ? Someone making a huge backup (some Gbytes) can slow down a lot the disk access.


-------------------------
Hi Jose, thanks for taking the time to respond.

We haven't tried the combination of settings you suggested for dead session handling. I'm willing to give a try, but am just wondering what the purpose of going with those settings is. My first concern in doing this stems from what was discussed in this earlier thread regarding a memory accumulation bug that happened when lock files were created in the db engine when a new session was established:
http://www.elevatesoft.com/forums?action=view&category=dbisam&id=dbisam_cs&page=1&msg=7982

If I understand the nature of that issue correctly, and knowing the version of DBISam we use pre-dates the update Tim applied to fix that problem, I'm wondering if this could make freezing occur more often as session recreation could ramp up. Just my thoughts on that.

As for your other questions, we've looked for scheduled tasks/backups/operations that could be contributing to the freezing and nothing is set to happen on that server except for backup via a utility which is scheduled to run at 2AM on weekends for full backups and shorted incremental backups scheduled at 2am to occur on week nights. That's the only task that occurs and it is most definitely after hours. The database and engine are run on their own dedicated 7200 rpm hard drive. When the program isn't freezing performance is actually very good considering the size of their office and amount of data being processed by up to 30 clients machines at any given time.

The actual freezing issue occurs randomly, typically later in the day, but within a range of four hours or so. The server is not rebooted nightly and I've noticed that memory used by DBISam is retained even if all connections to the server are closed. What I think happens is they'll do a certain amount of work in the system, memory usage will increase over time, they will encounter a freeze and restart, then proceed to do in effect the same quantity of work and the freezing will occur again in as much time. So if they freeze toward the end of their day one day they would restart the server and not encounter any freezing issues til much later in the day the next day, and the cycle continues.

Our application uses two unique logins when establishing sessions, one is an admin thread, the second is Admin. Technically the same but unique user names due to the capital a. The admin user is tied to the main session that the application uses while the Admin user is used for additional sessions which is connected depending on what the user is doing in our application, and from what the developer tells me is used for speeding up calculations our program performs.
Fri, Oct 3 2014 12:34 PMPermanent Link

KyleL

Roy Lambert wrote:

KyleL


If following Eduardo's ideas doesn't improve things then its very difficult.

You might want to try recompiling the server with MadExcept or EurekaLog to see if they can spot the problem.

The fact that you can log in and dump server logs indicates that the server itself isn't frozen so I wonder if there is some sort of race condition eg two queries both trying to set transaction locks and both waiting on the other to finish.

Finally, if there are other offices about the same size, about the same work pattern, then suspect the site rather than your software

Roy Lambert

-----------------------------

Thanks for the response Roy.

I'm willing to try the changes mentioned, but would like to know the reason for setting those settings like that, more for the purpose to better understand what those settings contribute to how the engine operates (my full response can be found in the above post). I will get back in touch to let you guys know how it goes though.

As for recompiling with MadExcept/EurekaLog, I'll talk to our developer about that.

This customer is on the upper end of the spectrum in terms of size, but we do have bigger ones. We don't really hear from the larger customers but they also have their own dedicated IT crew who I think has just learned how to quickly deal with these kind of issues. I don't know if that IS the case but from experience I'm inclined to believe that is the case. We've been working with this customer for awhile and they have made all the recommended adjustment and setup requests we've supplied, so I'm not terribly inclined to blame their setup either.
Fri, Oct 3 2014 5:42 PMPermanent Link

Raul

Team Elevate Team Elevate

<< KyleL wrote:
Any help that can be offered for this would be greatly appreciated as the customer is quite obviously becoming tired of having to deal with this problem and the things we've tried to address this do not seem to work.
>>

KyleL,

My initial guess would be around dead sessions or locking as well but its hard to guess.

You've gotten some good advice already so i'll add some different items - most are workarounds rather than solutions but might be of value.

Not sure from the description if this is daily issue or just regular. If not everyday occurrence then i would suggest stopping and restarting the dbisam server every night (if you can - meaning customer is not  24/7 operation) or even rebooting the actual server nightly. It won't solve the root of the problem but if this gets customer thru full day then at least i would say the server side is the problem and not app.

I don't know the app but my other suggestion would be to add some extra logging in the app - for example in a specific are where user has to update the data and post it and then log as much detail (with timestamps) to see where the slowdown happens (for example is it the post or start transaction, commit, etc). Somebody from customer side would have to use this area of the ap pwhen slowdown happens but it might help narrow it down to specific operation (as well as compare to normal timings).

FastMM - possible as you do consume 800MB but the FastMM generally is a good memory manager and i would have expected to see more posts (here and in delphi forums) if issue were widespread. Not saying to ignore it but maybe leave it lower in the list to check. i know our app has been running for years with FastMM without problems.

Any chance you can upgrade to newer dbisam version to see if it does not have the problem?

Something we're doing is running multiple dbsrvr instances on the same server (against same db) to both balance the load and provide redundant dbisam servers.  Your client app would need to know there are more than 1 available (and rotate thru them) but something longer term that might help.

Raul
Tue, Oct 7 2014 5:03 PMPermanent Link

KyleL

Hey guys I have an update.

The changes to disconnect settings did not result in a change.

I have appealed to our developer about compiling a test build with additional error logging built in for us to try.

The issue appears independent to memory usage. I noticed that memory usage by the engine was lower when no disconnected sessions were allowed to be created, but as the day went on and the client's business worked they eventually hit a network wide slow down and had to restart the server service.

One thing I'm looking to verify is that they are indeed using a build of the server app that has FastMM in it. We were sure it was, but nothing is as sure as recompiling the code and trying it. I'm wondering if this situation is symptomatic with issues FastMM helps resolve to begin with though.

At this time we need to stay with build 4.26 so upgrading isn't on the table.

Any other suggestions, help anyone would like to add would be greatly appreciated, and thanks again for the help so far.
Wed, Oct 8 2014 3:15 AMPermanent Link

Roy Lambert

NLH Associates

Team Elevate Team Elevate

KyleL


When the freeze happens can you still access the tables via DBSys? If not then I'd suspect some form of dead lock / race condition probably involving transaction locks.

Roy Lambert
Mon, Oct 27 2014 7:21 PMPermanent Link

KyleL

Roy Lambert wrote:

KyleL


When the freeze happens can you still access the tables via DBSys? If not then I'd suspect some form of dead lock / race condition probably involving transaction locks.

Roy Lambert

I had a chance to test for this. I was able to confirm that when testing connection to two prominent tables through dbsys when freezing occurred that dbsys was not able to complete the query and hung. So it does seem like something might be going on with how sessions are being handled?
Tue, Oct 28 2014 4:06 AMPermanent Link

Roy Lambert

NLH Associates

Team Elevate Team Elevate

KyleL



As I said, if that happens I'd suspect a race condition somewhere, probably involving transactions. These are very hard to track down.

Roy Lambert
Page 1 of 2Next Page »
Jump to Page:  1 2
Image