Hi,

I'm currently having an issue with one of my clients and a particular
server setup that is causing major performance issues with DBISam 4.

I have a table that is roughly 80MB in size, with indexes close to 70MB.

Recordsize is 1936, with 107 fields, and 31 indexes in total.

When I restructure the table (add or delete a field) on my computer (or
pretty much any other machines) it takes around 1 minute to accomplish
the task.

However they have 3 servers at different sites on the same network
configured similar to each other. This process can take anything from 13
to 76 minutes to complete the task. (Depending on which server I try it
on).

My machine is a virtual machine running Windows 7 and 4GB of RAM in
total. (The host has 16mb available to it).

Another computer on their network at another site has 8GB of RAM on
Server 2003, and it runs quite fast. (Similar to my machine - around 1
min to complete)

The 3 computers / servers that are having issues are configured as
virtual machines. The host has a total of 4GB of RAM, with 3GB given to
the guest OS which is Server 2008 R2.

When we run the restructure operation, we can see disk activity go up,
but it doesn't seem to be peaking all the time. Likewise memory usage
goes up a little bit, but once again, it doesn't peak / use all
available memory - yet it takes so much longer to accomplish the task.

Personally, I'd just love to throw more RAM into the machines, but
unfortunately these particular machines can't do that. They're
effectively an application server, and we'd have to replace the whole
board to accomplish this. (Which isn't going to happen unless I can
prove irrevocably that this will correct the issue, and their IT
department doesn't believe this is the case because they don't see the
RAM peak during my operations).

The server with 8GB of RAM that ran quickly (~1min) is running on the
same network, with the same Antivirus. (Although we did try turning off
the Antivirus application on the various servers which made no
difference to the performance).

I was just wondering - does anyone have any idea what would cause this
significant change in performance issues with DBISam on the servers
above, or can give insight into how DBISam actually operates and what
area we need to be looking into to trap the cause?

The differences between the machines / servers as I can see where the
slow servers are running are:

1) The slow servers are running Windows 2008 R2.

2) The slow servers are running with only 4GB physical RAM, and 3GB
passed through to the virtual machine.

Cheers

Adam.

Adam

I have had the same situation with one of my customers and the conclusion
is:

Windows 2008 = 8Gb or more

Eduardo

On 2/11/2013 8:03 PM, Adam H. wrote:

> However they have 3 servers at different sites on the same network
> configured similar to each other. This process can take anything from 13
> to 76 minutes to complete the task. (Depending on which server I try it
> on).

Every time we have run into something similar (and never as bad as what
you're seeing) it's been something interfering at file access layer -
AntiVirus, some backup software, etc.

Other thing to look for is temp folder dbisam would use - any chance
that is slow or included still in AV scanning.

You did not mention what virtualization solution is used - any chance
slowdown is on host side (rest of you VM OS should be impacted).

> When we run the restructure operation, we can see disk activity go up,
> but it doesn't seem to be peaking all the time. Likewise memory usage
> goes up a little bit, but once again, it doesn't peak / use all
> available memory - yet it takes so much longer to accomplish the task.

I personally use Process Monitor from sysinternals
(http://technet.microsoft.com/en-us/sysinternals/bb896645) to see what's
really going on. You can filter on dbisam table name in question and see
whether something else is also accessing files and how long do the reads
take.


> Personally, I'd just love to throw more RAM into the machines, but
> unfortunately these particular machines can't do that. They're

> The differences between the machines / servers as I can see where the
> slow servers are running are:
> 1) The slow servers are running Windows 2008 R2.
> 2) The slow servers are running with only 4GB physical RAM, and 3GB
> passed through to the virtual machine.

What's the virtualization solution?

For server 2008 R2 3GB is not optimal but also within minimum so it
should nto be this slow either. Anything else running or just base OS
and your app?  I'd also look at disk performance - could be just a slow
disk or even host consuming disk I/O.


Raul

Hi Eduardo,

> I have had the same situation with one of my customers and the conclusion
> is:
>
> Windows 2008 = 8Gb or more


I've got to admit, I'm very surprised that they're willing to run a
server under such low resources, but I guess considering they'd probably
have a few hundred servers within the corporation, they try to keep
costs low.

Unfortunately 3 of these servers also run my application, making it
difficult.

Cheers

Adam.

Hi Raul,

Thanks for your reply...

> Every time we have run into something similar (and never as bad as what
> you're seeing) it's been something interfering at file access layer -
> AntiVirus, some backup software, etc.

That was my thought too. We've disabled the antivirus, and also
confirmed that the backups only run at a specific time (and weren't
running at the time of testing). Pitty - that's normally the issue, and
normally a quick fix too.


> Other thing to look for is temp folder dbisam would use - any chance
> that is slow or included still in AV scanning.

Good thought, but no - as we've completely disabled the antivirus at one
point during our testing just to confirm there wasn't anything missed
such as this.

> You did not mention what virtualization solution is used - any chance
> slowdown is on host side (rest of you VM OS should be impacted).

There's only the one VM running on the server, and shouldn't be any
other processes there at all. Apparently, apart from my application -
the only other stuff the server is doing is basic file sharing for the
local site.

> I personally use Process Monitor from sysinternals
> (http://technet.microsoft.com/en-us/sysinternals/bb896645) to see what's
> really going on. You can filter on dbisam table name in question and see
> whether something else is also accessing files and how long do the reads
> take.

Thanks for that - something worth trying.

> What's the virtualization solution?

I believe they're running VMWare.

> For server 2008 R2 3GB is not optimal but also within minimum so it
> should nto be this slow either. Anything else running or just base OS
> and your app?  I'd also look at disk performance - could be just a slow
> disk or even host consuming disk I/O.

Thanks Raul - we'll continue to investigate.

Cheers

Adam.

On 2/13/2013 6:04 PM, Adam H. wrote:
>> What's the virtualization solution?
>
> I believe they're running VMWare.

i would check what they are using - if they are using hypervisor (ESX,
ESXi, etc) then i'm not sure why your VM does not get all 4GB (unless
there is another VM running).

If they use something like vmware server or other host based VM then 1GB
for host might be a bottleneck.

Raul

Thanks Raul,

I'm not sure which they're running, but I shall endeavor to find out.

Cheers

Adam.

>>
"Eduardo [HPro]" wrote:

Adam

I have had the same situation with one of my customers and the conclusion
is:

Windows 2008 = 8Gb or more

Eduardo
>>

What he said. At 3GB those Server 2008 R2 systems are crippled. It takes at least 8GB to get normal performance out of 2008. What's worse though I think is that the underlying hypervisor is overloaded. My suspicion is that there is a lot of page faulting/thrashing going on in the hypervisor as it tries to satisfy the memory demands of the guest, and that is in turn killing the performance of the guest.

I doubt there is going to be anything you can do about it.

Hi Arthur,

Thanks very much for that - I think you could be right in what you're
saying. The reason why we may not be seeing a whole lot of disk activity
might be due to the underlying hypervisor doing all the work instead.

I guess we've really reached a point where I can't give them a
guarantee, but a best guess. I'll be up to them to decide how they want
to continue now.

Cheers

Adam.