Hi Tim,

A few days ago I upgraded one webserver running 2.2.8 to 2.2.11 and it resulted in some strange memory issues.

Running 2.2.8 the edbserver runs with a memory useage of 40-80MB, however, the switch to 2.2.11 made the memory useage go sky-high.
Memory consumption climbs steadily until it reaches 2GB and the performance is not in the same league as 2.2.8.

This, somehow, also turns the wp3 service into a memory hog.

Well, long story short, I switched back to 2.2.8 last night and performance is back to normal.
(actually it is lightning fast, running on Intel X25-E SSD)

So, what happened between 2.2.8 and 2.2.11?

--
Jan-Erik

Jan-Erik,

<< A few days ago I upgraded one webserver running 2.2.8 to 2.2.11 and it
resulted in some strange memory issues.

Running 2.2.8 the edbserver runs with a memory useage of 40-80MB, however,
the switch to 2.2.11 made the memory useage go sky-high.  Memory consumption
climbs steadily until it reaches 2GB and the performance is not in the same
league as 2.2.8.

This, somehow, also turns the wp3 service into a memory hog.

Well, long story short, I switched back to 2.2.8 last night and performance
is back to normal. (actually it is lightning fast, running on Intel X25-E
SSD)

So, what happened between 2.2.8 and 2.2.11? >>

Query result sets were changed so that they can buffer more memory, but not
a very large increase and it should go away once the result set is closed.
This also affects views.

Are you opening up queries/views and keeping them open for long periods of
time ?

--
Tim Young
Elevate Software
www.elevatesoft.com

Hi Tim,

<<
Query result sets were changed so that they can buffer more memory, but not
a very large increase and it should go away once the result set is closed.
This also affects views.

Are you opening up queries/views and keeping them open for long periods of
time ?
>>

OK,
99% of the queries are standard SqlDataSources with EnableCaching turned On.
I'm not sure how this is handled internally in (ASP).NET but I expect they are closed as soon as the query returns the result set?

Most queries contain images so the memory "print" of the result set is probably large compared to "normal" queries.
But, should the cache grow beyond the size of the database?
Currently, the active part of the database is less than 200MB and the total size is less than 1GB.

I observed the memory consumption on 2.2.8 today and it never exceeded 100MB (TaskManager)
2.2.11 reached 1GB in 1-3 hours, depending on the workload.
I don't think I ever observed that the memory size decreased on 2.2.11. (TaskManager)

The server is running Windows 2003 Server x64 with 4GB memory.

What do you need to get this sorted out?

--
Jan-Erik

Jan-Erik,

<< 99% of the queries are standard SqlDataSources with EnableCaching turned
On. >>

<< I'm not sure how this is handled internally in (ASP).NET but I expect
they are closed as soon as the query returns the result set? >>

No, they are not closed until the EDBCommand is closed, re-executed,
unprepared, or disposed of, or a EDBDataReader being used with the
EDBCommand is closed or disposed of.

Also, just to confirm - are you using the EDB Server at all ?

<< Most queries contain images so the memory "print" of the result set is
probably large compared to "normal" queries. >>

How big are the images ?  And also, what is the block size of the table(s)
that contain the images ?

<< But, should the cache grow beyond the size of the database? >>

It can with address fragmentation in the memory manager, or in the case of
..NET, the garbage collector takes its sweet time releasing the memory.

<< What do you need to get this sorted out? >>

Just the above answers for now, but I may need more later.  I'll let you
know.

--
Tim Young
Elevate Software
www.elevatesoft.com

<< 99% of the queries are standard SqlDataSources with EnableCaching turned
On. >>

<< I'm not sure how this is handled internally in (ASP).NET but I expect
they are closed as soon as the query returns the result set? >>

No, they are not closed until the EDBCommand is closed, re-executed,
unprepared, or disposed of, or a EDBDataReader being used with the
EDBCommand is closed or disposed of.

Also, just to confirm - are you using the EDB Server at all ?

<< Most queries contain images so the memory "print" of the result set is
probably large compared to "normal" queries. >>

How big are the images ?  And also, what is the block size of the table(s)
that contain the images ?

<< But, should the cache grow beyond the size of the database? >>

It can with address fragmentation in the memory manager, or in the case of
..NET, the garbage collector takes its sweet time releasing the memory.

Tim,

The EDB server is used as a remote connection, although it is on the same physical server at the moment.

As I said, ~99% of the queries are used directly through components (DevExpress, Telerik) and there is no codebehind that handles the
SqlDataSources.
The remaining queries are opened, closed and disposed of in the code behind file.

At the moment EDB server 2.2.8 is using ~39MB of memory. (up time: 20hours, throughput ~10GB)
(It peaked at 130MB earlier today)

Typically the tables that store images have 2 blob columns, one of the columns is storing a thumbnail of the image (4-8KB) and the other blob
column is the image in full size (30-100KB, very few are outside of this range).

Blob block size : 4096-8192
Maximum Row Buffer Pool Size 32768-65636
Maximum Index Buffer Pool size 65536-131072
Maximum BLOB Buffer Pool size 32768-262144

I did find some warnings in the event log:
Exception message: ElevateDB Error #1100 A connection to the server at '127.0.0.1' cannot be established ('An existing connection was forcibly
closed by the remote host')

What could this mean?

--
Jan-Erik

Jan-Erik,

<< The EDB server is used as a remote connection, although it is on the same
physical server at the moment. >>

So, just to confirm, no local queries are being executed entirely in .NET,
rather they are all executed on the EDB Server ?

The only change that was made was to buffer up to 64k (count) worth of rows
before flushing to disk, but only during the execution of a query that
returns an insensitive result set, and after which the memory is freed.  So,
you should only be seeing peaks of memory consumption during the query
execution, and only for queries that generate that many rows during the
construction of the result set.  Plus, if you're using the EDB Server for
all query execution, then the memory should not be subject to any vagaries
of the .NET garbage collection, and will be using the FastMM4 memory
manager.

<< Typically the tables that store images have 2 blob columns, one of the
columns is storing a thumbnail of the image (4-8KB) and the other blob
column is the image in full size (30-100KB, very few are outside of this
range).

Blob block size : 4096-8192
Maximum Row Buffer Pool Size 32768-65636
Maximum Index Buffer Pool size 65536-131072
Maximum BLOB Buffer Pool size 32768-262144 >>

Could you possibly send me a sample copy of the database along with a query
that represents a typical query that is used often in your application ?
I'm obviously missing some little important item here, and my tests here are
not shedding any light on the situation.

<< I did find some warnings in the event log:
Exception message: ElevateDB Error #1100 A connection to the server at
'127.0.0.1' cannot be established ('An existing connection was forcibly
closed by the remote host')  >>

How many times did that error message appear ?  It could be a side-effect of
the memory issue where the lack of memory is causing failures in the
connections.

--
Tim Young
Elevate Software
www.elevatesoft.com

Tim,

<<So, just to confirm, no local queries are being executed entirely in .NET,
rather they are all executed on the EDB Server ?>>

All queries are done using a single ConnectionString which points to a "remote" server.

<<The only change that was made was to buffer up to 64k (count) worth of rows
before flushing to disk, but only during the execution of a query that
returns an insensitive result set, and after which the memory is freed.  So,
you should only be seeing peaks of memory consumption during the query
execution, and only for queries that generate that many rows during the
construction of the result set.  Plus, if you're using the EDB Server for
all query execution, then the memory should not be subject to any vagaries
of the .NET garbage collection, and will be using the FastMM4 memory
manager.>>

OK,
Once 2.2.11 consumed more than 1-1.5GB I had to restart EDBserver, it became slow and unresponsive.
The strange thing is that 2.2.8 performes so well, using the same code and components.

<<Could you possibly send me a sample copy of the database along with a query
that represents a typical query that is used often in your application ?
I'm obviously missing some little important item here, and my tests here are
not shedding any light on the situation.>>

I'll prepare something with the necessary tables and queries within a few days.
Depending on the size, you'll probably have to download it from one of my webservers.
(I'll get back to you on the details)

<<
Error #1100 A connection to the server at '127.0.0.1' cannot be established ('An existing connection was forcibly closed by the remote host')
How many times did that error message appear ?  It could be a side-effect of
the memory issue where the lack of memory is causing failures in the
connections.>>

At least a few times per hour when the server was busy, maybe more.
I'll send you an example of one of the exceptions when I get the tables ready.

--
Jan-Erik

Jan-Erik,

<< I'll prepare something with the necessary tables and queries within a few
days. Depending on the size, you'll probably have to download it from one of
my webservers. (I'll get back to you on the details) >>

Thanks.  I'll see if I can extrapolate the bigger picture from what you send
me.  2.02 B11 was designed to use more memory, but only on an extremely
short-term basis (after which the memory is completely freed), so I'm not
sure where all of the memory consumption is coming from, especially since
the Win32 FastMM4 memory manager is extremely good at avoiding fragmentation
of the address space for a given process.  Plus, if it was a blatant memory
leak, then it would be showing up in the leak tracking from FastMM4 that is
turned on for the EDB Manager and EDB Server.

--
Tim Young
Elevate Software
www.elevatesoft.com