Icon View Thread

The following is the text of the current message along with any replies.
Messages 1 to 10 of 10 total
Thread Odd Slowness
Thu, Nov 29 2018 5:44 PMPermanent Link

David

I have been noticing an odd slowness in my C/S application recently and wondering if anyone has any idea what it could be.

I have a customised server that has a couple of triggers to create an audit log for all new, deleted or edited records.  The system has been working fine for a number of years and not much has changed.

What I am noticing is on occasion if I edit a record it could take a 0.5 to 1 secondto add the record to the audit log, it should be instant.  Normal operations can have many records created at the exact same time so taking 0.5 to 1 second per record is the cause of the slowness.   As the Audit Log can get large after months of running I wondered if this was due to the size of the log so I cleared it out and this worked for some time but recently it has came back again.  

I have restarted the server when I noticed it running slow and it didn't make any difference, but if I open the Audit log first on any connected client, the audit log returns to normal operation.  I also noticed today that doing an optimize on the table without restarting appeared to return the Audit Log back to normal operation, but for how long I am not sure.

This is running on 4.25 at the moment on Server 2008 R2.  The only change are our corporate IT have forced use to use SentinelOne end point protection on the client and server, this sadly does not make it easy to put in any exclusions or even see any log file so I am a little in the dark with this software.  Also our server used to be restricted to SMB 1.0 because I had read about issues with SMB 2 on server 2008 but I think I picked up on this wrong as this wont apply to C/S so I enabled it recently as Windows 10 will be getting used soon and will not work with SMB 1.0


I was just wondering is there anything in an optimize that would release some kind of lock, potentially from SentinelOne or something to do with SMB 2?


It is possible to create an app to keep the Audit Log open all the time, but not my preferred option as this never used to be needed for years.


Hoping someone could maybe help shine some light on this.


Regards
David.
Fri, Nov 30 2018 2:34 AMPermanent Link

Roy Lambert

NLH Associates

Team Elevate Team Elevate

David


My bet would be on SentinelOne. Way back in the dim and distant past I do remember people doing speed tests on DBISAM into the millions of records and that did exhibit slowdown but nothing as drastic as a second (an may not be the case with the modern versions).

I think you're going to have to find out how to exclude directories from scanning. Frown

Roy Lambert
Fri, Nov 30 2018 4:25 AMPermanent Link

David

Hi Roy.

Yes that is my suspicions regarding the A/V as well but it is still on odd behaviour.  At one time there is a noticeable delay, but after doing an optimize itappears to start working again, it is almost like the optimize released a lock .  

This type of A/V is not supposed to scan files, but it does examine the processes and using machine learning to decide on what is suspicious activity.  I wonder if it is this that is happening, but sadly it is a black box and not easy to get a picture of what is happening.




Roy Lambert wrote:

David


My bet would be on SentinelOne. Way back in the dim and distant past I do remember people doing speed tests on DBISAM into the millions of records and that did exhibit slowdown but nothing as drastic as a second (an may not be the case with the modern versions).

I think you're going to have to find out how to exclude directories from scanning. Frown

Roy Lambert
Fri, Nov 30 2018 5:24 AMPermanent Link

Roy Lambert

NLH Associates

Team Elevate Team Elevate

David


Optimising a table essentially means writing a new table then deleting the old one (I think DBISAM renames the old table first). Its the only sane way to do it. I dread to think what would be required if you just shuffled bits up to overwrite the deleted stuff, especially if the index to optimise by was different to the entry order. What that sort of means is that if SentinelOne is "keeping an eye" on the original file because it thinks its suspicious it will probably take some time to reach the same decision about the new file.

Looking at SentinelOne's website it doesn't give many clues about what its actually doing (and I really hate sites where I can't easily, or at all, find a cost). My guess would be - optimise sets a pattern of high volume writes and then it "sees" a different pattern and is checking to make sure nothing nasty is happening. What they're actually doing if, as they claim, they're not using signatures is beyond my ability to determine from the bullshit bingo that has infested their website.

One thought that did occur to me is why not automate the creation of a new logfile every day (week or whatever period seems good)?



Roy Lambert
Fri, Nov 30 2018 7:33 AMPermanent Link

David

Hi Roy.

Yes I don't like Sentinel One either, I hate anything that is hidden from at least an admin into what it is getting up too.  Your thoughts on possible causes does make some sense, however I did notice that this seemed to persist even after restarting the DB server, do you think that the AV could still be hanging onto the file even after a DB server restart?  If so, your thoughts make more sense with the apparent freeing after an optimize.

I have thought about automating this and doing an optimize on the Audit Log when the DB server starts up.  I have a task that restarts the server every few days to free up memory creep in DBISam that I can't get rid of completely and this is the easiest way to resolve, so it could work.

Roy Lambert wrote:

David

One thought that did occur to me is why not automate the creation of a new logfile every day (week or whatever period seems good)?



Roy Lambert
Fri, Nov 30 2018 8:08 AMPermanent Link

Roy Lambert

NLH Associates

Team Elevate Team Elevate

David

Many, many years ago (think W95) there was a product called (I think) Defend-IT. This didn't use virus signatures but instead monitored activity for what it thought was suspicious behaviour. It did have some interesting heuristics built in and a bit of learning capability.  Its a long time back so memory may be wrong but it sounds familiar. Defend-IT was brilliant, over the time I used it (W95 & W98) I don't remember any false positives and I didn't get virus bitten. Microsoft bought it Frownyou can guess the rest of the story.

I'm bringing it up because it sounds as though SentinelOne is doing something similar but it has what they're please to call ML & AI built in (ie probably some noddy level rules) and its not working quite as it should. What can be done about it I have no idea - sorry. It might be worth opening a support ticket with Tim.


Roy Lambert
Fri, Nov 30 2018 8:29 AMPermanent Link

David

The problem with this issue is that it is intermittent so difficult to track down.  I might consider a support ticket later on if I can't get somewhere with other avenues, but I imagine this one will be difficult to track down, even harder if Tim isn't actually using Sentinel One in the first place.  But the main reason is I don't want to get in the way of a possible Unicode DBISam Smile

Thanks Roy.

Roy Lambert wrote:

David

It might be worth opening a support ticket with Tim.


Roy Lambert
Mon, Dec 10 2018 3:05 PMPermanent Link

Tim Young [Elevate Software]

Elevate Software, Inc.

Avatar

Email timyoung@elevatesoft.com

David,

<< I have a customised server that has a couple of triggers to create an audit log for all new, deleted or edited records.  The system has been working fine for a number of years and not much has changed.

What I am noticing is on occasion if I edit a record it could take a 0.5 to 1 secondto add the record to the audit log, it should be instant.  Normal operations can have many records created at the exact same time so taking 0.5 to 1 second per record is the cause of the slowness.   As the Audit Log can get large after months of running I wondered if this was due to the size of the log so I cleared it out and this worked for some time but recently it has came back again. >>

This might just be down to the normal burps (especially with disk) that occur in the OS that result in non-deterministic behaviors.  You can alleviate a lot of this by using a queue/separate thread for tracking audit changes and writing them to disk.  You can find out more about how to do this by looking at the source code to the mirroring version of the DBISAM Database Server:

https://www.elevatesoft.com/blog?action=view&id=mirroring_dbisam_database_server_code

It is included with the CS versions of DBISAM in the <InstallDir>\servers\dbsrvr\mirror installation directory.

Tim Young
Elevate Software
www.elevatesoft.com
Wed, Dec 12 2018 3:55 PMPermanent Link

David

Thanks Tim I will have a look at this.

Would this slowness persisting after a DB server restart still indicate it is likely something else (Sentinel one) could be interfering with the table access and causing a slow down.

Regards
David.

Tim Young [Elevate Software] wrote:


This might just be down to the normal burps (especially with disk) that occur in the OS that result in non-deterministic behaviors.  You can alleviate a lot of this by using a queue/separate thread for tracking audit changes and writing them to disk.  You can find out more about how to do this by looking at the source code to the mirroring version of the DBISAM Database Server:

https://www.elevatesoft.com/blog?action=view&id=mirroring_dbisam_database_server_code

It is included with the CS versions of DBISAM in the <InstallDir>\servers\dbsrvr\mirror installation directory.

Tim Young
Elevate Software
www.elevatesoft.com
Thu, Dec 13 2018 1:12 PMPermanent Link

Tim Young [Elevate Software]

Elevate Software, Inc.

Avatar

Email timyoung@elevatesoft.com

David,

<< Would this slowness persisting after a DB server restart still indicate it is likely something else (Sentinel one) could be interfering with the table access and causing a slow down. >>

It's always possible.  Any time I suspect something environmental that can't be directly profiled, I use another machine with a different environment to test whether the issue is actually environmental.

So, if you have a spare machine that doesn't have SentinelOne installed on it, that's what I would test with first.

Tim Young
Elevate Software
www.elevatesoft.com
Image