Icon View Incident Report

Serious Serious
Reported By: Krisztian Kepes
Reported On: 5/17/2010
For: Version 2.03 Build 13
# 3217 ElevateDB Server Can Deadlock with Frequent Updates to the Configuration or Database Catalogs

We migrated our biggest customer in sunday. Now they started the work.

Two times we got problem. The problem that server not responding.

The memory usage is: 1,2 GB.
The session count 40-50.

The server is not responding, we cannot start/restart it, the users are logged out because of timeout.

Schedule a job to run on the ElevateDB Server every minute, and then run the following code from a client:

unit stressconnectmain;

interface

uses
  Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
  Dialogs, StdCtrls, edbcomps, contnrs, edbutilcomps;

type

  TConnectThread = class(TThread)
   private
      FConnectSession: TEDBSession;
      FConnectDatabase: TEDBDatabase;
      FConnectTable: TEDBTable;
      FErrorMessage: string;
      procedure Initialize;
      procedure Reset;
      procedure IncNumber;
      procedure DecNumber;
      procedure LogError;
   protected
      procedure Execute; override;
   end;

  TForm1 = class(TForm)
    Button1: TButton;
    NumConnectionsEdit: TEdit;
    Label1: TLabel;
    Label2: TLabel;
    Memo1: TMemo;
    procedure FormCreate(Sender: TObject);
    procedure FormDestroy(Sender: TObject);
    procedure Button1Click(Sender: TObject);
  private
    ThreadList: TObjectList;
    TotalThreads: Integer;
  public
    procedure IncNumber;
    procedure DecNumber;
    procedure LogError(const ErrorMessage: string);
  end;

var
  Form1: TForm1;

implementation

procedure TConnectThread.Initialize;
begin
   Synchronize(IncNumber);
   FConnectSession:=TEDBSession.Create(nil);
   with FConnectSession do
      begin
      SessionType:=stRemote;
      SessionName:='StressTest_'+IntToStr(ThreadID);
      LoginUser:='Administrator';
      LoginPassword:='EDBDefault';
      end;
   FConnectDatabase:=TEDBDatabase.Create(nil);
   with FConnectDatabase do
      begin
      SessionName:=FConnectSession.SessionName;
      DatabaseName:='StressTestDB';
      Database:='Test';
      end;
   FConnectTable:=TEDBTable.Create(nil);
   with FConnectTable do
      begin
      SessionName:=FConnectSession.SessionName;
      DatabaseName:='StressTestDB';
      TableName:='customer';
      end;
end;

procedure TConnectThread.Reset;
begin
   FreeAndNil(FConnectTable);
   FreeAndNil(FConnectDatabase);
   FreeAndNil(FConnectSession);
   Synchronize(DecNumber);
end;

procedure TConnectThread.IncNumber;
begin
   Form1.IncNumber;
end;

procedure TConnectThread.DecNumber;
begin
   Form1.DecNumber;
end;

procedure TConnectThread.LogError;
begin
   Form1.LogError(FErrorMessage);
end;

procedure TConnectThread.Execute;
begin
   Initialize;
   try
      Sleep(Random(100));
      try
         with FConnectTable do
            begin
            Open;
            First;
            while (not EOF) do
               Next;
            end;
      except
         on E: Exception do
            begin
            FErrorMessage:=E.Message;
            Synchronize(LogError);
            end;
      end;
      Sleep(Random(500));
   finally
      Reset;
   end;
end;

{$R *.dfm}

procedure TForm1.FormCreate(Sender: TObject);
begin
   ThreadList:=TObjectList.Create;
end;

procedure TForm1.FormDestroy(Sender: TObject);
var
   I: Integer;
begin
   for I:=ThreadList.Count-1 downto 0 do
      begin
      with TConnectThread(ThreadList[I]) do
         begin
         FreeOnTerminate:=False;
         Terminate;
         WaitFor;
         end;
      ThreadList.Delete(I);
      end;
   FreeAndNil(ThreadList);
end;

procedure TForm1.IncNumber;
begin
   Inc(TotalThreads);
   Label2.Caption:=IntToStr(TotalThreads);
   Label2.Update;
end;

procedure TForm1.DecNumber;
begin
   Dec(TotalThreads);
   Label2.Caption:=IntToStr(TotalThreads);
   Label2.Update;
   if (TotalThreads=0) then
      Button1Click(nil);
end;

procedure TForm1.LogError(const ErrorMessage: string);
begin
   Memo1.Lines.Add(ErrorMessage);
end;

procedure TForm1.Button1Click(Sender: TObject);
var
   I: Integer;
   TempCount: Integer;
   TempThread: TConnectThread;
begin
   TempCount:=StrToInt(NumConnectionsEdit.Text);
   I:=0;
   while (I < TempCount) do
      begin
      TempThread:=TConnectThread.Create(True);
      ThreadList.Add(TempThread);
      with TempThread do
         begin
         Priority:=tpLower;
         Resume;
         end;
      Inc(I);
      end;
end;

end.



Comments Comments
The problem was caused by a deadlock condition between two completely separate critical sections in the engine code. It was most likely to occur under heavy connection/disconnection activity combined with updates to the configuration or database catalog(s).


Resolution Resolution
Fixed Problem on 5/22/2010 in version 2.03 build 14


Products Affected Products Affected
ElevateDB Additional Software and Utilities
ElevateDB DAC Client-Server
ElevateDB DAC Client-Server with Source
ElevateDB DAC Standard
ElevateDB DAC Standard with Source
ElevateDB DAC Trial
ElevateDB LCL Standard with Source
ElevateDB VCL Client-Server
ElevateDB VCL Client-Server with Source
ElevateDB VCL Standard
ElevateDB VCL Standard with Source
ElevateDB VCL Trial

Image