I have now recompiled the EDB Manager to debug what's realy going on. After a fresh start of EDBM the save update works right. At some point Save Update begins to fail and does nothing and messed up the EDBPbl file in some way. i have no steps yet as what exactly ust be done to make it broken.

If it doesn't work anymore the following line in edblocal.pas at line 33305 returns always 0 for TempTableManager.PublishedUpdatesSize:

    Inc(TempTotalSize,TempTableManager.PublishedUpdatesSize);

For some reasons this happens only in EDB Manager and stays in this state until I close and reopen the database. If  I debug my own small Save Update application, this lines returns allways the correct new UpdateSize. I do open the database in my Save updates program in the OnCreate Event of my form. So it should be open all the time and there seems not to be a need for reopen it there.

Any ideas or tips where I should look as next in the source?

Tim,

I have just found out how to reproduce this isse. If a do a SELECT * FROM VideoList (VideoList is the published table) after the first correct Save Update, any following Saves will fail and gives me this issues. TempTableManager.PublishedUpdatesSize gives me now all the time 0 back until I reopen the database in EDBM.

Tim,

I can reproduce it also in my own application if I call a SELECT on this published table 2 times. It looks as it must be called two times as normaly after the first open, it is still returning the correct update count. After this Select is called the 2nd time, TempTableManager.PublishedUpdatesSize returns allways 0.

I have a button with this Event on it. After the first call everything is fine. After the second call it is also fine, as the Select was only run one time yet. On the 3rd call, where the select was running two times now, the TempTableManager.PublishedUpdatesSize returns 0, which is wrong.

procedure TForm1.Button1Click(Sender: TObject);
begin
 qryTemp.SQL.Text := 'SAVE UPDATES FOR DATABASE "xupnpVideoList" ' +
   'AS "xupnpVideoList Updates ' + FormatDateTime('yyyymmdd hhnnss', Now) + '" ' +
   'TO STORE "LocalStore" ' +
   'TABLES "VideoList" IF NOT EMPTY';
 qryTemp.ExecSQL;

 qrySelect.SQL.Text := 'SELECT * FROM VideoList';
 qrySelect.Open;
end;

I hope very much you are able to reproduce this issue now with this infos.

On 4/3/2016 8:30 AM, Rolf Frei wrote:
> I have a button with this Event on it. After the first call everything is fine. After the second call it is also fine, as the Select was only run one time yet. On the 3rd call, where the select was running two times now, the TempTableManager.PublishedUpdatesSize returns 0, which is wrong.
>
> procedure TForm1.Button1Click(Sender: TObject);
> begin
>    qryTemp.SQL.Text := 'SAVE UPDATES FOR DATABASE "xupnpVideoList" ' +
>      'AS "xupnpVideoList Updates ' + FormatDateTime('yyyymmdd hhnnss', Now) + '" ' +
>      'TO STORE "LocalStore" ' +
>      'TABLES "VideoList" IF NOT EMPTY';
>    qryTemp.ExecSQL;
>
>    qrySelect.SQL.Text := 'SELECT * FROM VideoList';
>    qrySelect.Open;
> end;
>
> I hope very much you are able to reproduce this issue now with this infos.

Rolf,

I'm trying to understand how to reproduce this as i'm curious now but
this is still bit unclear !?

Why exactly is the PublishedUpdatesSize = 0 a problem ?

You're saving updates every time button is clicked (with qryTemp) and
assuming there are updates they get saved so pending updates would be
none and and hence updatesize would become 0 once there are no updates.

Can you expand on the problem little more ?

Raul

Raul,

There is another applicaiton running which adds and updates much records at the same time. The EDBPbl contains already new entries if I do a new Save Update, but it is not seen by the applicaiton who calls Save Update and it returns 0 as pending rupdates, but there are already new records as the EDBPbl files is just growing.

Rolf,

<< 2. Yes both EDBM and my application are using the same config path. You mean the TEDBSession.LocalConfigPath property right? >>

If you're using the TEDBSession.LocalConfigPath, then you *must* make sure that the the TEDBEngine.UseLocalSessionEngineSettings property is set to True, otherwise you'll run into the problem that I described in my last post about locks not being respected between sessions.

<< My further investigations brought me to the actual conclusion, that it seems to be a problem of the EDB Manager and not the engine itself, >>

That's not possible - the EDB Manager uses the EDB engine, and can't "subvert" it.

<< I have no idea yet what is different in EDBM which leads to this issue. I have 2 sessions in EDBM, one is the local one and the other is the C/S one to the server database. In both sessions the database has the same name. Can this be an issue? >>

The only thing that can cause this issue is what I previously described.

<< I believe the Manager gets sometimes some irritations of the two sessions and I have the fealing, that it doesnt correctly change all the configuration on switching between this sessions. >>

That is not correct - there is no such issue in the EDB Manager.  If you're new to the EDB Manager, then you may not be keeping track of your active session/database properly - you can always see it in the bottom right corner of the EDB Manager in the status bar.

Tim Young
Elevate Software
www.elevatesoft.com

Rolf,

<< I hope very much you are able to reproduce this issue now with this infos. >>

No, as I already indicated, I'm going to need an application from you that reproduces this completely.  I've already tried to reproduce it here, and it works fine.

Tim Young
Elevate Software
www.elevatesoft.com

Rolf,

Scratch that last response - I think I see what you're saying with this last bit of information.  Let me check it out and I'll see what I can find out.

Tim Young
Elevate Software
www.elevatesoft.com

On 4/4/2016 7:30 AM, Rolf Frei wrote:
> There is another applicaiton running which adds and updates much records at the same time. The EDBPbl contains already new entries if I do a new Save Update, but it is not seen by the applicaiton who calls Save Update and it returns 0 as pending rupdates, but there are already new records as the EDBPbl files is just growing.

Got it - i had tried this before within single app (one query inserting
1 record on timer (10 times a sec) and other queries  configured like
your button event to save updates and select) and did not run into the
issue.

However when running 2 instances of the same app (one doing timer and
other button event) I did run into something similar  - i was missing a
set of updates (in my case insert adds incrementing integer and same in
string foramt so updates files should include 1..x records in order). In
one case wfter loading updates into temporary create table one after
another i was missing one range.

Can't instantly reproduce it but definitely looks similar to your issue.

I hope Tim can track it down as well with his tests and i if i have time
tonight will try again also.

Raul

Tim,

I have sent you a test application and database to test with. For anyone else (Raul) I have attached that project here as well.

This is the instructional email I sent Tim:

---------------------------------------------------------
I have now done a test application, you can use to reproduce the bug. I have attached it including the database as a ZIP (rename the attached zzz-file to zip). The included databases (client and server) just shows the actual result of a test here. If you do a compare of the record count you will notice that there are now records missing on the server table between ID 551 and 760. They are missing in the Update 3 file. This is the file which was done after the Save Updates failed and didn’t create a file. After a reopen of the database I was able to genereade that Update 3 file, but it lost inserts and updates of records as you can see. (For your own testing, empty this two tables first)

The ZIP contains this structure:
<root>   
Delphi project and EDBConfig for the client database

<root>\Database   
Databasefolder of the client Database “BugTest”.

<root>\Store
Storefolder for the update files.

<root>\Server
Everything for the server database including the EDBConfig and the Database itself.


After unpacking the ZIP, setup two Sessions in EDB Manager. One for for <root> (BugTest Client) and on for the server <root>\Server (BugTest Server). After that, alter the database of the client session to point to <root>\Database and be sure the Database Name is “BugTest”. Also alter the Server Database to point it to <root>\Server.

Also create two Stores:
1. In the client Session and name it UpdateStore or what ever you want.
2. In the server Session and name it UpdateStore or what ever you want. I think you can point that to the same folder as the client session, so you don’t need to copy teh Update Files over every time.

If this is done, you should be able to open the VideoList Table and compare the two tables of the two databases.


I have also included two SQL files (in case you need them), with the create table of the Client (ClientTable.SQL), which has an AutoInc generator on the ID Field, and the Server (ServerTable.SQL), which should NOT have the ID as an autoinc field. There are never inserts or updates on the server in my case. There are only selects running on it. This is important, as else the ID’s get messed up on updating a incorrect Update File with missing records. This would make it harder to see what is different. And don’t forget to publish this table in the Client Database. The server database should not be published.


*** Now to the steps how to reperoduce it ***
Tip: Open the <root>\Store folder in Windows Explorer and let it open to see if a file is genereated or not. Also a folder of the Database-Directory <root>\Database is helpfull so you can see how the EDBPbl file is growing.

If you still have my test tables with data in it, do a Empty for both tables first to start clean.

1. Start the EDBBugGen.exe from my ZIP or build it yourself with my sources. It needs the Indy components to generate an SHA1 Hash of the VideoURL, which is the PK of the table.

2. Press the Start Button and let it run.

3. Go into EDB Manager and select the client database. Right click on it and Select “Save Database Updates”. Name the file as “Update 1” and save it in the Store “UpdateStore”.

Everything should be fine here and an Update file should be produced..

4. Now open an SQL window and run this SQL: “SELECT * FROM VideoList WHERE ID < 1000”.

5. Right click on the database and select again “Save Database Updates”. Name the file as “Update 2”.

Now it may be that it also just fine saved the Update file, in this case just run the SQL from step 4 again and call the update “Update 3”. Now you should not see an Update 3 in the store folder,as it now isn’t created. From now on you have no chance to write the Update File and “Save Updates” doesn’t do anything.

6. (optional but helps for testing) On the running test application press “Cancel” to stop it running. So we will have all records after the next Save Update in Update files for Importing on the server database.

7. Close the Database in EDB Manager and reopen it. Now right click on the Database and select “Save Database Updates”. Name the file as “Update 3”, which shouldn’t be existing yet. If it is, just name the new one “Update 4” so we do not overwrite anything.

This time the save does work again and you should be get an “Update 3” file now.

As you now should have 3 Updatefiles with all updates and inserts, it’s time to load them in the server Database:
1. If the Store of the Server doesn’t point to the same directory as the client, copy the files to the Server Store.

2. Select the server Session and right click on the server Database and select “Load Database Updates”. Now import all of the genereated Updates Files in the correct order.

3. If they are loaded, open the VideoList table on both sessions and look at the record count. If everything were run fine, both should now be identical.

You should notice that there are records missing on the server, which are lost completly. Also updates are missing if you make an export on both tables an to a compare of the two export files. The inserts and Updates which are missing are the ones since the last good Update and the point where no updatefiles was generated.

-----------------------------------------------