Icon View Thread

The following is the text of the current message along with any replies.
Messages 1 to 10 of 41 total
Thread 1000 Databases With 64 Datasets Each Crashes The EWB Server
Mon, Aug 6 2018 4:45 AMPermanent Link

Frederick Chin

I have pre-created 1,000 databases with 64 datasets each in EWB server's INI file but the EWB server is taking a long time to load and consumes 1.4GB of memory.

If I re-import the databases from EWB, I can save the imported database and datasets but when I tried to restart the server, it hangs.

The reason why I am creating this many databases is because I want to have the databases ready so that when a new user tries my program, they are allotted one of the databases already created.

At this stage, I am not even done with the number of datasets for one database. I expect to see it increase 3 times.

Is there a better way of allowing individual databases for 1,000 unique users?

--
Frederick
Mon, Aug 6 2018 6:35 AMPermanent Link

Uli Becker

Frederick,

> I have pre-created 1,000 databases with 64 datasets each in EWB server's INI file but the EWB server is taking a long time to load and consumes 1.4GB of memory.

I don't think that's a good idea. Smile

Do these databases/datasets have the same structure? If so, I would use just one database and filter the datasets by using an ID which is different for each user.

Uli
Mon, Aug 6 2018 11:09 AMPermanent Link

Frederick Chin

Uli,

/*
I don't think that's a good idea. Smile
*/

I guess not. Smile EWB seems to be loading everything in the INI file at startup. I think there will still be a problem even if I reduce the database count from 1,000 to a couple of hundred.

/*
Do these databases/datasets have the same structure? If so, I would use just one database and filter the datasets by using an ID which is different for each user.
*/

All the databases have the same dataset structures. It is only the database names and directory which separates them.

I can't use filtering because this is an accounting system and each user is actually a company with multiple users. It is unlikely that the companies would want to have their data in a single database. There is also the data security and confidentiality issues.

If EWB allowed me to change the directory name of the database at runtime, I would not have this problem.

--
Frederick
Mon, Aug 6 2018 12:52 PMPermanent Link

Uli Becker

Frederick,

> I can't use filtering because this is an accounting system and each user is actually a company with multiple users. It is unlikely that the companies would want to have their data in a single database. There is also the data security and confidentiality issues.
>
> If EWB allowed me to change the directory name of the database at runtime, I would not have this problem.

Maybe a module would be a good solution for you. The module could either
alter the database path or access different databases. That can be
managed by passing a simple param.

Uli
Mon, Aug 6 2018 2:16 PMPermanent Link

Tim Young [Elevate Software]

Elevate Software, Inc.

Avatar

Email timyoung@elevatesoft.com

Frederick,

<< If EWB allowed me to change the directory name of the database at runtime, I would not have this problem. >>

From the *client*, or on the *server* ?  I imagine that you can figure out why the former is a bad idea.  EWB 2 can't do anything like this server-side without using an EWB Web Server module, and the reason is obvious: there isn't really any "server-side" code execution without using such a module.

EWB 3, with its server-side code execution, will offer more flexibility in this regard.

Tim Young
Elevate Software
www.elevatesoft.com
Mon, Aug 6 2018 7:38 PMPermanent Link

Frederick Chin

Uli,

/*
Maybe a module would be a good solution for you. The module could either
alter the database path or access different databases. That can be
managed by passing a simple param.
*/

Do you have an example where I could change the database path or access a different database based on the param? I know how to do the param passing part but the other part seems to require getting the data from a different database and passing the result back to the EWB client.

--
Frederick
Mon, Aug 6 2018 7:46 PMPermanent Link

Frederick Chin

Tim,

/*
From the *client*, or on the *server* ?  I imagine that you can figure out why the former is a bad idea.  EWB 2 can't do anything like this server-side without using an EWB Web Server module, and the reason is obvious: there isn't really any "server-side" code execution without using such a module.
*/

It is from the client side. If you are saying that the web module needs to be written so that the data from a different database is passed back to the EWB client, it is going to be a huge job with the 64 datasets I have currently

If it is simpler, examples would be appreciated.

/*
EWB 3, with its server-side code execution, will offer more flexibility in this regard.
*/

Don't tease us already. Release it next week! Smile

--
Frederick
Wed, Aug 8 2018 3:28 AMPermanent Link

Matthew Jones

Frederick Chin wrote:

> Is there a better way of allowing individual databases for 1,000 unique users?

I pondered this, and the answer is simple - yes, you should do your own server. That said, it may be that the new server will be able to do what you want out of the box. The key is that the WebBuilder product is designed to allow you to write applications for your browser, and includes a server that is ideal for someone wanting to make an in-house application "out of the box". It isn't designed to do what you are wanting, and for that, you'd be a lot better off writing your own server that uses REST or such calls to do all the work, and to determine which database to load when a user logs in, etc. You are wanting to do more than the product is designed for, and need to either customise it a lot, or do your own. I can't see how you could do a database selection safely for a user if the server doesn't allow it as standard.

It may be that the new server can do what you want - with secure database selection per user login. If so, you should wait for it before doing that part. If not, then DIY is the only sensible way.

Of course if your application doesn't matter a lot if people see other's data, then not really an issue, but if it is important that they never see the wrong data, you should be sure to use a solution that can make that guarantee. Tim's comments imply that version 3 will improve that a lot.

--

Matthew Jones
Wed, Aug 8 2018 5:07 AMPermanent Link

Uli Becker

Frederick,

> Do you have an example where I could change the database path or access a different database based on the param? I know how to do the param passing part but the other part seems to require getting the data from a different database and passing the result back to the EWB client.

Scratch the part ("Change the database path"). That won't work in a
multi user environment.

If you have several databases defined in EDB, this would be a possible
approach:

1. Name all databases "DBCustomer" + CustomerID
2. Create a login in EWB and after getting back the CustomerID, store
this ID in memory and use it for all requests (using a module off course)
3. In the module uses something like this:

procedure TDataSetModule.EWBModuleExecute(Request: TEWBServerRequest);
var
   CurrentCustomerID: string;
   CurrentDatabaseName: string;
begin
   CurrentCustomerID := RequestParams.Values['CustomerID'];
   CurrentDatabaseName := 'DBCustomer' + CurrentCustomerID;
   MyEDBDatabase.Database := CurrentDatabaseName;
   ...
end;

But generally I doubt that it's the right way to create 1.000 databases.
Why not use just one database and uses a CustomerID to query the
database in a secure way? What are the security issues you mentioned?

Uli
Wed, Aug 8 2018 11:23 AMPermanent Link

Frederick Chin

"Matthew Jones" wrote:

/*
Of course if your application doesn't matter a lot if people see other's data, then not really an issue, but if it is important that they never see the wrong data, you should be sure to use a solution that can make that guarantee. Tim's comments imply that version 3 will improve that a lot.
*/

Since it is an accounting system, it is imperative that data integrity be present. Lumping data from multiple clients in the same table and filtering does not cut it. In addition, the volume of data can be huge and speed will be an issue later.

I've reduced the database count to 100 but the EWB server is still taking up 156MB of memory.

Actually, the EWB server, as-is now, works well. The major flaw I think is that it cannot handle a large database/dataset count. All the definitions are in the INI file and the entire file is loaded into memory.

I would have liked the database/dataset definitions be stored in a table and the EWB server pick up only what is required. This will scale well and I could have 1 million databases and as many datasets per database as I want without causing the server to grind to halt because of memory consumption. I also do not need to write a server just for this.

As for version 3, with no release timeframe to look forward to, its expected benefits will be difficult to access.

--
Frederick
Page 1 of 5Next Page »
Jump to Page:  1 2 3 4 5
Image