Some time ago, Ralf asked for the best technique for adding many records
in client/server mode.  As I understand things, using DBISAM's ordinary
mechanisms will be very slow because each insert requires a round trip
to the server for the Post/Execute.  That is, time cost will be

 latency * num_records * 2

Eryk suggested writing a server-side procedure to do the inserts.  I
have done this, but I was disappointed to find that my DBISAM's string
remote parameters are not 8-bit clean.  That is to say, a string
containing null characters was truncated.  Luckily it works using blob
parameters.


   Oliver

This code marshals an array of array of Variants into a string buffer.

var
   buf: string;
   buf_pos, buf_len: Integer;
   procedure AppendBytes(const source; size: Integer);
   begin
      if buf_pos + size > buf_len then begin
         buf_len := buf_len * 2;
         SetLength(buf, buf_len);
      end;
      Move(source, buf[buf_pos + 1], size);
      buf_pos := buf_pos + size;
   end;
   procedure Append(v: Variant);
   var
      s: string;
      l: Integer;
   begin
      // We cannot marshal varAny, varDispatch, varUnknown, varArray,
      // varByRef or varOleStr (sorry Ole).
      if varString = VarType(v) then begin
         // String variants have their data following the TVarData.
         // Since the VString member is irrelevant, we use VInteger
         // to store the length.
         s := string(TVarData(v).VString);
         VarClear(v);
         l := Length(s);
         v := l;
         TVarData(v).VType := varString;
         AppendBytes(TVarData(v), SizeOf(TVarData));
         TVarData(v).VType := varInteger;
         AppendBytes(s[1], l);
      end else begin
         // For simple types we can just marshal the TVarData.
         AppendBytes(TVarData(v), SizeOf(TVarData));
      end;
   end;
var
   row: TVariantDynArray;
   i, j: Integer;
begin
   // Encode the rows in a big buffer for transmission over the
   // client/server link.  Each element is a variant.  The first
   // contains the number of rows, and is followed by each row.
   // Each row starts with the number of its elements and
   // continues with their contents.
   buf_pos := 0;
   buf_len := 10240;
   SetLength(buf, buf_len);
   Append(Length(rows));
   for i := 0 to High(rows) do begin
      row := rows[i];
      Append(Length(row));
      for j := 0 to High(row) do
         Append(row[j]);
   end;
   SetLength(buf, buf_pos);

And this code unpacks it:

var
   buf: string;
   buf_pos: Integer;
   function ExtractVariant: Variant;
   var
      s: string;
      pv: PVarData;
      l: Integer;
   begin
      pv := PVarData(Addr(buf[buf_pos + 1]));
      buf_pos := buf_pos + SizeOf(TVarData);
      if pv.VType = varString then begin
         l := pv.VInteger;
         SetLength(s, l);
         Move(buf[buf_pos + 1], s[1], l);
         result := s;
         buf_pos := buf_pos + l;
      end else
         Move(pv^, result, SizeOf(TVarData));
   end;
var
   rows: TVariantRows;
   row: TVariantDynArray;
   row_count, col_count, i, j: Integer;
begin
   buf := session.RemoteParamByName('row_buffer').AsBlob;
   buf_pos := 0;
   row_count := ExtractVariant;
   SetLength(rows, row_count);
   for i := 0 to High(rows) do begin
      col_count := ExtractVariant;
      SetLength(rows[i], col_count);
      row := rows[i];
      for j := 0 to High(row) do
         row[j] := ExtractVariant;
   end;

P.S. There is a subtly memory leak of strings in the unpacking code.
Can you find it?

Oliver,

<< P.S. There is a subtly memory leak of strings in the unpacking code. Can
you find it? >>

Are we going to be graded on this, or do you really need to know where the
leak is because you can't find it ?

--
Tim Young
Elevate Software
www.elevatesoft.com

Tim Young [Elevate Software] wrote:
> << P.S. There is a subtly memory leak of strings in the unpacking code. Can
> you find it? >>
>
> Are we going to be graded on this, or do you really need to know where the
> leak is because you can't find it ?

You have discovered my cunning plan.  I admit that I just don't know
where the leak is, but I trust FastMM's report.


Regards,
  Oliver