Icon View Thread

The following is the text of the current message along with any replies.
Messages 1 to 10 of 15 total
Thread OT - Pointers, comparing memory and TxQuery
Thu, Jun 13 2013 9:54 PMPermanent Link

Adam H.

Hi,

Sorry for the off topic post - but I know there are a few guru's around
here that may be able to assist me with a problem I'm having with DBISam
and TxQuery.

I'll post another message with more detail - but the short of it is
this. I have traced the problem down to comparing memory pointers.

The code is:

Result := Comparemem( ( Buffer1 + SizeOf( Integer ) ), ( Buffer2 +
SizeOf( Integer ) ), FRecordBufferSize - SizeOf( Integer ) )

The problem is that I have no idea how to view the actual contents of
the memory during debugging. (Since it's a pointer and not a variable).

I'd like to be able to determine what the data is that it's seeing - and
why it is different.

Can anyone advise if it's possible to view the data in memory given the
information above?

Thanks & Regards

Adam.
Thu, Jun 13 2013 10:28 PMPermanent Link

Adam H.

Firstly, summarising - the problem only shows its head when:

- A certain range of records is in the source data.

- It only occurs when using DBISAM. (v4.35 Build 2 at present) but not
when using BDE

- It will perform correctly if the result set has the same data in it -
but also a few additional records.

-------------

As promised, here's a more detailed explanation of my problem (for
anyone who's interested and/or wants more information).

I've been using txQuery for some time now, but recently have ran into a
bit of a problem where it is not pivoting / grouping correctly.

As an example of what's happening, if I have the following data:

TICKET   PRODUCT      QA   Value
114661   CBALES      
114662   SORGHUM      ADM   0.20
114662   SORGHUM      MOI   14.9
114662   SORGHUM      SCR   1.10
114663   SORGHUM      ADM   0.30
114663   SORGHUM      MOI   14.5
114663   SORGHUM      SCR   1.00

and then run my TxQuery - I SHOULD get the following result:


TICKET   PRODUCT      ADM   MOI   SCR
114661   CBALES      0.00   0.00   0.00   
114662   SORGHUM      0.20   14.9   1.10
114663   SORGHUM      0.30   14.5   1.00


However, when I run my query - I am getting the following:
   
TICKET   PRODUCT      ADM   MOI   SCR
114661   CBALES      0.00   0.00   0.00   
114662   SORGHUM      0.20   0.00   0.00
114662   SORGHUM      0.00   14.9   0.00
114662   SORGHUM      0.00   0.00   1.10
114663   SORGHUM      0.30   0.00   0.00
114663   SORGHUM      0.00   14.5   0.00
114663   SORGHUM      0.00   0.00   1.00

But where it gets very strange is that if I change the result set and
have one (or more) additional records in it (so same data, but include
data for ticket 114664 - it performs exactly as expected for tickets
114662 and 114663 and condenses them to the one record).

The info above is an extract from what the query really sorts through.

Adding to the strange behaviour - I have a demonstration project of the
error using the BDE instead of DBISAM (so I can submit a bug), using
exactly the same data, and it performs correctly as expected. I have the
demo project running both a TQuery, and a TDBISAMQuery. The TQuery works
correctly, whereas when using the TDBISAMQuery component it misbehaves
and shows the above problems.

If anyone has TxQuery (freely available from the web now I believe) and
would like me to send through a copy of my demo project with the
offending data, please drop me an email and I'll send you a copy.

At this stage though, I'd just be happy to view the data that it's
comparing that's sitting in the memory to see if I can get a glimpse of
why it's misbehaving.

Cheers

Adam.
Fri, Jun 14 2013 4:17 AMPermanent Link

Roy Lambert

NLH Associates

Team Elevate Team Elevate

Adam


You don't mention which version of Delphi you're using and that might have some impact.

As far as displaying memory is concerned you should be able to do that by using the ^ or @ operators (again I'm incompetent with them). A start point to look at is http://rvelthuis.de/articles/articles-pointers.html

Another little thought. The test

Result := Comparemem( ( Buffer1 + SizeOf( Integer ) ), ( Buffer2 +SizeOf( Integer ) ), FRecordBufferSize - SizeOf( Integer ) )

From my limited understanding of memory operations and all things pointers it looks as though you're trying to compare a full record less the record number. Buffer1 & Buffer2 are the records? If I'm right you should be able to walk along the "fields" until you find which one(s) fail the test eg if the first "field" after the record number was an integer


if Comparemem( ( Buffer1 +SizeOf( Integer )+ SizeOf( Integer ) ), ( Buffer2 +SizeOf( Integer ) +SizeOf( Integer ) ), FRecordBufferSize -( SizeOf( Integer +SizeOf( Integer ) )) )
then ShowMessage('ok')
else ShowMessage('found it');

The akward one might be string fields since the size of string in the post D2009 days insn't a simple column size but column size * SizeOf(Char) or somesuch.

Not a lot of help but it may get you going in some direction right or wrong.

Roy Lambert
Fri, Jun 14 2013 2:23 PMPermanent Link

Tim Young [Elevate Software]

Elevate Software, Inc.

Avatar

Email timyoung@elevatesoft.com

Adam,

<< If anyone has TxQuery (freely available from the web now I believe) and
would like me to send through a copy of my demo project with the offending
data, please drop me an email and I'll send you a copy. >>

Fire one off to me - if I get a chance, I'll take a look.  I suspect that if
TxQuery is doing any straight-up memory comparisons, that it may run into
issues since each database can use vastly different buffer arrangement
schemes.

Tim Young
Elevate Software
www.elevatesoft.com
Sun, Jun 16 2013 10:38 PMPermanent Link

Adam H.

> Fire one off to me - if I get a chance, I'll take a look.  I suspect
> that if TxQuery is doing any straight-up memory comparisons, that it may
> run into issues since each database can use vastly different buffer
> arrangement schemes.

Thanks for that. I have sent through an email with TXquery (just incase
you don't have it), along with the example project that contains data
for both DBISAM and BDE and a demonstration of the conversion with both.

Best Regards

Adam.
Sun, Jun 16 2013 10:56 PMPermanent Link

Adam H.

Hi Roy,


> You don't mention which version of Delphi you're using and that might have some impact.
>
> As far as displaying memory is concerned you should be able to do that by using the ^ or @ operators (again I'm incompetent with them). A start point to look at is http://rvelthuis.de/articles/articles-pointers.html

Thanks for your reply..

Sorry - I'm using Delphi 2007.

I've tried using the ^ operator, and only get the result of #2 each
time. (Even when comparemem returns false).

I believe you are right about comparing less the record number, but I'm
not sure how to walk along the field numbers. (I'm not sure what part of
that actually even refers to a field number).

I fear I'm a little out of my depth when it comes to pointers and
buffers. I was hoping that ^buffer1 would return some sort of result
where I could physically see the actual content, but I think my
expectations are incorrect.

Cheers

Adam.
Mon, Jun 17 2013 3:52 AMPermanent Link

Roy Lambert

NLH Associates

Team Elevate Team Elevate

Adam


>Sorry - I'm using Delphi 2007.

That takes one piece out of the puzzle.

>I've tried using the ^ operator, and only get the result of #2 each
>time. (Even when comparemem returns false).
>
>I believe you are right about comparing less the record number, but I'm
>not sure how to walk along the field numbers. (I'm not sure what part of
>that actually even refers to a field number).

I think Tim could be right here (surprise, surprise) and that the two systems could be arranging things and storing then differently. I know you've sent the info to Tim but out of interest can you send it to me as well. I'd just like to have a look.

>I fear I'm a little out of my depth when it comes to pointers and
>buffers.

I'm not a little out of my depth - I'm drowning, waving my arms and legs and taking in about a gallon of water a minute!

>I was hoping that ^buffer1 would return some sort of result
>where I could physically see the actual content, but I think my
>expectations are incorrect.

All right thinking languages refuse to allow programmers direct access to memory. A good thing in my mind.

Roy Lambert [Team Elevate]
Mon, Jun 17 2013 7:12 PMPermanent Link

Adam H.

Hi Roy,

>> I believe you are right about comparing less the record number, but I'm
>> not sure how to walk along the field numbers. (I'm not sure what part of
>> that actually even refers to a field number).
>
> I think Tim could be right here (surprise, surprise) and that the two systems could be arranging things and storing then differently. I know you've sent the info to Tim but out of interest can you send it to me as well. I'd just like to have a look.

Too easy. I've sent a copy to your email address.

Tim may be right. I don't even know what a buffer is in regards to
databases, so that's a bit of a mystery to me. From what I can tell in
the code, it loads data from two records into a component of it's own,
and then does a comparison on that data.

My thought would be if it loads the data into it's own component first -
then maybe that should eliminate any differences between databases, so
maybe the problem is elsewhere in the code. (Maybe there's a problem
when it copies the data).

The hurdle I face is that when I look at this code, it took me the good
part of a day just to find the comparemem function due to my struggle
with tracing pointers and a whole lot of other coding in there I have
nothing about.


>> I fear I'm a little out of my depth when it comes to pointers and
>> buffers.
>
> I'm not a little out of my depth - I'm drowning, waving my arms and legs and taking in about a gallon of water a minute!

Nice. I'm not even sure why people use pointers. To me, it seems easier
just to pass the variable through the function. Maybe it is faster to
use pointers, I'm not sure.

>> I was hoping that ^buffer1 would return some sort of result
>> where I could physically see the actual content, but I think my
>> expectations are incorrect.
>
> All right thinking languages refuse to allow programmers direct access to memory. A good thing in my mind.

Probably - except when you want to see what the results are. Wink

Cheers

Adam.
Tue, Jun 18 2013 4:23 AMPermanent Link

Roy Lambert

NLH Associates

Team Elevate Team Elevate

Adam

>Too easy. I've sent a copy to your email address.

Its arrived  - I'll have a look later on.

>Tim may be right. I don't even know what a buffer is in regards to
>databases, so that's a bit of a mystery to me. From what I can tell in
>the code, it loads data from two records into a component of it's own,
>and then does a comparison on that data.

There's lots of different buffers involved with databases but this one is the record buffer (I think). Its generated when you read a record from disk or wherever and plonk it into memory so you can work on it. Its what's looked at when you do something like tablefield.asstring.


>Nice. I'm not even sure why people use pointers. To me, it seems easier
>just to pass the variable through the function. Maybe it is faster to
>use pointers, I'm not sure.

Speed was (probably still is) the main issue but also you can do things when accessing memory directly that you can't do otherwise, or not very easily eg if you know where screen memory is then altering it directly, especially in the early days of PCs, was a lot faster than calling an OS routine to do it for you.

>>> I was hoping that ^buffer1 would return some sort of result
>>> where I could physically see the actual content, but I think my
>>> expectations are incorrect.
>>
>> All right thinking languages refuse to allow programmers direct access to memory. A good thing in my mind.
>
>Probably - except when you want to see what the results are. Wink

Yup, but you should never have to do that. I may be wrong but in the languages I used before Delphi (ie mainly pre PC days) I had no way to access memory directly. That was the domain of the systems programmers. You could recognise them by the open toed sandals, beards, scraggy hair do and bemused expressions.

Roy
Tue, Jun 18 2013 7:34 AMPermanent Link

Roy Lambert

NLH Associates

Team Elevate Team Elevate

Adam


I've started to have a look and I don't think its the comparemem routine.

IsEqual (the function that calls comparemem)  is called from DoGroupBy, DoTransform, and CreateResultSet.

I added some debug code to IsEqual

if result then sl.Add(IntToSTr(Recno1)+'x'+IntToStr(recno2)+' .T.') else sl.add(IntToStr(Recno1)+'x'+IntToStr(recno2)+'.F.');

and altered the parameters to pass in a string indicating where it came from and it seems to indicate that the problem comes from DoTransform.

Only DoTransform and DoGroupBy get called so its got to be one of those.

Looking through the code I'd guess its somewhere in between the two calls to IsEqual but it could also be in Transf_CreateSelectValues. Without spending a lot of time tracking through it its hard for me to say.

If you look at the table below C and D are the first & second call to IsEqual. In all cases its comparing a record 1 and 2 but what seems to have happened is that the number 2 record isn't being changed properly. My advice is to forget the comparemem thing and concentrate on DoTransform. Could be totally wrong though.


BDE        DBISAM
   C   1x2.F.    C   1x2.F.
   D   1x2.F.    D   1x2.F.
   C   1x2.F.    C   1x2.F.
   D   1x2 .T.    D   1x2.F.
   C   1x2.F.    C   1x2.F.
   D   1x2 .T.    D   1x2.F.
   C   1x2.F.    C   1x2.F.
   D   1x2 .T.    D   1x2.F.
   C   1x2.F.    C   1x2.F.
   D   1x2 .T.    D   1x2.F.
   C   1x2.F.    C   1x2.F.
   D   1x2 .T.    D   1x2.F.
   C   1x2.F.    C   1x2.F.
   D   1x2 .T.    D   1x2.F.
   C   1x2.F.    C   1x2.F.
   D   1x2 .T.    D   1x2.F.
   C   1x2.F.    C   1x2.F.
   D   1x2.F.    D   1x2.F.
   C   1x2.F.    C   1x2.F.
   D   1x2 .T.    D   1x2.F.
   C   1x2.F.    C   1x2.F.
   D   1x2 .T.    D   1x2.F.
   C   1x2.F.    C   1x2.F.
   D   1x2 .T.    D   1x2.F.
   C   1x2.F.    C   1x2.F.
   D   1x2 .T.    D   1x2.F.
   C   1x2.F.    C   1x2.F.
   D   1x2 .T.    D   1x2.F.
   C   1x2.F.    C   1x2.F.
   D   1x2 .T.    D   1x2.F.
   C   1x2.F.    C   1x2.F.
   D   1x2 .T.    D   1x2.F.


Roy Lambert
Page 1 of 2Next Page »
Jump to Page:  1 2
Image