Urgent Help needed with binary reading....

[MopàrBCN]

Hi,

I am currently working on a program which allows me to compare two cal's and list me exactly those tables, bytes, words which either are different or don't exist in one or the other.

I will post up the tool once done. The Problem, I am a good .net/t-sql and even pl-sql programmer but I have never even touched binary file reading.

The Way I understand Chem2 is that it uses the xml presentation of the cal as this, a mere description of the tables etc. However, the values for the grafical presentation of the table (etc) data it obviously reads/writes directly from the .bin.

The Problem I am facing right now, how to actually read the binary. I have it open in a binary reader but now I am facing the problem to where to search.

I know there is an offset to each value. So I suppose this is the address inside my binary file where to find my values. Then I have a base offset of hex8000. If I translate this to decimal I get at 32768 which is the file length if I get this right. Now, the table offsets translate all to numbers higher then this offset of 8000. So I guess I have to calculate dataoffset - cal offset => position of first value of table to search??? Here I am getting confused.

1. Even if this would be the starting point how would I know when the table ends?

2. I would expect to find at this address the first byte value of what I am looking for, wouldn't I???

So PLEASE HELP:

Let's suppose I look at Ladybug60.bin (which I am using for testing this prog) at the Table AdvanceWarmFullThrottle_FromMap.

Offset for this Table is a057 (hex) = > 41047 (which is greater then my file and I understand this since this is the position where it looks at the chip)

Now, the cal has an offset of 8000 = > 32768

So to search the first byte value of this table I should be searching using an offset from 41047 - 32768 ????

This is my first big question!

The second one is, once I now where to get the first byte, how do I know until where I have to look???????

Chem's calx doesn't seem to provide me with sufficient information to this. So here I need help.

Thanks!!

If I resolve this, the little tool is nearly ready!!

[MopàrBCN]

Hi, Ok, I actually managed to find the table in the byte array from it's offset. What I need to understand now is to determine until what point to look!! How can I find programatically where to stop looking???

So any help here would be catapulting me to finish this tonight!!!

[risen]

.

I know there is an offset to each value. So I suppose this is the address inside my binary file where to find my values. Then I have a base offset of hex8000. If I translate this to decimal I get at 32768 which is the file length if I get this right. Now, the table offsets translate all to numbers higher then this offset of 8000. So I guess I have to calculate dataoffset - cal offset => position of first value of table to search??? Here I am getting confused.

1. Even if this would be the starting point how would I know when the table ends?

2. I would expect to find at this address the first byte value of what I am looking for, wouldn't I???

So PLEASE HELP:

Let's suppose I look at Ladybug60.bin (which I am using for testing this prog) at the Table AdvanceWarmFullThrottle_FromMap.

Offset for this Table is a057 (hex) = > 41047 (which is greater then my file and I understand this since this is the position where it looks at the chip)

Now, the cal has an offset of 8000 = > 32768

So to search the first byte value of this table I should be searching using an offset from 41047 - 32768 ????

This is my first big question!

The second one is, once I now where to get the first byte, how do I know until where I have to look???????

Chem's calx doesn't seem to provide me with sufficient information to this. So here I need help.

Thanks!!

If I resolve this, the little tool is nearly ready!!

You're well on your way to having it figured out.

I wrote a perl script to calcualte an advance map from the various spark tables (and to force me to understand how things were laid out in the cal). So the code below is in perl, but it should be readable enough to give you an idea of what needs to be done. I can provide you the perl script if you want it as a reference. Some parts of it are *UGLY* but LMK if you want it.

I'll tackle the offset question first, since you need to know how to hit an offset to read the tables from the .bin. Basically, you can take the vaule in the .tbl or the xml and subtract 0xc00 from it. A simplified bit of code is below. Basically take the address from the .calx/.tbl, subtract 0xc00 from it, and seek to that location. That byte will be the length of your table.

       $offset = hex($offset_from_tbl_file) - hex(C000);
        seek( $CALFILE, $offset, SEEK_SET );
        read( $CALFILE, $in, 1);

The next thing you need to know about is the table layout. The basic info is below. I took it from a comment in the program, you can ignore the part after unused in the slope comment (I'm calculating slope, not reading it).

#table entry layout is: aa bb cc dd dd bb cc dd dd.....
#where aa = number of elements in table
#bb is compare point for input (X)
#cc is output point at that input (Y)
#dd dd is slope, unused, delta(Y)/delta(X) is used to find slope and interpolate between points

So for each table, you lookup the offset, read the first byte. That first byte is the length of the table. You can then take the value of that byte you just read and read the table by reading (value of that byte * 4). This is for 8 bit tables. The 16 bit would be (value of first byte * 5) I believe. I never tried reading any of the 16 bit (fueling mostly) tables so don't quote me.

[MopàrBCN]

Hey THANKS! That does help and I can work from that!!!

Never thought that my car eventually would make me a propper programmer

BTW: The comment where you said that you did it in part to force you to understand how the cal works, this is part of my excercise as well.

[MopàrBCN]

PS: here is a preview of what I want to get (ignore the data, the comparison is still buggy!)

There are two purposes for which I want to use this:
1. To keep track of changes I've made between versions (I am too lazy to document)
2. To see at a glance the difference of two cal's - as far as they are comparable.

This will be a work in progress. As soon as I have a first working version I'will post it with source code.

[MopàrBCN]

Another Question:

Can I assume, that the Tables with the Type of "fuelSMEC" are 16 bit tables and the rest are 8 bit tables??

I found the following types for tables:

fuelSMEC
normal
sqrt


Anotherone:

To do the same for WORD type data, is it that the first byte will tell me the length of the WORD as well???

Then I would have it solved.

[risen]

Another Question:

Can I assume, that the Tables with the Type of "fuelSMEC" are 16 bit tables and the rest are 8 bit tables??

I found the following types for tables:

fuelSMEC
normal
sqrt


Anotherone:

To do the same for WORD type data, is it that the first byte will tell me the length of the WORD as well???

Then I would have it solved.

I should mention that I have an error above. The last entry in each table has no slope. So to read the whole table in you would need to subtract 2 bytes from the value I gave you. It didn't matter in my program because I'm calculating slope, not reading it. Or, more consicely, you want to read (4*table_len -2 ) bytes for single byte tables.

The first byte only tells you the number of entries in the table I don't believe it tells you anything about the type. In the basic tables (the ones that use 1 byte as input and 1 byte as output) it means that each entry in the table, except for the last one, takes 4 bytes. 1 byte input, 1 byte output, and 2 bytes for slope. This is the reason for the 4 in the equation above. For fueling tables you have 1 byte input, 2 bytes output, and 2 bytes for slope (sans last entry).

I should also mention that I only have played with single byte tables for LM based cals. What you could do, and this is what I did, is open the cal in d-cal and select a table. Then hit the cal data button, and it should give you the bytes in hex. You could also look through the ASM, the tables in there are very clearly defined and each line contains an entry for the table, so you can just count the bytes for each type.

Also, again I haven't messed with the smec tables, I'd imagine that the fueling tables are the only place that the 16 bit return values are used, so I'd guess that's what the fuelSMEC type means. I have no idea what the sqrt tables look like, but it seems to me that they're usually used for inverse values (1/X), I believe.

[MopàrBCN]

Hi, yes I figured this out :-) (4*table_len -2 ) ...

I am left with the problem of the Fuel Tables. Indeed, they seem to be the only 16 Bit tables and follow a strange pattern: Byte, Word, Word, Byte, Word,Word,Byte

Interesting is, that it seems, that the last byte always seems to be "FF". If that is the delimiter for those tables it would be easy.

Does anyone know if that is the case???

If not I am not sure how to deal with them.

Also I would be curious as to what the function would be of the preceeding byte before the 2 word values.

In total, THANK YOU for your help on this. I got from you the decisive inputs, either directly or indirectly. I really apreciate your help!!

[risen]

Hi, yes I figured this out :-) (4*table_len -2 ) ...

I am left with the problem of the Fuel Tables. Indeed, they seem to be the only 16 Bit tables and follow a strange pattern: Byte, Word, Word, Byte, Word,Word,Byte

Interesting is, that it seems, that the last byte always seems to be "FF". If that is the delimiter for those tables it would be easy.

Does anyone know if that is the case???

If not I am not sure how to deal with them.

Also I would be curious as to what the function would be of the preceeding byte before the 2 word values.

In total, THANK YOU for your help on this. I got from you the decisive inputs, either directly or indirectly. I really apreciate your help!!

The 0xff bytes are usually left at the ends of the table to allow you to add points to a table in a .bin. My understanding is that they're ignored if they're outside the number of entries in the table. You should be able to get things to work without a delimiter by using the length of the table and only reading a number of entries equal to the length.

I believe that there should be 5 bytes in the fuel tables for each entry. You should have 1 byte for the input, 2 bytes for the output and 2 bytes for the slope. So, to use the equation above, you should be able to read (5*table_len -2) bytes to read the full table. So the Byte,Word,Word sequence you list above is one entry, and your listing above is actually 2 entries.

Do you have an example?

[MopàrBCN]

Hi,

this would be one example of a 16 bit table:

This is the fuel at full throttle from map as in the .asm file:

.byte 0x00
.word 0x0000, 0x26b3
.byte 0x68
.word 0x0fb9, 0x2371
.byte 0x83
.word 0x1376, 0x3780
.byte 0x89
.word 0x14c3, 0x2368
.byte 0xff

Now, see the first byte is 0. So it's function must be different then indicating the total length of the table (in this case 4 points). It must be part of the point itself it describes.

If I would use the same principle to calculate it's length as I would do with the 8 bit tables I would get a length of 0.

Your equation would however work, if I had beforehand a information for how many points I must look. (5 * 4(Length) ) => 20

The only constant I have found looking at all fuel tables is that the last byte always terminates in FF

My current aproach is, to read from the first byte, treating it as part of the first point and iterate through my byte array using the schema Byte,word,word (=> block of 5 bytes) until I find after reading a complete block a byte with a value of "FF".

Well this is what I will be doing today to see how it works.

[risen]

Now that I acaually sit down and look at the fueling tables they're all like that (LM included). The first byte can't be the length. I guess the only way to know when you've reached the end is to look for 0xff. Maybe if Rob or someone who actually knows sees this they'll chime in.

I suppose you could look through the code and see how the call to do fueling table lookups works. I'm not that good with ASM so it would take me a couple days worth of spare time to figure it out.

[MopàrBCN]

Hi, yep this is how I have done it at the end.

I can do anything with vb.net pl/sql or t-sql. But asm I never really understood. I am amazed that I seem to understand part of how the cals are build but that it's about it. c++ is the same. I am reading the chem source code since I thought I will get a hint there, and even found where the routines are, but do not have a clue how to interpret the code....

But to get back to what I'm doing right now, the looking up of the 0xff seems to do the trick. I have done the algorithm in a way that it does not stop at any 0xff but only once it has read 5 bytes worth of data. So right now I got it to work like this.

Now I am doing stuff to look up the .tbl files directly and then I will release again....

[risen]

I can do anything with vb.net pl/sql or t-sql. But asm I never really understood. I am amazed that I seem to understand part of how the cals are build but that it's about it. c++ is the same. I am reading the chem source code since I thought I will get a hint there, and even found where the routines are, but do not have a clue how to interpret the code....

I can look through the chem2 source. I'm pretty proficent in c++ (more-so in ansi c, but that's besides the point). It should only take me a minute or two to understand it in c++. Do you know what the name of the function is?