try 2 send a struct - problem with padding bytes & endian

[tricio67]

Hi
we are trying to implement an interprocess comunication (IPC) throught ethernet using tcpip sockets (trying to gain SW and HW independece and need not to treat each element of the structure independently nor write a procedure for each type of structure_data we need to send to the remote machine ).
the idea is:
- from MachineA memcpyed the information of a varA (type structure X) to send_array of char
- keeping in mind (sizeof(varA) < sizeof(send_array))
- send the array to MachineB using sockets
- in MachineB memcpyed the information of recv_array to varB (type structure X)

We have come to the point that every thing works IF we use the same HW and OS. Two things (till now.. may be you will add some more "@@" ) are not solved for our goal SW & HW freedom.
- difference between big endian format and little endianformat
- padding bytes - sizeof(type structure X)in MachineA !=sizeof(type structure X)in MachineB
any suggestions or different ways to solve this are well received
- is there any way to analize the structure of the struct or the memory allocated for the struct var...maybe a function like ntohl() and htonl() could be impemented... ideas???

thanks in advance
(note: we work in RT so fast code is must!!!)
PS sorry for my english ...but system I am using has no ... special tools and I didn't fin an online one...

[RobSeace]

It sounds like what you're doing now is really no different than simply directly
sending the raw struct itself... Ie: the "memcpy to/from array" steps you talk about
seem rather pointless, currently...

Sadly, there's no magic function that can parse an arbitrary struct and break down
the elements into machine-independant versions for transport, and then reassemble
that back on the other end... It'd be nice if there were, but I think it's probably an
impossibility for any normal code... To really do it, you'd need to basically parse the
structure definition to learn the fields and their types, so you'd basically need to
build in a C code parser, which isn't a pleasant prospect... (And, it'd surely be slow,
which violates one of your conditions, anyway...) The best you could really do, I
think, is come up with a way of identifying structs' layouts (field offsets and lengths)
via some array of separate structs, then have a function would could read that
definition and do the appropriate conversions... Sounds doable, certainly, but in the
end, I think you'll find the code required to define the struct layout would be just as
ugly as having to do the conversions by hand for every given struct, anyway...
Maybe not, if you made appropriately clever use of GCC's typeof() along with
offsetof() and sizeof(); then, you might be able to just supply all of the field names,
and have some macro do all the magic needed to define the struct layout for you...
I don't know... *shrug*

[i3839]

Being able to transfer arbitrary structures with such IPC is impossible because your code doesn't know how those structures look like, so it has no idea what it must do to rebuild the thing on the other end.

But luckily, this isn't needed as the structures are known at compile time. So all you need to do is automatically generating the code which transforms a structure in a portable way so that it can be used by the other side.

I'm repeating Rob here with different words.

What I'd probably do if I were in your position is writing a little script or program which takes a structure definition and produces the needed conversion functions. I wouldn't try to do it with only macro's, as that seems very hard and ugly. Then I'd make one macro which calls the appropriate conversion function for a given structure by using typeof and string concatenation. E.g. pack_struct(src, dst) and unpack_struct(src, dst).

So only things needed to do is writing a structure parser which takes as input a structure name and the file where that structure is defined, and which outputs C code with the conversion functions. Then you'd have many autogenerated conversion functions, but only two macro functions which you use everywhere. And if there is no conversion function for some structure you'll get a linker error.

Parsing structures isn't that hard as their format is simple, not as hard as writing a full blown C parser anyway. To make it easier for yourself you could also make a special identifier which makes the structure packed and tells your auto code generator that that sturcture needs conversion functions. Then you can run the thing once over your whole code.

[tricio67]

thanks a lot...
the need of the memcpy to string is because we send not only one type of structure but many (different size and types) ... and we wanted a generic procedure so we send an id_messege "int" as the head of the string and then the data structure.
For now we are going to put a restricction to the types in the structure (all doubles) and workout a function to hton_double and ntoh_double, that will be applied to all the string in blocks of 8 starting over the head id_messege "int".

[duncang]

One alternative not mentioned so far is XDR - eXternal Data Representation, which is
part of the RPC framework. The headers & libraries should be on most Unix-like &
Windows systems & there are plenty of tutorials on the web. Obviously it's a general
purpose approach so there may be some performance overhead.

Having said that, I tend to use the approach suggested by i3839, depending on
how many different structs you need to transfer between systems. I do something
like this:

- declare typedef's for each data structure
typedef struct _msg1 {int f1; int f2;} FBMSG0;
typedef struct _msg2 {float f1; float f2;} FBMSG1;

- declare a union of the above
typedef union _msg {FBMSG0 msg0; FBMSG1 msg1;} FBMSG;

- create encode & decode functions for each type, using sprintf & sscanf to pack the fields
to/from char arrays.

- create sendmessage() & recvmessage() functions. My send function writes 4 ASCII chars
indicating the length of the following msg. The next 2 chars are the msg number which is
used as an index into a lookup table on the recipient to find the appropriate decode function.
Then just send the encoded char array.

int ipcsendmsg(int sockfd, int msgnum, FBMSG *msg)
int ipcrecvmsg(int sockfd, int *msgnum, FBMSG *msg)

In use, it goes something like:

Sender:

int i, msgnum;
FBMSG msg;

msg.msg0.f1 = 99;
msg.msg0.f2 = 100;
msgnum = 0;
i = ipcsendmsg(fd, msgnum, &msg);

Receiver:

int i, msgnum;
FBMSG msg;

i = ipcrecvmsg(fd, &msgnum, &msg);

if (msgnum == 0) {
printf("f1 = %d, f2 = %d\n", msg.msg0.f1, msg.msg0.f2);
}

The work then is in coding & testing the encode/decode functions for each struct,
but it works well & is pretty quick. With some effort you can send quite complex structs.

PM me if you'd like to see some real code.

hth

D
-