Loco
07-24-2002, 08:35 PM
Taken from the original FAQ.
2.15 - How do I send [this] over a socket?
Anything other than single bytes of data will probably get mangled unless you take care. For integer values you can use htons() and friends, and strings are really just a bunch of single bytes, so those should be OK. Be careful not to send a pointer to a string though, since the pointer will be meaningless on another machine. If you need to send a struct, you should write sendthisstruct() and readthisstruct() functions for it that do all the work of taking the structure apart on one side, and putting it back together on the other. If you need to send floats, you may have a lot of work ahead of you. You should read RFC 1014 which is about portable ways of getting data from one machine to another (thanks to Andrew Gabriel for pointing this out).
From: Gregory
Where can i find sendthisstruct and readthisstruct. What library are they in and are there definitions of what arguments they take and return.These functions seem like they could solve alot of my problems Thanks
From: Rob Seace
No, I think the point was that YOU are supposed to write
those functions yourself... Because, only you will know
the contents of your structures, and how to properly
assemble and disassemble them...
However, let me just add a semi-heretical suggestion: if
you know for a fact that your client and server programs
are going to be running on the same platform, it's really
ok to simply send your structs raw, doing absolutely no
magic transformations to them, at all... I do it all the
time... Simply pass a pointer to your struct to
recv()/send(), read()/write(), whatever... (Or, memcpy()
to/from a char buffer...) The only real restriction is that
your structs must NOT contain any pointers, at all... Of
course, those simply can't be sent... (Or, rather, it would
be pointless to send them, as they are meaningless on the
remote machine...) But, pretty much anything else is fair
game... But, again, let me repeat: this will ONLY work if
both ends are on the same platform... If you intend your
program to be used on a multiple platforms, avoid this
method at all costs... But, if you know it's only going to
be used on a specific fixed platform, it makes for a nice
simple method of transfering your structs...
From: Michael Lampkin
Just read this question for the first time... It was referenced by another question that had a recent post...
Anyway... a couple of my thoughts...
XDR (the ref to rfc 1014) is / was designed for RPC... where the value(s) type are implicitly known and matched to a "procedure" type call... For passing (this is a IMHO) more complex / nested types (or defined types of arbitrary length) where a "packet" may or may not require certain data fields, it becomes wasteful... That isn't to say it isn't useful - it certainly is and is well suited for it's intended purpose, RPC...
Another alternative are that current pop encoding called XML... Very popular... very very popular... because a brain dead chimp can write up XML (oops, that was another IMHO)... but it is up to the person(s) writing the system to match the data types to the platforms... is extremely verbose (a boolean takes 8 (minimum ?) bytes to represent instead of one bit)... has no truly uniform method of passing binary data (um, ok... base64 encoding (?) which will cause the data to blow up several times its original size with truly random data)... Anyway... it leaves a lot to be desired for data transfers of numeric / binary data... and shouldn't be used as such... it is, after all, a MARKUP LANGUAGE (in case you don't know what that means, it means used for delineating TEXT data)...
Sorry... always get myself worked up on the XML thing... Its just that I have seen way too many projects in the past couple years get twisted by upper management types that it be used... because they can understand it... and then I get to listen to complaints about systems being slow / bandwidth hogs....
Oops... there I went again... lol.
So my personal favorite? Easy enough... ASN.1 (ITU-T Rec. X.680)... Allows you to pass pretty much any type you want (including arbitrary length bit stream, reals, etc.)... Is the industry standard (though XML is somehow making inroads into it's domain ~ god help us)... In fact, check out all the network crypto standards and you'll see ASN.1 everywhere... It also has several well defined compression schemes... The specifications are human readable (albeit, sometimes hard for a novice when dealing with legacy specifications of "packets" because of back references to previous specs)... and there is a free way to formally record your specifications for everyone to see / use if you are making an open system... Btw, if anyone is interested you can find some of the docs for it at http://asn1.elibel.tm.fr/en/standards/ if you want to take a look...
So I don't start sounding like a groupy for ASN.1 I will close with the following observation... I do NOT normally recommend it for small projects such as a system passing a couple of ints back and forth...
Anyway... just my thoughts ;-)
ML
From: Michael Lampkin
Added on: 2002-06-18 13:56:53
Ok...
Before I get blasted by anyone who has asked me this question in email... and has gotten the ~ convert it to a string and then transmit [str len][str chars] answer (the easiest way)... or the XDR one (more complicated but less data expansion)...
I have an excuse for not doing ASN.1!!! That is, I have tried to walk a few folks through ASN.1 but failed miserably... too complex for someone just starting out I guess... or perhaps I am just bad at explaining how to use it... either way, a HORRIBLE experience... :-/
I must state that I do STILL prefer ASN.1 since it allows you do put "holes" inside your data structures for future expansion etc. where XDR does not (well, you technically can but you have more overhead)... but please don't ask for an ASN.1 walkthru... lol :-P
ML
From: Michael Lampkin
Added on: 2002-06-19 07:22:02
Um.. people...
PLEASE stop sending questions to my email addresss... if you need / want examples, they can be asked for and provided here... thanks :-)
Now, since I have gotten 3 requests (actually 4, but who is counting) for the XDR stuff... the following is a very simply example with no guarantees as to being 100% correct... in other words, use at your own risk!
So the assumption is (for the question previously asked in another thread) we are trying to send something like an array of unknown size containing double values. The first thing to do is create file (we will name it xdr_stuff.x ~ yes, .x) and put the following in it:
struct data
{
double value<>;
};
Next, run the command rpcgen xdr_stuff.x and it will create two files, one named xdr_stuff.h and xdr_stuff_xdr.c for you. If you look into xdr_stuff.h file, you will see that among other things it generated the following:
struct data {
struct {
u_int value_len;
double *value_val;
} value;
};
This structure is what we will use in our code...
Now following is the code for a simple program showing how to use xdr and the generated file... a couple of quick items... normally you would actually compile the xdr_stuff_xdr.c file... and just include xdr_stuff.h in our header... and then link the .o file... I am just including BOTH for simplicities sake so I do not have to walk through the proper way...
Hopefully the comments will explain what is going on...
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "xdr_stuff.h"
#include "xdr_stuff_xdr.c"
/*
Arbitrary... see RNDUP info for proper way to calculate.
I believe this value should work unless someone passes a
double array larger than 4 k - 1 in size.
*/
#define BUFFER_SIZE 16 * 1024
int main( )
{
XDR xdr_handle;
char * buffer;
data outgoing_xdr_struct;
data incoming_xdr_struct;
double outgoing_test_data[100];
double incoming_test_data[100];
int result;
int size;
int index;
for (index = 0; index < 10; index ++)
{
outgoing_test_data[index] = (double) ((index + 1) * 2) / (double) ((index + 4) * 3);
printf("Index: %i Value:%e\n", index, outgoing_test_data[index]);
}
outgoing_xdr_struct.value.value_len = 10;
outgoing_xdr_struct.value.value_val = outgoing_test_data;
/*
We have test outgoing data setup, an array of 10
double values. Start xdr stuff.
*/
buffer = (char *) malloc(BUFFER_SIZE);
xdrmem_create(& xdr_handle, buffer, BUFFER_SIZE, XDR_ENCODE);
/*
Encodes the data (including the held array) in the
struct outgoing_test_data and puts it into buffer.
*/
result = xdr_data(& xdr_handle, & outgoing_xdr_struct);
if (result != TRUE)
{
printf("Error during encoding!!!\n");
exit(0);
}
size = xdr_getpos(& xdr_handle);
printf("Size of encoded data: %i\n", size);
/*
At this point the data is encoded and in buffer. You
can send without worrying about endianism etc. and just
use a write(descriptor, buffer, size).
You know the size of the buffer of encoded data but you
have to tell the other side so you will probably want to
write htonl(size) then the buffer data... The receiving
end would read 4 bytes, do a ntohl to get the size, then
read that size / number of bytes...
There are more efficient ways of doing it such as embedding
the data directly at the start of the buffer but I'll leave
that to you ;-)
*/
/* DO NETWORK WRITE HERE */
/*
Since we are only sending once, just destroy our handle. In
real life, you would probably call xdr_setpos(& xdr_handle, 0)
to reset the pointer in buffer back to 0 so you could re-use
it for the next transmit.
*/
xdr_destroy(& xdr_handle);
/*
Now we will reverse the process (like on the receiving
end) and decode the data in buffer and place it into
incoming_xdr_struct. We must create an XDR stream that does
decoding (this would be on the client / rcv side). Note
it is the same as above but uses the constant XDR_DECODE for
the last parameter.
We are assuming that buffer is the buffer used in a call on
the client for read(descriptor, buffer, count).
*/
xdrmem_create(& xdr_handle, buffer, BUFFER_SIZE, XDR_DECODE);
/*
Just to "prove" incoming_xdr_struct is empty we'll zero
it out before doing anything else.
*/
memset(& incoming_xdr_struct, 0 , sizeof(struct data));
result = xdr_data(& xdr_handle, & incoming_xdr_struct);
if (result != TRUE)
{
printf("Error during decoding!!!\n");
exit(0);
}
size = xdr_getpos(& xdr_handle);
printf("Ate %i bytes of encoded data: %i\n", size);
/*
Now lets print our incoming structure to see if it matches
what was printed for outgoing! Notice we are using our value_len
field of the struct to check when we are finished indexing.
*/
for (index = 0; index < incoming_xdr_struct.value.value_len; index ++)
{
printf("Index: %i Value:%e\n", index, incoming_xdr_struct.value.value_val[index]);
}
/*
Last but not least ~ during the call to xdr_data XDR_DECODE, memory
was automatically allocated for the pointer to our doubles array. We
must deallocate this after we have used the numbers or copied them
to a new location. So do the following (notice the cast to (char *)!) and
if you are wondering where xdr_data is at, look at the header file
that was generated by rpcgen at the start.:
*/
xdr_free((xdrproc_t) xdr_data, (char *) & incoming_xdr_struct);
/*
...and if we are completely done with the handle e.g. exiting
the program or some such we should free / destroy that also...
*/
xdr_destroy(& xdr_handle);
}
...and thats all folks... :-/
2.15 - How do I send [this] over a socket?
Anything other than single bytes of data will probably get mangled unless you take care. For integer values you can use htons() and friends, and strings are really just a bunch of single bytes, so those should be OK. Be careful not to send a pointer to a string though, since the pointer will be meaningless on another machine. If you need to send a struct, you should write sendthisstruct() and readthisstruct() functions for it that do all the work of taking the structure apart on one side, and putting it back together on the other. If you need to send floats, you may have a lot of work ahead of you. You should read RFC 1014 which is about portable ways of getting data from one machine to another (thanks to Andrew Gabriel for pointing this out).
From: Gregory
Where can i find sendthisstruct and readthisstruct. What library are they in and are there definitions of what arguments they take and return.These functions seem like they could solve alot of my problems Thanks
From: Rob Seace
No, I think the point was that YOU are supposed to write
those functions yourself... Because, only you will know
the contents of your structures, and how to properly
assemble and disassemble them...
However, let me just add a semi-heretical suggestion: if
you know for a fact that your client and server programs
are going to be running on the same platform, it's really
ok to simply send your structs raw, doing absolutely no
magic transformations to them, at all... I do it all the
time... Simply pass a pointer to your struct to
recv()/send(), read()/write(), whatever... (Or, memcpy()
to/from a char buffer...) The only real restriction is that
your structs must NOT contain any pointers, at all... Of
course, those simply can't be sent... (Or, rather, it would
be pointless to send them, as they are meaningless on the
remote machine...) But, pretty much anything else is fair
game... But, again, let me repeat: this will ONLY work if
both ends are on the same platform... If you intend your
program to be used on a multiple platforms, avoid this
method at all costs... But, if you know it's only going to
be used on a specific fixed platform, it makes for a nice
simple method of transfering your structs...
From: Michael Lampkin
Just read this question for the first time... It was referenced by another question that had a recent post...
Anyway... a couple of my thoughts...
XDR (the ref to rfc 1014) is / was designed for RPC... where the value(s) type are implicitly known and matched to a "procedure" type call... For passing (this is a IMHO) more complex / nested types (or defined types of arbitrary length) where a "packet" may or may not require certain data fields, it becomes wasteful... That isn't to say it isn't useful - it certainly is and is well suited for it's intended purpose, RPC...
Another alternative are that current pop encoding called XML... Very popular... very very popular... because a brain dead chimp can write up XML (oops, that was another IMHO)... but it is up to the person(s) writing the system to match the data types to the platforms... is extremely verbose (a boolean takes 8 (minimum ?) bytes to represent instead of one bit)... has no truly uniform method of passing binary data (um, ok... base64 encoding (?) which will cause the data to blow up several times its original size with truly random data)... Anyway... it leaves a lot to be desired for data transfers of numeric / binary data... and shouldn't be used as such... it is, after all, a MARKUP LANGUAGE (in case you don't know what that means, it means used for delineating TEXT data)...
Sorry... always get myself worked up on the XML thing... Its just that I have seen way too many projects in the past couple years get twisted by upper management types that it be used... because they can understand it... and then I get to listen to complaints about systems being slow / bandwidth hogs....
Oops... there I went again... lol.
So my personal favorite? Easy enough... ASN.1 (ITU-T Rec. X.680)... Allows you to pass pretty much any type you want (including arbitrary length bit stream, reals, etc.)... Is the industry standard (though XML is somehow making inroads into it's domain ~ god help us)... In fact, check out all the network crypto standards and you'll see ASN.1 everywhere... It also has several well defined compression schemes... The specifications are human readable (albeit, sometimes hard for a novice when dealing with legacy specifications of "packets" because of back references to previous specs)... and there is a free way to formally record your specifications for everyone to see / use if you are making an open system... Btw, if anyone is interested you can find some of the docs for it at http://asn1.elibel.tm.fr/en/standards/ if you want to take a look...
So I don't start sounding like a groupy for ASN.1 I will close with the following observation... I do NOT normally recommend it for small projects such as a system passing a couple of ints back and forth...
Anyway... just my thoughts ;-)
ML
From: Michael Lampkin
Added on: 2002-06-18 13:56:53
Ok...
Before I get blasted by anyone who has asked me this question in email... and has gotten the ~ convert it to a string and then transmit [str len][str chars] answer (the easiest way)... or the XDR one (more complicated but less data expansion)...
I have an excuse for not doing ASN.1!!! That is, I have tried to walk a few folks through ASN.1 but failed miserably... too complex for someone just starting out I guess... or perhaps I am just bad at explaining how to use it... either way, a HORRIBLE experience... :-/
I must state that I do STILL prefer ASN.1 since it allows you do put "holes" inside your data structures for future expansion etc. where XDR does not (well, you technically can but you have more overhead)... but please don't ask for an ASN.1 walkthru... lol :-P
ML
From: Michael Lampkin
Added on: 2002-06-19 07:22:02
Um.. people...
PLEASE stop sending questions to my email addresss... if you need / want examples, they can be asked for and provided here... thanks :-)
Now, since I have gotten 3 requests (actually 4, but who is counting) for the XDR stuff... the following is a very simply example with no guarantees as to being 100% correct... in other words, use at your own risk!
So the assumption is (for the question previously asked in another thread) we are trying to send something like an array of unknown size containing double values. The first thing to do is create file (we will name it xdr_stuff.x ~ yes, .x) and put the following in it:
struct data
{
double value<>;
};
Next, run the command rpcgen xdr_stuff.x and it will create two files, one named xdr_stuff.h and xdr_stuff_xdr.c for you. If you look into xdr_stuff.h file, you will see that among other things it generated the following:
struct data {
struct {
u_int value_len;
double *value_val;
} value;
};
This structure is what we will use in our code...
Now following is the code for a simple program showing how to use xdr and the generated file... a couple of quick items... normally you would actually compile the xdr_stuff_xdr.c file... and just include xdr_stuff.h in our header... and then link the .o file... I am just including BOTH for simplicities sake so I do not have to walk through the proper way...
Hopefully the comments will explain what is going on...
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "xdr_stuff.h"
#include "xdr_stuff_xdr.c"
/*
Arbitrary... see RNDUP info for proper way to calculate.
I believe this value should work unless someone passes a
double array larger than 4 k - 1 in size.
*/
#define BUFFER_SIZE 16 * 1024
int main( )
{
XDR xdr_handle;
char * buffer;
data outgoing_xdr_struct;
data incoming_xdr_struct;
double outgoing_test_data[100];
double incoming_test_data[100];
int result;
int size;
int index;
for (index = 0; index < 10; index ++)
{
outgoing_test_data[index] = (double) ((index + 1) * 2) / (double) ((index + 4) * 3);
printf("Index: %i Value:%e\n", index, outgoing_test_data[index]);
}
outgoing_xdr_struct.value.value_len = 10;
outgoing_xdr_struct.value.value_val = outgoing_test_data;
/*
We have test outgoing data setup, an array of 10
double values. Start xdr stuff.
*/
buffer = (char *) malloc(BUFFER_SIZE);
xdrmem_create(& xdr_handle, buffer, BUFFER_SIZE, XDR_ENCODE);
/*
Encodes the data (including the held array) in the
struct outgoing_test_data and puts it into buffer.
*/
result = xdr_data(& xdr_handle, & outgoing_xdr_struct);
if (result != TRUE)
{
printf("Error during encoding!!!\n");
exit(0);
}
size = xdr_getpos(& xdr_handle);
printf("Size of encoded data: %i\n", size);
/*
At this point the data is encoded and in buffer. You
can send without worrying about endianism etc. and just
use a write(descriptor, buffer, size).
You know the size of the buffer of encoded data but you
have to tell the other side so you will probably want to
write htonl(size) then the buffer data... The receiving
end would read 4 bytes, do a ntohl to get the size, then
read that size / number of bytes...
There are more efficient ways of doing it such as embedding
the data directly at the start of the buffer but I'll leave
that to you ;-)
*/
/* DO NETWORK WRITE HERE */
/*
Since we are only sending once, just destroy our handle. In
real life, you would probably call xdr_setpos(& xdr_handle, 0)
to reset the pointer in buffer back to 0 so you could re-use
it for the next transmit.
*/
xdr_destroy(& xdr_handle);
/*
Now we will reverse the process (like on the receiving
end) and decode the data in buffer and place it into
incoming_xdr_struct. We must create an XDR stream that does
decoding (this would be on the client / rcv side). Note
it is the same as above but uses the constant XDR_DECODE for
the last parameter.
We are assuming that buffer is the buffer used in a call on
the client for read(descriptor, buffer, count).
*/
xdrmem_create(& xdr_handle, buffer, BUFFER_SIZE, XDR_DECODE);
/*
Just to "prove" incoming_xdr_struct is empty we'll zero
it out before doing anything else.
*/
memset(& incoming_xdr_struct, 0 , sizeof(struct data));
result = xdr_data(& xdr_handle, & incoming_xdr_struct);
if (result != TRUE)
{
printf("Error during decoding!!!\n");
exit(0);
}
size = xdr_getpos(& xdr_handle);
printf("Ate %i bytes of encoded data: %i\n", size);
/*
Now lets print our incoming structure to see if it matches
what was printed for outgoing! Notice we are using our value_len
field of the struct to check when we are finished indexing.
*/
for (index = 0; index < incoming_xdr_struct.value.value_len; index ++)
{
printf("Index: %i Value:%e\n", index, incoming_xdr_struct.value.value_val[index]);
}
/*
Last but not least ~ during the call to xdr_data XDR_DECODE, memory
was automatically allocated for the pointer to our doubles array. We
must deallocate this after we have used the numbers or copied them
to a new location. So do the following (notice the cast to (char *)!) and
if you are wondering where xdr_data is at, look at the header file
that was generated by rpcgen at the start.:
*/
xdr_free((xdrproc_t) xdr_data, (char *) & incoming_xdr_struct);
/*
...and if we are completely done with the handle e.g. exiting
the program or some such we should free / destroy that also...
*/
xdr_destroy(& xdr_handle);
}
...and thats all folks... :-/