Send raw HTTP requests

[Encrypted]

Hi,

Now i have had some experience with making raw packets, i'd like to make some packet for a specific protocol, and i picked HTTP.
Now, in the RFC i read about various header fields and stuff, but i cant seem to find a header fil for this :?
I mean, does a linux os contain those header files standard?
or do i have to download some new libraries or something?

And btw, i have looked in the Apache source (assuming it would hold some raw stuff), but i didnt find anything usefull (or at least, i didnt think it was usefull)

Any help appreciated ;)

thanks,
encrypted

[RobSeace]

I don't think you'll find too many apps that use raw sockets for
dealing with HTTP... Kinda pointless thing to do, really... ;-)

But, HTTP "headers" are not raw binary headers, of the sort
you're probably thinking of (ie: TCP, UDP, IP, etc. headers)...
They're just arbitrary-length, carriage-return/line-feed delimited
lines of text... They don't really lend themselves to definition
as any sort of struct, or anything that you seem to be looking
for...

[Encrypted]

I see,

Let me get this right,
In the RFC, a request is declared like this:

Request = Request-Line ; Section 5.1
*(( general-header ; Section 4.5
| request-header ; Section 5.3
| entity-header ) CRLF) ; Section 7.1
CRLF
[ message-body ] ; Section 4.3

The Request line is something like
GET /index.html HTTP/1.1
The General Header

general-header = Cache-Control ; Section 14.9
| Connection ; Section 14.10
| Date ; Section 14.18
| Pragma ; Section 14.32
| Trailer ; Section 14.40
| Transfer-Encoding ; Section 14.41
| Upgrade ; Section 14.42
| Via ; Section 14.45
| Warning ; Section 14.46

And so you have to fill in this stuff like

char *general_header = "Cache-Control = = \"Cache-Control\" \":\" 1#cache-directive"
"Connection: close"
"Content-Encoding: gzip"
"......

and so on.... you mean like this?
If so, i assume you have to do this with all the headers, like

char *request_header;
char *entity_header;


btw, yeah i was looking for some kind of struct, but this is also nice : )

thanks,
encrypted

[vikasgp]

The way to do it is this:

First, open a TCP socket to the HTTP server(port 80).

All the headers, requests, responses defined in the RFC are just plain text sent through this socket. For example, if you wanted to get a page, you use something like:

...
buf = "GET index.html HTTP/1.1\r\n";
len = strlen(buf);
send(sockfd, buf, len, ...)

I can't remember the order of args to send() though,... but you get the idea.

So no need to use struct's, just send plain text. If you want examples, look at the sources of apps like wget(GNU), or download accelerators like ProZilla.

[Encrypted]

yeah thanks,

but cant you edit all the header fields also?
(like i showed in my post before?)

Because theres this challenge, where you have to fill in some fields by yourself, in order to get the password.

thanks,
encrypted

[RobSeace]

Yes, the headers are just lines of text like that... You can send
anything you want... (Whether or not it makes sense to the
server is another matter...)

Your best bet would probably be sniffing a normal browser->httpd
session... Then, you can see exactly what is being sent, and it
will probably make more sense to you...

[Encrypted]

Oke, so i sniffed a session from my computer with some httpd, (with ethereal)

at first it, obviously, does the handshake and all.
And in C connect() does this for me ( i get your point that using raw sockets with HTTP is rather useless :))

Then the client sends a GET / HTTP/1.1 .
If i send out some raw request, do i declare it like this?:

request = "GET / HTTP/1.1\r\n"
"Accept: image/gif\r\n"
"Accept-Language:nl\r\n"
"Accept-Encoding: gzip, deflate\r\n"
"User-Agent: Mozilla/4.0\r\n"
"Host: www.netric.org\r\n"
"Connection: Keep-Alive\r\n"
"\r\n";


And some other thing,
when the client sends a request, the server answers with HTTP/1.1 200 OK,
and after that, when i take a closer look at the HTTP header, it only says:
data(1420 bytes)

And so i guess that is where the transaction actually takes place ?
well it actually makes sense to me now...


And a last question (nothing to do with http really..),
If you've used ethereal before, you can see that the package you select is displayed beneath there, in hex and plain text, and i know the data is going over the hardware as binary,
but my question, where is the data being converted to hex? which layer i mean.

thanks again ;)
encrypted

[vikasgp]

If it is of any use to you, I post here a small program I wrote a couple of days ago while trying to understand HTTP. The program is invoked as
$ nget <host> /<path>

The slash before the path is very important.

The file pointed to by <path> is saved in the current directory.

/*************** Cut here *****************************/
/*
* nget.c
* A rudimentary HTTP file downloader.
*/


#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <stdio.h>
#include <netdb.h>
#include <string.h>

#define BUFLEN 4096
#define HTTP_PORT 80
#define REQ_LEN 1024

int main(int argc, char **argv)
{
int s, nbytes;
struct sockaddr_in serv;
struct hostent *h_serv;
char *program_name = argv[0];
char buf[BUFLEN]; /* The recieve buffer */
char req[REQ_LEN]; /* The request buffer */

FILE *local_file;
char *localf_name; /* Name of local file */
char *remote_path; /* Path to remote resource */

int ntransmitted = 0;

if( argc < 2 ){
printf("usage: %s host [/path]\n", program_name);
return -1;
}


if(( s = socket(AF_INET, SOCK_STREAM, 0)) < 0 ){
printf("\n%s: Couldn't create socket\n", program_name);
return -1;
}


bzero(&serv, sizeof(serv));

serv.sin_family = AF_INET;
serv.sin_port = htons(HTTP_PORT);

if( (serv.sin_addr.s_addr = inet_addr(argv[1])) == INADDR_NONE){
/* Need to look up */
printf("%s: Looking up %s...", program_name, argv[1]);
h_serv = gethostbyname(argv[1]);
if( h_serv == NULL ){
printf("\n%s: Couldn't resolve %s\n", program_name, argv[1]);
return -1;
}
else{
printf("resolved\n", program_name);
}
serv.sin_addr.s_addr = *(long *)(h_serv->h_addr);
}

if( connect(s, (struct sockaddr *)&serv, sizeof(serv)) < 0 ){
printf("%s: Connection refused\n", program_name);
return -1;
}
printf("%s: Connected to %d.%d.%d.%d\n", program_name,
h_serv->h_addr[0],
h_serv->h_addr[1],
h_serv->h_addr[2],
h_serv->h_addr[3]);

/* Set the local file name */

if( argc == 2 ){ /* Assume / */
remote_path = "/";
localf_name = "index.html";
}
else{
remote_path = argv[2];

/* Char next to the slash */
localf_name = strrchr(remote_path, '/') + 1;

}

/* Create the local file */
if( (local_file = fopen(localf_name, "w")) == NULL ){
printf("%s: Couldn't create file\n", program_name);
return -1;
}

/* Send the request */
sprintf(req, "GET %s\r\n\r\n", remote_path);

if( send(s, req, strlen(req), 0) < 0 ){
printf("%s: Unable to send request\n", program_name);
return -1;
}

/* Recieve and save data */
printf("%s: Recieving data\n", program_name);

do{
nbytes = recv(s, buf, BUFLEN-1, 0);
ntransmitted += nbytes;
printf(".");
fwrite((void *)buf, nbytes, sizeof(char), local_file);
}while( nbytes == BUFLEN-1 );

printf("\n%d bytes written to %s\n", ntransmitted, localf_name);
close(s);
return 0;
}

/********************* Cut here *****************************/

[vikasgp]

Oops, the indentation is all gone. Is there a way to retain it ?

[mlampkin]

Just use the bb code items and to retain formatting...

As for the question proper... and the headers you were showing...

I would drop the "Connection: Keep-Alive" since its a pre 1.1 remnant that really shouldn't be used... declaring HTTP version 1.1 (in the main request) says you want to keep the connection open anyway... and you actually have to explicitly terminate the keep alive (that is default for 1.1) with a "Connection: close"...

You also don't need to really bother with ethereal... just telnet to the host at port 8 and cut / paste the requests / headers in... you will be able to see everything the server responds with...

Michael

[RobSeace]

Heh. That would be and Michael tried to say... ;-)

As for the question about hex... Well, hex is just a common
method of displaying raw binary data, because its more compact
and easier to deal with than actual binary... And, there's a nice
4-to-1 relationship between digits, so it's easy to convert from
one to the other... But, it doesn't really matter; ethereal could
show you the dump in binary, or octal, or base 36, for that matter...
It just doesn't make a bit of difference; it's just an arbitrarily
chosen representation of the data... It just so happens that hex
is a well-recognized and understood representation, so that's
what it uses... But, nothing is being "converted" to hex at any
networking "layer", as you seem to suggest... It's merely being
displayed as hex for interpretation by us lowly humans... ;-)