Description of the minssl protocol

Say there is a zoinxd server on machine B, listening on IP bip and port bport, launched with the command minsslserver bip bport zoinxd. (zoinxd is a UCSPI server application.)

Say a user on machine A wants to connect to B's zoinxd service via minssl, and enters the command minsslclient bip bport zoinx. (zoinx is a UCSPI client application.)

Here is the minssl protocol: what both sides compute and send each other.

Connection establishment

Phase	Program in charge on A	Action on A	Program in charge on B	Action on B
Listening state			tcpserver	listens on IP bip, port bport
TCP connection	tcpclient	connects to bip:bport	tcpserver	performs TCP access control accepts the connection
Greeting and cookie exchange: from A to B	minssl	makes a 28-byte random cc. sends 64 bytes. The first 8 bytes are "`minssl1-`", the next 28 bytes are cc, the next 28 bytes are irrelevant.	minssld	receives 64 bytes. ignores the first 8 bytes. stores the next 28 bytes as the client cookie, cc.
Greeting and cookie exchange: from B to A	minssl	receives 64 bytes. dies if the first 8 bytes are not equal to "`minssl1-`". stores the next 28 bytes as the server cookie, sc.	minssld	makes a 28-byte random sc sends 64 bytes. The first 8 bytes are "`minssl1-`", the next 28 bytes are sc, the next 28 bytes are irrelevant.
Diffie-Hellmann exchange (part 1) and server host key check	minssl	receives 112 bytes: 28 bytes hpk, 28 bytes sspk, 56 bytes sig. checks, against hpk, whether sig is a valid signature for the concatenation of hpk, sspk and cc; dies if it is not the case. checks hpk against the local server host public key database, via a "key manager".	minssld	reads hsk, the 28-byte host private key for the zoinxd service, from a file. reads hpk, the 28-byte host public key for the zoinxd service, from a file; or computes it from hsk. makes a random server session private key sssk and computes the corresponding server session public key sspk computes sig, a kcdsa224 signature of the concatenation of hpk, sspk and cc. Those bytes are signed with hsk. sends 112 bytes: hpk (28), sspk (28), and sig (56).
Diffie-Hellmann exchange (part 2)	minssl	makes a random client session private key cssk and computes the corresponding client session public key cspk computes the shared secret sh with cssk and sspk sends 28 bytes: cspk	minssld	receives 28 bytes: cspk computes the shared secret sh with sssk and cspk

Both sides then use sh, cc and sc to compute a 160-byte shared secret big, via the following formula:

  big = SHA1(sh ## cc ## sc ## 0) ## SHA1(sh ## cc ## sc ## 1) ## ...

## is the concatenation operator. 0, 1, 2... are single bytes: 0x00, 0x01, 0x02...

big is then transmitted to the minssl-tunnel program, that uses it to initialize:
- the "encryption" RC4 pseudorandom number generator
- the "generation" hash127mac message authenticator
- the "decryption" RC4 pseudorandom number generator
- the "verification" hash127mac message authenticator

When the connection has been made

At this point, the minssl-tunnel program is in charge, on both sides of the tunnel. From a protocol point of view, there is no more distinction between the client and the server: A and B are peers. The only data that flows between the peers is the encrypted data that the application programs zoinx and zoinxd need to exchange. Here is what happens when a side X sends data to its peer Y.

The sending side

The application program X writes len bytes of cleartext on its network-write file descriptor. This fd is 7 if X is the client zoinx, or 1 if X is the server zoinxd.

This fd is actually a pipe. minssl-tunnel reads the cleartext from the reading end of the pipe. If len>(BUFSIZE-18) then the text will be processed in chunks of (BUFSIZE-18) bytes. Normally BUFSIZE is 4096 bytes; it is never greater than 8192. Let n be the length of the read chunk.

minssl-tunnel packs n into two little-endian bytes, which are prepended to the cleartext.

It computes a hash127mac h of these 2+n bytes (padded with zeros until the next multiple of 4).
The 2+n bytes are encrypted via RC4, and the resulting 2+n bytes of ciphertext are sent to the network.
h is a 128-bit integer, with the most significant bit always equal to 1. minssl-tunnel packs it into 16 little-endian bytes, which are sent to the network.

EOF management: If the cleartext-reading pipe closes, then minssl-tunnel sends one more message following the previous description, with n equal to zero: this is a "close" message, and its length is 18 bytes. minssl-tunnel then shuts down its network-write file descriptor.

The receiving side

minssl-tunnel reads 2 bytes of ciphertext from the network.
It decrypts these 2 bytes via RC4 and unpacks these 2 little-endian bytes into an integer n.
It reads n bytes from the network and decrypts them via RC4.
It computes a hash127mac authenticator for the 2+n deciphered bytes, padded with zeros as needed; it packs the result into 16 little-endian bytes.
It reads 16 bytes from the network.
It checks these 16 bytes against the recomputed message authenticator. If they do not match, it dies instantly, shutting down the connection.
It writes the n bytes of cleartext to a pipe communicating with the application program Y.
Y reads the clear bytes from its network-read file descriptor. This fd is 6 if Y is the client zoinx, or 0 if Y is the server zoinxd.

EOF management: If n is zero, then minssl-tunnel reads the network-read file descriptor again. If there is more data to read, it exits with an error message; if it reads EOF, the connection has been properly closed, and minssl-tunnel closes its pipe sending cleartext to Y.

Unlike SSL, minssl allows one-way closes. minssl-tunnel can close its "receiving" and its "sending" series of fds separately. It does not interfere with the way the application protocol handles EOF.

When both sides of the connection are shut down by its child or its peer, minssl-tunnel waits for its child, then exits with the same exit code - unless there has been some serious error, in which case the exit code indicates the error.