i2p.www/www.i2p2/pages/streaming.html

{% extends "_layout.html" %}
{% block title %}Streaming Lib{% endblock %}
{% block content %}
<h2>Overview</h2>

<p>
The streaming library is technically part of the "application" layer,
as it is not a core router function.
In practice, however, it provides a vital function for almost all
existing I2P applications, by providing a TCP-like
streams over I2P, and allowing exising apps to be easily ported to I2P.
</p>

<p>The streaming library is a layer on top of the core 
<a href="i2cp">I2CP</a> that allows reliable, in order, and authenticated streams
of messages to operate across an unreliable, unordered, and unauthenticated 
message layer.  Just like the TCP to IP relationship, this streaming 
functionality has a whole series of tradeoffs and optimizations available, but
rather than embed that functionality into the base I2P code, it has been factored
off into its own library both to keep the TCP-esque complexities separate and to
allow alternative optimized implemenations.</p>

<h2>History</h2>
<p>
The streaming library has grown organically for I2P - first mihi implemented the
"mini streaming library" as part of I2PTunnel, which was limited to a window
size of 1 message (requiring an ACK before sending the next one), and then it was
refactored out into a generic streaming interface (mirroring TCP sockets) and the
full streaming implementation was deployed with a sliding window protocol and 
optimizations to take into account the high bandwidth x delay product.  Individual
streams may adjust the maximum packet size and other options. The default
message size is selected to fit precisely in two 1K I2NP tunnel messages,
and is a reasonable tradeoff between the bandwidth costs of 
retransmitting lost messages and the latency of multiple messages.
</p>

<p>
In addition, in consideration of the relatively high cost of subsequent messages, 
the streaming library's protocol for scheduling and delivering messages has been optimized to
allow individual messages passed to contain as much information as is available.
For instance, a small HTTP transaction proxied through the streaming library can
be completed in a single round trip - the first messages bundle a SYN, FIN, and
the small HTTP request payload, and the reply bundles the SYN,
FIN, ACK, and the HTTP response payload.  While an additional
ACK must be transmitted to tell the HTTP server that the SYN/FIN/ACK has been
received, the local HTTP proxy can often deliver the full response to the browser 
immediately.  
</p>

<p>
On the whole, however, the streaming library bears much resemblance to an 
abstraction of TCP, with its sliding windows, congestion control algorithms
(both slow start and congestion avoidance), and general packet behavior (ACK,
SYN, FIN, RST, rto calculation, etc).  
</p>


<h2>Usage</h2>
<p>
The standard interface to the streaming lib is for the application to set up
a I2PSocketManagerFactory from the <a href="ministreaming.html">ministreaming lib</a>.
Only I2PSocketManagerFactory is used here - everything else is from the full streaming lib
(I2PSocketManagerFull, not I2PSocketManagerImpl, and I2PSocketFull, not I2PSocketImpl).
The remainder of the ministreaming lib is not normally used - don't be confused.
</p>

<p>
For a good example of usage, see the i2psnark code.
</p>


<h2>Advantages</h2>
<p>The streaming lib has many advantages over the ministreaming library written by mihi as a part of his 
<a href="i2ptunnel">I2PTunnel</a> application.
The streaming library is
a more robust streaming library
which is further optimized for operation over I2P.  The two main issues with 
the ministreaming library are its use of the traditional TCP two phase 
establishment protocol and the current fixed window size of 1.
The streaming lib fixes both of these issues - it has a one-phase setup, and
it contains a full windowing implementation.
</p>

<p>
Significant tuning of the streaming lib parameters,
greatly increasing outbound performance, was implemented in 0.6.1.28.
Subsequent releases include additional tuning and bug fixes.

<h2>Default Parameters</h2>
The current default values are listed below.
Lower case values are streaming lib parameters that can changed on a
per-connection basis.
These values are tuned for HTTP performance over typical I2P conditions. Other applications such
as peer-to-peer services are strongly encouraged to
modify as necessary, by setting the options and passing them via the call to
I2PSocketManagerFactory.createManager(_i2cpHost, _i2cpPort, opts).
Time values are in ms.
<ul>
<li>MIN_RESEND_DELAY = 2*1000
<li>MAX_RESEND_DELAY = 45*1000
<li>i2p.streaming.connectTimeout = 5*60*1000
<li>i2p.streaming.initialReceiveWindow = 1
<li>i2p.streaming.initialWindowSize = 12
<li>MIN_WINDOW_SIZE = 1
<li>i2p.streaming.maxWindowSize = 128 // as of release 0.6.3 (was 64)
<li>TREND_COUNT = 3
<li>i2p.streaming.maxResends = 8
<li>RTT_DAMPENING = 0.875 // as of release 0.6.5 (was 0.9)
<li>i2p.streaming.profile = 1 (bulk) (2=interactive not suported)
<li>MIN_MESSAGE_SIZE = 512 // as of release 0.6.5
<li>i2p.streaming.maxMessageSize = 1730 // as of release 0.6.5 (was 960)
<li>INBOUND_BUFFER_SIZE = maxMessageSize * (maxWindowSize + 2)
<li>i2p.streaming.initialRTT = 10*1000
<li>INITIAL_TIMEOUT = 1.5 * initialRTT
<li>i2p.streaming.initialResendDelay = 1000
<li>i2p.streaming.initialAckDelay = 2000
<li>i2p.streaming.inactivityTimeout = 90*1000
<li>i2p.streaming.inactivityAction = 2 (send) (0=noop, 1=disconnect)
<li>i2p.streaming.congestionAvoidanceGrowthRateFactor = 1
<li>i2p.streaming.slowStartGrowthRateFactor = 1
<li>PASSIVE_FLUSH_DELAY = 250 // as of release 0.6.5 (was 500)
<li>i2p.streaming.answerPings = true // new option as of release 0.7.7
</ul>
</p>

<p>
The streaming lib uses standard slow-start (exponential window growth) and congestion avoidance (linear window growth)
phases. However, before the 0.6.1.33 release, window growth was substantially slower than optimal;
these issues were fixed in release 0.6.1.33.
</p>

<p>
The maximum message size (also called the MTU / MRU) is negotiated to the lower value supported by
the two peers. As tunnel messages are padded to 1KB, a poor MTU selection will lead to
a large amount of overhead.
The MTU is chosen to fit precisely in an integral number of 1K I2NP tunnel messages,
including overhead for the typical case.
The first message in a connection includes a 387 byte (typical) Destination added by the streaming layer,
and usually a 898 byte (typical) LeaseSet bundled in the Garlic message.
Therefore, the goal of fitting a complete HTTP request in a single 1KB I2NP message is not realistic.
However, the selection of the MTU, together with careful implementation of fragmentation
and batching strategies in the tunnel gateway procesor, are important factors in network bandwidth,
latency, reliability, and efficiency, especially for long-lived connections.
</p>

<p>
The interaction of the routing algorithms with the streaming lib strongly affects performance.
In particular, random distribtion of messages to multiple tunnels in a pool
leads to a high degree of out-of-order delivery which results in smaller window
sizes than would otherwise be the case.
In release 0.6.1.30, the routing of messages to the outbound tunnels was made
consistent, with pushback when a tunnel was backlogged.
This had a significant positive impact on bandwidths.
The pushback code was reverted in release 0.6.1.31 due to anonymity concerns.
Consistent message routing to inbound tunnels
was implemented in release 0.6.1.32.
</p>

<p>
Another area for research is the interaction of the streaming lib with the
NTCP and SSU transport layers.
See <a href="ntcp.html">the NTCP page</a> for a discussion.
</p>


<h2>Packet Format</h2>
<p>
Here is the format of a single packet transferred as part of a streaming connection.  
<table>
<tr><th>Field<th>Length<th>Contents
<tr><td>endStreamId <td>4 byte value<td>Random number selected by the connection recipient
and constant for the life of the connection.
0 in the SYN message sent by the originator.
<tr><td>receiveStreamId <td>4 byte value<td>Random number selected by the connection originator
and constant for the life of the connection.

<tr><td>sequenceNum <td>4 byte unsigned integer<td>
The sequence for this message, starting at 0 in the SYN message,
and incremented by 1 in each message except for plain ACKs and retransmissions.
If the sequenceNum is 0 and the SYN is not set, this is a plain ACK 
packet that should not be ACKed.

<tr><td>ackThrough <td>4 byte unsigned integer<td>
The highest packet sequence number that was received
on the receiveStreamId.  This field is ignored on the initial
connection packet (where receiveStreamId is the unknown id) or
if FLAG_NO_ACK is set.
All packets up to and including this sequence number are acked,
EXCEPT for those listed in NACKs below.

<tr><td>number of NACKs <td>1 byte unsigned integer<td>

<tr><td>that many NACKs <td>n * 4 byte unsigned integers<td>
Sequence numbers less than ackThrough that are not yet received.

<tr><td>resendDelay <td>1 byte unsigned integer<td>
How long is the creator of this packet going to wait before
resending this packet (if it hasn't yet been ACKed).  The 
value is seconds since the packet was created.
Ignored on receive. Broken on send before release 0.7.8 (the sender did not divide by 1000,
and the default is 1000 ms, so the included value was 1000 &amp 0xff = 0xe8 = 232 seconds.

<tr><td>flags <td>2 byte value<td>
See below.

<tr><td>option data size <td>2 byte unsigned integer<td>
See below.

<tr><td>option data specified by those flags <td>0 or more bytes<td>
See below.

<tr><td>payload <td>remaining packet size<td>
</table>

<p>The flags field above specifies some metadata about the packet, and in
turn may require certain additional data to be included.  The flags are
as follows (with any data structures specified added to the options area
in the given order):</p>
<table>
<tr><th>Bit Number<th>Flag<th>Option Data<th>Function
<tr><td>0<td>FLAG_SYNCHRONIZE<td>no option data<td>
Similar to TCP SYN.
<tr><td>1<td>FLAG_CLOSE<td>no option data<td>
Similar to TCP FIN. If the response to a SYN fits in a single message, the response
will contain both FLAG_SYNCHRONIZE and FLAG_CLOSE.
<tr><td>2<td>FLAG_RESET<td>no option data<td>
Abnormal close.
<tr><td>3<td>FLAG_SIGNATURE_INCLUDED<td>40 bytes<td>net.i2p.data.Signature
Typically sent only with FLAG_SYNCHRONIZE.
If the signature is included, it uses the Destination's DSA key 
to sign the entire header and payload with the space in the options 
for the signature being set to all zeroes.
<td>
<tr><td>4<td>FLAG_SIGNATURE_REQUESTED<td>no option data<td>
Unused. Requests every packet in the other direction to have FLAG_SIGNATURE_INCLUDED
<td>
<tr><td>5<td>FLAG_FROM_INCLUDED<td>typ. 387 bytes<td>net.i2p.data.Destination
Typically sent only with FLAG_SYNCHRONIZE.
<tr><td>6<td>FLAG_DELAY_REQUESTED<td>2 byte integer<td>
Optional delay.
How many milliseconds the sender of this packet wants the recipient
to wait before sending any more data
<td>
<tr><td>7<td>FLAG_MAX_PACKET_SIZE_INCLUDED<td>2 byte integer<td>
Apparently always included, could be optimized to only send with a SYN
<td>
<tr><td>8<td>FLAG_PROFILE_INTERACTIVE<td>no option data<td>
Apparently unused or ignored
<td>
<tr><td>9<td>FLAG_ECHO<td>no option data<td>
Unused except by ping programs
<tr><td>10<td>FLAG_NO_ACK<td>no option data<td>
Apparently unused, an ack is always included.
This flag simply tells the recipient to ignore the ackThrough field in the header.
<tr><td>11-15<td>unused<td><td>
</table>



<h2>Control Block Sharing</h2>
<p>
As of release 0.7.1, the streaming lib supports "TCP" Control Block sharing.
This shares two important streaming lib parameters
(window size and round trip time)
across connections to the same remote peer.
This is used for "temporal" sharing at connection open/close time,
not "ensemble" sharing during a connection (See RFC 2140).
There is a separate share per ConnectionManager (i.e. per local Destination)
so that there is no information leakage to other Destinations on the
same router.
</p>

<h2>Future Work and Proposals</h2>
<p>
There are proposals to replace the streaming lib with standard TCP
(or perhaps a null layer together with raw sockets).
This would unfortunately be incompatible with the streaming lib
but it would be good to compare the performance of the two.
</p>

{% endblock %}