Q Protocol Specification

(first draft by aum)

Return to Q Homepage

Introduction | XML-RPC | Architecture | Commands | Example Code | Metadata | Security | Contact us

1. Introduction

This document describes details of the interfaces between the various entities in the Q network - server nodes, client nodes and client applications.

Purpose is to:

2. XML-RPC Interface

2.1. WTF? All those ugly complicated angle-brackets?!?

If you haven't come across XML-RPC before, the whole concept might seem frightening, like you've gotta write thousands of lines of code for parsing and encoding XML, and negotiate some mind-numbingly complex multi-layered protocol.

This is most certainly not the case. XML-RPC libraries are way simple to use.

XML-RPC client and server libraries are available for all major (and most minor) programming languages, and are structured in a way that hides all the intricate details and presents an extremely simple and quickly learnable API over the top.

2.2. Why XML-RPC??

I've chosen XML-RPC as the node interface framework because:

3. Architectural Overview

The Q network is structured as a two-level hierarchy of server nodes and client nodes. Additionally, client applications are run by users, and form the human interface to Q.

Let's quickly overview the difference between these three entities:

4. Q Command Interface Description

4.1. Overview

As mentioned earlier, communication between all Q entities takes place via an XML-RPC mechanism.

This chapter describes the actual primitives which are supported by both server nodes and client nodes.

Although the primitives are the same for both server and client, the way they are actioned internally may vary.
For example, with the getItem primitive, server nodes will only look in their local content store for the item, returning either that item's data and metadata, or a failure reply. On the other hand, client nodes will try their local content store first, and if the item is not found, will look in their peer catalogues. If the item is found in a peer catalogue, the client node will then on-send getItem calls to all server nodes believed to hold that item, until or unless it retrieves a verifiable copy of that item

4.2. XML-RPC Data Types

It's possibly a good idea here to get a hold of the XML-RPC library for your favourite programming language, as well as the manual, and look up the description of data types. Also, if you're especially keen, you might like to read up on XML-RPC in general:
XML-RPC supports a canonical set of data types, which are seamlessly integrated into all its high level language implementations. A quick overview of the XML-RPC data types used in Q appears below.

XML-RPC Data Type Description
int Plain 32-bit integer
string Sequence of ASCII bytes, viewed as java.lang.String objects in java, and str objects (strings) in Python. Note that ASCII control chars, and high-bit-set chars, are highly illegal and will cause failure.
binary data Raw binary data, viewed as byte [] in java, and xmlrpclib.Binary objects in Python. This is the format used for raw content data.
list Sequence of objects, viewed as java.util.Vector in java, and list objects in Python.
struct An unordered set of (key, value) pairs. Represented as java.util.Hashtable objects in java, and dict objects in Python, (associative array in perl, ...)

4.3. General Command/Response Format

With Q's XML-RPC usage, all commands are a sequence of zero or more arguments. All responses are a struct with at least the key status, whose value, a string, is one of: Note that all commands are also implemented with an alternative entry point, one which takes a single Hashtable (struct/dict/assoc-array) argument. Refer to the javadocs for further info:

4.4. Exceptions - XML-RPC and Otherwise

In certain cases, XML-RPC calls to Q nodes may return an exception.

For instance, any attempt to invoke any primitive other than those listed below will most definitely cause an exception, because in the Q XML-RPC implementation, no provision is made for default handlers.

Apart from this, it's possible that calls to known legal methods may trigger an exception. This is not supposed to happen, and the author will be working over time to intercept all such exceptions and wrap them in appropriate response structures. But in the meantime, client app developers should catch any exceptions resulting from their XML-RPC calls and recover appropriately.

4.5. Overview of Q XML-RPC Primitives

The XML-RPC primitives supported by Q server and client nodes include:

4.6. i2p.q.ping

Overview

The i2p.q.ping primitive is used to test if a given server or client node is presently online. It can be sent by server nodes, client nodes and client apps.

Arguments

This primitive accepts no arguments, and will fail if any arguments are given.

Server Behaviour

No action on the part of the receiving server is required, apart from sending back:
KeyTypeDescription
status string "ok"
id string The node's nodeId, as a base64 string
dest string Node's destination, represented as base64 string
uptime int The number of seconds that this node has been running for
load float Current load this node is experiencing, as a float between 0.0 (no load) to 1.0 (impossibly flatlined)

Client Behaviour

Same as server.

4.7. i2p.q.hello

Overview

The i2p.q.hello primitive is sent by new server nodes to advise other existing server nodes of their existence. It is only sent by server nodes to other server nodes. It is considered an abuse for a client node to send this command.

Arguments

Server Behaviour

If the destination is valid, the receiving server will reply with:
KeyTypeDescription
status string "ok"
If the destination is invalid, the receiving server will send back:
KeyTypeDescription
status string "error"
error string "baddest"

Client Behaviour

i2p.q.hello calls to clients are illegal. Client nodes receiving such calls will respond with:
KeyTypeDescription
status string "error"
error string "unimplemented"

4.8. i2p.q.getItem

Overview

The i2p.q.getItem primitive is used to attempt retrieval of an item of content from a client or server node.

Arguments

Server Behaviour

Servers receiving this command will only search their own datastore for the item. They will never attempt to on-request this item from other servers.

If the server possesses the requested item in its datastore, it will respond with:
KeyTypeDescription
status string "ok"
metadata struct A nested struct, containing the metadata for the key. (Refer section on metadata).
data binary data The raw data.
If the server doesn't possess the data, it will respond with:
KeyTypeDescription
status string "error"
error string "notfound"

Client Behaviour

If the client possesses the key in its own local datastore, it will send back the full data immediately:
KeyTypeDescription
status string "ok"
metadata struct A nested struct, containing the metadata for the key. (Refer section on metadata).
data binary data The raw data.
If the client doesn't possess the key, it will search its internal catalogues for a server which does have the key.

If one or more servers possessing the key are found, the client will on-send an i2p.q.getItem command to each of those servers in turn, until it either successfully retrieves the data, or fails.

If the client successfully retrieves the data from one or more of its servers, it will add the data to its internal cache, and reply with the above success response.

If the client was unable to source the complete data from any of its servers, it will reply with:
KeyTypeDescription
status string "error"
error string "notfound"

4.9. i2p.q.putItem

Overview

The i2p.q.putItem primitive is used by client nodes to insert a new item of content onto a server node.

It is also used by client apps to insert a new item onto their client node.

Also, if a server node is receiving a high traffic of requests for a given item, it may at its discretion send i2p.q.putItem commands to peer servers to mirror the item on those servers, and spread the load.

Arguments

Server Behaviour

If the server successfully received and stored the data (and optionally provided metadata), it will reply with:
KeyTypeDescription
status string "ok"
key string The base64 key under which this item has been stored, and which should be used for any subsequent i2p.q.getItem requests for that item within the Q network.
However, if the server's datastore is full, the server will not be able to store this item, in which case it will respond with:
KeyTypeDescription
status string "error"
error string "storefull"

Client Behaviour

Client nodes receiving this command will attempt to store the item in their own datastore, and respond immediately with one of the above server responses.

In addition, client nodes will enqueue a background job to upload this item to one or more selected server nodes.

4.10. i2p.q.getUpdate

Overview

The i2p.q.getUpdate primitive is used to request a differential peers list update (which optionally can include a catalog update as well).

Client apps invoke this primitive on client nodes to get up-to-date listings of items available in the network. Note that client apps will not hand over any peers list.

Client nodes periodically schedule a background job to invoke this primitive on their known servers, such that they keep the most recent possible view of available data and other servers.

Arguments

Server Behaviour

On receiving this command, a server node will send back lists of metadata records for all new content (and/or all new peers) it has become aware of since the given date. The full response is formatted as follows:
KeyTypeDescription
status string "ok"
items list A list of metadata records for new items. Refer to the section on metadata for more information. If the server has not become aware of any new data since the given date (or if the includeCatalog argument was 0), this list will be empty.
peers list A list of destinations of new peers. If the server has not discovered any new peers since the given date (or if the includePeers argument was 0), this list will be empty.
timeUpdateEnds int unixtime in secs that this update ends. The peer receiving this response should note this time, and quote it as the since argument in the next getUpdate request
timeNextContact int Advised time (unixtime in sec) for sending the next getUpdate command. The sending peer should not issue any getCatalog commands before this time, but is welcome to issue them after this time. The actual time value is guesstimated by the server node, depending on its current load.

4.11. i2p.q.search

Overview

The i2p.q.search primitive is invoked by client apps to search a client node for data items matching a set of criteria.
Only client nodes support this primitive. Server nodes will return an empty result set and an error response.

Arguments

Server Behaviour

Servers receiving this command will send back an error response:
KeyTypeDescription
status string "error"
error string "unimplemented"

Client Behaviour

Client nodes receiving this command will send back the following response:
KeyTypeDescription
status string "ok"
items vector A list of metadata records (Hashtables) for items which match the given search criteria, and are retrievable through this client node (ie, the client node either possesses the item, or knows one or more servers which possess the item).

5. Client Program Examples

5.1. Overview

This section provides a couple of simple examples of client app programming.

At present, only Python and Java examples are given.

(If you don't know either of these languages, you should be able to get the general drift by studying the examples, sufficient to map the concepts to the XML-RPC API available to your preferred language.)

The examples below communicate with a client node XML-RPC server (running on the local machine and listening on its default port of 7651), and perform simple operations of data insertion, catalog fetching and data retrieval.

5.2. Java Example

To run this example, you'll need: Now for the code (heavily annotated, so you don't necessarily need to know or understand Java), which should be written to a source file called QDemo.java. Note that this client would be a significantly shorter if it instantiated a QClientNode class directly and invoked its methods, but that is not what we're showing here - we're demonstrating the use of the client node's XML-RPC interface.
// QDemo.java
//
// A simple demo example of a Q client application, which 
// communicates with a running Q client node on the local
// machine via its TCP XML-RPC interface
//
// If your client node is not running on localhost, or
// if it's listening on a port other than the default
// 7651, you'll need to change the code below.
//
// Note that this demo is bloated by the fact we're using
// raw XML-RPC.
// 
// The following exercises are left to the reader:
//  1. Modify this app so that instead of using the XML-RPC
//     interface, it instantiates a QClientNode, and
//     invokes its methods directly.
//  2. Write a thin wrapper class which instantiates an XML-RPC
//     client, and offers simpler access methods (thus avoiding
//     the need to create and populate Vectors of args before
//     calling, and pick through a reply Hashtable after the call),
//     and create a version of this demo which uses the wrapper.

// pull in some standard java stuff
import java.*;
import java.lang.*;
import java.util.*;
import java.net.*;
import java.io.*;

// pull in some xml-rpc stuff
import org.apache.xmlrpc.*;

// since we're talking to the node via xmlrpc, and talking to
// it in a separate VM, we don't need to import any Q packages

// Define a minimal demo class, which kust defines a
// main method enabling us to run the demo from a shell.
//
// For the purposes of this demo, we're assuming that your Q client node is
// running on your local machine, and that you haven't altered the
// listening port (default 7651) for the client's XML-RPC interface.

public class QDemo {

    // just define a main so we can run this from a shell
    static public void main(String [] args)
        throws MalformedURLException, XmlRpcException, IOException
    {
        // for getting and analysing replies from node
        Hashtable result;
        String status;

        // Create a new client app object
        XmlRpcClient myClient = new XmlRpcClient("http://127.0.0.1:7651");

        // -------------------------------------
        // First action - execute a 'ping' on this peer
        // -------------------------------------

        Vector noArgs = new Vector();
        result = (Hashtable)myClient.execute("i2p.q.ping", noArgs);
        print("ping: result=" + result);

        // -------------------------------------
        // Second action - insert an item of data
        // -------------------------------------

        // mark the current time, we'll use this later
        Integer then = new Integer((int)(new Date().getTime() / 1000));

        // create metadata
        // (note from previous chapter that metadata is optional)
        Hashtable meta = new Hashtable();
        meta.put("type", "text");
        meta.put("abstract", "a simple piece of demo data");
        meta.put("mimetype", "text/plain");

        // create some data
        String data = "Hello, world";

        // set up the arguments list
        Vector insertArgs = new Vector();
        insertArgs.addElement(meta);
        insertArgs.addElement(data.getBytes()); // must insert data as byte[]
            
        // and do the insert
        result = (Hashtable)myClient.execute("i2p.q.putItem", insertArgs);
        print("putItem: result=" + result);

        // check what happened
        status = (String)result.get("status");
        String key;
        if (status.equals("ok")) {
            // insert succeeded
            key = (String)result.get("key");
            print("Insert successful");
        } else {
            // insert failed, bail
            print("Insert failed: error=" + (String)result.get("error"));
            return;
        }

        // -------------------------------------
        // Third action - check for catalog updates
        // (which should include what we've just inserted)
        // -------------------------------------

        // create an args list, with just the date we noted before the insert
        Vector updateArgs = new Vector();
        updateArgs.addElement(then);
        // add the flags
        updateArgs.addElement(new Integer(0));   // 'includePeers'
        updateArgs.addElement(new Integer(1));   // 'includeCatalog'

        // execute the 'getCatalog'
        result = (Hashtable)myClient.execute("i2p.q.getUpdate", updateArgs);
        print("getUpdate: result="+result);

        // pick out the results, and search for what we just inserted
        int i;
        Vector items = (Vector)result.get("items");
        int nitems = items.size();
        boolean foundit = false;
        for (i = 0; i < nitems; i++) {
            // get the nth item
            Hashtable metaRec = (Hashtable)items.get(i);
            String thisKey = (String)metaRec.get("key");
            if (thisKey.equals(key)) {
                // yay, got it!
                foundit = true;
                break;
            }
        }

        // did we get it?
        if (!foundit) {
            print("wtf? we inserted it but it's not in the catalog!");
            return;
        }

        // yep, we got it, so try to retrieve it back
        Vector getArgs = new Vector();
        getArgs.addElement(key);
        result = (Hashtable)myClient.execute("i2p.q.getItem", getArgs);
        print("getItem: result=" + result);

        // did we get it?
        status = (String)result.get("status");
        if (!status.equals("ok")) {
            print("getItem failed: " + (String)result.get("error"));
            return;
        }

        // yep, got it
        byte [] binData = (byte [])result.get("data");
        String strData = new String(binData);
        print("getItem: success, data='"+strData+"'");

        print("--- END OF Q CLIENT DEMO ---");
    }

    // a convenient shorthand method for printing stuff to stdout
    static void print(String msg) {
        System.out.println(msg);
    }
}
        

5.3. Python Example

To run this example, you will need a running I2P installation, including a running instance of a Q client node.

Note that, in contrast to Java, Python 2.3 and later have all the necessary XML-RPC libraries built in.
Now for some code (again, heavily annotated). This, together with the previous example, present an interesting comparison between some of Java and Python's ways of doing things.
#!/usr/bin/env python
"""
QDemo.py

A simple demo example of a Q client application, which 
communicates with a running Q client node on the local
machine via its TCP XML-RPC interface

If your client node is not running on localhost, or
if it's listening on a port other than the default
7651, you'll need to change the code below.

Note that this demo is bloated by the fact we're using
raw XML-RPC.

The following exercise is left to the reader:
 * Write a thin wrapper class which instantiates an XML-RPC
   client, and offers simpler access methods (thus avoiding
   the need to pick through a reply dict after the call),
   and create a version of this demo which uses the wrapper.
"""

# a coupla needed imports
from time import time
from xmlrpclib import ServerProxy, Binary

# For the purposes of this demo, we're assuming that your Q client node is
# running on your local machine, and that you haven't altered the
# listening port (default 7651) for the client's XML-RPC interface.

def qdemo():
    # Create a new client app object
    myClient = ServerProxy("http://127.0.0.1:7651")

    # -------------------------------------
    # First action - execute a 'ping' on this peer
    # -------------------------------------

    result = myClient.i2p.q.ping()
    print "ping: result=%s" % result

    # -------------------------------------
    # Second action - insert an item of data
    # -------------------------------------

    # mark the current time, we'll use this later
    then = int(time())

    # create metadata
    # (note from previous chapter that metadata is optional)
    meta = {
        "type" : "text",
        "abstract" : "a simple piece of demo data",
        "mimetype" : "text/plain",
        }

    # create some data, and binary-wrap it
    data = "Hello, world"
    binData = Binary(data)

    # and do the insert
    result = myClient.i2p.q.putItem(meta, binData)
    print "putItem: result=%s" % result

    # check what happened
    if result["status"] == "ok":
        # insert succeeded
        key = result["key"]
        print "Insert successful"
    else:
        # insert failed, bail
        print "Insert failed: error=%s" % result['error']
        return;

    # -------------------------------------
    # Third action - check for catalog updates
    # (which should include what we've just inserted)
    # -------------------------------------

    # execute the 'getUpdate'
    result = myClient.i2p.q.getUpdate(then, 0, 1)
    print "getUpdate: result=%s" % result

    # pick out the results, and search for what we just inserted
    foundit = False
    for metaRec in result['items']:
        if metaRec['key'] == key:
            # yay, got it!
            foundit = True
            break

    # did we get it?
    if not foundit:
        print "wtf? we inserted it but it's not in the catalog!"
        return;

    # yep, we got it, so try to retrieve it back
    print "getCatalog: found the item we just inserted"
    result = myClient.i2p.q.getItem(key)
    print "getItem: result=%s" % result

    # did we get it?
    if result["status"] != "ok":
        print "getItem failed: %s" + result["error"]
        return;

    # yep, got it (note that data is an xmlrpclib.Binary object,
    # and the raw data we want is in its .data attribute)
    print "getItem: success, data='%s'" % result['data'].data

    print "--- END OF Q CLIENT DEMO ---"

# run the demo func if this script is executed directly
if __name__ == '__main__':
    qdemo()
        

6. Keys and Metadata

6.1. Overview

Like Freenet, content is stored in Q as (data, metadata) pairs.

However, there's a difference. On Freenet, metadata is stored as a string of up to 32k length, and must be parsed (and sometimes executed) by client code. On the other hand, metadata is exposed in Q as an XML-RPC struct (Java Hashtable or Properties object, or Python dict, or Perl associative array etc).

If a content item gets inserted to the Q network without metadata, a minimal metadata set will be transparently generated, and is guaranteed to contain at least the following elements:

KeyTypeDescription
size int Size of the stored data item, in bytes
dataHash string a base64 representation of the SHA256 hash of the full raw data, using the I2P base64 alphabet


6.2. Node IDs

When Q nodes are first created, they generate themselves a random I2P privKey/dest keypair using the in-I2P services.

The I2P destination gets converted to what we call a Q Node ID, as follows:

6.3. Keys

Here, 'key' means the unique short string, by which items of content can be retrieved, and which is returned from an i2p.q.putItem command.

Like Freenet's CHK@ keytype, Q keys are hashes of the key's content and metadata.

The recipe for calculating the 'key' of a particular item of metadata+data is:
  1. If no metadata is submitted with the data, create a minimal metadata as per above
  2. Serialise out the metadata into a string representation, with the fieldnames in alphanumeric order. The format of such string is one line per metadata field/value pair per line, in the format:
    metadatakeyname=metadatakeyvalue\n
  3. Calculate the binary SHA1 digest of this serialised metadata string
  4. Base64-encode this binary digest via the I2P Base64 alphabet

6.4. Q Metadata Conventions

Additional to the core metadata defined above, there is a convention in Q that the following optional extra metadata keys be provided on insert, and recognised and honoured on retrieve.

It is highly recommended that these keys be included in metadata when content is inserted:

KeyTypeDescription
title string A short and descriptive title for the item, preferably formatted as a filename which is legal and convenient on all main operating systems, ie, containing only alphanumerics, '-', '_' and '.'.

It is highly advisable that an appropriate file extension appear at the end of the title. Refer to the Security Considerations section below.
It is expected that client applications will use this title field when displaying available content lists to users.
type string Generic type of material, using the following superset of the eMule/Donkey classifications:
  • text
  • html
  • image
  • audio
  • video
  • software
  • archive
  • misc
mimetype string A recognised mime-type, as per RFC1341, RFC1521, RFC1522, such as audio/mpeg, text/plain etc.

This will help client app developers devise ways of disposing with data items they request from client nodes.

For instance, client apps with http front ends may send back this mimetype as the value of the Content-type: header, (and possibly take preventative action with potentially hazardous mimetypes, such as those which some browsers such as IE might trust and execute blindly as binary code).

Alternatively, gui-based or cli-based client apps may convert this mimetype to an appropriate benign file extension (such as .txt, .ogg, .jpg etc). See Security Considerations below.
keywords string A set of space-separated keywords describing this item, intended for human reading, as well as automatic parsing by client apps.
abstract string A short descriptive summary of the nature of the data, intended for human reading, as well as automatic pattern matching searches by client apps.



6.5. One Data Item, Many Metadata Sets?

It is perfectly possible, and legal, for one item of data to be referenced by two completely different items of metadata.

Since content keys are a hash of metadata, which in turn contains a hash of the data, then two pieces of metadata referencing the same data item, but containing different metadata values, will end up with different keys.

So as far as key addresses go, there will be a many-to-1 relationship between raw content keys, and the data returned under these keys.


7. Security Considerations

All Peer2Peer software (as with all networked software in general) carries with it a set of devastating security risks which should be respected to the utmost.

This applies in no small part to Q.

So this brief sermon is addressed to anyone writing any client applications or APIs talking to the Q network.

Any material which involves the execution of code on a client machine is risky. However, much of the risk can be managed if the code is open source and peer-reviewed.

Perhaps the biggest issue as far as Q is concerned is this:
Client app developers should never, NEVER implicitly trust incoming content, and should always assume that malicious remote users will insert content which attempts to compromise other users' systems.
If a Q client app wants to offer filetype-specific support, then perhaps a good strategy is for the client app to use a whitelist of known low-risk file extensions, such as .txt, or (possibly) .ogg, .png etc. Recall that in some Windows configurations, even .jpg can carry an arbitrary code execution attack!

Note that .html (text/html) is especially dangerous, and should be respected accordingly.

Support for .html could be a real boon. For instance, it could allow I2P users to publish an I2P equivalent of freenet's freesites - static HTML websites which are accessible even when the author goes offline.

However, if a client app chooses to recognise .html, it should either use a code-screening mechanism like freenet's fproxy and keep it up to date with all the latest advisories, or use a mandatory-proxy mechanism like I2P's eepProxy.

One possiblility is to serve up such content via a totally in-I2P http interface, such that Joe can view the content via his regular eeproxy-configured browser.

This is a typical case where security and ease/convenience can end up in direct conflict. Automatic handling of content according to data type is great from a Joe Sixpack Windows User point of view, but it is a snake-pit of risks that can potentially result in any of the following (or worse): The crux of this lecture is that client app writers have a huge responsibility to ensure their apps are safe against malicious content.

Perhaps the best and most practical solution is to just store downloaded material into a directory known to and owned by the user, and make it the user's task and responsibility to manually copy materials out of this directory and take responsibility for how s/he uses this content thereafter.


8. Contacting the Author

I am aum, and can be reached as aum on in-I2P IRC networks, and also at the in-I2P email address of aum@mail.i2p.


Introduction | XML-RPC | Architecture | Commands | Example Code | Metadata | Security | Contact us


Last modified: Sat Apr 2 13:31:08 NZST 2005