For example, with the getItem primitive, server nodes will only look in their local content store for the item, returning either that item's data and metadata, or a failure reply. On the other hand, client nodes will try their local content store first, and if the item is not found, will look in their peer catalogues. If the item is found in a peer catalogue, the client node will then on-send getItem calls to all server nodes believed to hold that item, until or unless it retrieves a verifiable copy of that item
XML-RPC supports a canonical set of data types, which are seamlessly integrated into all its high level language implementations. A quick overview of the XML-RPC data types used in Q appears below.It's possibly a good idea here to get a hold of the XML-RPC library for your favourite programming language, as well as the manual, and look up the description of data types. Also, if you're especially keen, you might like to read up on XML-RPC in general:
| XML-RPC Data Type | Description |
| int | Plain 32-bit integer |
| string | Sequence of ASCII bytes, viewed as java.lang.String objects in java, and str objects (strings) in Python. Note that ASCII control chars, and high-bit-set chars, are highly illegal and will cause failure. |
| binary data | Raw binary data, viewed as byte [] in java, and xmlrpclib.Binary objects in Python. This is the format used for raw content data. |
| list | Sequence of objects, viewed as java.util.Vector in java, and list objects in Python. |
| struct | An unordered set of (key, value) pairs. Represented as java.util.Hashtable objects in java, and dict objects in Python, (associative array in perl, ...) |
In certain cases, XML-RPC calls to Q nodes may return an exception.
For instance, any attempt to invoke any primitive other than those listed below will most definitely cause an exception, because in the Q XML-RPC implementation, no provision is made for default handlers.
Apart from this, it's possible that calls to known legal methods may trigger an exception. This is not supposed to happen, and the author will be working over time to intercept all such exceptions and wrap them in appropriate response structures. But in the meantime, client app developers should catch any exceptions resulting from their XML-RPC calls and recover appropriately.
Overview
The i2p.q.ping primitive is used to test if a given server or client node is presently online. It can be sent by server nodes, client nodes and client apps.Arguments
This primitive accepts no arguments, and will fail if any arguments are given.Server Behaviour
No action on the part of the receiving server is required, apart from sending back:
Key Type Description status string "ok" id string The node's nodeId, as a base64 string dest string Node's destination, represented as base64 string uptime int The number of seconds that this node has been running for load float Current load this node is experiencing, as a float between 0.0 (no load) to 1.0 (impossibly flatlined) Client Behaviour
Same as server.
Overview
The i2p.q.hello primitive is sent by new server nodes to advise other existing server nodes of their existence. It is only sent by server nodes to other server nodes. It is considered an abuse for a client node to send this command.Arguments
- destination (string) - the base64 representation of the calling node's I2P destination (on which the calling node's in-I2P XML-RPC server may be subsequently reached). Same format as the I2P hosts.txt listing.
Server Behaviour
If the destination is valid, the receiving server will reply with:
Key Type Description status string "ok" If the destination is invalid, the receiving server will send back:
Key Type Description status string "error" error string "baddest" Client Behaviour
i2p.q.hello calls to clients are illegal. Client nodes receiving such calls will respond with:
Key Type Description status string "error" error string "unimplemented"
Overview
The i2p.q.getItem primitive is used to attempt retrieval of an item of content from a client or server node.Arguments
- key (string) - the base64 key under which the item in question is stored
Server Behaviour
Servers receiving this command will only search their own datastore for the item. They will never attempt to on-request this item from other servers.
If the server possesses the requested item in its datastore, it will respond with:
Key Type Description status string "ok" metadata struct A nested struct, containing the metadata for the key. (Refer section on metadata). data binary data The raw data. If the server doesn't possess the data, it will respond with:
Key Type Description status string "error" error string "notfound" Client Behaviour
If the client possesses the key in its own local datastore, it will send back the full data immediately:
Key Type Description status string "ok" metadata struct A nested struct, containing the metadata for the key. (Refer section on metadata). data binary data The raw data. If the client doesn't possess the key, it will search its internal catalogues for a server which does have the key.
If one or more servers possessing the key are found, the client will on-send an i2p.q.getItem command to each of those servers in turn, until it either successfully retrieves the data, or fails.
If the client successfully retrieves the data from one or more of its servers, it will add the data to its internal cache, and reply with the above success response.
If the client was unable to source the complete data from any of its servers, it will reply with:
Key Type Description status string "error" error string "notfound"
Overview
The i2p.q.putItem primitive is used by client nodes to insert a new item of content onto a server node.
It is also used by client apps to insert a new item onto their client node.
Also, if a server node is receiving a high traffic of requests for a given item, it may at its discretion send i2p.q.putItem commands to peer servers to mirror the item on those servers, and spread the load.Arguments
- data - (binary) - the raw data to insert. Refer earlier - the compatible Java datatype is byte[], and Python datatype is xmlrpclib.Binary.
- metadata - (struct) - optional - a struct of metadata to insert alongside the data. If this is not given, a minimal metadata set will be automatically created by the recipient. See the section on metadata.
Server Behaviour
If the server successfully received and stored the data (and optionally provided metadata), it will reply with:
Key Type Description status string "ok" key string The base64 key under which this item has been stored, and which should be used for any subsequent i2p.q.getItem requests for that item within the Q network. However, if the server's datastore is full, the server will not be able to store this item, in which case it will respond with:
Key Type Description status string "error" error string "storefull" Client Behaviour
Client nodes receiving this command will attempt to store the item in their own datastore, and respond immediately with one of the above server responses.
In addition, client nodes will enqueue a background job to upload this item to one or more selected server nodes.
Overview
The i2p.q.getUpdate primitive is used to request a differential peers list update (which optionally can include a catalog update as well).
Client apps invoke this primitive on client nodes to get up-to-date listings of items available in the network. Note that client apps will not hand over any peers list.
Client nodes periodically schedule a background job to invoke this primitive on their known servers, such that they keep the most recent possible view of available data and other servers.
Arguments
- since - (int) - unix time in seconds to update from. The recipient will send back a list of all content it has become aware of since this time.
- includePeers - (int) - set to 1 to include peer list update in the return data, 0 to omit.
- includeCatalog - (int) - set to 1 to include catalog update in the return data, 0 to omit.
Server Behaviour
On receiving this command, a server node will send back lists of metadata records for all new content (and/or all new peers) it has become aware of since the given date. The full response is formatted as follows:
Key Type Description status string "ok" items list A list of metadata records for new items. Refer to the section on metadata for more information. If the server has not become aware of any new data since the given date (or if the includeCatalog argument was 0), this list will be empty. peers list A list of destinations of new peers. If the server has not discovered any new peers since the given date (or if the includePeers argument was 0), this list will be empty. timeUpdateEnds int unixtime in secs that this update ends. The peer receiving this response should note this time, and quote it as the since argument in the next getUpdate request timeNextContact int Advised time (unixtime in sec) for sending the next getUpdate command. The sending peer should not issue any getCatalog commands before this time, but is welcome to issue them after this time. The actual time value is guesstimated by the server node, depending on its current load. 4.11. i2p.q.search
Overview
The i2p.q.search primitive is invoked by client apps to search a client node for data items matching a set of criteria.
Only client nodes support this primitive. Server nodes will return an empty result set and an error response.Arguments
- criteria - (hashtable) - a set of metadata criteria to match. Each key in this hashtable is a metadata key (eg title, type etc), and the corresponding value is a regular expression string to match. Regular expression syntax is documented in the java API in the section on class 'Pattern'.
The search criteria work 'AND-style', in that if more than one metadata key match pattern is given, then only items matching all of the given criteria will be returned.
Python example (using XML-RPC proxy - see code samples below):Java Example (using XML-RPC proxy - see code examples below):result = mynode.i2p.q.search({"type":"text", "summary":"^War.*"}) metaRecs = result['items']Note that if the criteria argument is empty (no keys/values), then the client node will send back metadata for every item of content it knows of, which (depending on the size of the Q network), could be quite a resource-hungry operation.Hashtable criteria = new Hashtable(); criteria.put("type", "text"); criteria.put("summary", "^War.*"); Vector args = new Vector(); args.addElement(criteria); Hashtable result = (Hashtable)mynode.execute("i2p.q.search", args); Vector metaRecs = (Vector)result.get("items");Server Behaviour
Servers receiving this command will send back an error response:
Key Type Description status string "error" error string "unimplemented" Client Behaviour
Client nodes receiving this command will send back the following response:
Key Type Description status string "ok" items vector A list of metadata records (Hashtables) for items which match the given search criteria, and are retrievable through this client node (ie, the client node either possesses the item, or knows one or more servers which possess the item).
5. Client Program Examples
5.1. Overview
This section provides a couple of simple examples of client app programming.
At present, only Python and Java examples are given.
(If you don't know either of these languages, you should be able to get the general drift by studying the examples, sufficient to map the concepts to the XML-RPC API available to your preferred language.)
The examples below communicate with a client node XML-RPC server (running on the local machine and listening on its default port of 7651), and perform simple operations of data insertion, catalog fetching and data retrieval.
5.2. Java Example
To run this example, you'll need:Now for the code (heavily annotated, so you don't necessarily need to know or understand Java), which should be written to a source file called QDemo.java. Note that this client would be a significantly shorter if it instantiated a QClientNode class directly and invoked its methods, but that is not what we're showing here - we're demonstrating the use of the client node's XML-RPC interface.
- A running I2P installation, with an instance of a Q client node.
- The I2P standard jarfiles declared in your java CLASSPATH
- The standard Apache XML-RPC library jarfile in your CLASSPATH (which you will already have on your CLASSPATH, because this is part of installing Q). Recall that you can get a copy of Apache java XML-RPC lib jarfile from http://ws.apache.org/xmlrpc).
// QDemo.java // // A simple demo example of a Q client application, which // communicates with a running Q client node on the local // machine via its TCP XML-RPC interface // // If your client node is not running on localhost, or // if it's listening on a port other than the default // 7651, you'll need to change the code below. // // Note that this demo is bloated by the fact we're using // raw XML-RPC. // // The following exercises are left to the reader: // 1. Modify this app so that instead of using the XML-RPC // interface, it instantiates a QClientNode, and // invokes its methods directly. // 2. Write a thin wrapper class which instantiates an XML-RPC // client, and offers simpler access methods (thus avoiding // the need to create and populate Vectors of args before // calling, and pick through a reply Hashtable after the call), // and create a version of this demo which uses the wrapper. // pull in some standard java stuff import java.*; import java.lang.*; import java.util.*; import java.net.*; import java.io.*; // pull in some xml-rpc stuff import org.apache.xmlrpc.*; // since we're talking to the node via xmlrpc, and talking to // it in a separate VM, we don't need to import any Q packages // Define a minimal demo class, which kust defines a // main method enabling us to run the demo from a shell. // // For the purposes of this demo, we're assuming that your Q client node is // running on your local machine, and that you haven't altered the // listening port (default 7651) for the client's XML-RPC interface. public class QDemo { // just define a main so we can run this from a shell static public void main(String [] args) throws MalformedURLException, XmlRpcException, IOException { // for getting and analysing replies from node Hashtable result; String status; // Create a new client app object XmlRpcClient myClient = new XmlRpcClient("http://127.0.0.1:7651"); // ------------------------------------- // First action - execute a 'ping' on this peer // ------------------------------------- Vector noArgs = new Vector(); result = (Hashtable)myClient.execute("i2p.q.ping", noArgs); print("ping: result=" + result); // ------------------------------------- // Second action - insert an item of data // ------------------------------------- // mark the current time, we'll use this later Integer then = new Integer((int)(new Date().getTime() / 1000)); // create metadata // (note from previous chapter that metadata is optional) Hashtable meta = new Hashtable(); meta.put("type", "text"); meta.put("abstract", "a simple piece of demo data"); meta.put("mimetype", "text/plain"); // create some data String data = "Hello, world"; // set up the arguments list Vector insertArgs = new Vector(); insertArgs.addElement(meta); insertArgs.addElement(data.getBytes()); // must insert data as byte[] // and do the insert result = (Hashtable)myClient.execute("i2p.q.putItem", insertArgs); print("putItem: result=" + result); // check what happened status = (String)result.get("status"); String key; if (status.equals("ok")) { // insert succeeded key = (String)result.get("key"); print("Insert successful"); } else { // insert failed, bail print("Insert failed: error=" + (String)result.get("error")); return; } // ------------------------------------- // Third action - check for catalog updates // (which should include what we've just inserted) // ------------------------------------- // create an args list, with just the date we noted before the insert Vector updateArgs = new Vector(); updateArgs.addElement(then); // add the flags updateArgs.addElement(new Integer(0)); // 'includePeers' updateArgs.addElement(new Integer(1)); // 'includeCatalog' // execute the 'getCatalog' result = (Hashtable)myClient.execute("i2p.q.getUpdate", updateArgs); print("getUpdate: result="+result); // pick out the results, and search for what we just inserted int i; Vector items = (Vector)result.get("items"); int nitems = items.size(); boolean foundit = false; for (i = 0; i < nitems; i++) { // get the nth item Hashtable metaRec = (Hashtable)items.get(i); String thisKey = (String)metaRec.get("key"); if (thisKey.equals(key)) { // yay, got it! foundit = true; break; } } // did we get it? if (!foundit) { print("wtf? we inserted it but it's not in the catalog!"); return; } // yep, we got it, so try to retrieve it back Vector getArgs = new Vector(); getArgs.addElement(key); result = (Hashtable)myClient.execute("i2p.q.getItem", getArgs); print("getItem: result=" + result); // did we get it? status = (String)result.get("status"); if (!status.equals("ok")) { print("getItem failed: " + (String)result.get("error")); return; } // yep, got it byte [] binData = (byte [])result.get("data"); String strData = new String(binData); print("getItem: success, data='"+strData+"'"); print("--- END OF Q CLIENT DEMO ---"); } // a convenient shorthand method for printing stuff to stdout static void print(String msg) { System.out.println(msg); } }
5.3. Python Example
To run this example, you will need a running I2P installation, including a running instance of a Q client node.
Note that, in contrast to Java, Python 2.3 and later have all the necessary XML-RPC libraries built in.
Now for some code (again, heavily annotated). This, together with the previous example, present an interesting comparison between some of Java and Python's ways of doing things.#!/usr/bin/env python """ QDemo.py A simple demo example of a Q client application, which communicates with a running Q client node on the local machine via its TCP XML-RPC interface If your client node is not running on localhost, or if it's listening on a port other than the default 7651, you'll need to change the code below. Note that this demo is bloated by the fact we're using raw XML-RPC. The following exercise is left to the reader: * Write a thin wrapper class which instantiates an XML-RPC client, and offers simpler access methods (thus avoiding the need to pick through a reply dict after the call), and create a version of this demo which uses the wrapper. """ # a coupla needed imports from time import time from xmlrpclib import ServerProxy, Binary # For the purposes of this demo, we're assuming that your Q client node is # running on your local machine, and that you haven't altered the # listening port (default 7651) for the client's XML-RPC interface. def qdemo(): # Create a new client app object myClient = ServerProxy("http://127.0.0.1:7651") # ------------------------------------- # First action - execute a 'ping' on this peer # ------------------------------------- result = myClient.i2p.q.ping() print "ping: result=%s" % result # ------------------------------------- # Second action - insert an item of data # ------------------------------------- # mark the current time, we'll use this later then = int(time()) # create metadata # (note from previous chapter that metadata is optional) meta = { "type" : "text", "abstract" : "a simple piece of demo data", "mimetype" : "text/plain", } # create some data, and binary-wrap it data = "Hello, world" binData = Binary(data) # and do the insert result = myClient.i2p.q.putItem(meta, binData) print "putItem: result=%s" % result # check what happened if result["status"] == "ok": # insert succeeded key = result["key"] print "Insert successful" else: # insert failed, bail print "Insert failed: error=%s" % result['error'] return; # ------------------------------------- # Third action - check for catalog updates # (which should include what we've just inserted) # ------------------------------------- # execute the 'getUpdate' result = myClient.i2p.q.getUpdate(then, 0, 1) print "getUpdate: result=%s" % result # pick out the results, and search for what we just inserted foundit = False for metaRec in result['items']: if metaRec['key'] == key: # yay, got it! foundit = True break # did we get it? if not foundit: print "wtf? we inserted it but it's not in the catalog!" return; # yep, we got it, so try to retrieve it back print "getCatalog: found the item we just inserted" result = myClient.i2p.q.getItem(key) print "getItem: result=%s" % result # did we get it? if result["status"] != "ok": print "getItem failed: %s" + result["error"] return; # yep, got it (note that data is an xmlrpclib.Binary object, # and the raw data we want is in its .data attribute) print "getItem: success, data='%s'" % result['data'].data print "--- END OF Q CLIENT DEMO ---" # run the demo func if this script is executed directly if __name__ == '__main__': qdemo()
6. Keys and Metadata
6.1. Overview
Like Freenet, content is stored in Q as (data, metadata) pairs.
However, there's a difference. On Freenet, metadata is stored as a string of up to 32k length, and must be parsed (and sometimes executed) by client code. On the other hand, metadata is exposed in Q as an XML-RPC struct (Java Hashtable or Properties object, or Python dict, or Perl associative array etc).
If a content item gets inserted to the Q network without metadata, a minimal metadata set will be transparently generated, and is guaranteed to contain at least the following elements:
Key Type Description size int Size of the stored data item, in bytes dataHash string a base64 representation of the SHA256 hash of the full raw data, using the I2P base64 alphabet
6.2. Node IDs
When Q nodes are first created, they generate themselves a random I2P privKey/dest keypair using the in-I2P services.
The I2P destination gets converted to what we call a Q Node ID, as follows:
- Start with binary destination (not base64)
- Determine the SHA256 binary digest of this dest
- Encode the resulting binary string via I2P's base64 alphabet
6.3. Keys
Here, 'key' means the unique short string, by which items of content can be retrieved, and which is returned from an i2p.q.putItem command.
Like Freenet's CHK@ keytype, Q keys are hashes of the key's content and metadata.
The recipe for calculating the 'key' of a particular item of metadata+data is:
- If no metadata is submitted with the data, create a minimal metadata as per above
- Serialise out the metadata into a string representation, with the fieldnames in alphanumeric order. The format of such string is one line per metadata field/value pair per line, in the format:
metadatakeyname=metadatakeyvalue\n- Calculate the binary SHA1 digest of this serialised metadata string
- Base64-encode this binary digest via the I2P Base64 alphabet
6.4. Q Metadata Conventions
Additional to the core metadata defined above, there is a convention in Q that the following optional extra metadata keys be provided on insert, and recognised and honoured on retrieve.
It is highly recommended that these keys be included in metadata when content is inserted:
Key Type Description title string A short and descriptive title for the item, preferably formatted as a filename which is legal and convenient on all main operating systems, ie, containing only alphanumerics, '-', '_' and '.'.
It is highly advisable that an appropriate file extension appear at the end of the title. Refer to the Security Considerations section below.
It is expected that client applications will use this title field when displaying available content lists to users.type string Generic type of material, using the following superset of the eMule/Donkey classifications:
- text
- html
- image
- audio
- video
- software
- archive
- misc
mimetype string A recognised mime-type, as per RFC1341, RFC1521, RFC1522, such as audio/mpeg, text/plain etc.
This will help client app developers devise ways of disposing with data items they request from client nodes.
For instance, client apps with http front ends may send back this mimetype as the value of the Content-type: header, (and possibly take preventative action with potentially hazardous mimetypes, such as those which some browsers such as IE might trust and execute blindly as binary code).
Alternatively, gui-based or cli-based client apps may convert this mimetype to an appropriate benign file extension (such as .txt, .ogg, .jpg etc). See Security Considerations below.keywords string A set of space-separated keywords describing this item, intended for human reading, as well as automatic parsing by client apps. abstract string A short descriptive summary of the nature of the data, intended for human reading, as well as automatic pattern matching searches by client apps.
6.5. One Data Item, Many Metadata Sets?
It is perfectly possible, and legal, for one item of data to be referenced by two completely different items of metadata.
Since content keys are a hash of metadata, which in turn contains a hash of the data, then two pieces of metadata referencing the same data item, but containing different metadata values, will end up with different keys.
So as far as key addresses go, there will be a many-to-1 relationship between raw content keys, and the data returned under these keys.
7. Security Considerations
All Peer2Peer software (as with all networked software in general) carries with it a set of devastating security risks which should be respected to the utmost.
This applies in no small part to Q.
So this brief sermon is addressed to anyone writing any client applications or APIs talking to the Q network.
Any material which involves the execution of code on a client machine is risky. However, much of the risk can be managed if the code is open source and peer-reviewed.
Perhaps the biggest issue as far as Q is concerned is this:Client app developers should never, NEVER implicitly trust incoming content, and should always assume that malicious remote users will insert content which attempts to compromise other users' systems.If a Q client app wants to offer filetype-specific support, then perhaps a good strategy is for the client app to use a whitelist of known low-risk file extensions, such as .txt, or (possibly) .ogg, .png etc. Recall that in some Windows configurations, even .jpg can carry an arbitrary code execution attack!
Note that .html (text/html) is especially dangerous, and should be respected accordingly.
Support for .html could be a real boon. For instance, it could allow I2P users to publish an I2P equivalent of freenet's freesites - static HTML websites which are accessible even when the author goes offline.
However, if a client app chooses to recognise .html, it should either use a code-screening mechanism like freenet's fproxy and keep it up to date with all the latest advisories, or use a mandatory-proxy mechanism like I2P's eepProxy.
One possiblility is to serve up such content via a totally in-I2P http interface, such that Joe can view the content via his regular eeproxy-configured browser.
This is a typical case where security and ease/convenience can end up in direct conflict. Automatic handling of content according to data type is great from a Joe Sixpack Windows User point of view, but it is a snake-pit of risks that can potentially result in any of the following (or worse):The crux of this lecture is that client app writers have a huge responsibility to ensure their apps are safe against malicious content.
- Set up Joe's computer as a spambot
- Get Joe's personal credit card and other info, and use this criminally
- Download child pornography or terrorist information onto Joe's PC, use an exploit to get Joe's IP address and/or identity details, and report this to authorities, thus framing Joe and sending him off undeservedly to Club Fed or Her Majesty's
- Mount a DDoS, anonymity or other attack on the I2P network
- Further spread additional content for achieving more of the above on other unsuspecting users.
Perhaps the best and most practical solution is to just store downloaded material into a directory known to and owned by the user, and make it the user's task and responsibility to manually copy materials out of this directory and take responsibility for how s/he uses this content thereafter.
8. Contacting the Author
I am aum, and can be reached as aum on in-I2P IRC networks, and also at the in-I2P email address of aum@mail.i2p.
Introduction | XML-RPC | Architecture | Commands | Example Code | Metadata | Security | Contact us
Last modified: Sat Apr 2 13:31:08 NZST 2005