Q User/Programmer Manual

Note from the prior qmgr status command the line:

numPeers=0

This means that your Q client node is running standalone, and doesn't have any contact with any Q network. As such, your node is effectively useless. We need to hook up your node with other nodes in the Q network.

Q doesn't ship with any means for new client nodes to automatically connect to any Q server nodes. This is deliberate.

In all likelihood, there will be one 'main' Q network running within I2P, largely based around the author's own Q server node, and most people will likely want to use this Q network. But the author doesn't want to stop other people running their own private Q networks, for whatever purpose has meaning for them.

---

This is especially relevant for Q as opposed to Freenet. With Freenet, there's no way for a user to know of the existence of any item of content without first being given its 'key'. However, since Q works with published catalogs, any user can know everything that's available on a Q network, which might not be desirable to those wishing to share content in a private situation.

The Q author anticipates, and warmly supports, people running their own private Q networks within I2P, in addition to accessing the mainstream 'official' Q network.

The way Q is designed and implemented, there is no way for anyone, including Q's author, to know of the existence of anyone else's private Q network. It is beyond the author's control, (and thus arguably the author's legal responsibility), what private Q networks people set up, and what kind of content is trafficked on these networks. This claim of plausible deniability on the part of Q's author parallels that of a hardware retailer denying responsibility for what people do with tools that they purchase.

---

Ok, getting back on topic - your brand new virgin Q client node is useless and lonely, and desperately needs some Q server nodes to talk to. So let's hook up your node to the mainstream Q network.

You'll need to get one or more 'noderefs' for Q server nodes.

There's nothing fancy about a Q noderef. It's just a regular I2P 'destination', with which your Q Client Node can connect with a Q Server Node.

A 'semi-official' list of noderefs for the mainstream Q network can be downloaded from the url: http://aum.i2p/q/qnoderefs.txt.

Download this file, save it as (say) qnoderefs.txt. (Alternatively, if you're wanting to subscribe into a private Q network, then get a noderef for at least one of that network's server nodes from someone on that network who trusts you).

Import these noderefs into your Q client node via the command:

qmgr addref qnoderefs.txt

If all goes well, you should see no output from this command, or (possibly) a brief line or two suggesting success.

Your client node is now subscribed into the Q network of your choice. Verify this with the command:

qmgr status

In the output from that command, you should see the numPeers= line showing at least 1 peer.

If there is more than one Q Server Node on the Q network you've just subscribed to, then your local node should sooner or later discover all these server nodes, and the numPeers value should increase over time.

---

While Q is in its early development and testing stages, the author may abdicate the mainstream Q network, and publish nodrefs for a whole new mainstream Q network. This will especially happen if the author makes any substantial changes to the inter-node protocol, and/or releases incompatible new versions of Q client/server nodes. Remember that http://aum.i2p/q/qnoderefs.txt will serve as the authoritative source for noderefs for the mainstream Q network within the mainstream I2P network.

---

When your client node gets its noderefs to a Q network, it will periodically, from then on, retrieve differential peer list and catalog updates from servers it knows about.

Even if you only feed your client just one ref for a single server node, it will in time discover all other operating server nodes on that Q network, and will build up a full local catalog of everything that's available on that Q network.

Provided that your client is running ok, and has been fed with at least one ref for a live Q network that contains content, then over time, successive:

qmgr status

commands should report increasing values in the fields:

• numPeers - number of peers this client node knows about
• numLocalItems - number of locally stored content items, ie items which you have either inserted to, or retrieved from, your client node
• numRemoteItems - number of unique data items which are available on remote server nodes in the Q network, and which can be retrieved through your local client node.

---

4.7.1. One Big Warning

If you are participating in more than one distinct Q network, then do not insert noderefs for different networks into the same running instance of a local Q client, unless you don't plan on inserting content via that client.

For instance, let's say you are participating in two different Q networks:

• The 'mainstream' Q netowrk
• A secret Q network - "My friends' teen angst diaries"

If you get a noderef for both these networks, and insert both of these into the same running Q client node, then this local client node will be transparently connected to both networks.

If you only ever plan on retrieving content, and never inserting content, this won't be a problem, except that you won't be able to tell which content resides on the mainstream Q network, and which resides in the secret Q network.

The big problem arises from inserting content. Whenever you insert data through this 'contaminated' Q client node, this node picks 3 different servers to which upload a copy of this data. You won't have any control over whether the data gets inserted to the mainstream Q network, or your secret Q network. You might insert something sensitive, intending it to go only into the secret Q network, where in fact it also ends up in the mainstream network, with consequences you might not want.

4.8. Content Data and Metadata

Whenever content gets stored on Q, it is actually stored as two separate items:

• The raw data - whether a text file, or the raw bytes of image files, audio files etc
• The metadata, which contains human-readable and machine-readable descriptions of the data

Metadata consists of a set of category=value pairs.

Confused yet? Don't worry, I'm confused as well. Let's illustrate this with an example of metadata for an MP3 audio recording:

• title=Fight_Last_Thursday.mp3
• type=audio
• mimetype=audio/mpeg
• abstract=upcoming single recorded in our garage last April
• keywords=grunge,country,indie
• artist=Ring of Fire
• size=4379443
• contact=ring-of-fire@mail.i2p
• key=blah37blah24-yada23hfhyada

All metadata categories are optional. In fact, you can insert content with no metadata at all.

If you fail to provide metadata when inserting an item, a blank set of metadata will be created with at least the following categories:

• key - the derived key, under which the item will later be retrievable by yourself and others
• title - if not provided at insert time, this will be set to the key
• size - size of the item's raw data, in bytes

Within Q, there is a convention to supply a minimal amount of metadata. While this is not expected or enforced, including all these categories is most strongly recommended. These core categories are:

• title - a meaningful title for the data item, consisting only of characters which are legal in filenames on all platforms, and which ends with a file extension.
• type - one of a superset of eMule classifiers, such as:
  • text - plain text
  • html - HTML content
  • image - content is in an image format, such as .png, .jpg, .gif etc
  • audio - content is an audio sample, such as .ogg, .mp3, .wav etc
  • video - due to the sheer size of video files, and Q's present design, it's unlikely people will be inserting video content anytime soon (unless it's very short)
  • archive - packed file collections, such as .tar.gz, .zip, .rar etc
  • misc - content does not fit into any of the above categories
• mimetype - not as important as the type category, but providing this category in your metadata is still strongly encouraged. Value for this category should be one of the standard mimetypes, eg text/html, audio/ogg etc.
• abstract - a short description (<255 characters), intended for human reading
• keywords - a comma-separated list of keywords, intended for machine-readability, should be all lowercase, no spaces

Note that you can supply extra metadata categories in addition to the above, and that people searching for content can search on these extra categories if they know about them.

4.9. Searching For Content

As mentioned earlier - in constrast with Freenet, local Q nodes build up a complete catalog of all available content on whatever Q network they are connected to.

This is a design decision, based on the choice to eliminate query traffic.

The author hopes that this will result in a distributed storage network with a high retrievability guarantee, in contrast with freenet which offers no such guarantee.

With Freenet, you only ever know of the existence of something if someone tells you about it.

But with Q, your local client node builds up a global catalog of everything that's available within the whole network.

The QMgr client has a command for searching your Q client node:

qmgr search -m category1=pattern1 category2=pattern2 ...

For example:

qmgr search -m type=audio artist=Mozart keywords=symphony

or:

qmgr search -m type=text title="bible|biblical|(Nag Hammadi)" keywords="apocrypha|Magdalene"

As implied in the latter example, search patterns are regular expressions. This example will locate all text items, whose title metadata category contains one of bible, biblical or Nag Hammadi, and whose keywords category contains either or both the words apocrypha or Magdalene.

Please use the search function carefully, otherwise (if and when Q usage grows) you could be inundated with thousands or even millions of entries.

If a search turns up nothing, qmgr will simply exit. But if it turns up one or more items, it will the items out one at a time, with the key first, then each metadata entry on an indented line following.

4.10. Retrieving Content

Now, we're actually going to retrieve something.

Presumably, after following the previous section, you will have seen one or more search results come up, with the 'keys' under which the items can be accessed.

Now, choose one of the keys, preferably for a short text item. Try either of the following commands:

qmgr get <keystring> something.txt

or:

qmgr get <keystring> > something.txt

(both have the same effect - the first one explicitly writes to the named file, the second one dumps the raw data to stdout, which we shell-redirect into the file.

Note - redirection of fetched data to a file via shell is not working at present. Use only the first form till we fix the bug.

4.11. Inserting Content

Our last example in this walkthrough relates to inserting content.

Firstly, create a small text file with 2-3 lines of text, and save it as (say) myqinsert.txt.

Now, think of some metadata to insert along with the file. Or, you can just use the set:

type=text
keywords=test
abstract=My simple test of inserting into Q
title=myqinsert.txt

Now, let's insert the file. Ensure your Q client node is running, then type:

qmgr put myqinsert.txt -m type=text keywords=test title="myqinsert.txt" \
 abstract="My simple test of inserting into Q"

If all went well, this command should produce half a line of gibberish, followed immediately by your shell prompt, eg:

aRoFC~9MU~pM2C-uCTDBp5B7j79spFD8gUeu~BNkUf0=$

The '$' at the end is your shell prompt, and all the characters before it are the 'key' which was derived from the content you just inserted.

To avoid the hassle of copying/pasting the key, you could just add output redirection to the above command, eg:

qmgr put myqinsert.txt -m type=text keywords=test title="myqinsert.txt" \
 abstract="My simple test of inserting into Q" \
 > myqinsert.key

This will cause the generated key to be written safe and sound into the file myqinsert.key.

You can verify that this insert worked by a 'get' command, as in:

qmgr get `cat myqinsert.key` somefilename.ext

(Note that this won't work on windows because the DOS shell is irredeemably brain-damaged. If you're using Windows, you will have to cut/paste the key.

4.12. Shutting Down your Node

If you've worked through to here, then congratulations! You've got your Q Client Node set up and working, and ready to meet all your distributed file storage and retrieval needs.

You can leave your client node running 24/7 if you want. In fact, we recommend you keep your client node running as much of the time as possible, so that you get prompt catalog updates, and can more quickly stay in touch with new content.

However, if you need to shut down your node, the command for doing this is:

qmgr stop

This command will take a while to complete (since the node has to wait for the I2P java shutdown hooks to complete before it can rest in peace). But once your node is shut down, you can start it up again at any time and pick up where you left off.

When a Q Server node starts up for the first time, it is in a private network all by itself.

If you want to link your server into an existing Q network, you'll have to add a noderef for at least one other server on that network. The command to do this is similar to that for subscribing a client node to a network:

qmgr server addref <noderef-file>

where <noderef-file> is a file into which you've saved the noderef for the network you want to join.

---

Recall from the section on client nodes that the authoritative noderefs for the mainstream Q network can be downloaded from http://aum.i2p/q/qnoderefs.txt.

---

After you've added the noderef, subsequent qmgr server status commands should show numPeers having a value of at least 1 (and growing, as more server nodes come online in the mainstream Q network.)

5.5. Private Networks - Exporting Your Server's Noderef

If you're planning to start your own private Q network, and want to include other server operators in this network, then you'll have to export your server's noderef and make it available to the others you want to invite into your network.

The command to export your Q Server noderef is:

qmgr server getref <noderef-file>

This will extract the I2P Destination of your running server node, and write it into <noderef-file>. You can then privately share this file with others who you want to invite into your private network. Each recipient of this file will do a qmgr server addref <noderef-file> command to import your ref into their servers.

Don't forget that if you're running, or participating in, a private Q network, then you'll need to run a separate client for accessing this network, separate from any mainstream Q network client you may already be running.

To start this extra client, you'll have to choose a directory where you want this client to reside, a port number you want your client to listen on locally for user commands, and run the command:

qmgr -dir /path/to/my/new/client -port <portnum> start

You need the -port <portnum> command, because otherwise it'll fail to launch (if you already have a client node running off the mainstream Q network).

This will create, and launch, a new instance of a Q client, accessing your private Q network. Don't forget to import your server's noderef into this client. Also, note that you'll have to use this same -port <portnum> argument when doing any operation on this client instance, such as get, put, status, search.