Q Metadata Specification

1. Introduction

This document lists the standard metadata keys for Q data items, discussing the rules of metadata insertion, processing and validation.

1.1. Definitions

To avoid confusions in terminology, this document will strictly abide the following definitions:

Term Definition
key A metadata category name, technically a key as the word is used with Java Hashtable and Python dict objects.
uri A Uniform Resource Indicator for an item of content stored within the Q network.
Q URIs have the form: Q:<basename>[,<cryptoKey>][<path>]

Some examples:
  • Q:fhvnr3HFSK234khsf90sdh42fsh (a plain hash uri, no cryptoKey)
  • Q:e54fhjeo39schr2kcy4osEH478D/files/johnny.mp3 (a secure space URI, no cryptoKey)
  • Q:vhfh4se987WwfkhwWFEwkh3234S,47fhh2dkhseiyu (a plain hash URI, with a cryptoKey)
basename The basic element of a Q uri. This will be a base64-encoded hash - refer below to URI calculation procedures
cryptoKey An optional session encryption key for the stored data, encoded as base64. This affords some protection to server node operators, and gives them a level of plausible deniability for whatever gets stored in their server's datastore without their direct human awareness.
path Whever an item of content is inserted in secure space mode, this path serves as a pseudo-pathname, and is conceptually similar to the path component in (for example) standard HTTP URLs http://<domainname>[:<port>][<path>], such as http://slashdot.org/faq/editorial.shtml (whose path is /faq/editorial.shtml).

Paths, if not empty, should contain a leading slash ("/"). If an application specifies a non-empty path that doesn't begin with a leading '/', a '/' will be automatically prepended by the receiving node.
plain hash A mode of inserting items, whereby the security of the resulting URI comes from computing the URI from a hash of the item's data and metadata (and imposing a mathematical barrier against spoofing content under a given URI). Corresponds to Freenet's CHK@ keys.
secure space A mode of inserting items where the security of the URI is based not on a hash of the item's data and metadata (as with plain hash mode), but on the privateKey provided by the application, and a content signature created from that private key. Corresponds to Freenet's SSK@ keys. Within a secure space, you can insert any number of items under different pseudo-pathnames (as is the case with Freenet SSK keys).


2.1. Keys Inserted By Application Before sending putItem RPCs

As the heading suggests, this is a list of metadata keys which should be inserted by a Q application prior to invoking a putItem RPC on the local Q client node.

Key Data Type Description
title String Optional but strongly recommended. A free-text short description of the item, should be less than 80 characters. The idea is that applications should support a 'view' of catalogue data that shows item titles. (Prior Q convention of titles expressed as valid filename syntax has been abandoned).
path String Optional but strongly recommended. A virtual 'pathname' for the item, which should be in valid *nix absolute pathname syntax (beginning with '/', containing no '//', consisting only of alphanumerics, '-', '_', '.' and '/'.

In Q web interfaces, the filename component of this path will serve as the recommended filename when downloading/saving the item.

If the application also provides a privateKey key, the path is used in conjunction with the private key to generate publicKey and signature keys (see below), and ultimately the final uri under which the item can be retrieved by others.

Refer also to mimetype below.
encrypt String Optional. If this key is present, and has a value "1", "yes" or "true", this indicates that the application wishes the data to be stored on servers in encrypted form.

If this key is present and set to a positive value, the Q node, on receiving the putItem RPC, will:
  1. Generate a random symmetric encryption key
  2. Encrypt the item's data using this encryption key
  3. Delete the encrypt key from the metadata
  4. Enclose a base64 representation of this encryption key in the RPC response it sends back to the application (embedded in the uri
type String Optional but strongly recommended. A standard ed2k specifier, one of text html image audio video archive other
mimetype String Optional but moderately recommended. Mimetype designation of data, eg text/html, image/jpeg etc. If not specified, an attempt will be made to guess a mometype from the value of the path key. If this attempt fails, then this key will be set to application/x-octet-stream by the node receiving the putItem RPC.
keywords String Optional but moderately recommended. A set of keywords, under which the inserting app would like this item to be discoverable. Keywords should be entirely lower case and comma-separated. Content inserts should consider keywords carefully, and only use space characters inside keywords when necessary (eg, for flagging a distinctive phrase containing very common words).
privateKey String Optional. A Base64-encoded signing private key, in cases where the application wishes to insert an item in signed space mode. This can be accompanied by another key, path, indicating a 'path' within the signed space. If 'path' is not given, it will default to '/'.

Either way, when a node receives a putItem RPC containing a privateKey in its metadata, it removes this key and replaces it with publicKey and signature.
path String Optional. The virtual pathname, within signed space, under which to store the item. This gets ignored and deleted unless the application also provides a privateKey as well. But if the private key is given, the path is used in conjunction with the private key to generate publicKey and signature keys (see below).
path should be a 'unix-style pathname', ie, containing only slashes as (pseudo) directory delimiters, and alphanumeric, '-', '_' and '.' characters, and preferably ending in a meaningful file extension such as .html
expiry int Unixtime at which the inserted item should expire. When this expiry time is reached, the item won't necessarily be deleted straight away, but may be deleted whenever a node's data store is full.

If this is not provided, it will default to a given duration according to the client node's configuration.

If it is provided, by an application, then the client node will transparently generate the required 'rent payment' before caching the data item and uploading it to servers.


2.2. Keys Inserted By Node Upon Receipt Of putItem RPC

Key Data Type Description
size Integer Size of the data to be inserted, in bytes.
dataHash String base64-encoded SHA256 hash of data.
uri String This depends on whether the item is being inserted in plain or signed space mode.

If inserting in plain mode, then the uri is in the form Q:somebase64hash, where the hash is computed according to the plain hash calculation procedure.

If inserting in signed space mode, then the uri will be in the form Q:somebase64hash/path.ext, where the hash is computed as per the signed space hash calculation procedure, and the /path.ext is the verbatim value of the app-supplied path key.
publicKey String Base64-encoded signing public key. In cases where app provides privateKey, a node will derive the signing public key from the private key, delete the private key from the metadata, and replace it with its corresponding public key key.
signature String Base64-encoded signature of path+dataHash, created using the app-provided privateKey.
rent String A rent payment for the data's accommodation on the server.
Intention is to support a variety of payment tokens. Initially, the only acceptable form of payment will be a hashcash-like token, in the form hashcash:base64string. The hashcash: prefix indicates that this payment is in hashcash currency, in which case the base64String should decode to a 16-byte string whose SHA256 hash partially collides with dataHash. The greater the number of bits in the collision, the longer the data's accommodation will be 'paid up for'.

If this key is already present, a Q node will verify the hashcash, and adjust the expiry key value to the time the item's accommodation is paid up till.

If the key is not present:
  • A client node will generate a value for this key with enough collision bits to pay the accommodation up till the given app-specified expiry date.
  • A server node will grant temporary free accommodation, and adjust the expiry key to the end of the free accommodation period.


3. URI Determination Procedures

3.1. Plain Hash URI Calculation Procedure

When items are inserted in plain mode, the final URI is determined from a hash of the data and metadata. Security of the item is based on the mathematical difficulty of creating an arbitrary data+metadata set whose hash collides with the target URI.

Specifically, the recipe for calculating plain hash URIs is:
  1. If the key size is missing, set this to the size of the data, in bytes
  2. If the key dataHash is missing, set this to the base64-encoded SHA256(data)
  3. If the key title is missing, set this to the value of dataHash
  4. From the metadata, create a set of strings, each in the form key=value, where each line contains a metadata key and its value, and is terminated by an ASCII linefeed (\n, 0x10).
  5. Ensure that key uri is omitted
  6. Sort the strings into ascending ASCII sort order
  7. Concatenate the strings together into one big string
  8. Calculate the SHA256 hash of this string
  9. Encode the hash into Base64
  10. Prepend the string Q: to this

3.2. Signed Space URI Calculation Procedure

This is much simpler than determining plain hash URI, since the security of the URI is based not on hashes of data and metadata, but on the cryptographic privateKey given by the application.

Calculation recipe for Signed Space URIs is:
  1. Calculate the SHA256 hash of the private key's binary data (not its base64 representation)
  2. Encode this hash into base64, dropping any trailing '=' characters
  3. Append to this the value of metadata item path (recall that path, if not empty, must begin with a '/')
  4. Prepend the string Q: to this
The resulting URI then is in the form Q:pubkeyHash/path.ext
Last modified: Wed Apr 6 00:36:37 NZST 2005