373 lines
16 KiB
HTML
373 lines
16 KiB
HTML
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||
|
|
<html>
|
||
|
|
<head>
|
||
|
|
<title>Q Metadata Specification</title>
|
||
|
|
|
||
|
|
<style type="text/css">
|
||
|
|
<!--
|
||
|
|
td { vertical-align: top; }
|
||
|
|
code { font-family: courier, monospace; font-weight: bolder; font-size:smaller }
|
||
|
|
-->
|
||
|
|
</style>
|
||
|
|
|
||
|
|
</head>
|
||
|
|
|
||
|
|
<body>
|
||
|
|
<h1>Q Metadata Specification</h1>
|
||
|
|
|
||
|
|
<h2>1. Introduction</h2>
|
||
|
|
|
||
|
|
This document lists the standard metadata keys for Q data items,
|
||
|
|
discussing the rules of metadata insertion, processing and validation.<br>
|
||
|
|
|
||
|
|
<hr>
|
||
|
|
|
||
|
|
<h3>1.1. Definitions</h3>
|
||
|
|
|
||
|
|
To avoid confusions in terminology, this document will strictly abide the following definitions:
|
||
|
|
<br>
|
||
|
|
<br>
|
||
|
|
<table width=80% cellspacing=0 cellpadding=4 border=1 align=center>
|
||
|
|
<tr style="font-weight: bold">
|
||
|
|
<td>Term</td>
|
||
|
|
<td>Definition</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td><code>key</code></td>
|
||
|
|
<td>A metadata category name, technically a <code>key</code> as the word is used with
|
||
|
|
Java <code>Hashtable</code> and Python <code>dict</code> objects.</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td><code>uri</code></td>
|
||
|
|
<td>A Uniform Resource Indicator for an item of content stored within the Q network.<br>
|
||
|
|
Q URIs have the form: <code>Q:<basename>[,<cryptoKey>][<path>]</code>
|
||
|
|
<br>
|
||
|
|
<br>
|
||
|
|
Some examples:
|
||
|
|
<ul>
|
||
|
|
<li><code>Q:fhvnr3HFSK234khsf90sdh42fsh</code> (a plain hash uri, no cryptoKey)</li>
|
||
|
|
<li><code>Q:e54fhjeo39schr2kcy4osEH478D/files/johnny.mp3</code> (a secure space URI,
|
||
|
|
no cryptoKey)</li>
|
||
|
|
<li><code>Q:vhfh4se987WwfkhwWFEwkh3234S,47fhh2dkhseiyu</code> (a plain hash URI, with
|
||
|
|
a cryptoKey)</li>
|
||
|
|
</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td><code>basename</code></td>
|
||
|
|
<td>The basic element of a Q uri. This will be a base64-encoded hash - refer below to
|
||
|
|
URI calculation procedures</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td><code>cryptoKey</code></td>
|
||
|
|
<td>An optional session encryption key for the stored data, encoded as base64.
|
||
|
|
This affords some protection to server node operators, and gives them a level
|
||
|
|
of plausible deniability for whatever gets stored in their server's
|
||
|
|
datastore without their direct human awareness.</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td><code>path</code></td>
|
||
|
|
<td>Whever an item of content is inserted in <code>secure space</code> mode, this path
|
||
|
|
serves as a pseudo-pathname, and is conceptually similar to the <code>path</code>
|
||
|
|
component in (for example) standard HTTP URLs
|
||
|
|
<code>http://<domainname>[:<port>][<path>]</code>, such as
|
||
|
|
<code>http://slashdot.org/faq/editorial.shtml</code> (whose <code>path</code>
|
||
|
|
is <code>/faq/editorial.shtml</code>).<br>
|
||
|
|
<br>
|
||
|
|
Paths, if not empty, should contain a leading slash ("/").
|
||
|
|
If an application specifies a non-empty <code>path</code> that doesn't begin with a
|
||
|
|
leading '/', a '/' will be automatically prepended by the receiving node.
|
||
|
|
</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td><code>plain hash</code></td>
|
||
|
|
<td>A mode of inserting items, whereby the security of the resulting URI comes from
|
||
|
|
computing the URI from a hash of the item's data and metadata (and imposing a
|
||
|
|
mathematical barrier against spoofing content under a given URI). Corresponds to
|
||
|
|
Freenet's <code>CHK@</code> keys.</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td><code>secure space</code></td>
|
||
|
|
<td>A mode of inserting items where the security of the URI is based not on a hash of the
|
||
|
|
item's data and metadata (as with <code>plain hash</code> mode),
|
||
|
|
but on the <code>privateKey</code> provided by the
|
||
|
|
application, and a content signature created from that private key.
|
||
|
|
Corresponds to Freenet's <code>SSK@</code> keys. Within a secure space, you
|
||
|
|
can insert any number of items under different pseudo-pathnames (as is the case
|
||
|
|
with Freenet SSK keys).
|
||
|
|
</li>
|
||
|
|
</table>
|
||
|
|
|
||
|
|
<br><br>
|
||
|
|
|
||
|
|
<hr>
|
||
|
|
|
||
|
|
<h3>2.1. Keys Inserted By Application Before sending <code>putItem</code> RPCs</h3>
|
||
|
|
|
||
|
|
As the heading suggests, this is a list of metadata keys which should be inserted by a
|
||
|
|
Q application prior to invoking a <code>putItem</code> RPC on the local Q client node.<br>
|
||
|
|
<br>
|
||
|
|
<table width=80% cellspacing=0 cellpadding=4 border=1 align=center>
|
||
|
|
<tr style="font-weight: bold">
|
||
|
|
<td>Key</td>
|
||
|
|
<td>Data Type</td>
|
||
|
|
<td>Description</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td><code>title</code></td>
|
||
|
|
<td>String</td>
|
||
|
|
<td>Optional but strongly recommended. A free-text short description of the item,
|
||
|
|
should be less than 80 characters. The idea is that applications should
|
||
|
|
support a 'view' of catalogue data that shows item titles. (Prior Q convention of
|
||
|
|
titles expressed as valid filename syntax has been abandoned).
|
||
|
|
</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td><code>path</code></td>
|
||
|
|
<td>String</td>
|
||
|
|
<td>Optional but strongly recommended.
|
||
|
|
A virtual 'pathname' for the item, which should be in valid *nix
|
||
|
|
absolute pathname syntax (beginning with '/', containing no '//', consisting
|
||
|
|
only of alphanumerics, '-', '_', '.' and '/'.<br>
|
||
|
|
<br>
|
||
|
|
In Q web interfaces, the <code>filename</code> component of this path will
|
||
|
|
serve as the recommended filename when downloading/saving the item.<br>
|
||
|
|
<br>
|
||
|
|
If the application also provides a
|
||
|
|
<code>privateKey</code> key, the path
|
||
|
|
is used in conjunction with the private key to generate <code>publicKey</code>
|
||
|
|
and <code>signature</code> keys (see below), and ultimately the final <code>uri</code>
|
||
|
|
under which the item can be retrieved by others.<br>
|
||
|
|
<br>
|
||
|
|
Refer also to <code>mimetype</code> below.
|
||
|
|
</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td><code>encrypt</code></td>
|
||
|
|
<td>String</td>
|
||
|
|
<td>Optional. If this key is present, and has a value "1", "yes" or "true",
|
||
|
|
this indicates that the application wishes the data to be stored on servers in
|
||
|
|
encrypted form.<br>
|
||
|
|
<br>
|
||
|
|
If this key is present and set to a positive value, the Q node, on receiving the
|
||
|
|
<code>putItem</code> RPC, will:
|
||
|
|
<ol>
|
||
|
|
<li>Generate a random symmetric encryption key</li>
|
||
|
|
<li>Encrypt the item's data using this encryption key</li>
|
||
|
|
<li>Delete the <code>encrypt</code> key from the metadata</li>
|
||
|
|
<li>Enclose a base64 representation of this encryption key in the RPC response
|
||
|
|
it sends back to the application (embedded in the <code>uri</code></li>
|
||
|
|
</ol>
|
||
|
|
</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td><code>type</code></td>
|
||
|
|
<td>String</td>
|
||
|
|
<td>Optional but strongly recommended. A standard ed2k specifier, one of <code>text html image
|
||
|
|
audio video archive other</code></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td><code>mimetype</code></td>
|
||
|
|
<td>String</td>
|
||
|
|
<td>Optional but moderately recommended. Mimetype designation of data, eg <code>text/html</code>,
|
||
|
|
<code>image/jpeg</code> etc. If not specified, an attempt will be made to guess
|
||
|
|
a mometype from the value of the <code>path</code> key. If this attempt fails, then
|
||
|
|
this key will be set to <code>application/x-octet-stream</code> by the node receiving
|
||
|
|
the <code>putItem</code> RPC.</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td><code>keywords</code></td>
|
||
|
|
<td>String</td>
|
||
|
|
<td>Optional but moderately recommended.
|
||
|
|
A set of keywords, under which the inserting app would like this item to be
|
||
|
|
discoverable. Keywords should be entirely lower case and comma-separated. Content
|
||
|
|
inserts should consider keywords carefully, and only use space characters inside
|
||
|
|
keywords when necessary (eg, for flagging a distinctive phrase containing very
|
||
|
|
common words).</td>
|
||
|
|
<tr>
|
||
|
|
<td><code>privateKey</code></td>
|
||
|
|
<td>String</td>
|
||
|
|
<td>Optional. A Base64-encoded signing private key, in cases where the application wishes
|
||
|
|
to insert an item in <code>signed space</code> mode. This can be accompanied by another key,
|
||
|
|
<code>path</code>, indicating a 'path' within the signed space. If 'path'
|
||
|
|
is not given, it will default to '/'.<br>
|
||
|
|
<br>
|
||
|
|
Either way, when a node receives a
|
||
|
|
<code>putItem</code> RPC containing a <code>privateKey</code> in its metadata,
|
||
|
|
it removes this key and replaces it with <code>publicKey</code> and
|
||
|
|
<code>signature</code>.
|
||
|
|
</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td><code>path</code></td>
|
||
|
|
<td>String</td>
|
||
|
|
<td>Optional. The virtual pathname, within signed space, under which to store the item.
|
||
|
|
This gets ignored and deleted unless the application also provides a
|
||
|
|
<code>privateKey</code> as well. But if the private key is given, the path
|
||
|
|
is used in conjunction with the private key to generate <code>publicKey</code>
|
||
|
|
and <code>signature</code> keys (see below).<br>
|
||
|
|
<code>path</code> should be a 'unix-style pathname', ie, containing only slashes
|
||
|
|
as (pseudo) directory delimiters, and alphanumeric, '-', '_' and '.' characters,
|
||
|
|
and preferably ending in a meaningful file extension such as <code>.html</code>
|
||
|
|
</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td><code>expiry</code></td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>Unixtime at which the inserted item should expire. When this expiry time
|
||
|
|
is reached, the item won't necessarily be deleted straight away, but may
|
||
|
|
be deleted whenever a node's data store is full.<br>
|
||
|
|
<br>
|
||
|
|
If this is not provided, it will default to a given duration according to
|
||
|
|
the client node's configuration.<br>
|
||
|
|
<br>
|
||
|
|
If it is provided, by an application, then the client node will transparently
|
||
|
|
generate the required 'rent payment' before caching the data item and uploading
|
||
|
|
it to servers.
|
||
|
|
</td>
|
||
|
|
</tr>
|
||
|
|
</table>
|
||
|
|
|
||
|
|
<br><br>
|
||
|
|
|
||
|
|
<hr>
|
||
|
|
|
||
|
|
<h3>2.2. Keys Inserted By Node Upon Receipt Of <code>putItem</code> RPC</h3>
|
||
|
|
|
||
|
|
<table width=80% cellspacing=0 cellpadding=4 border=1 align=center>
|
||
|
|
<tr style="font-weight: bold">
|
||
|
|
<td>Key</td>
|
||
|
|
<td>Data Type</td>
|
||
|
|
<td>Description</td>
|
||
|
|
</tr>
|
||
|
|
|
||
|
|
<tr>
|
||
|
|
<td><code>size</code></td>
|
||
|
|
<td>Integer</td>
|
||
|
|
<td>Size of the data to be inserted, in bytes.</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td><code>dataHash</code></td>
|
||
|
|
<td>String</td>
|
||
|
|
<td>base64-encoded SHA256 hash of data.</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td><code>uri</code></td>
|
||
|
|
<td>String</td>
|
||
|
|
<td>This depends on whether the item is being inserted in <i>plain</i> or
|
||
|
|
<i>signed space</i> mode.<br>
|
||
|
|
<br>
|
||
|
|
If inserting in <i>plain</i> mode, then the uri is in the form
|
||
|
|
<code>Q:somebase64hash</code>, where the hash is computed according to
|
||
|
|
the <a href="#plainhash">plain hash calculation procedure</a>.<br>
|
||
|
|
<br>
|
||
|
|
If inserting in <i>signed space</i> mode, then the uri will be in the form
|
||
|
|
<code>Q:somebase64hash/path.ext</code>, where the hash is computed as per
|
||
|
|
the <a href="#signedhash">signed space hash calculation procedure</a>, and
|
||
|
|
the <code>/path.ext</code> is the verbatim value of the app-supplied
|
||
|
|
<code>path</code> key.
|
||
|
|
</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td><code>publicKey</code></td>
|
||
|
|
<td>String</td>
|
||
|
|
<td>Base64-encoded signing public key. In cases where app provides
|
||
|
|
<code>privateKey</code>,
|
||
|
|
a node will derive the signing public key from the private key,
|
||
|
|
delete the private key from the metadata, and replace it with its corresponding
|
||
|
|
public key
|
||
|
|
key.</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td><code>signature</code></td>
|
||
|
|
<td>String</td>
|
||
|
|
<td>Base64-encoded signature of <code>path+dataHash</code>, created using
|
||
|
|
the app-provided <code>privateKey</code>.</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td><code>rent</code></td>
|
||
|
|
<td>String</td>
|
||
|
|
<td>A rent payment for the data's accommodation on the server.<br>
|
||
|
|
Intention is to support a variety of payment tokens. Initially, the
|
||
|
|
only acceptable form of payment will be a hashcash-like token,
|
||
|
|
in the form <code>hashcash:base64string</code>. The <code>hashcash:</code>
|
||
|
|
prefix indicates that this payment is in hashcash currency, in which case
|
||
|
|
the <code>base64String</code> should decode to a 16-byte string whose
|
||
|
|
SHA256 hash partially collides with <code>dataHash</code>.
|
||
|
|
The greater the number of bits in the collision,
|
||
|
|
the longer the data's accommodation will be 'paid up for'.<br>
|
||
|
|
<br>
|
||
|
|
If this key is already present, a Q node will verify the hashcash,
|
||
|
|
and adjust the <code>expiry</code> key value to the time the item's accommodation
|
||
|
|
is paid up till.<br>
|
||
|
|
<br>
|
||
|
|
If the key is not present:
|
||
|
|
<ul>
|
||
|
|
<li>A client node will generate a value for this key with enough collision bits
|
||
|
|
to pay the accommodation up till the given app-specified <code>expiry</code> date.</li>
|
||
|
|
<li>A server node will grant temporary free accommodation, and adjust the <code>expiry</code>
|
||
|
|
key to the end of the free accommodation period.</li>
|
||
|
|
</ul>
|
||
|
|
</td>
|
||
|
|
</tr>
|
||
|
|
</table>
|
||
|
|
|
||
|
|
<br><br>
|
||
|
|
|
||
|
|
<a name="plainhash"/>
|
||
|
|
|
||
|
|
<hr>
|
||
|
|
|
||
|
|
<h2>3. URI Determination Procedures</h2>
|
||
|
|
|
||
|
|
<h3>3.1. Plain Hash URI Calculation Procedure</h3>
|
||
|
|
|
||
|
|
When items are inserted in <code>plain</code> mode, the final URI is determined from
|
||
|
|
a hash of the data and metadata. Security of the item is based on the mathematical difficulty
|
||
|
|
of creating an arbitrary data+metadata set whose hash collides with the target URI.<br>
|
||
|
|
<br>
|
||
|
|
Specifically, the recipe for calculating plain hash URIs is:
|
||
|
|
<ol>
|
||
|
|
<li>If the key <code>size</code> is missing, set this to the size of the data,
|
||
|
|
in bytes</li>
|
||
|
|
<li>If the key <code>dataHash</code> is missing, set this to the base64-encoded
|
||
|
|
SHA256(data)</li>
|
||
|
|
<li>If the key <code>title</code> is missing, set this to the value of <code>dataHash</code></li>
|
||
|
|
<li>From the metadata, create a set of strings, each in the form <code>key=value</code>,
|
||
|
|
where each line contains a metadata <code>key</code> and its <code>value</code>, and
|
||
|
|
is terminated by an ASCII linefeed (\n, 0x10).</li>
|
||
|
|
<li>Ensure that key <code>uri</code> is omitted</li>
|
||
|
|
<li>Sort the strings into ascending ASCII sort order</li>
|
||
|
|
<li>Concatenate the strings together into one big string</li>
|
||
|
|
<li>Calculate the SHA256 hash of this string</li>
|
||
|
|
<li>Encode the hash into Base64</li>
|
||
|
|
<li>Prepend the string <code>Q:</code> to this</li>
|
||
|
|
</ol>
|
||
|
|
|
||
|
|
<a name="signedhash"/>
|
||
|
|
|
||
|
|
<hr>
|
||
|
|
|
||
|
|
<h3>3.2. Signed Space URI Calculation Procedure</h3>
|
||
|
|
|
||
|
|
This is much simpler than determining plain hash URI, since the security of the URI
|
||
|
|
is based not on hashes of data and metadata, but on the cryptographic <code>privateKey</code>
|
||
|
|
given by the application.<br>
|
||
|
|
<br>
|
||
|
|
Calculation recipe for Signed Space URIs is:
|
||
|
|
<ol>
|
||
|
|
<li>Calculate the SHA256 hash of the private key's binary data (not its base64 representation)</li>
|
||
|
|
<li>Encode this hash into base64, dropping any trailing '=' characters</li>
|
||
|
|
<li>Append to this the value of metadata item <code>path</code> (recall that <code>path</code>,
|
||
|
|
if not empty, must begin with a '/')</li>
|
||
|
|
<li>Prepend the string <code>Q:</code> to this</li>
|
||
|
|
</ol>
|
||
|
|
The resulting URI then is in the form <code>Q:pubkeyHash/path.ext</code>
|
||
|
|
|
||
|
|
<hr>
|
||
|
|
<!-- Created: Tue Apr 5 00:56:45 NZST 2005 -->
|
||
|
|
<!-- hhmts start -->
|
||
|
|
Last modified: Wed Apr 6 00:36:37 NZST 2005
|
||
|
|
<!-- hhmts end -->
|
||
|
|
</body>
|
||
|
|
</html>
|