Merge branch 'multi-peer-reader' into master.

This includes a new working implementation of multi-peer downloads using a
two-phase protocol for the concurrent retrieval of content seeded by several
other clients.

Ouinet protocol v6 and signed HTTP storage v3 are introduced.  They both are
interim verions until some cleanups afecting protocol consistency are applied,
specs updated, and v7 is released.

doc/arch-drafts/http-roll-sigs-1.md

-# Partial content signing
+# Streamable content signing
 
   - Allow streaming content from origin to client.
       - Less memory usage in injector: do not "slurp" whole response.
@@ -56,7 +56,7 @@ Please note that neither the initial signature nor framing headers (`Transfer-En
 
 ```
 HTTP/1.1 200 OK
-X-Ouinet-Version: 5
+X-Ouinet-Version: 6
 X-Ouinet-URI: https://example.com/foo
 X-Ouinet-Injection: id=d6076384-2295-462b-a047-fe2c9274e58d,ts=1516048310
 Date: Mon, 15 Jan 2018 20:31:50 GMT
@@ -74,9 +74,9 @@ Trailer: Digest, X-Ouinet-Data-Size, X-Ouinet-Sig1
 0123456789...
 100000;ouisig=BASE64(BSIG(d607…e58d NUL 0 NUL CHASH[0]=SHA2-512(SHA2-512(BLOCK[0]))))
 0123456789...
-4;ouisig=BASE64(BSIG(d607…e58d NUL 1048576 NUL CHASH[1]=SHA2-512(CHASH[0] SHA2-512(BLOCK[1]))))
+4;ouisig=BASE64(BSIG(d607…e58d NUL 1048576 NUL CHASH[1]=SHA2-512(SIG[0] CHASH[0] SHA2-512(BLOCK[1]))))
 abcd
-0;ouisig=BASE64(BSIG(d607…e58d NUL 2097152 NUL CHASH[2]=SHA2-512(CHASH[1] SHA2-512(BLOCK[2]))))
+0;ouisig=BASE64(BSIG(d607…e58d NUL 2097152 NUL CHASH[2]=SHA2-512(SIG[1] CHASH[1] SHA2-512(BLOCK[2]))))
 Digest: SHA-256=BASE64(SHA2-256(COMPLETE_BODY))
 X-Ouinet-Data-Size: 1048580
 X-Ouinet-Sig1: keyId="ed25519=????",algorithm="hs2019",created=1516048311,
@@ -96,13 +96,15 @@ The signature string for each block covers the following values (separated by nu
 
     This helps detecting an attacker which replies to a range request with a range of the expected length, with correctly signed and ordered blocks, that however starts at the wrong offset.
 
-  - A **chain hash** (binary) computed from the chain hash of the previous block and the **data hash** of the block itself: for the i-th block, `DHASH[i]=SHA2-512(BLOCK[i])` and `CHASH[i]=SHA2-512(CHASH[i-1] DHASH[i])`, with `CHASH[0]=SHA2-512(DHASH[0])`.
+  - A **chain hash** (binary) computed from the chain hash of the previous block and the **data hash** of the block itself: for the i-th block, `DHASH[i]=SHA2-512(BLOCK[i])` and `CHASH[i]=SHA2-512(SIG[i-1] CHASH[i-1] DHASH[i])`, with `CHASH[0]=SHA2-512(DHASH[0])`.
 
     Signing the hash instead of block data itself spares the signer from keeping the whole block in memory for producing the signature (the hash algorithm can be fed as data comes in from the origin).
 
     Using the data block hash instead of its data allows to independently verify the signatures without needing to be in possession of the data itself, just the hashes.
 
-    Keeping the injection identifier out of the hash allows to compare the hashes at particular blocks of different injections (if transmitted independently) to ascertain that their data is the same up to that block.
+    **TODOv6 REVIEW,OBSOLETE** Keeping the injection identifier out of the hash allows to compare the hashes at particular blocks of different injections (if transmitted independently) to ascertain that their data is the same up to that block. **TODO contradicts below**
+
+    **TODOv6 REVIEW** Including the previous signature in the hash allows to transitively verify the signatures of previous blocks by verifying the last signature (in case signatures and hashes are retrieved by themselves without the data beforehand). **TODO contradicts above**
 
     The chaining precludes the attacker from reordering correctly signed blocks for this injection.  SHA2-512 is used as a compromise between security and speed on 64-bit platforms; although the hash is longer than the slower SHA2-256, it will be seldom transmitted (e.g. for range requests as indicated below).
 
@@ -132,7 +134,7 @@ For example, a client having stored the complete response shown above may reply
 
 ```
 HTTP/1.1 200 OK
-X-Ouinet-Version: 4
+X-Ouinet-Version: 6
 X-Ouinet-URI: https://example.com/foo
 X-Ouinet-Injection: id=d6076384-2295-462b-a047-fe2c9274e58d,ts=1516048310
 Date: Mon, 15 Jan 2018 20:31:50 GMT
@@ -157,7 +159,7 @@ In contrast, a client having stored only an incomplete response from the injecto
 
 ```
 HTTP/1.1 200 OK
-X-Ouinet-Version: 4
+X-Ouinet-Version: 6
 X-Ouinet-URI: https://example.com/foo
 X-Ouinet-Injection: id=d6076384-2295-462b-a047-fe2c9274e58d,ts=1516048310
 Date: Mon, 15 Jan 2018 20:31:50 GMT

doc/ouinet-network-whitepaper.md

@@ -375,6 +375,8 @@ A cache entry signed using implementations of these primitives different from th
 
 #### Examples
 
+**TODOv6 OBSOLETE**
+
 An injector server using Ed25519 private key `KEY` might construct the following as-yet unsigned cache entry:
 
 ```
@@ -398,7 +400,7 @@ X-Ouinet-Data-Size: 12
 The injector server would create the following complete cache entry signature:
 
 ```
-keyId="ed25519=<key>",algorithm="hs2019",created=1584748800, headers="(response-status) (created) x-Ouinet-version x-Ouinet-uri x-Ouinet-injection date content-type digest x-Ouinet-data-size",signature="<signature-base64>"
+keyId="ed25519=<key>",algorithm="hs2019",created=1584748800, headers="(response-status) (created) x-ouinet-version x-ouinet-uri x-ouinet-injection date content-type digest x-ouinet-data-size",signature="<signature-base64>"
 ```
 
 In this signature, `<key>` stands for the public key associated with the `KEY` private key, and `<signature-base64>` is the base64 encoding of the Ed25519 signature of the following string:
@@ -406,13 +408,13 @@ In this signature, `<key>` stands for the public key associated with the `KEY` p
 ```
 (response-status): 200
 (created): 1584748800
-x-Ouinet-version: 4
-x-Ouinet-uri: https://example.com/hello
-x-Ouinet-injection: id=qwertyuiop-12345,ts=1584748800
+x-ouinet-version: 4
+x-ouinet-uri: https://example.com/hello
+x-ouinet-injection: id=qwertyuiop-12345,ts=1584748800
 date: Sat, 21 Mar 2020 00:00:00 GMT
 content-type: text/plain
 digest: SHA-256=wFNeS+K3n/2TKRMFQ2v4iTFOSj+uwF7P/Lt98xrZ5Ro=
-x-Ouinet-data-size: 12
+x-ouinet-data-size: 12
 ```
 
 Lines in this string are separated by newline `\n` characters. The string does not begin with or end in a newline character.
@@ -421,7 +423,7 @@ The injector server might choose not to create a signature stream for this cache
 
 * `block_size`: `5`
 * `injection-id`: `qwertyuiop-12345`
-* `header-signature`: `keyId="ed25519=<key>",algorithm="hs2019",created=1584748800, headers="(response-status) (created) x-Ouinet-version x-Ouinet-uri x-Ouinet-injection date content-type",signature="<header-signature-base64>"`
+* `header-signature`: `keyId="ed25519=<key>",algorithm="hs2019",created=1584748800, headers="(response-status) (created) x-ouinet-version x-ouinet-uri x-ouinet-injection date content-type",signature="<header-signature-base64>"`
 * `block(0)`: `Hello`
 * `block(1)`: ` worl`
 * `block(2)`: `d!`
@@ -439,9 +441,9 @@ In the computation of `header-signature` in the above, `<key>` stands for the pu
 ```
 (response-status): 200
 (created): 1584748800
-x-Ouinet-version: 4
-x-Ouinet-uri: https://example.com/hello
-x-Ouinet-injection: id=qwertyuiop-12345,ts=1584748800
+x-ouinet-version: 4
+x-ouinet-uri: https://example.com/hello
+x-ouinet-injection: id=qwertyuiop-12345,ts=1584748800
 date: Sat, 21 Mar 2020 00:00:00 GMT
 content-type: text/plain
 ```
@@ -574,6 +576,8 @@ Of these three examples, the last two would be considered equivalent by a recipi
 
 ### Peer-to-peer cache entry exchange
 
+**TODOv6 OBSOLETE,INCOMPLETE(multi-peer)**
+
 When a Ouinet client stores a collection of cache entries in its device local storage, it can share these cache entries with other users that wish to access them. By fetching cache entries from other users in this way, without involvement of the injector servers, a Ouinet client can access web content even in cases when it cannot reach the injector servers.
 
 A Ouinet client willing to share its cache entries with others can serve HTTP requests using a protocol very similar to that used by the injector servers. Unlike injector servers, a Ouinet client participating in the distributed cache will only respond to such requests by serving a copy of a cache entry it has stored in its local device storage. Using this system, a client wishing to fetch a cached resource from another client that stores a cache entry for that resource can establish a peer-to-peer connection to that client, send an HTTP request for the cached resource, and retrieve the cache entry. The recipient can then verify the legitimacy of the cache entry, use the resource in a user application, and optionally store the resource in its own local storage.

asio-utp @ 95888ecd

-Subproject commit 318c1f0f41594532317bcd03dd8d993da4e390d6
+Subproject commit 95888ecdd232f705551b377a0b7afd2fa79c8631

src/cache/chain_hasher.h

+#pragma once
+
+#include "../util/crypto.h"
+
+namespace ouinet { namespace cache {
+
+class ChainHash {
+public:
+    using PrivateKey = util::Ed25519PrivateKey;
+    using PublicKey  = util::Ed25519PublicKey;
+    using Signature  = PublicKey::sig_array_t;
+    using Hash       = util::SHA512;
+    using Digest     = Hash::digest_type;
+
+    size_t    offset;
+    Digest    chain_digest;
+    Signature chain_signature;
+
+    bool verify(const PublicKey& pk, const std::string& injection_id) const {
+        return pk.verify(str_to_sign(injection_id, offset, chain_digest), chain_signature);
+    }
+
+private:
+    friend class ChainHasher;
+
+    static
+    std::string str_to_sign(
+            const std::string& injection_id,
+            size_t offset,
+            Digest digest)
+    {
+        static const auto fmt_ = "%s%c%d%c%s";
+        return ( boost::format(fmt_)
+               % injection_id % '\0'
+               % offset % '\0'
+               % util::bytes::to_string_view(digest)).str();
+    }
+};
+
+class ChainHasher {
+public:
+    using PrivateKey = ChainHash::PrivateKey;
+    using Signature  = ChainHash::Signature;
+    using Hash       = ChainHash::Hash;
+    using Digest     = ChainHash::Digest;
+
+    struct Signer {
+        const std::string& injection_id;
+        const PrivateKey&  key;
+
+        Signature sign(size_t offset, const Digest& chained_digest) const {
+            return key.sign(ChainHash::str_to_sign(injection_id, offset, chained_digest));
+        }
+    };
+
+    using SigOrSigner = boost::variant<Signature, Signer>;
+
+public:
+    ChainHasher()
+        : _offset(0)
+    {}
+
+    ChainHash calculate_block(size_t data_size, Digest data_digest, SigOrSigner sig_or_signer)
+    {
+        Hash chained_hasher;
+
+        if (_prev_chained_signature) {
+            chained_hasher.update(*_prev_chained_signature);
+        }
+
+        if (_prev_chained_digest) {
+            chained_hasher.update(*_prev_chained_digest);
+        }
+
+        chained_hasher.update(data_digest);
+
+        Digest chained_digest = chained_hasher.close();
+
+        Signature chained_signature = util::apply(sig_or_signer,
+                [&] (const Signature& s) { return s; },
+                [&] (const Signer& s)    { return s.sign(_offset, chained_digest); });
+
+        size_t old_offset = _offset;
+
+        // Prepare for next block
+        _offset += data_size;
+        _prev_chained_digest    = chained_digest;
+        _prev_chained_signature = chained_signature;
+
+        return {old_offset, chained_digest, chained_signature};
+    }
+
+    void set_prev_chained_digest(Digest prev_chained_digest) {
+        _prev_chained_digest = prev_chained_digest;
+    }
+
+    void set_offset(size_t offset) {
+        _offset = offset;
+    }
+
+    const boost::optional<Digest>& prev_chained_digest() const {
+        return _prev_chained_digest;
+    }
+
+private:
+    size_t _offset;
+    boost::optional<Digest> _prev_chained_digest;
+    boost::optional<Signature> _prev_chained_signature;
+};
+
+}} // namespaces

src/cache/client.h

@@ -43,7 +43,8 @@ public:
               , Cancel
               , asio::yield_context);
 
-    void serve_local( const http::request<http::empty_body>&
+    // Returns true if both request and response had keep-alive == true
+    bool serve_local( const http::request<http::empty_body>&
                     , GenericStream& sink
                     , Cancel&
                     , Yield);

src/cache/dht_lookup.h

@@ -3,6 +3,8 @@
 #include <functional>
 #include <set>
 #include "../../util/async_job.h"
+#include "../../util/hash.h"
+#include "../../bittorrent/dht.h"
 
 namespace std {
     template<> struct hash<ouinet::bittorrent::NodeID> {
@@ -49,11 +51,12 @@ private:
 public:
     DhtLookup(DhtLookup&&) = delete;
 
-    DhtLookup(std::weak_ptr<bittorrent::MainlineDht> dht_w, NodeID infohash)
-        : infohash(infohash)
-        , exec(dht_w.lock()->get_executor())
-        , dht_w(dht_w)
-        , cv(exec)
+    DhtLookup(std::weak_ptr<bittorrent::MainlineDht> dht_w, std::string swarm_name)
+        : _swarm_name(std::move(swarm_name))
+        , _infohash(util::sha1_digest(_swarm_name))
+        , _exec(dht_w.lock()->get_executor())
+        , _dht_w(dht_w)
+        , _cv(_exec)
     { }
 
     Ret get(Cancel c, asio::yield_context y) {
@@ -61,54 +64,62 @@ public:
         // * Use previously returned result if it's not older than 5mins
         // * Otherwise wait for the running job to finish
 
-        auto cancel_con = lifetime_cancel.connect([&] { c(); });
+        auto cancel_con = _lifetime_cancel.connect([&] { c(); });
 
-        if (!job) {
-            job = make_job();
+        if (!_job) {
+            _job = make_job();
         }
 
-        if (last_result.is_fresh()) {
-            return last_result.value;
+        if (_last_result.is_fresh()) {
+            return _last_result.value;
         }
 
 #ifndef NDEBUG
-        WatchDog wd(exec, timeout() + std::chrono::seconds(5), [&] {
+        WatchDog wd(_exec, timeout() + std::chrono::seconds(5), [&] {
                 LOG_ERROR("DHT BEP5 DhtLookup::get failed to time out");
             });
 #endif
 
         sys::error_code ec;
-        cv.wait(c, y[ec]);
+        _cv.wait(c, y[ec]);
 
         return_or_throw_on_error(y, c, ec, Ret{});
 
         // (ec == operation_aborted) implies (c == true)
-        assert(last_result.ec != asio::error::operation_aborted || c);
+        assert(_last_result.ec != asio::error::operation_aborted || c);
 
-        return or_throw(y, last_result.ec, last_result.value);
+        return or_throw(y, _last_result.ec, _last_result.value);
     }
 
-    ~DhtLookup() { lifetime_cancel(); }
+    ~DhtLookup() { _lifetime_cancel(); }
+
+    NodeID infohash() const {
+        return _infohash;
+    }
+
+    const std::string& swarm_name() const {
+        return _swarm_name;
+    }
 
 private:
 
     std::unique_ptr<Job> make_job() {
-        auto job = std::make_unique<Job>(exec);
+        auto job = std::make_unique<Job>(_exec);
 
         job->start([ self = this
-                   , dht_w = dht_w
-                   , infohash = infohash
-                   , lc = std::make_shared<Cancel>(lifetime_cancel)
+                   , dht_w = _dht_w
+                   , infohash = _infohash
+                   , lc = std::make_shared<Cancel>(_lifetime_cancel)
                    ] (Cancel c, asio::yield_context y) mutable {
             auto cancel_con = lc->connect([&] { c(); });
 
             auto on_exit = defer([&] {
                     if (*lc) return;
-                    self->cv.notify();
-                    self->job = nullptr;
+                    self->_cv.notify();
+                    self->_job = nullptr;
                 });
 
-            WatchDog wd(self->exec, timeout(), [&] {
+            WatchDog wd(self->_exec, timeout(), [&] {
                     LOG_WARN("DHT BEP5 lookup ", infohash, " timed out");
                     c();
                 });
@@ -126,9 +137,9 @@ private:
             auto eps = dht->tracker_get_peers(infohash, c, y[ec]);
 
             if (!c && !ec) {
-                self->last_result.ec    = ec;
-                self->last_result.value = move(eps);
-                self->last_result.time  = Clock::now();
+                self->_last_result.ec    = ec;
+                self->_last_result.value = move(eps);
+                self->_last_result.time  = Clock::now();
             }
 
             return or_throw(y, ec, boost::none);
@@ -138,13 +149,14 @@ private:
     }
 
 private:
-    NodeID infohash;
-    asio::executor exec;
-    std::weak_ptr<bittorrent::MainlineDht> dht_w;
-    std::unique_ptr<Job> job;
-    ConditionVariable cv;
-    Result last_result;
-    Cancel lifetime_cancel;
+    std::string _swarm_name;
+    NodeID _infohash;
+    asio::executor _exec;
+    std::weak_ptr<bittorrent::MainlineDht> _dht_w;
+    std::unique_ptr<Job> _job;
+    ConditionVariable _cv;
+    Result _last_result;
+    Cancel _lifetime_cancel;
 };
 
 

src/cache/hash_list.cpp

 #include "hash_list.h"
 #include "http_sign.h"
+#include "chain_hasher.h"
 
+using namespace std;
 using namespace ouinet;
 using namespace ouinet::cache;
 
-using boost::optional;
+#define _LOG_PFX "HashList: "
+#define _WARN(...) LOG_WARN(_LOG_PFX, __VA_ARGS__)
 
-bool HashList::verify() const {
-    using Digest = util::SHA512::digest_type;
+static const size_t MAX_LINE_SIZE_BYTES = 512;
+
+static const std::string MAGIC = "OUINET_HASH_LIST_V1";
+static const char* ORIGINAL_STATUS = "X-Ouinet-Original-Status";
 
-    optional<Digest> last_digest;
+using Digest = util::SHA512::digest_type;
 
+bool HashList::verify() const {
     size_t block_size = signed_head.block_size();
-    size_t last_offset = 0;
 
-    bool first = true;
+    ChainHasher chain_hasher;
+
+    // Even responses with empty body have at least one block hash
+    if (blocks.empty()) return false;
+
+    ChainHash chain_hash;
+
+    for (auto& block : blocks) {
+        chain_hash = chain_hasher.calculate_block(
+                block_size, block.data_hash, block.chained_hash_signature);
+    }
+
+    return chain_hash.verify( signed_head.public_key()
+                            , signed_head.injection_id());
+}
+
+struct Parser {
+    using Data = std::vector<uint8_t>;
+
+    Data buffer;
 
-    for (auto& digest : block_hashes) {
-        util::SHA512 sha;
+    void append_data(const Data& data) {
+        buffer.insert(buffer.end(), data.begin(), data.end());
+    }
 
-        if (last_digest) {
-            sha.update(*last_digest);
+    // Returns a line of data (optionally)
+    boost::optional<string> read_line() {
+        auto nl_i = find_nl(buffer);
+
+        if (nl_i == buffer.end()) {
+            return boost::none;
         }
 
-        sha.update(digest);
+        string ret(buffer.begin(), nl_i);
+        buffer.erase(buffer.begin(), std::next(nl_i));
+
+        return ret;
+    }
+
+    boost::optional<util::Ed25519PublicKey::sig_array_t>
+    read_signature() {
+        return read_array<util::Ed25519PublicKey::sig_size>();
+    }
+
+    boost::optional<Digest>
+    read_hash() {
+        return read_array<util::SHA512::size()>();
+    }
+
+    template<size_t N>
+    boost::optional<std::array<uint8_t, N>> read_array() {
+        if (buffer.size() < N) return boost::none;
+        auto b = buffer.begin();
+        auto e = b + N;
+        std::array<uint8_t, N> ret;
+        std::copy(b, e, ret.begin());
+        buffer.erase(b, e);
+        return ret;
+    }
+
+    Data::iterator find_nl(Data& data) const {
+        return std::find(data.begin(), data.end(), '\n');
+    }
+};
+
+/* static */
+HashList HashList::load(
+        http_response::Reader& r,
+        const PubKey& pk,
+        Cancel& c,
+        asio::yield_context y)
+{
+    using namespace std::chrono_literals;
+    static const auto bad_msg = sys::errc::make_error_code(sys::errc::bad_message);
+
+    assert(!c);
+
+    sys::error_code ec;
+
+    auto part = r.timed_async_read_part(5s, c, y[ec]);
 
-        last_digest = sha.close();
+    if (!ec && !part) {
+        assert(0);
+        ec = c ? asio::error::operation_aborted
+               : sys::errc::make_error_code(sys::errc::bad_message);
+    }
+
+    if (c) ec = asio::error::operation_aborted;
+    if (ec) return or_throw<HashList>(y, ec);
+
+    return_or_throw_on_error(y, c, ec, HashList{});
+
+    if (!part->is_head()) return or_throw<HashList>(y, bad_msg);
+
+    auto raw_head = move(*part->as_head());
 
-        if (first) {
-            first = false;
+    if (raw_head.result() == http::status::not_found) {
+        return or_throw<HashList>(y, asio::error::not_found);
+    }
+
+    auto orig_status_sv = raw_head[ORIGINAL_STATUS];
+    auto orig_status = parse::number<unsigned>(orig_status_sv);
+    raw_head.erase(ORIGINAL_STATUS);
+
+    if (!orig_status) {
+        return or_throw<HashList>(y, bad_msg);
+    }
+
+    raw_head.result(*orig_status);
+
+    auto head_o = SignedHead::verify_and_create(move(raw_head), pk);
+
+    if (!head_o) return or_throw<HashList>(y, bad_msg);
+
+    head_o->erase(http::field::content_length);
+    head_o->set(http::field::transfer_encoding, "chunked");
+
+    Parser parser;
+
+    using Signature = PubKey::sig_array_t;
+
+    bool magic_checked = false;
+
+    boost::optional<Digest> digest;
+    boost::optional<Signature> signature;
+
+    std::vector<Block> blocks;
+
+    while (true) {
+        part = r.timed_async_read_part(5s, c, y[ec]);
+        return_or_throw_on_error(y, c, ec, HashList{});
+
+        if (!part) break;
+
+        if (part->is_body()) {
+            parser.append_data(*part->as_body());
+        } else if (part->is_chunk_body()) {
+            parser.append_data(*part->as_chunk_body());
         } else {
-            last_offset += block_size;
+            continue;
+        }
+
+        while (true) {
+            bool progress = false;
+
+            if (!magic_checked) {
+                auto magic_line = parser.read_line();
+                if (magic_line) {
+                    if (*magic_line != MAGIC)
+                        return or_throw<HashList>(y, bad_msg);
+                    magic_checked = true;
+                    progress = true;
+                }
+            } else {
+                if (!digest) {
+                    digest = parser.read_hash();
+                    if (digest) progress = true;
+                } else {
+                    assert(!signature);
+                    signature = parser.read_signature();
+
+                    if (signature) {
+                        progress = true;
+
+                        blocks.push_back({*digest, *signature});
+
+                        digest    = boost::none;
+                        signature = boost::none;
+                    }
+                }
+            }
+
+            if (!progress) {
+                if (parser.buffer.size() > MAX_LINE_SIZE_BYTES) {
+                    _WARN("Line too long");
+                    return or_throw<HashList>(y, bad_msg);
+                }
+                break;
+            }
         }
     }
 
-    if (!last_digest) return false;
+    if (blocks.empty()) return or_throw<HashList>(y, bad_msg);
+