Persistent signer mappings

* Replaces the KMIP specific in-memory `KeyMap` with a signer agnostic event sourcing aggregate store backed `SignerInfo` and `SignerMapper` (#537, #540).

* Factored `KrillSigner`, `SignerRouter` and `SignerProvider` types out to their own modules.

* Extends `SignerRouter` with the concept of active vs pending signers and registration/binding of signers based on generating & signing with a signer identity key.

* Extends signers with operator friendly names, internal aggregate store handles, backend "info" metadata, and support for signer registration/binding.

* Various KMIP error handling & logging tweaks including:
    * Raise new `SignerRouter` related `SignerUnavailable` and `SignerUnusable` internal error codes.
    * Upgrade `kmip-protocol` dependency to gain and handle a new `ItemNotFound` error.

.github/workflows/ci.yml

@@ -71,6 +71,9 @@ jobs:
   hsmtest:
     name: hsmtest
     runs-on: ubuntu-18.04
+    strategy:
+      matrix:
+        features: ["hsm", "hsm,hsm-tests"]
     steps:
     - name: Checkout repository
       uses: actions/checkout@v1
@@ -91,7 +94,7 @@ jobs:
 
     - name: Compile the tests
       run: |
-        cargo build --tests --no-default-features --features hsm,hsm-tests
+        cargo build --tests --no-default-features --features ${{ matrix.features }}
 
     - name: Run the tests against the PyKMIP server
       run: |

Cargo.toml

@@ -36,7 +36,7 @@ hex                   = "^0.4"
 hyper                 = { version = "^0.14", features = ["server"] }
 intervaltree          = "0.2.6"
 jmespatch             = { version = "^0.3", features = ["sync"], optional = true }
-kmip                  = { version = "0.3.1", package = "kmip-protocol", features = ["tls-with-openssl"], optional = true }
+kmip                  = { version = "0.4.0", package = "kmip-protocol", features = ["tls-with-openssl"], optional = true }
 libflate              = "^1"
 log                   = "^0.4"
 openidconnect         = { version = "^2.0.0", optional = true, default_features = false }
@@ -69,14 +69,18 @@ rustc_version   = "0.2.3"
 
 [features]
 default = [ "multi-user" ]
-rta = []
-aspa = []
-multi-user = [ "basic-cookies", "jmespatch/sync", "regex", "oso", "openidconnect", "rpassword", "scrypt", "unicode-normalization", "urlparse" ]
-ui-tests = []
 extra-debug = [ "rpki/extra-debug" ]
+multi-user = [ "basic-cookies", "jmespatch/sync", "regex", "oso", "openidconnect", "rpassword", "scrypt", "unicode-normalization", "urlparse" ]
 static-openssl = [ "openssl/vendored" ]
-all-except-ui-tests = [ "multi-user", "rta", "static-openssl", "aspa" ]
+
+# Preview features - not ready for production use
+aspa = []
+rta = []
 hsm = ["backoff", "kmip", "r2d2"]
+
+# Internal features - not for external use
+all-except-ui-tests = [ "multi-user", "rta", "static-openssl", "aspa" ]
+ui-tests = []
 hsm-tests = []
 
 # Make sure that Krill crashes on panics, rather than losing threads and
@@ -85,7 +89,7 @@ hsm-tests = []
 panic = "abort"
 
 [dev-dependencies]
-# for user management
+# For user management
 tiny_http = { version = "^0.8", features = ["ssl"] }
 ctrlc = "^3.1"
 
@@ -181,7 +185,7 @@ assets = [
 post_install_script = "/usr/share/krill/rpm/postinst $*"
 post_uninstall_script = "/usr/share/krill/rpm/postrm $*"
 
-# ensure that the useradd tool is present by depending on its package
+# Ensure that the useradd tool is present by depending on its package
 [package.metadata.generate-rpm.requires]
 shadow-utils = "*"
 

doc/development/hsm/architecture.md

@@ -80,18 +80,79 @@ With the addition of HSM support there may be multiple concurrently active Signe
 ```
 Krill calling code -> KrillSigner -> SignerRouter -> SignerProvider -> One of: OpenSslSigner, KmipSigner or Pkcs11Signer.
                                           |                              |
-                                          +--------> KeyMapper <---------+
+                                          +-------> SignerMapper <-------+
 ```
 
-_(note: in the current version of the code there is only an in-memory KeyMap used only by the KmipSigner and only ever
- one active Signer impl)_
-
 The SignerRouter either:
-- Uses the KeyMapper to determine which Signer owns the KeyIdentifier being used, OR
+- Uses the SignerMapper to determine which Signer owns the KeyIdentifier being used, OR
 - Uses internal rules to decide which Signer to invoke (e.g. always generate random values and do one-off signing using
   the OpenSslSigner for example)
 
 The Signers:
-- Update the KeyMapper when new keys are created to indicate that this Signer owns the key.
-- Store KeyIdentifier to HSM internal key identifier mappings in the KeyMapper.
-- Retrieve HSM internal key identifiers from the KeyMapper based on the KeyIdentifier being used.
+- Update the SignerMapper when new keys are created to indicate that this Signer owns the key.
+- Store KeyIdentifier to HSM internal key identifier mappings in the SignerMapper.
+- Retrieve HSM internal key identifiers from the SignerMapper based on the KeyIdentifier being used.
+
+The SignerRouter also manages signer instances based on configuration settings:
+  (note: in the current version the configuration is hard-coded)
+- Create instances of the appropriate signers (structs that implement the Signer trait) based on configuration.
+- On attempts to use signers:
+  1. Bind all pending signers.
+  2. Select the appropriate signer for the request.
+  3. Delegate to the signer, if bound.
+
+Binding of signers is done by cryptographically connecting the signer configuration/backend to the SignerMapper stored
+keys:
+- Created signers have a reference to the `SignerMapper` and an uninitialized `Handle`.
+- Created signers are added to an in-memory collection of "pending" signers in the `SignerRouter`.
+- When the `SignerRouter` receives a signing request, when no signers previously existed:
+  - For each pending signer request that it create a key pair. If it is not yet contactable we will try again later.
+  - Combine the string forms of the KeyIdentifier from the public key and the signer internal id of the private half of
+    the key and use that as a "handle" for the newly created signer. This proves that we can connect to the signer and
+    use it and that it can create RSA key pairs.
+  - Verify that the signer is able to correctly sign a given challenge string such that verification of the created
+    signature using the public key works. This proves that the signer can perform signing operations and that the 
+    created signatures are valid.
+  - Add a `SignerInfo` to the `SignerMapper` internal `AggregateStore` using the new signer handle, and store the entire
+    created public key as metadata attached to the `SignerInfo`. We also attach at this point any details about the
+    signer backend, e.g. type, vendor, FQDN and port number (for KMIP) or path on disk (for OpenSSL) etc.
+  - Tell the `Signer` instance the `Handle` it should use with the `SignerMapper` reference it has to lookup existing
+    keys and record new keys. The `SignerMapper` store is located on disk as a `signers` subdirectory under the Krill
+    data directory.
+  - Remove the signer from the pending collection.
+
+On subsequent restarts of Krill:
+- When the `SignerRouter` receives a signing request and signers previously existed:
+  - For each `SignerInfo` `Handle` stored in the `SignerMapper` extract the signer internal private key id from the
+    `Handle` and ask the signer to sign a challenge string using the specified private key.
+    - If we can't contact the signer at this point we will try again later.
+    - If we can contact the signer and it doesn't know the private key id then this signer configuration/signe
+      instance is not associated with this `SignerInfo` `Handle` and key store.
+    - If the signer returns signed data, verify it using the public key stored with this `Handle` `SignerInfo`. If
+      the verification fails it might be that the signer used simple internal key ids (e.g. PyKMIP uses ascending
+      integer numbers) and so we accidentally found the "right" key but this isn't the right signer.
+    - Otherwise this is the right signer so tell the `Signer` instance the `Handle` it should use with the
+     `SignerMapper` reference that it has.
+     - We also record a "name" metadata property with each `SignerInfo`. If the name and/or signer backend details
+       have changed then they are updated in the `SignerStore` 
+    - Remove the signer from the pending collection.
+
+This gives us verified connections and confirmation of capability in a standard way irrespective of signer type,
+between the configuration specified by an operator and the actual signer backend such as a local OpenSSL key store or
+a "remote" PKCS#11 or KMIP HSM service. The operator can rename the instance without impacting the mapping and
+doesn't need to manage a signer specific identifier correctly, and we can correctly route signing requests to the
+right signer without knowing anything about how to relate the configuration to the actual signer backend. If the
+operator replaces a HSM server and restores data from backup and the new one is on a different IP address and/or
+uses different authentication credentials we can work out for ourself which existing keys it should be used with.
+If the operator connects a new HSM not used by this Krill instance before, even if used by a different Krill instance,
+we can use it. We're also capable of using HSMs when they come online without blocking the operation of Krill.
+
+If we were to also drop signers from the active set when they are unreachable we could automagically migrate from
+a live HSM to a cold spare (assuming the spare has most of the data from the live HSM, in particular the "registration"
+key that we created).
+
+No connection details or secrets are stored in the `SignerInfo` store. If later the `AggregateStore` mechanism is
+upgraded to work with some central key value store service used between multiple Krill instances then each Krill
+instance could have its own connection details and secrets and connect to separate HSMs in a cluster and both
+think that they are talking to the same "signer" because of the presence of the "registration" signing key in
+both HSM nodes and the public half being available to both Krill nodes via the central KV store.

doc/development/hsm/overview.md

@@ -15,6 +15,10 @@ The feature adds (or will add) the following to Krill:
     - To sign with a key the request must be delegated to the correct signer.
     - Therefore we must keep track of which signer "owns" each key.
 
+  - Signer tracking:
+    - To delegate a request to the correct signer we must know which signer "instance" corresponds to which signer
+      "configuration".
+
   - Random value generation fallback support:
     - Not all PKCS#11 or KMIP compatible devices support generating random values.
     - Fallback in such cases to the OpenSSL signer (or to a user specified signer?)

src/commons/api/ca.rs

@@ -2664,12 +2664,9 @@ impl ResourceSetError {
 #[cfg(test)]
 mod test {
     use bytes::Bytes;
-
     use rpki::repository::crypto::PublicKeyFormat;
 
-    use crate::commons::crypto::signers::softsigner::OpenSslSigner;
-
-    use crate::test;
+    use crate::{commons::crypto::OpenSslSigner, test};
 
     use super::*;
 
@@ -2706,7 +2703,12 @@ mod test {
     #[test]
     fn mft_uri() {
         test::test_under_tmp(|d| {
+            #[cfg(not(feature = "hsm"))]
             let signer = OpenSslSigner::build(&d).unwrap();
+
+            #[cfg(feature = "hsm")]
+            let signer = OpenSslSigner::build(&d, "dummy", None).unwrap();
+
             let key_id = signer.create_key(PublicKeyFormat::Rsa).unwrap();
             let pub_key = signer.get_key_info(&key_id).unwrap();
 

src/commons/crypto/mod.rs

@@ -10,6 +10,4 @@ pub use self::error::*;
 mod signing;
 pub use self::signing::*;
 
-pub mod signers;
-
 pub type CryptoResult<T> = std::result::Result<T, self::error::Error>;