Implement activation registry (closes #17)

[daniil-berg]

As described in #17, this adds the ability for the user to register his own activation functions and have them (de-)serialized properly.

The interface is comprised of two associated functions for the Activation struct:

• register takes a name, a pointer to the actual activation function and a pointer to the derivative function (just like the new constructor), creates a new Activation with those and inserts it into the registry. If an instance with the same name was already present, it is replaced and the old one is returned (wrapped in Some(...)). Otherwise None is returned.

  tensorevo/src/activation.rs

  Lines 108 to 113 in 4895317

  	pub fn register<S: Into<String>>(name: S, function: TFunc<T>, derivative: TFunc<T>) -> Option<Self> {
  	let name: String = name.into();
  	let registry_lock = Self::get_activation_registry();
  	registry_lock.write().unwrap()
  	.insert(name.clone(), Activation::new(name, function, derivative))
  	}

• from_name now returns a clone of the Activation instance registered under the provided name (wrapped in Some(...)). If none was registered under that name, None is returned. Since from_name used to return an Activation proper, this is technically a breaking change.

  tensorevo/src/activation.rs

  Lines 131 to 136 in 4895317

  	pub fn from_name<S: Into<String>>(name: S) -> Option<Self> {
  	let registry_lock = Self::get_activation_registry();
  	registry_lock.read().unwrap()
  	.get(&name.into())
  	.and_then(|activation| Some(activation.clone()))
  	}

The implementation relies on the generic_singleton::get_or_init macro and the registry (for a specific type Activation<T>) is a static singleton. The ActivationRegistry<T> is simply a type alias for HashMap<String, Activation<T>>.

tensorevo/src/activation.rs

Lines 67 to 73 in 4895317

	fn get_activation_registry() -> &'static RwLock<ActivationRegistry<T>> {
	get_or_init!(|| {
	let registry_lock = RwLock::new(ActivationRegistry::<T>::new());
	Self::register_common(&registry_lock);
	registry_lock
	})
	}

During initialization of the registry singleton, the pre-implemented common activation functions (currently only sigmoid and relu) are added automatically. Those are then available by default, but can be replaced by custom implementations via register at any point.

tensorevo/src/activation.rs

Lines 79 to 89 in 4895317

	fn register_common(registry_lock: &RwLock<ActivationRegistry<T>>) {
	let mut registry = registry_lock.write().unwrap();
	let _ = registry.insert(
	"sigmoid".to_owned(),
	Self::new("sigmoid".to_owned(), sigmoid, sigmoid_prime),
	);
	let _ = registry.insert(
	"relu".to_owned(),
	Self::new("relu".to_owned(), relu, relu_prime),
	);
	}

There is still a lot of unwrapping going on there with the RwLock, but since we have not cleaned that up anywhere else in the code yet, I did not bother to handle potential errors more cleanly yet.