Support different cluster topologies

[derekkraan]

Currently every node is connected to every other node (via delta_crdt). It should be possible to define other topologies to make synchronization more efficient.

[beardedeagle]

Has any thought been given about how libraries like libcluster or partisan would come into play here?

[derekkraan]

I think libcluster is complimentary to Horde.

If it's going to mean anything for this feature, then it'll have to be a new feature in libcluster I think.

I haven't given much thought to partisan, I'm not 100% sure what it actually is / does.

[shikanime]

If I understood correctly, here is an example of how i use the libcluster callbacks to form my Horde cluster and connect to the rest as another service in a microservices cluster using another topology:

topologies = [
  replicas: [
    strategy: Cluster.Strategy.Gossip,
    config: [secret: "my-secret"],
    connect: {MyApp.HordeFederation, :connect, [MyApp.HordeFederation]},
    disconnect: {MyApp.HordeFederation, :disconnect, [MyApp.HordeFederation]}
  ],
  my_other_service: [
    strategy: Cluster.Strategy.Gossip,
    config: [secret: "my-secret-for-other-service"],
  ]
]

defmodule MyApp.HordeFederation do
  use GenServer

  defstruct cluster: nil,
            nodes: []

  def start_link(options \\ []) do
    keys = [:cluster]
    {server_opts, start_opts} = Keyword.split(options, keys)
    GenServer.start_link(__MODULE__, server_opts, start_opts)
  end

  def connect(federation, node) do
    GenServer.cast(federation, {:nodeup, node})
  end

  def disconnect(federation, node) do
    GenServer.cast(federation, {:nodedown, node})
  end

  def init(options) do
    :net_kernel.monitor_nodes(true)

    nodes = [Node.self()]
    cluster = Keyword.fetch!(options, :cluster)

    state = %__MODULE__{
      cluster: cluster,
      nodes: nodes
    }

    {:ok, state, {:continue, :recluster}}
  end

  def handle_continue(:recluster, state) do
    Horde.Cluster.set_members(
      state.cluster,
      Enum.map(state.nodes, &{state.cluster, &1})
    )

    {:noreply, state}
  end

  def handle_info({:nodedown, node}, state) do
    nodes = state.nodes -- node
    state = %{state | nodes: nodes}
    {:noreply, state, {:continue, :recluster}}
  end

  def handle_info({:nodeup, node}, state) do
    nodes = [node | state.nodes]
    state = %{state | nodes: nodes}
    {:noreply, state, {:continue, :recluster}}
  end
end

[derekkraan]

To clarify this issue, currently Horde sets up delta_crdt in a mesh network that mirrors the erlang cluster 1:1. This is not necessarily the most efficient way to set up a network, and that's why you have a project like partisan, which can work with other topologies.

Because a mesh network has O(n^2) connections, it does not scale well. There are other topologies possible, such as a ring topology, with O(n) connections, or a tree topology, also O(n). Partially connected mesh is also possible, with something in between.

Anyways, using Partisan could be a possibility in the future for even larger clusters, but we could start with this feature, adding the ability to hook up the underlying delta_crdt in different cluster topologies.

So this issue isn't about libcluster topologies, but about the topology of the delta_crdt network underpinning Horde.