Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Network architecture #9

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions authors.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,6 @@
"simon": "Simon Masson",
"naqib": "Naqib Zarin",
"tobias": "Tobias Heindel",
"tg-x": "TG x Thoth",
"yulia": "Yulia Khalniyazova"
}
Binary file added tg-x/media/avatar.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added tg-x/media/network-architecture.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
266 changes: 266 additions & 0 deletions tg-x/network-architecture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,266 @@
---
title: Anoma Network Architecture
category: distributed-systems
publish_date: 2024-07-07
image: media/network-architecture.jpg
excerpt: A brief introduction to the network architecture of Anoma.
---

## Introduction
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why should reader read this?

Is it cool, new, amazing, ... (I know, but reader doesn't ...


Anoma is researching and developing a set of network protocols
that facilitate message passing in a distributed system.

The network protocol stack is designed as a standalone component
generically useful for building distributed systems.

This article serves as a brief introduction and high-level overview of the network architecture.

## Node architecture

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Diagram would be great to add here to help the reader understand.

The system is inspired by the *actor model*,
where each node runs a set of processes, or actors.
We call these *engines* in our design.
Engines have state and communicate with each other via asynchronous message passing:.
An engine may send and respond to messages,
react to timer events, and spawn other engines.

Each node has a cryptographic identity that uniquely identifies it in the network.
A node consists of a collection of running engines.
Local engines communicate with each other directly,
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

directly
→ unencrypted message passing

or do you mean

directly
→ directly (in principle via shared memory)

not sure how to avoid the idea that we suddenly do not do message passing any more

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

both, though in this context it refers to direct messages between local engines,
and thus not via the router/transport engines

while they reach remote nodes via the Router and Transport engines.

The Transport engine is responsible for maintaining authenticated and encrypted network connections between nodes
via a modular transport system that supports various transport protocols with different reliability, latency, and privacy guarantees.
Transport protocols initially supported include QUIC and TLS,
later this can be extended with onion-routed, mix, mesh, and delay-tolerant network protocols.

The Router engine routes incoming and outgoing messages from and to local engines,
based on the destination address of each message, which consists of a node and engine identity.

<figure>

```mermaid
flowchart LR
T((Transport))
R(((Router)))
X((Engine_X))
Y((Engine_Y))

X & Y --- R --- T
```

<figcaption>A node with its engines.</figcaption>
</figure>

## Communication patterns

The network supports unicast, multicast, and anycast communication patterns.

- *Unicast*: direct messages between two engines.
- *Multicast*: few-to-many messages sent to a publish-subscribe topic by authorized publishers and delivered to all subscribers.
- *Anycast*: messages sent to any known member of a domain (see the next sections).

<figure>

```mermaid
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in this figure,
ideally we would have two types of edges

  • links/connections/direct acquaintances
  • message hops

sth. like this
image

https://excalidraw.com/#json=qcijAZi3QPhjvbSiTntcl,yxzrxo8AKJKJ-E9gJS4_3w

flowchart LR

subgraph NA[Node_A]
AT((Transport_A))
AR(((Router_A)))
AX((Engine_AX))
AY((Engine_AY))

AX --> AR --> AT
AY -.- AR

end

AT --> BT

subgraph NB[Node_B]
BT((Transport_B))
BR(((Router_B)))
BX((Engine_BX))
BY((Engine_BY))

BT --> BR --> BX
BY -.- BR
end
```

<figcaption>Unicast communication: Engine AX sends a message to BX via the Router And Transport engines.</figcaption>
</figure>

## Network Architecture

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here would be an excellent place to add a diagram that matches the text below.

The network consists of nodes that may communicate with each other either directly or via relays,
and may participate in any number of domains.

Each node has cryptographic identity and a number of transport address(es)
specified in their signed node advertisements.

The core network has low churn where nodes have publicly reachable IP addresses,
while mobile nodes of end-users participate in high-churn edge networks,
where they communicate directly with other mobile nodes,
and do not advertise their public IP address,
instead reach the core network via relays
that store and forward messages for them.

This results in a two-tier network architecture,
where relays provide reliable message delivery and basic location privacy for mobile nodes.
Mobile nodes with higher location privacy requirements
may use an onion-routed or mix networks to connect to the core network.

<figure>

```mermaid
flowchart LR

subgraph Core
A((A))
B((B))
C((C))
D((D))
R1((R1))
R2((R2))

A --- B & C --- D
C --- R1
D --- R2
end

subgraph Edge_A
K((K))
L((L))
M((M))
N((N))
O((O))

K --- L --- M & N --- O
end

subgraph Edge_B
P((P))
Q((Q))
R((R))
S((S))

P --- Q & R --- S
end

R1 -.- L & P
R2 -.- O & R
```

<figcaption>Two-tier network architecture with core & edge networks, connected via relays R1 & R2.</figcaption>
</figure>

## Domains

The network consists of sovereign domains.
A domain is a sub-network with its own *cryptographic identity*.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nodes also have crypto ids

  • what's the difference?
  • what are the commonalities

Domain creation is permissionless, it only requires a domain advertisement signed by the domain owner(s),
which contains the domain configuration.
Domain identity is based on a public key, and may use threshold cryptography.
The domain configuration specifies the authentication mechanism used to join the domain,
the protocols used in the domain, and the list of nodes that serve external requests.
This enables *pluralistic interoperability*,
i.e. domains may run different protocols internally,
but able to respond to external requests nevertheless.

The domain's *membership* is the list of nodes
that are allowed to participate in the network protocols inside the domain.
A domain may be instantiated as one or more overlay networks that may or may not be inter-connected,
depending on the protocols used in the domain.
This allows both single-instance domains with globally synchronized data structures,
as well as [grassroots](https://arxiv.org/abs/2301.04391) domains
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a brief compare contrast b/w grassroots domains and single domains beyond what is stated would be helpful

that may have a number of instances in the core network and on various edge networks,
and use mergeable data structures, such as [CRDTs](https://crdt.tech).

The initial set of protocols that run in a domain consists of
a membership protocol, a topic-based publish-subscribe protocol,
and a block storage protocol.
Later, domains with higher privacy requirements
will be able to opt to be part of a mix network
that provides pub/sub and block storage services.

A *domain advertisement* serves the purpose of advertising
a list of nodes that serve join requests and external requests from non-members.
Join requests may require an authentication mechanism,
such as a pre-shared key, public-key certificates, or other forms of membership proof.
External requests are used by non-members
to request information from the domain without joining,
and may require an authentication token.

Anycast routing is used to send messages to any known member of a domain that serve requests.

A topic-based publish-subscribe (pub/sub) protocol
allows domain members to subscribe to topics of interest,
where authorized publishers for a given topic may publish messages.

A block storage protocol allows limited persistence of data,
which is integrated with the pub/sub protocol
such that storage nodes willing to store a published message
publish an acknowledgement that indicates how long they're willing to store the message.

<figure>

```mermaid
flowchart LR

subgraph DA[Domain_A]
A(((A)))
B(((B)))
C((C))
D((D))
E((E))
F((F))

A --- E & C
B --- C --- D
B --- F
C --- F
D --- E
F --- E
end

M((M))
N((N))

M -. JoinReq .-> A
N -. ExtReq .-> B
```

<figcaption>A domain overlay where nodes A & B handle external requests.</figcaption>
</figure>

## Summary

The network architecture of Anoma consists of
nodes in a two-tier network architecture,
sovereign domains that allow pluralistic interoperability,
messaging inspired by the actor model with unicast, multicast, and anycast communication pattern,
and a modular transport system with various transport protocols
with varying levels of reliability, latency, and privacy guarantees.

More details will be published in the upcoming report at [Anoma Research Topics](https://art.anoma.net).

<style>
.language-mermaid {
width: 100%;
}

.language-mermaid .cluster > rect {
stroke: #caa !important;
fill: #eee !important;
fill-opacity: 50%;
}

.language-mermaid .node > :first-child,
.language-mermaid .node > :first-child * {
stroke: #c33 !important;
fill: #fee !important;
}
</style>