Brainstorming: atproto Dev Tooling and Experience

[bnewbold]

Gathering ideas to make it easier to work with atproto: building entire new Lexicions, working with data, etc.

[bnewbold]

Infra Enumeration

Endpoints on existing infra (relay, appview) could make it easier to discover other infra services in the network:

• Relays could enumerate all the PDS instances they know of
• Relays already can enumerate all the accounts they know of
• PLC directory could have a way to enumerate DIDs, instead of just exporting an audit log
• AppView could enumerate feed generators
• AppView could enumerate labelers

These should be relatively cheap queries to service, and make it so folks don't need to backfill the whole network for some basic tasks.

[itaru2622]

PDS can enumerate its managing accounts with emails it knows of (for admin use cases)

[bnewbold]

Query Accounts and Filtered Firehose

When working with new Lexicons/apps, there are probably only dozens/hundreds of folks using it and writing records to their repos. Right now services need to backfill the entire network of millions of accounts just to find that tiny fraction of users.

Backfilling everything will probably always be the most complete and correct option, but it would be really helpful to just get a list of accounts which have any of the records included.

Related, possibly a way to filter the firehose to only accounts which have touched certain record types. There are some concerns about validating repo content if only commits with specific collections are returned, but just filtering to accounts would already go a long way.

(this has been tossed around as an idea for a very long time)

[emilyhunt]

@MarshalX and I have done some profiling of the ATProto SDK for Python's peak firehose performance and tried to increase it more. Maybe the numbers are helpful for this discussion to know what the current state of 3rd party infra is at!

Right now, we can consume 25,000 ops/second peak [1] when running on a server with dedicated CPU cores [2], at which point it gets CPU-bound and we'd need a fancier multithreaded solution. Still though, this is really far above the current ops/second speed on the network, and suggests that there's some headroom left.

Thinking ahead to the future, it's really difficult to find out how many tweets are sent per second on Twitter (all the data I can find is really old), but (pre-Musk) it seemed to average at 6000/second. This implies that existing 3rd party firehose infra could already cope with a Twitter-like amount of posts if we had a filtered firehose that just shows posts. In practice, I think most feeds don't use likes, so filtering by record type may already go a long way in a pinch. No clue if different record types take more/less time to process (probably not, as the extra text field in posts isn't that big).

[1] this is only for a firehose consumer that does nothing, Python's GIL and difficulties with true multithreading make this slower for a real server that actually does things with the commits (7000 ops/sec is more realistic right now), but we can probably improve this and get closer to the 25k/sec theoretical single webhook speed.

[2] Hetzner CCX23 with 4 dedicated CPUs. This costs $30/month, which isn't much for something that could keep up with this much content, and I think is not an unreasonable amount of funding for 3rd party devs to find.

[snarfed]

This is great to hear, @emilyhunt. Thank you! My Python stack for Bridgy Fed is currently topping out at ~200 firehose ops/sec, even after I've moved all logic and even CBOR parsing (!) off the websocket client thread. Clearly not acceptable. Did your profiling include a real websocket client? If so, I expect I'll be looking at it in detail soon.

[snarfed]

@bnewbold I'm obviously very interested in this. Bridgy Fed's use case isn't filtered by record type, though, it's looking for records from, to, or otherwise referencing (eg mentioning) a set of DIDs. Right now that set is 15-20k, but it will grow. Not the easiest candidate use case for a filtering service, I know. 😐

[DavidBuchanan314]

re: subset of DIDs, one way to do the filtering relatively efficiently would be a bloom filter. false-positives don't matter much, just a small portion of wasted bandwidth. A hypothetical service offering a filtered firehose could accept a custom bloom filter as part of the request body, maybe with some protocol for modifying it over time.

[emilyhunt]

This is great to hear, @emilyhunt. Thank you! My Python stack for Bridgy Fed is currently topping out at ~200 firehose ops/sec, even after I've moved all logic and even CBOR parsing (!) off the websocket client thread. Clearly not acceptable. Did your profiling include a real websocket client? If so, I expect I'll be looking at it in detail soon.

Yes, it used the AsyncFirehoseSubscribeReposClient in the Python ATProto SDK. Try running this example script on your hardware and see what speed you can get, which actually does a lot of work all on one thread.

Are you constrained by CPU or internet? I can only get the best speeds on dedicated hosting. I can get just over 2k ops/second at home, where it gets limited by my 50 Mbps internet, but 200 sounds slow. If you're CPU constrained then definitely try the Python implementation, it's very fast for CBOR processing as it uses a Rust implementation for decoding.

[bnewbold]

Lexicon Hub

A service which discovers and aggregates Lexicons. Could help track changes, discover related/interesting lexicons, validate/lint Lexicons themselves, etc.

Could maybe consume from firehose to discover new lexicons being used, and give estimates of popularity/adoption.

Would be an archive/mirror in case upstream hosting disappears.

Would probably need to be combined with (long-planned) Lexicon resolution mechanism.

Could be built on atproto (lexicons in repos!) or just on the web.

[bnewbold]

Interop Test Suite

Language-neutral test vectors and network-level test scripts for checking correctness of atproto code: PDS implementations, SDKs, relay implementations, labelers, etc.

There are some existing test vectors with CAR files (repo and MST structure), and syntax corner-cases (text files with weird handles, DIDs).

HTTP APIs could be tested with https://hurl.dev/, or python scripts.

[bnewbold]

Rich Text Helpers

Parsing post text in to richtext facets is a common pain point.

• test cases for developers: input strings and output richtext
• SDK helpers in more languages
• possibly an appview endpoint to parse text, returning facets

[bnewbold]

Lexicon Styleguide

• simple syntax and type guidelines
• describe how different types can be used: unknown, open unions, etc
• ways of solving common problems
• patterns for specific applications
• possibly linter to check against styles

[snarfed]

These are all 🔥🔥🔥

Wish I had more free time and I could work on some of them! Maybe I should anyway. (That way lie dragons... 😆)

[itaru2622]

requests end to end tests supporting for CI including requestPlcOperationSignature, which responds token by mail(outbound of REST/XRPC)

[itaru2622]

requests enrich HACKING.md for how to

• FROM adding custom API to lexicon TO regenerating/building components to support custom API, because social-app web edition depends on both atproto/packages/api and indigo/api
• debug social-app web edition, because social-app/src/logger seems not working under web edition.

[itaru2622]

request re-birth (recovering) ingester feature in appview v2
to support starting subscribelabels from all labelers with detecting their joining.