adds registries and plugins

[rudolfix]

Description

Introduced changes

1. Represents secret types as Annotated, not a NewType which allows to use regular ie. string as initializers without typechecker complaining. also secret types can be dynamically created without breaking any rules

TSecretValue = Annotated[Any, SecretSentinel]

2. dtl.source decorator creates DltSourceFactoryWrapper instead of wrapping source function. this class is callable, preserves signature and allows to rename the source and change other parameters at runtime (with_args). see how this is applied in rest_api
3. RunContext is a new context available via Container that tells dlt where is the current run root, settings folder (ie. toml files), data folder (with pipelines dir) etc. a default context is using the same paths that were "hardcoded" before.
4. A plugin system based on pluggy is added with one hook active that allows to override the default run context with plugged class. There's also an example and a test. works pretty well.
5. Mind that pluggy is also an injected context. so it is created on demand and in theory can also be a plugin :)
6. Sources Registry is greatly improved. It keeps source factories now (so you can create renamed wrappers out of it and then call them at will). it introduces following reference format

<run context name>.<section/module>.<function/name>

with shorthands
<section/module>.<function/name> - run context name taken from run context
<section/module> if equals function name (ie sql_database is a valid reference)

when source is being added to a registry, the name of current run context is used
references not in registry will be assumed to be importable python modules with fortmat exactly like we use in the destination.

Less important changes:

1. I fixed init scripts
2. I refactored providers (so they can use run_context)

TODO in this PR:

1. finalize from_reference in SourceReference. to

• try to import from dlt (defaullt) run context if we cannot resolve for current run context (so sql_database will resolve in any context)
• import python modules
• maybe extract base class for EnttityReference to reuse destination code.

2. add some tests (maybe) - ie. to test run context in runtime tests

TODO in next PR:

1. add destination registry and register our "standard" destinations as too, via destination decorator
2. add a decorator to register config providers instread of current code. maybe a pluggy plugin
3. rethink where to keep the hook specs for pluggy (many files or just one that declares all hooks)
4. rewrite airflow helper with its own run context (or proxying one that modified the behavior of the current one) and create context manager that will activate it
5. allow type dicts to be used in resolve_configuration

• use those to validate dicts if available from config
• bind secret values declared in dicts
• IMO it can be implemented as specialized validator that also resolves secret values

[netlify]

✅ Deploy Preview for dlt-hub-docs ready!

Name	Link
🔨 Latest commit	6a216b7
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/67064a151cdcbc00080be85b
😎 Deploy Preview	https://deploy-preview-1894--dlt-hub-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.