Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dlt pipeline -v <pipeline> trace source password not redacted #1687

Closed
devcovato opened this issue Aug 14, 2024 · 1 comment · Fixed by #1797
Closed

dlt pipeline -v <pipeline> trace source password not redacted #1687

devcovato opened this issue Aug 14, 2024 · 1 comment · Fixed by #1797
Assignees
Labels
bug Something isn't working community This issue came from slack community workspace sprint Marks group of tasks with core team focus at this moment

Comments

@devcovato
Copy link

dlt version

0.5.2

Describe the problem

Running dlt pipeline -v <pipeline> trace the source password is not redacted listing the resolved config and secret values.

Expected behavior

I expect the source password to be redacted.

Steps to reproduce

Create the pipeline sql_database using MariaDB as source and Postgres as destination.

Operating system

Linux

Runtime environment

Docker, Docker Compose

Python version

3.11

dlt data source

No response

dlt destination

Postgres

Other deployment details

No response

Additional information

I'm using Python 3.12. The Bug Report doesn't list it.

@rudolfix rudolfix added bug Something isn't working community This issue came from slack community workspace labels Aug 14, 2024
@rudolfix
Copy link
Collaborator

rudolfix commented Aug 14, 2024

  1. We should stop dumping secrets in traces altogether. Currently we hide them when trace is converted into dict so secrets are not loaded, but they indeed will appear in strings
  2. deployment scripts use dumps to generate deployment instructions. we need to update them to take secrets from configs (via dlt.secrets.get() accessor). if not available - we should show error message but not raise

some implementation detials:

  1. there's util method is_secret_hint that tells if given hint contains secret or not
  2. SerializableResolvedValueTrace contains value and default_value. those should be set to empty strings when is_secret_hint is True
  3. fixing the deployment flow is the most tricky because you'll need to test it. find _echo_secrets we are displaying value. try to get this value from dlt.secrets getter (using the stored sections`) if not found display a red info to the user tha s/he needs to do it themselves

@rudolfix rudolfix self-assigned this Aug 19, 2024
@rudolfix rudolfix added the sprint Marks group of tasks with core team focus at this moment label Aug 26, 2024
@rudolfix rudolfix assigned willi-mueller and unassigned rudolfix Sep 9, 2024
@willi-mueller willi-mueller linked a pull request Sep 10, 2024 that will close this issue
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working community This issue came from slack community workspace sprint Marks group of tasks with core team focus at this moment
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants