target-s3

target-s3 is a Singer target for s3.

Build with the Meltano Target SDK.

Configuration

Accepted Config Options

Capabilities

• about
• stream-maps
• schema-flattening

A full list of supported settings and capabilities for this target is available by running:

target-s3 --about

Settings

Setting	Required	Default	Description
aws_access_key	False	None	The aws secret access key for auth to S3.
aws_secret_access_key	False	None	The aws secret access key for auth to S3.
aws_region	True	None	The aws region to target
bucket	True	None	The aws bucket to target.
prefix	False	None	The prefix for the key.
append_date_to_prefix	False	None	A flag to append the date to the key prefix.
append_date_to_prefix_grain	False	None	The grain of the date to append to the prefix.
append_date_to_filename	False	None	A flag to append the date to the key filename.
append_date_to_filename_grain	False	None	The grain of the date to append to the filename.
object_format	False	None	The format of the storage object.
flatten_records	False	None	A flag indictating to flatten records.
stream_maps	False	None	Config object for stream maps capability. For more information check out Stream Maps.
stream_map_config	False	None	User-defined config values to be used within map expressions.
flattening_enabled	False	None	'True' to enable schema flattening and automatically expand nested properties.
flattening_max_depth	False	None	The max depth to flatten schemas.

Configure using environment variables

This Singer target will automatically import any environment variables within the working directory's .env if the --config=ENV is provided, such that config values will be considered if a matching environment variable is set either in the terminal context or in the .env file.

Source Authentication and Authorization

Usage

You can easily run target-s3 by itself or in a pipeline using Meltano.

Executing the Target Directly

target-s3 --version
target-s3 --help
# Test using the "Carbon Intensity" sample:
tap-carbon-intensity | target-s3 --config /path/to/target-s3-config.json

Developer Resources

Follow these instructions to contribute to this project.

Initialize your Development Environment

pipx install poetry
poetry install

Create and Run Tests

Create tests within the target_s3/tests subfolder and then run:

poetry run pytest

You can also test the target-s3 CLI interface directly using poetry run:

poetry run target-s3 --help

Testing with Meltano

Note: This target will work in any Singer environment and does not require Meltano. Examples here are for convenience and to streamline end-to-end orchestration scenarios.

Next, install Meltano (if you haven't already) and any needed plugins:

# Install meltano
pipx install meltano
# Initialize meltano within this directory
cd target-s3
meltano install

Now you can test and orchestrate using Meltano:

# Test invocation:
meltano invoke target-s3 --version
# OR run a test `elt` pipeline with the Carbon Intensity sample tap:
meltano elt tap-carbon-intensity target-s3

SDK Dev Guide

See the dev guide for more instructions on how to use the Meltano Singer SDK to develop your own Singer taps and targets.