v0.7.2

What's Changed

• Revert missing #53 from #63 by @firehist in #65
• feat: check for bucket's existence before creation by @tomasvotava in #67
• fix: use bigquery_client for gcs_stage methods by @tomasvotava in #70
• refactor(core): UUID suffix instead of Unix timestamp by @emesar in #72
• feat: add date suffix in addition to UUID suffix for a well organized data lake (GCS Load Patten) by @maamoonhussain in #77
• Fix "TypeError: unhashable type: 'list'" when cluster_on_key_properties is enabled by @buu-nguyen in #78
• ♻️ Remove duplicate line in BigQueryGcsStagingSink constructor by @buu-nguyen in #79
• escaped table names while dedup by @vrumjantsev in #80
• added Forbidden in dataset create error catching by @loveeklund-osttra in #85
• Edge case for object properties with empty schemas by @rubenvereecken in #86
• 🔧 add timeout also for batch jobs in load_table_from_uri by @AlejandroUPC in #90
• 🔧 change logic to see if ds or table exists by @AlejandroUPC in #91

New Contributors

• @firehist made their first contribution in #65
• @tomasvotava made their first contribution in #67
• @emesar made their first contribution in #72
• @maamoonhussain made their first contribution in #77
• @buu-nguyen made their first contribution in #78
• @vrumjantsev made their first contribution in #80
• @loveeklund-osttra made their first contribution in #85
• @rubenvereecken made their first contribution in #86
• @AlejandroUPC made their first contribution in #90

Full Changelog: 0.6.8...0.7.2

v0.6.8

Full Changelog: 0.6.7...0.6.8

chore: relax python version to allow 3.11

v0.6.7

What's Changed

• fix: Import Sink from singer_sdk package root by @edgarrmondragon in #20
• fix: View Creation fails with with reserved words in project name by @chollinger93 in #22
• Patch/filter null arrays by @chollinger93 in #24
• Always pass the project ID to Google API clients by @pdfrod in #31
• Wrap column name in quotes by @jcbmllgn in #33
• fix: Escaping table identifiers for deduplication and overwrite cases by @emishas in #38
• matf/stop-overwriting-state-on-empty-state by @shopmatfournier in #53
• Fixing Job class (attempts) + Waiting for workers to kill BigQuery streams + Fixing workers streams/offsets + Regulate global queue by @levavaq in #63

New Contributors

• @edgarrmondragon made their first contribution in #20
• @chollinger93 made their first contribution in #22
• @pdfrod made their first contribution in #31
• @jcbmllgn made their first contribution in #33
• @emishas made their first contribution in #38
• @shopmatfournier made their first contribution in #53
• @levavaq made their first contribution in #63

Full Changelog: 0.6.3...0.6.7

v0.6.3

What's Changed

• See CHANGELOG for details.

New Contributors

• @mattsax made their first contribution in #19

Full Changelog: 0.6.2...0.6.3

v0.6.2

What's Changed

• Add support for configuration of Bigquery dataset location by @VinceLegendre in #13
• Correct stream type for storage_write_api method in batch by @VinceLegendre in #17
• Implemented changie for changelog management moving forward
• Implemented overwrite and upsert methods for loading tables controllable via config, was previously append only

New Contributors

• @VinceLegendre made their first contribution in #13

Full Changelog: 0.5.2...0.6.2

NOTE:
We will look to automate generating github releases moving forward.

v0.5.2

Full Changelog: 0.5.0...0.5.2

This release is a patch release since there is no breaking change on the surface level but under the hood we have moved all worker management logic up from the stream level to the target level creating much more intuitive worker pool. Architecturally, it simplifies all of the sink implementations as can be observed in the code. This also includes general improvements in the storage write API implementation and dynamic protocol buffer schema generation. The worker pool by default is based on CPU cores.

v0.5.0

Full Changelog: 0.4.0...0.5.0

This release marks the completion of a large refactor to further boost performance, make the architecture of the sinks more scalable, and introduces denormalized loading for the Storage Write API making it the preferred method in most situations. Too many changes to list out. But most of it is encapsulated in the README. A brave soul may inspect the diff if they feel so inclined.

Our test suite provided a high degree of confidence all throughout the updates. Acceptance testing was done in a staging followed by a production environment at reasonable scale preceding this release.

v0.4.0

This release marks a milestone in target production worthiness. Accumulated updates include a full test suite with integration tests and many hours of production exposure across various systems, with various extractors, delivering data at scale. This is the most versatile BigQuery loader available to my knowledge.