incremental `scd2` with `merge_key` #1818

jorritsandbrink · 2024-09-16T13:40:10Z

Description

Adds merge_key support for scd2 to enable incremental extracts.

When a merge_key is provided, absent records are only retired if their merge_key value is present in the source extract.

@dlt.resource(
    merge_key="my_natural_key",
    write_disposition={"disposition": "merge", "strategy": "scd2"}
)
def my_incremental_dim_data():
    ...

Related Issues

Closes #1789

netlify · 2024-09-16T13:40:27Z

✅ Deploy Preview for dlt-hub-docs ready!

Name	Link
🔨 Latest commit	`84c101b`
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/66f7cfb2d32b140008588a65
😎 Deploy Preview	https://deploy-preview-1818--dlt-hub-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

sh-rp · 2024-09-18T12:46:10Z

tests/load/pipeline/test_scd2.py

+    assert_load_info(info)
+    assert load_table_counts(p, "dim_test")["dim_test"] == 3
+    boundary_ts = get_load_package_created_at(p, info)
+    # natural key 1 should now have two records (one retired, one active)


we should also test here that when the row with natural key 1 reverts back to the first version (and has the same row hash as it had in the first load) it will still create a new entry.

sh-rp · 2024-09-18T12:53:27Z

dlt/common/schema/typing.py

    validity_column_names: Optional[List[str]]
    active_record_timestamp: Optional[TAnyDateTime]
    boundary_timestamp: Optional[TAnyDateTime]
    row_version_column_name: Optional[str]
+    retire_if_absent: Optional[bool]


maybe call this "retire_absent_rows"? then it is a bit more clear just from the naming what it does. wdyt?

Yes! Changed it.

…-scd2-incremental

sh-rp · 2024-09-24T11:31:39Z

dlt/common/schema/typing.py

    validity_column_names: Optional[List[str]]
    active_record_timestamp: Optional[TAnyDateTime]
    boundary_timestamp: Optional[TAnyDateTime]
    row_version_column_name: Optional[str]
+    retire_absent_rows: Optional[bool]
+    natural_key: Optional[str]


this is no longer needed, is it?

Whoops. Removed now.

sh-rp · 2024-09-24T11:32:04Z

dlt/extract/hints.py

+                dict_["x-boundary-timestamp"] = md_dict["boundary_timestamp"]
+            if "retire_absent_rows" in md_dict:
+                dict_["x-retire-absent-rows"] = md_dict["retire_absent_rows"]
+            if "natural_key" in md_dict:


also no longer needed

also removed

sh-rp · 2024-09-24T11:34:23Z

docs/website/docs/general-usage/incremental-loading.md

+    ...
+...
+```
+Using this setting, records are not retired in the destination if their corresponding natural keys are not present in the source extract. This allows for incremental extracts that only contain updated records. You need to specify the natural key as `merge_key` when `retire_absent_rows` is `False`. Compound natural keys are allowed and can be specified by providing a list of column names as `merge_key`.


@rudolfix we are using only the merge_key as opposed to a combination of primary key and merge key here, let us know if this is ok.

@jorritsandbrink @sh-rp I think there's some kind of conceptual confusion. merge_key is correct but it is not a natural key for the source data. it is the same merge_key we have in delete-insert strategy.

Example 1: I load data day by day. I set the merge key to day column. I load the same day twice. Then I retire records only from the given day.
Example 2: I set the merge key to the same value as primary (natural) key (or to the content hash if no natural key is present). Then I have the behavior as I'd set the retire_absent_rows to False. (which is not needed anymore btw.)
Example 3: I update chunked documents and use merge key to be doc_id. Then I retire all the missing chunks for those doc ids, not touching the others.

My take:

drop the flag

if the merge key is present, always limit the set of records to retire

update the documentation to explain two basic cases (do not retire deleted records at all, retire deleted records only for given partition)

WDYT? IMO this is really powerful functionality if done this way

I think you're right :) I'll adapt per your suggestions.

sh-rp

Small changes requested and one question that @rudolfix should answer.

jorritsandbrink · 2024-09-28T14:30:17Z

@rudolfix I've adapted the code per your suggestions. Can you review?

rudolfix

this is so good! thanks!

jorritsandbrink added 2 commits September 16, 2024 17:37

remove unused imports

a3a29ac

add scd2 retire_if_absent option

aa62983

jorritsandbrink added the enhancement New feature or request label Sep 16, 2024

jorritsandbrink linked an issue Sep 16, 2024 that may be closed by this pull request

Support incremental loading for scd2 #1789

Closed

rewrite scd2 retire logic

bcbd0c7

sh-rp force-pushed the devel branch from ec730e8 to fcc4c45 Compare September 17, 2024 10:04

jorritsandbrink added 3 commits September 17, 2024 15:31

include new keys in typing

70d036d

finetune scd2 typing

5cfc5c8

update typeddict validation test

54dfe14

jorritsandbrink marked this pull request as ready for review September 17, 2024 15:09

sh-rp reviewed Sep 18, 2024

View reviewed changes

jorritsandbrink added 3 commits September 18, 2024 23:20

rename to retire_absent_rows

c28f8ba

add reinsert test case

7d1aad9

replace natural_key with merge_key

a0aa99c

jorritsandbrink changed the title ~~incremental scd2 with retire_if_absent flag~~ incremental scd2 with retire_absent_rows flag Sep 19, 2024

jorritsandbrink added 5 commits September 19, 2024 17:17

rewrite natural key presence check

65838e4

simplify scd2 test and remove redundancy

16e52cd

set constants once

c826afc

document incremental scd2

ae7cd00

Merge branch 'devel' of https://github.com/dlt-hub/dlt into feat/1789…

5888a07

…-scd2-incremental

jorritsandbrink requested a review from sh-rp September 20, 2024 07:13

sh-rp reviewed Sep 24, 2024

View reviewed changes

sh-rp requested changes Sep 24, 2024

View reviewed changes

jorritsandbrink added 3 commits September 24, 2024 15:54

remove natural_key remnants

8ecf7c6

remove retire_absent_rows flag

1035ffa

add scd2 merge key partition test

c5be436

jorritsandbrink added 2 commits September 28, 2024 13:41

fix typos

21ce4f8

update incremental scd2 docs

84c101b

jorritsandbrink changed the title ~~incremental scd2 with retire_absent_rows flag~~ incremental scd2 with merge_key Sep 28, 2024

jorritsandbrink requested a review from rudolfix September 28, 2024 14:29

rudolfix approved these changes Sep 29, 2024

View reviewed changes

rudolfix merged commit cb45046 into devel Sep 29, 2024
60 of 61 checks passed

rudolfix deleted the feat/1789-scd2-incremental branch September 29, 2024 19:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

incremental `scd2` with `merge_key` #1818

incremental `scd2` with `merge_key` #1818

jorritsandbrink commented Sep 16, 2024 •

edited

Loading

netlify bot commented Sep 16, 2024 •

edited

Loading

sh-rp Sep 18, 2024

jorritsandbrink Sep 19, 2024

sh-rp Sep 18, 2024

jorritsandbrink Sep 19, 2024

sh-rp Sep 24, 2024

jorritsandbrink Sep 24, 2024

sh-rp Sep 24, 2024

jorritsandbrink Sep 24, 2024

sh-rp Sep 24, 2024

rudolfix Sep 26, 2024

jorritsandbrink Sep 27, 2024

sh-rp left a comment

jorritsandbrink commented Sep 28, 2024

rudolfix left a comment

incremental scd2 with merge_key #1818

incremental scd2 with merge_key #1818

Conversation

jorritsandbrink commented Sep 16, 2024 • edited Loading

Description

Related Issues

netlify bot commented Sep 16, 2024 • edited Loading

✅ Deploy Preview for dlt-hub-docs ready!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sh-rp left a comment

Choose a reason for hiding this comment

jorritsandbrink commented Sep 28, 2024

rudolfix left a comment

Choose a reason for hiding this comment

incremental `scd2` with `merge_key` #1818

incremental `scd2` with `merge_key` #1818

jorritsandbrink commented Sep 16, 2024 •

edited

Loading

netlify bot commented Sep 16, 2024 •

edited

Loading