BFD-3550: Add cross-reference ID and cross-reference switch to CCW pipeline #2391
Conversation
| public void verifySampleAColumns() { | ||
| verifyColumns(StaticRifResourceGroup.SAMPLE_A); | ||
| public void verifySampleBColumns() { | ||
| verifyColumns(StaticRifResourceGroup.SAMPLE_B); |
There was a problem hiding this comment.
I left the sample A files alone for now so we don't mess with the existing tests. Sample B contains the new xref columns which we need to use for this test since we're checking that all columns in the file are in the schema.
| @@ -56,20 +58,17 @@ private void verifyColumns(StaticRifResourceGroup sampleGroup) { | |||
| Enum<?>[] columnsInEnum = getColumnsInEnum(sampleFile.getRifFileType()); | |||
|
|
|||
| // Use a CSVParser to parse the header out of the sample file. | |||
| CSVFormat parserFormat = CSVFormat.DEFAULT.withDelimiter('|'); | |||
| CSVFormat parserFormat = CSVFormat.DEFAULT.builder().setDelimiter('|').build(); | |||
There was a problem hiding this comment.
All the changes to .builder() are to fix the deprecation warnings on withDelimiter
| toHeaderFormat(columnsInSample, c -> c), | ||
| toHeaderFormat(columnsInEnum, c -> c.name()))); | ||
|
|
||
| HashSet<String> sampleSet = new HashSet<>(Arrays.asList(columnsInSample)); |
There was a problem hiding this comment.
Changes here are to prevent the order from mattering. Instead of checking that the columns have the same index, we only care that all columns are included.
| @@ -493,7 +493,7 @@ private void tweakIfBeneficiary( | |||
| * state as a BeneficiaryHistory record. (Note: this has to be done AFTER the secret hashes | |||
| * have been updated, as per above. | |||
| */ | |||
| updateBeneficaryHistory( | |||
| updateBeneficiaryHistory( | |||
There was a problem hiding this comment.
This is just fixing a typo
| @@ -869,6 +871,19 @@ static boolean isBeneficiaryHistoryEqual( | |||
| newBeneficiaryRecord.getMbiEffectiveDate(), oldBeneficiaryRecord.getMbiEffectiveDate())) { | |||
| return false; | |||
| } | |||
|
|
|||
| if (!Objects.equals( | |||
There was a problem hiding this comment.
This will instruct the pipeline to add a new history record if the xref id or switch has changed
| .toArray(String[]::new); | ||
| CSVFormat csvFormat = RifParsingUtils.CSV_FORMAT.withHeader(csvHeader); | ||
| // Get the header from the original file | ||
| List<String> csvHeader = RifParsingUtils.createCsvParser(inputFile).getHeaderNames(); |
There was a problem hiding this comment.
Instead of trying to create a new header with everything in the schema, we instead pull the header from the original file since the order could be different.
aschey-forpeople
requested review from
mjburling,
dondevun,
meliGuzman and
malessi
as code owners
| @@ -0,0 +1,128 @@ | |||
| /* | |||
There was a problem hiding this comment.
Created a runbook that explains how to use this script for re-running a pipeline load: https://github.com/CMSgov/beneficiary-fhir-data/wiki/How-to-clean%E2%80%90up-and-reload-test-data-from-a-CCW-pipeline-load
JIRA Ticket:
BFD-3550
What Does This PR Do?
xref_grp_idandxref_swcolumns to the database and the CCW pipelinewithHeadermethod for creating CSVs has been throwing deprecation warnings for a while. I replaced all instances of these with the builder pattern that they now want you to use.What Should Reviewers Watch For?
If you're reviewing this PR, please check for these things in particular:
What Security Implications Does This PR Have?
Please indicate if this PR does any of the following:
Adds any new software dependencies
Modifies any security controls
Adds new transmission or storage of data
Any other changes that could possibly affect security?
I have considered the above security implications as it relates to this PR. (If one or more of the above apply, it cannot be merged without the ISSO or team security engineer's (
@sb-benohe) approval.)Validation
Created an ephemeral environment from test, ran database migrations, and re-ran an old pipeline load that contained a beneficiary file.