VCFHeader and VCFHeaderLine rewrite/refactoring #1581

cmnbroad · 2021-11-09T22:06:23Z

This PR is resurrected from the PR graveyard, updated and rebased. It contains:

significant updates to VCFHeader and VCFHeaderLine class hierarchy
significantly updated VCFHeader merging
prerequisites for writing VCF4.3 and BCF2.2
some source-level incompatibilities for VCFHeader and VCFHeaderLine (a detailed document is being prepared)

Fixes numerous issues, including:

codecov-commenter · 2021-11-09T22:33:13Z

Codecov Report

Merging #1581 (3084903) into master (347c0ac) will increase coverage by 0.022%.
The diff coverage is 79.088%.

❗ Current head 3084903 differs from pull request most recent head a4c529d. Consider uploading reports for the commit a4c529d to get more accurate results

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@               Coverage Diff               @@
##              master     #1581       +/-   ##
===============================================
+ Coverage     69.856%   69.878%   +0.022%     
- Complexity      9695      9832      +137     
===============================================
  Files            703       706        +3     
  Lines          37772     38022      +250     
  Branches        6139      6155       +16     
===============================================
+ Hits           26386     26569      +183     
- Misses          8929      8965       +36     
- Partials        2457      2488       +31

Impacted Files	Coverage Δ
...in/java/htsjdk/samtools/SAMSequenceDictionary.java	`75.000% <ø> (ø)`
...c/main/java/htsjdk/variant/vcf/VCFRecordCodec.java	`0.000% <0.000%> (ø)`
src/main/java/htsjdk/variant/vcf/VCFCodec.java	`25.000% <22.222%> (-52.273%)`	⬇️
src/main/java/htsjdk/variant/vcf/VCFUtils.java	`45.570% <28.571%> (-3.706%)`	⬇️
src/main/java/htsjdk/variant/vcf/VCF3Codec.java	`57.895% <50.000%> (-5.520%)`	⬇️
...n/java/htsjdk/variant/vcf/VCFFormatHeaderLine.java	`66.667% <57.895%> (-3.333%)`	⬇️
...tsjdk/variant/variantcontext/writer/VCFWriter.java	`77.451% <65.000%> (-5.345%)`	⬇️
...main/java/htsjdk/variant/vcf/VCFHeaderVersion.java	`75.000% <66.667%> (+3.125%)`	⬆️
...n/java/htsjdk/variant/vcf/VCFSampleHeaderLine.java	`66.667% <66.667%> (-33.333%)`	⬇️
...main/java/htsjdk/variant/vcf/VCFAltHeaderLine.java	`72.727% <68.421%> (-27.273%)`	⬇️
... and 22 more

... and 69 files with indirect coverage changes

andersleung

A little nitpicky detail, some uses of "its" and "it's" in comments are backwards.

andersleung · 2021-11-11T15:39:21Z

src/main/java/htsjdk/variant/vcf/VCFHeaderMerger.java

+        for (final VCFHeader sourceHeader : headers) {
+            for (final VCFHeaderLine line : sourceHeader.getMetaDataInSortedOrder()) {
+                final String key = line.getKey();
+                if (VCFHeaderVersion.isFormatString(key) || key.equals(VCFHeader.CONTIG_KEY)) {


I know we've had this conversation before, but just putting this here to see what the other reviewers think of this idea. This seems like another case where internally representing the fileformat line as just another header line doesn't seem to make much sense, because its semantics are special.

VCFHeader::getMetaDataIn{Input,Sorted}Order returns the fileformat line along with all other lines, but it needs to be dropped, special-cased, or converted back into a VCFHeaderVersion for a lot of our applications. A similar pattern (scanning for the fileformat line and dropping it) occurs in VCFMetaDataLines::validateMetaDataLines and VCFWriter:: writeHeader. The functionality of storing the version is duplicated by the VCFHeaderVersion field as well, and the two have to be kept in sync (technically triplicated also by the version field in VCFHeader).

Not storing it as a line and only as aVCFHeaderVersion field, and converting fileformat lines to version objects when they’re added to the header might be cleaner in my opinion. VCFHeader::getMetaDataIn{Input,Sorted}Order would still have to include the fileformat line to keep behaviour the same.

Obviously we'll still have to accept fileformat lines at the API boundary.

Removing the redundant VCFHeaderVersion from VCFHeader makes sense now I think - we'll just need to query VCFMetaDataLines before and after every change, but thats cheap.

Changing VCFMetaData lines to not model the version as a line in the list will result in awkward semantics and implementation for things like removeMetaDataLine, findEquivalentHeaderLine, getOtherHeaderLines, but I don't have a strong opinion on that - as long as we keep the awkward public header API, there will be awkward semantics one way or another. I'll make the change and we can see which way we think is better.

None of that will address the external consumers though, such as VCFWriter, which is still going to have to strip the file format line manually, since we need to hand them out at the public boundary to be compatible with past behavior. The right long term fix is to have a real codec that encodes as well as decodes, such as we have in the CZI package.

andersleung · 2021-11-11T15:40:18Z

src/main/java/htsjdk/variant/bcf2/BCF2Utils.java

-                    dict.add(idLine.getID());
-                    seen.add(idLine.getID());
+                if (!line.isIDHeaderLine()) {
+                    //is there a better way to ensure that shouldBeAddedToDictionary==true only when isIDHeaderLine==true


I'm wondering whether shouldBeAddedToDictionary should be removed from VCFHeaderLine and replaced with something like a static method here, since it's a BCF 2.2 specific implementation detail.

We should do that in the BCF branch.

andersleung · 2021-11-11T15:41:08Z

src/main/java/htsjdk/variant/vcf/VCFCompoundHeaderLine.java

+                    return Optional.of(validationFailure);
+                } else {
+                    // warn for older versions - this line can't be used as a v4.3 line
+                    logger.warn(validationFailure.getFailureMessage());


I'm wondering whether it makes more sense to move the logging (and checking validation strictness) out of these functions making them pure, and into the functions that actually consume them and have side effects, which can choose to throw or log based on the validation strictness.
These functions also aren't totally consistent, the one in VCFHeaderLine doesn't check validation strictness.

I like the idea of passing back the error and letting the caller handle it, but it looks like the superclass handler is intended to ignore validation strictness so I'm not sure that it would be easy to redesign.

I would lean towards making the validation stringency a parameter instead of grabbing it through defaults because it makes that dependency much less hidden and also easier to test.

andersleung · 2021-11-11T15:41:21Z

src/main/java/htsjdk/variant/vcf/VCFCompoundHeaderLine.java

+                    // to retain backward compatibility with previous implementations, we accept (and repair) and the line here.
+                    updateGenericField(NUMBER_ATTRIBUTE, "0");
+                    lineCount = 0;
+                    logger.warn(String.format("FLAG fields must have a count value of 0, but saw count %d for header line %s. A value of 0 will be used",


Small detail, but this logging should come before lineCount is updated, otherwise the message will print the contradictory warning "FLAG fields must have a count value of 0, but saw count 0".

Yeah good catch - I've been meaning to fix that since it looks weird every time I see it.

andersleung · 2021-11-11T15:45:57Z

src/main/java/htsjdk/variant/vcf/VCFHeader.java

+            // We only allow going forward to a newer version, not backwards to an older one, since there
+            // is really no way to validate old header lines (pre vcfV4.2). The only way to create a header with
+            // an old version is to create it that way from the start.
+            // to be created with the old version from the start.


Comment looks like an old line got left in accidentally

andersleung · 2021-11-11T17:08:12Z

src/main/java/htsjdk/variant/vcf/VCFUtils.java

-        final Set<VCFHeaderVersion> headerVersions = new HashSet<>(2);
+    // a global mutable static - is there an alternative ?
+    // there isn't any other reasonable place to keep this state
+    private static boolean vcfStrictVersionValidation = true;


I think it might be better to have this in Defaults making it immutable over the course of one process because the semantics of turning it on or off during one run might be unexpected. Starting with strict validation off then turning it on doesn't trigger validation of existing header lines for example, so we might still be outputting sloppily validated lines.

This setting is already hooked up to Defaults, but I forgot to delete this variable - so this just needs to be removed.

Done (removed).

andersleung · 2021-11-11T17:09:08Z

src/test/java/htsjdk/variant/variantcontext/writer/VCFWriterUnitTest.java

@@ -330,6 +326,7 @@ public void TestWritingLargeVCF(final String extension) throws FileNotFoundExcep
    @DataProvider(name = "vcfExtensionsDataProvider")
    public Object[][]vcfExtensionsDataProvider() {
        return new Object[][] {
+                //TODO: fix this BCF problem!


I'll make a note to fix these tests in my BCF branch, but from what I can tell, these failures are expected, 2 of these (TestWritingLargeVCF, testBasicWriteAndRead) make sense to fix and are from a missing standard header line (DP), because BCF requires all VC attributes to have a corresponding line in the header.

The other 2 (testWriteAndReadVCFHeaderless and testWriteWithEmptyHeader) don't make sense at all to test for BCF, because headerless or empty header BCFs aren't well formed.

andersleung · 2021-11-11T17:09:28Z

src/test/java/htsjdk/variant/vcf/AbstractVCFCodecTest.java

+                { "key=<", new VCFHeaderLine("key", "<") },
+                // taken from Funcotator test file as ##ID=<Description="ClinVar Variation ID">
+                // technically, this is invalid due to the lack of an "ID" attribute, but it should still parse
+                // into a VCFHeaderLine (but noa VCFSimpleHeaderLine


Typo in "noa", missing closing )

andersleung · 2021-11-30T16:15:32Z

src/main/java/htsjdk/variant/vcf/VCFHeader.java

+            return null;
+        } else if (lineList.size() > 1) {
+            throw new TribbleException(
+                    String.format(


This format string is missing a %s placeholder for the list of duplicated "other" lines argument

cmnbroad · 2022-10-17T13:11:39Z

Note: this needs to be resolved/updated to account for the final disposition of structured vs. unstructured lines, attribute ordering, and quoting, as described in samtools/hts-specs#642 #1610, and also samtools/hts-specs#620.

ohofmann · 2023-03-02T23:11:59Z

We've been discussing the lack of support for VCF 4.4 in IGV (igvteam/igv#1289) and started looking at the current htsjdk / VCF interaction. This PR looks like a major step forward to resolve issues around VCF 4.3 - thanks for all your work on this! Any chance of merging this soon so we can consider what changes would be required for 4.4?

cmnbroad · 2023-03-06T19:52:50Z

@ohofmann Hi Oliver - yes, this PR addresses quite a few issues with VCF support in htsjdk, and is a prerequisite for a couple of other large PRs for VCF/BCF support. The team has been sidetracked for quite a while with upgrading several of these repos to Java 17, which turned out to be a lot more work than anticipated, but we finally got the main PR for that merged last week. We discussed this PR in our team meeting this morning, and we do plan to get it merged sometime in the next quarter. It has had a fair amount of review/discussion already, but its quite large and will require a major version release, and has significant downstream test impact on projects such as GATK (I haven't tried to build IGV with it, but I expect the IGV fallout to be minimal). Hope that help.s

ohofmann · 2023-03-06T22:54:38Z

@cmnbroad Thanks for the update! It does help, though we may have to consider postponing the VCF 4.4 announcement / PR drive. I wouldn't want the first experience of many users to be an htsjdk error. Will keep an eye on this PR in the meantime.

…ilsTest.

…BCF2.2 and bug fixes.

…or customs serialization.

cmnbroad · 2023-03-14T14:14:42Z

@lbergelson I've rebased this on master/Java 17.

lbergelson

@cmnbroad I have a lot of comments. Some are more important than others. Some of them are much older than others also... There's a bunch of places where we can make use of new 17 (and soon 21) styles stuff to clean up code and I've mentioned some of them. Feel free to ignore that though, we can always revisit.

I think it looks good overall. I found some things that I think need fixing but I think the overall approach is a huge improvement.

lbergelson · 2022-06-21T19:44:43Z

src/main/java/htsjdk/samtools/Defaults.java

@@ -134,6 +139,7 @@ public class Defaults {
        SAM_FLAG_FIELD_FORMAT = SamFlagField.valueOf(getStringProperty("sam_flag_field_format", SamFlagField.DECIMAL.name()));
        SRA_LIBRARIES_DOWNLOAD = getBooleanProperty("sra_libraries_download", false);
        DISABLE_SNAPPY_COMPRESSOR = getBooleanProperty(DISABLE_SNAPPY_PROPERTY_NAME, false);
+        STRICT_VCF_VERSION_VALIDATION = getBooleanProperty("strict_version_validation", true);


I would make the property name also "strict_vcf_version_validation".

lbergelson · 2022-06-21T19:46:29Z

src/main/java/htsjdk/samtools/SAMSequenceDictionary.java

+     *                       order in the list
+     */
+    //TODO: this method ignores (and actually mutates) the sequenceRecord's contig index to make it match
+    // the record's relative placement in the dictionary's internal list


Wow, that's a nasty side effect that we've been living with forever.

lbergelson · 2022-06-21T21:46:59Z

src/main/java/htsjdk/variant/variantcontext/writer/VCFWriter.java

-            }
+            writer.write(header.getHeaderFields().stream()
+                    .map(f -> f.name())
+                    .collect(Collectors.joining(VCFConstants.FIELD_SEPARATOR)).toString());


I think this too string is unecessary?

lbergelson · 2022-06-21T21:54:01Z

src/main/java/htsjdk/variant/vcf/AbstractVCFCodec.java

+        // collect metadata lines until we hit the required header line, or a non-metadata line,
+        // in which case throw since there was no header line
+        while (lineIterator.hasNext()) {
+            final String line = lineIterator.peek();


BUG! This goes into an infinite loop if it hits a line that doesn't start with either of these.

lbergelson · 2022-06-21T22:04:33Z

src/main/java/htsjdk/variant/vcf/AbstractVCFCodec.java

+                return this.header;
+            }
+        }
+        throw new TribbleException.InvalidHeader(


It would be nice if there was information available at this point to say WHICH vcf file had the error but I don't see anyway to get that at the moment.

lbergelson · 2024-02-01T17:55:57Z

src/main/java/htsjdk/variant/vcf/VCFSimpleHeaderLine.java

+
+    // List of expected tags (for this base class, its ID only; subclasses with more required tags
+    // should use a custom tag order if more required tags are expected
+    protected static final List<String> expectedTagOrder = Collections.unmodifiableList(


yay for List.of()

lbergelson · 2024-02-01T18:08:42Z

src/main/java/htsjdk/variant/vcf/VCFValidationFailure.java

+ * A class representing a VCF validation failure.
+ * @param <T> a type representing the object that is being validated
+ */
+class VCFValidationFailure<T> {


This is returned by the public method validateForVersion so probably it should be public. Or maybe the validate method should be protected.

lbergelson · 2024-02-01T18:15:10Z

src/main/java/htsjdk/variant/vcf/VCFSimpleHeaderLine.java

+    // have to be able to "repair" header lines (via a call to updateGenericField) during constructor validation.
+    //
+    // Otherwise the values here should never change during the lifetime of the header line.
+    private final Map<String, String> genericFields = new LinkedHashMap();


lbergelson · 2024-02-01T18:24:30Z

src/main/java/htsjdk/variant/vcf/VCFSimpleHeaderLine.java

+        // without string literal escaping and quoting the regex would be: replaceAll( ([^\])" , $1\" )
+        // ie replace: something that's not a backslash ([^\]) followed by a double quote
+        // with: the thing that wasn't a backslash ($1), followed by a backslash, followed by a double quote
+        return value.replaceAll("([^\\\\])\"", "$1\\\\\"");


It might be possible to write this more legibly with text blocks nowadays?

lbergelson · 2024-02-01T18:31:10Z

src/main/java/htsjdk/variant/vcf/VCFSampleHeaderLine.java

+
+    @Override
+    public Optional<VCFValidationFailure<VCFHeaderLine>> validateForVersion(final VCFHeaderVersion vcfTargetVersion) {
+        if (!vcfTargetVersion.isAtLeastAsRecentAs(VCFHeaderVersion.VCF4_0)) {


You use the pattern here of either warning or returning an optional here. It seems like it's done a number of times.
It seem a bit weird, an extra layer of optionalness. Should the decision to log a warning or not be left up to the caller? If some options are only errors during strict validation maybe that should be recorded in the ValidationFailure object and then the caller can decide what to do with all of them. If you don't like that idea, maybe pull out a method that either returns the option or warns and swallows it.

I guess the reason behind it is probably because it's necessary since the super call might return a higher priority error, but that could be reworked so it's called first or the two are compared, or even both are returned.

Not super important, but it seems awkward.

cmnbroad requested review from droazen and lbergelson November 9, 2021 22:06

cmnbroad force-pushed the cn_vcf_header branch from 9a9ea52 to f66252a Compare November 9, 2021 23:07

andersleung suggested changes Nov 11, 2021

View reviewed changes

andersleung reviewed Nov 30, 2021

View reviewed changes

cmnbroad mentioned this pull request Mar 1, 2022

VCF 4.3 writing. #1596

Draft

cmnbroad mentioned this pull request May 31, 2022

Remove validation of VCF Header line field order? #1610

Open

lbergelson mentioned this pull request Nov 18, 2022

Make AbstractVCFCodec thread-safe, allowing for multi-threaded VariantContext genotype decoding #1636

Open

5 tasks

cmnbroad added 10 commits March 13, 2023 10:22

Commit with raw GATK SequenceDictionaryUtils and SequenceDictionaryUt…

ce7cdcf

…ilsTest.

VCFHeader and VCFHeaderLine refactoring to enable support for VCF4.3/…

4d08d5d

…BCF2.2 and bug fixes.

Eliminate redundant modeling of VCFHeaderVersion in VCFHeader.

210adb2

Eliminate redundant modeling of file format lines in VCFMetaDataLines.

f3b9001

More code review comments.

2fe930c

Changes needed to port GATK over to htsjdk SAMSequenceDictionaryUtils.

88bdf78

One more code review comment fix.

d93546f

Make VCFMetaDataLines public/@internalapi to allow consumers access f…

f9a0c08

…or customs serialization.

Versioned header line validation framework.

29c854f

Remove obsolete variable.

5700958

cmnbroad force-pushed the cn_vcf_header branch from 3084903 to 5700958 Compare March 14, 2023 14:14

cmnbroad mentioned this pull request Jun 1, 2023

Error when reading a VCF 4.3 file with spaces in INFO field values #1667

Open

cmnbroad added 2 commits June 27, 2023 09:53

Properly handle info fields with embedded spaces by VCF version.

5e09eb3

Remove unnecessary null test.

860cab6

cmnbroad force-pushed the cn_vcf_header branch from a4c529d to 860cab6 Compare June 27, 2023 13:55

cmnbroad mentioned this pull request Nov 7, 2023

WIP BCF 2.2 #1550

Draft

5 tasks

lbergelson requested changes Feb 1, 2024

View reviewed changes

VCFHeader and VCFHeaderLine rewrite/refactoring #1581

Are you sure you want to change the base?

VCFHeader and VCFHeaderLine rewrite/refactoring #1581

Conversation

cmnbroad commented Nov 9, 2021 • edited Loading

codecov-commenter commented Nov 9, 2021 • edited Loading

Codecov Report

andersleung left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmnbroad commented Oct 17, 2022

ohofmann commented Mar 2, 2023

cmnbroad commented Mar 6, 2023

ohofmann commented Mar 6, 2023

cmnbroad commented Mar 14, 2023

lbergelson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmnbroad commented Nov 9, 2021 •

edited

Loading

codecov-commenter commented Nov 9, 2021 •

edited

Loading