[5pt] Skill regression in v4.4.13.1

[CarsonPruitt-NOAA]

No description provided.

[mluck]

For background: #1006 (comment)

Since we were trying to find the source of the errors between 4.4.9.0 and 4.4.13.1, I ran two BLE HUCs (12040101 and 12090301) and it looks like this FB did change the CSI in both of the HUCs I looked at (red = decrease, green = increase). The number of branch errors (exit status 61) in two HUCs (09010004 and 12100303) also seem to have increased in this FB.

[CarsonPruitt-NOAA]

@RyanSpies-NOAA @mluck @EmilyDeardorff @RobHanna-NOAA @derekgiardino @hhs732

Alright folks, I had an epiphany while walking the dog last night. All the work you guys have done have helped to figure it out, the dog and I were just able to connect the dots. Emily's backpools PR (#1006) caused a skill regression, but it didn't actually make the quality of FIM any worse. In fact, that PR has significantly improved our SRCs at the partner FIM (AHPS FIM) locations.

The first clue was Matt finding out about the underprediction in this new version of FIM compared to previous versions. The spatial calibration coefficient was significantly different. Next, seemingly unrelated, ras2fim v2 was producing horrible skill scores (321). Rob and Heidi were able to figure out that our AHPS benchmark flow files have been wrong this whole time; some of them include tributaries with flows the same as the mainstem (see cbst2). You might now be wondering why we have never noticed. Well, our spatial calibration is so good that it tuned those tributaries to match the AHPS benchmark extents even though the flows are way too high. The downside is that the calibration has totally screwed up our tributary SRCs at these locations, causing extreme underprediction. I haven't checked, but this may have also contributed to the underprediction seen at Montpelier, VT in last year's floods.

The final piece to the puzzle is Emily's backpools fix. It removed a significant portion of the tributary catchments that were actually part of the mainstem river. This caused the spatial calibration to be altered so much that we are no longer over-calibrating the tributaries to the wrong flows, exposing the faulty benchmark flow files.

I don't have time this morning to add screenshots, but I will add some here to illustrate what I'm trying to communication. The next steps will be to fix all of the benchmark flows that have tributaries incorrectly included and to make sure that spatial calibration doesn't occur at all in tributaries. Thank you all for your hard work on this!

[CarsonPruitt-NOAA]

v4.4.11.0

This is cbst2 in 12090301. The agreement is great, even though the tributary coming in from the north (Cummins Creek) is being prescribed the same flow as the Colorado River.

If you take a look at the same tributary in the BLE eval, it's significantly underpredicting because of the calibration of Cummins Creek to the Colorado River flow. This also means that are live FIM layers are currently significantly underpredicting.

v4.4.13.1

After the backpools fix, the Cummins Creek terminal catchment doesn't include the Colorado River anymore, so it wasn't being calibrated as much (or at all, I haven't checked the SRC). You can see the overprediction here caused by the massive flow being routed down Cummins Creek, but now isn't being corrected by spatial calibration.

And the BLE agreement no longer shows the significant underprediction, meaning live FIM will be better with this version.