Make it easier to capture SCP messages

[MonsieurNicolas]

Right now SCP messages go to 2 places:

• meta, as soon as we close a ledger
• history archives (via SCPHistory, as soon as we close a ledger)

This issue tracks improving the quality of the SCP messages uploaded to archives as it's the easiest of the two (we could emit SCP updates from older ledgers in the meta, but it seems a bit more involved).

The issue can be summarized as:
SCP messages that are emitted are the ones used by that node to externalize.
To externalize, the node needs to receive either CONFIRM or EXTERNALIZE messages from nodes in its quorum set.

Some systems (like bridges) rely on SCP messages, so the more EXTERNALIZE messages we can gather, the better it is (they're smaller and easier to process from contracts).

The proposal tracked here is to add a mechanism to core to improve the chances that we have EXTERNALIZE messages from as many validators in the transitive quorum as possible published to archive.

Given how the system works right now the following may work:

• audit herder/SCP filtering logic to ensure that we continue to store EXTERNALIZE messages in memory after a ledger is closed. I think this is already the case (right now we accumulate those so that we can serve them to other peers).
• add a new setting that controls a delay to publish to archive.
  • the delay could be expressed in terms of number of ledgers (anything less than MAX_SLOTS_TO_REMEMBER) or in time (maybe harder to configure/implement?) - in practice we probably need only a few more seconds to receive all messages (validators tend to track the same ledger)
• continue to save SCP messages in the database for older slots. Right now this only occurs right when we externalize
  * after the delay, update the scphistory table to reflect the internal SCP state. we could compute the differences in terms of SCP messages (or just rewrite the whole thing? -- I don't know how much data we're really talking about here) for the effected slot
  * note that we must write both when we externalize and later as we want to have in the database at least the messages that were used when externalizing in case of a crash.
  * the delay is also used to delay the actual snapshot (that produces files from database state) & publish step
  • this setting should probably not be set on validators, and only for watchers (we may fail to start on validators) as the delay can potentially put the network at risk of an outage

[marta-lokhova]

The delay implementation (second bullet) was introduced in #3832