Investigate loss spikes

[dlwh]

We occasionally get large spikes in the training loss during training. For example:

The system ends up recovering, but it can take quite a long time. This may not be worth worrying about because the system does recover, but we should understand this.

One hypothesis is we get "bad batches" where the data is garbage, is thus high perplexity, and gives us bad gradients.

Another is instability related to fp16.