Improve post-dial instrumentation

[tallclair]

We've added a lot of instrumentation around dials, but we have very little instrumentation once the connection is established. Some ideas for metrics to add:

• Total data packet counts in each direction (rate is the metric we care about)
• Per-connection packet counts (histogram)
• Connection lifespan (histogram)
• Packet send failure count
• Connection close instrumentation, similar to the "dial failure reason" instrumenting we did (who closed the connection, why)

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[tallclair]

/lifecycle frozen