[Logging] Document on-chain logging best practices

[bryanchriswhite]

Objective

Document on-chain logging best practices; e.g.:

 logger = logger.With(
 	"session_id", sessionHeader.GetSessionId(),
 	"session_end_height", sessionHeader.GetSessionEndBlockHeight(),
 	"supplier", supplierAddr)
 logger.Info("validated the submitProof message ")

Consider which logger should be used:

1. The default cosmos-sdk logger is not very practical when used at the "debug" level as it's overwhelmed with cosmos-sdk specific output.
2. We should prefer using the "debug" level for any log that doesn't require node operator attention (e.g. handling invalid requests). I think this represents (or it should) the majority of logger calls.

Search each module for usages of logger.Info() and logger.Error(). Places where an error is logged and returned are prime examples of mis-leveled logs.

Origin Document

• [Logging] Document on-chain logging best practices #856
• [Gateway] Enforce minimum stake when staking #843

Goals

• Establish and document the purpose of each log level and when each SHOULD be used, on-chain.
• Document the validator UX stakeholders and their overlapping and distinct/competing concerns.

Deliverables

• A PR which updates the docusaurus docs to include a breakdown of the on-chain logging levels and their conventional usage.
• A PR which updates the docusaurus docs to include an overview (and possibly a diagram) of validator UX stakeholders and their concerns.

Non-goals / Non-deliverables

• Updating any logs to be consistent with this documentation.

General deliverables

• Comments: Add/update TODOs and comments alongside the source code so it is easier to follow.
• Documentation: Update architectural or development READMEs; use mermaid diagrams where appropriate.

---

Creator: @bryanchriswhite
Co-Owners: @Olshansk