Fix SO timeout log issue when the worker pool is exhausted.

[malakaganga]

Fix SO timeout log issue when the worker pool is exhausted. Change SO timeout log structure.
Define new logs to identify secondary worker pool exhausting scenarios (by defining new max queueing time for that as well + at SO timeout) Change the implementation of max queueing time since it's error-prone with a new thread pool.

Improve the logs to indicate secondary passthrough pool is exhausted when all threads are busy.

Fixes: wso2/api-manager#1923

Add discard logic to clientWorker when secondary pool is not available

Move thread state to the relevant worker thread itself

Done a fix to move the thread state to the relevant worker thread itself rather than keeping it in the connection.

Fixes: wso2/api-manager#1923
Related to: https://github.com/wso2-support/wso2-synapse/pull/2135

Purpose

Describe the problems, issues, or needs driving this feature/fix and include links to related issues in the following format: Resolves issue1, issue2, etc.

Goals

Describe the solutions that this feature/fix will introduce to resolve the problems described above

Approach

Describe how you are implementing the solutions. Include an animated GIF or screenshot if the change affects the UI (email [email protected] to review all UI text). Include a link to a Markdown file or Google doc if the feature write-up is too long to paste here.

User stories

Summary of user stories addressed by this change>

Release note

Brief description of the new feature or bug fix as it will appear in the release notes

Documentation

Link(s) to product documentation that addresses the changes of this PR. If no doc impact, enter “N/A” plus brief explanation of why there’s no doc impact

Training

Link to the PR for changes to the training content in https://github.com/wso2/WSO2-Training, if applicable

Certification

Type “Sent” when you have provided new/updated certification questions, plus four answers for each question (correct answer highlighted in bold), based on this change. Certification questions/answers should be sent to [email protected] and NOT pasted in this PR. If there is no impact on certification exams, type “N/A” and explain why.

Marketing

Link to drafts of marketing content that will describe and promote this feature, including product page changes, technical articles, blog posts, videos, etc., if applicable

Automation tests

• Unit tests

  Code coverage information

• Integration tests

  Details about the test cases and coverage

Security checks

• Followed secure coding standards in http://wso2.com/technical-reports/wso2-secure-engineering-guidelines? yes/no
• Ran FindSecurityBugs plugin and verified report? yes/no
• Confirmed that this PR doesn't commit any keys, passwords, tokens, usernames, or other secrets? yes/no

Samples

Provide high-level details about the samples related to this feature

Related PRs

List any other related PRs

Migrations (if applicable)

Describe migration steps and platforms on which migration has been tested

Test environment

List all JDK versions, operating systems, databases, and browser/versions on which this feature/fix was tested

Learning

Describe the research phase and any blog posts, patterns, libraries, or add-ons you used to solve the problem.