feat(webserver): Middleware with default middleware for cors, authc, curl-like logging

[mathieucarbou]

This PR improves WebServer with the following additions:

• Add Middleware support (aka Expressif) with some built-in middlewares like cors, authc and logging
• Add ability to collect all incoming request headers
• Added response code and info on server

As discussed in #10185, I tested the example thanks to a pio config.

Ideally it would be nice if @me-no-dev and @ayushsharma82 could review this PR, both were involved in these parts.

[github-actions]

	Messages
📖	🎉 Good Job! All checks are passing!

👋 Hello mathieucarbou, we appreciate your contribution to this project!

---

Click to see more instructions ...

This automated output is generated by the PR linter DangerJS, which checks if your Pull Request meets the project's requirements and helps you fix potential issues.

DangerJS is triggered with each push event to a Pull Request and modify the contents of this comment.

Please consider the following:
- Danger mainly focuses on the PR structure and formatting and can't understand the meaning behind your code or changes.
- Danger is not a substitute for human code reviews; it's still important to request a code review from your colleagues.
- To manually retry these Danger checks, please navigate to the Actions tab and re-run last Danger workflow.

Review and merge process you can expect ...

We do welcome contributions in the form of bug reports, feature requests and pull requests.

1. An internal issue has been created for the PR, we assign it to the relevant engineer.
2. They review the PR and either approve it or ask you for changes or clarifications.
3. Once the GitHub PR is approved we do the final review, collect approvals from core owners and make sure all the automated tests are passing.
- At this point we may do some adjustments to the proposed change, or extend it by adding tests or documentation.
4. If the change is approved and passes the tests it is merged into the default branch.

Generated by 🚫 dangerJS against 4668e78

[github-actions]

Test Results

 56 files   -  83   56 suites   - 83   4m 15s ⏱️ - 1h 38m 23s
 21 tests  -   9   21 ✅  -   9  0 💤 ±0  0 ❌ ±0 
135 runs   - 168  135 ✅  - 168  0 💤 ±0  0 ❌ ±0 

Results for commit 4668e78. ± Comparison against base commit b05f18d.

This pull request removes 9 tests.

performance.coremark.test_coremark ‑ test_coremark
performance.fibonacci.test_fibonacci ‑ test_fibonacci
performance.psramspeed.test_psramspeed ‑ test_psramspeed
performance.ramspeed.test_ramspeed ‑ test_ramspeed
performance.superpi.test_superpi ‑ test_superpi
test_touch_errors
test_touch_interrtupt
test_touch_read
validation.periman.test_periman ‑ test_periman

♻️ This comment has been updated with latest results.

[github-actions]

Memory usage test (comparing PR against master branch)

The table below shows the summary of memory usage change (decrease - increase) in bytes and percentage for each target.

Memory	FLASH [bytes]		FLASH [%]		RAM [bytes]		RAM [%]
Target	DEC	INC	DEC	INC	DEC	INC	DEC	INC
ESP32S3	0	‼️ +3K	0.00	⚠️ +0.33	0	0	0.00	0.00
ESP32S2	0	‼️ +3K	0.00	⚠️ +0.34	0	0	0.00	0.00
ESP32C3	0	‼️ +2K	0.00	⚠️ +0.21	0	⚠️ +16	0.00	⚠️ +0.04
ESP32C6	0	‼️ +2K	0.00	⚠️ +0.22	0	⚠️ +16	0.00	⚠️ +0.04
ESP32	0	‼️ +3K	0.00	⚠️ +0.32	0	0	0.00	0.00

Click to expand the detailed deltas report [usage change in BYTES]

Target	ESP32S3		ESP32S2		ESP32C3		ESP32C6		ESP32
Example	FLASH	RAM	FLASH	RAM	FLASH	RAM	FLASH	RAM	FLASH	RAM
WebServer/examples/AdvancedWebServer	‼️ +3K	0	‼️ +3K	0	⚠️ +2044	⚠️ +16	‼️ +2K	0	‼️ +3K	0
WebServer/examples/FSBrowser	‼️ +3K	0	‼️ +3K	0	⚠️ +2040	0	‼️ +2048	⚠️ +16	‼️ +3K	0
WebServer/examples/Filters	‼️ +3K	0	‼️ +3K	0	‼️ +2K	⚠️ +16	⚠️ +2046	0	‼️ +3K	0
WebServer/examples/HelloServer	‼️ +3K	0	‼️ +3K	0	‼️ +2K	⚠️ +16	⚠️ +2046	0	‼️ +3K	0
WebServer/examples/HttpAdvancedAuth	‼️ +3K	0	‼️ +3K	0	⚠️ +2028	0	⚠️ +2036	0	‼️ +3K	0
WebServer/examples/HttpAuthCallback	‼️ +3K	0	‼️ +3K	0	⚠️ +2038	0	⚠️ +2028	0	‼️ +3K	0
WebServer/examples/HttpAuthCallbackInline	‼️ +3K	0	‼️ +3K	0	⚠️ +2036	0	⚠️ +2028	0	‼️ +3K	0
WebServer/examples/HttpBasicAuth	‼️ +3K	0	‼️ +3K	0	⚠️ +2036	0	⚠️ +2030	0	‼️ +3K	0
WebServer/examples/HttpBasicAuthSHA1	‼️ +3K	0	‼️ +3K	0	⚠️ +2036	0	⚠️ +2028	0	‼️ +3K	0
WebServer/examples/HttpBasicAuthSHA1orBearerToken	‼️ +3K	0	‼️ +3K	0	⚠️ +2038	⚠️ +16	⚠️ +2028	⚠️ +16	‼️ +3K	0
WebServer/examples/Middleware	-	-	-	-	-	-	-	-	-	-
WebServer/examples/MultiHomedServers	‼️ +3K	0	‼️ +3K	0	⚠️ +2032	0	⚠️ +2032	0	‼️ +3K	0
WebServer/examples/PathArgServer	‼️ +3K	0	‼️ +3K	0	⚠️ +2040	0	⚠️ +2040	0	‼️ +3K	0
WebServer/examples/SDWebServer	‼️ +3K	0	‼️ +3K	0	‼️ +2K	0	⚠️ +2044	⚠️ +16	‼️ +3K	0
WebServer/examples/SimpleAuthentification	‼️ +3K	0	‼️ +3K	0	⚠️ +2010	0	⚠️ +2018	⚠️ +16	‼️ +3K	0
WebServer/examples/UploadHugeFile	‼️ +3K	0	‼️ +3K	0	⚠️ +2044	⚠️ +16	‼️ +2K	0	‼️ +3K	0
WebServer/examples/WebServer	‼️ +3K	0	‼️ +3K	0	‼️ +2K	0	‼️ +2K	0	‼️ +3K	0
WebServer/examples/WebUpdate	‼️ +3K	0	‼️ +3K	0	⚠️ +2040	0	⚠️ +2032	0	‼️ +3K	0

[Jason2866]

@mathieucarbou Do you see the possibility to make this enhancement optional?
The size increase is not just a few bytes.

[mathieucarbou]

@mathieucarbou Do you see the possibility to make this enhancement optional? The size increase is not just a few bytes.

I saw that and I was surprised by it... I don't feel to have written 2k of code...

[mathieucarbou]

Adding a ref to: #10221 (comment)

[mathieucarbou]

@safocl : I would be glad to revisit this PR but only if there is a real motivation coming from the Espressif team to merge it.

Otherwise I won't spend my time updating a PR for nothing.

I have already implemented middleware support in the fork of https://github.com/mathieucarbou/ESPAsyncWebServer we maintain (https://github.com/mathieucarbou/ESPAsyncWebServer) and also in v2 branch of PsychicHttp (https://github.com/hoeken/PsychicHttp) which is the future version, with a lot of supported middlewares for CORS, Authc, authz, curl like logging, etc.

[me-no-dev]

I would be glad to revisit this PR but only if there is a real motivation coming from the Espressif team to merge it.

There is interest. we are trying to finish up some big tasks and have a good look at this PR

[mathieucarbou]

@safocl : thank your for your review

@me-no-dev : thank you for your confirmation

I have reworked the PR to clean it - as it was it was more a PoC waiting for some interest. So I correctly did it and cleaned up the things saw by @safocl and much more.

I saw several things that I really don't like in WebServer:

1. Returning String instead of const String&: I saw a bunch of placed where a String copy would be done in the return instead of returning a ref. Sadly, fixing that would break backward compatibility

2. Use of chained pointers: this just makes the code error prone and harder to read with all these boilerplate loops. I would rather use std::list instead, but since I didn't see any usage of that (and only a few about vector), I don't know if there's a reason why the Arduino core code would not use it ? Especially that in the case of WebServer, the number of elements in such list would not be high enough to trigger any downsides of using them.

3. Last point is more about organisation: the file structure inside src folder is quite fuzzy I find... Also WebServer.h would need a bit of re-ordering / api doc : there are so many public methods, and a mix of them for request and response.