Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nginx, Brotli, and hich CPU usage #1187

Open
bradp-wordkeeper opened this issue Aug 14, 2024 · 2 comments
Open

Nginx, Brotli, and hich CPU usage #1187

bradp-wordkeeper opened this issue Aug 14, 2024 · 2 comments

Comments

@bradp-wordkeeper
Copy link

Hi All,

We started seeing some odd CPU issues on 1 of our servers about 2 months ago and I'm digging for answers. The ultimate effect that we're seeing is that Nginx started using about 6x the power to serve the same number of requests on the same site as it did before.

Using perf top, I was able to determine that something in Brotli is causing this issue. Also, if I disable Brotli in the Nginx config and let it fall back to Gzip, CPU usage goes back to normal.

The issue started while using Nginx 1.25.5 and we're on 1.27.0 now. We've also gone from Brotli 1.0.9 to 1.1.0. No change during upgrades or recompiles of Nginx, Brotli, or the ngx_brotli module either.

One more data point is that I ran a test in a Fedora 40 container and that seemed to solve the problem. I compiled Nginx in the Fedora container using the exact same settings and build process. Then I just stopped the normal instance of Nginx that we use and started up the containerized version in its place. The CPU went right back to normal during that test and that's with Brotli enabled, but I couldn't leave it like that for various reasons. The only difference being the underlying OS and libraries that it uses in the container, I guess.

So that all makes me think that it could be either a bug in the OS libraries somewhere or a bug in Brotli that only happens under certain conditions. This server OS is CentOS Stream 9.

Any ideas what could be causing this high CPU usage when using Brotli specifically?

Here is Nginx build info if that helps:

nginx version: nginx/1.27.0
built by gcc 11.4.1 20231218 (Red Hat 11.4.1-3) (GCC)
built with OpenSSL 3.2.2 4 Jun 2024
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -march=x86-64-v3 -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -ftree-vectorize -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2 -DTCP_FASTOPEN=23' --with-ld-opt='-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -fPIC' --with-openssl-opt='enable-tls1_3 enable-ec_nistp_64_gcc_128 no-nextprotoneg no-weak-ssl-ciphers no-ssl3 enable-ktls' --prefix=/usr/share/nginx --conf-path=/opt/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --user=nobody --group=nobody --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-openssl=../openssl-3.2.2 --with-zlib=../zlib-ng --with-pcre=../pcre2-10.44 --with-http_realip_module --with-http_auth_request_module --with-http_gzip_static_module --with-http_v2_module --with-http_v3_module --with-http_sub_module --with-libatomic=../libatomic --with-file-aio --with-http_xslt_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --without-http_autoindex_module --without-http_ssi_module --without-http_scgi_module --without-http_uwsgi_module --add-module=../ngx_brotli --add-module=../ngx_http_geoip2_module-3.4 --add-module=../njs-0.8.5/nginx
@bradp-wordkeeper
Copy link
Author

One more detail that I meant to mention. The perf top command shows the specific function call causing the overhead is CreateBackwardReferencesNH5.

I can add a ton more detail if needed too. :)

@lancedockins
Copy link

For context, this high CPU use occurred with Brotli comp levels as low as 4 and the Nginx and Brotli types config is the same list of supported MIME types. There is no reason that Zlib at level 6 should take substantially less CPU horsepower than Brotli at level 4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants