send() failed (32: Broken pipe) when performing HTTP POST to Immich

[Matthias-vdE]

When running Immich (https://github.com/immich-app/immich) behind NPM(plus) and enabling Crowdsec/Appsec, it is not possible to upload files to the server via HTTP POST:

2024-09-27T11:33:13.859502771Z 2024/09/27 13:33:13 [error] 32855#32855: *10529 send() failed (32: Broken pipe), client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

2024-09-27T11:33:13.859545312Z 2024/09/27 13:33:13 [error] 32855#32855: *10529 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: broken pipe, client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

2024-09-27T11:33:13.859553917Z 2024/09/27 13:33:13 [error] 32855#32855: *10529 [lua] crowdsec.lua:651: Allow(): AppSec check: broken pipe, client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

2024-09-27T11:33:13.859564759Z 2024/09/27 13:33:13 [alert] 32855#32855: *10529 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied 'MY_IPADDRESS' with 'ban' (by appsec), client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

The issue was initially reported at ZoeyVid/NPMplus#1123.

[LaurenceJJones]

When running Immich (https://github.com/immich-app/immich) behind NPM(plus) and enabling Crowdsec/Appsec, it is not possible to upload files to the server via HTTP POST:

2024-09-27T11:33:13.859502771Z 2024/09/27 13:33:13 [error] 32855#32855: *10529 send() failed (32: Broken pipe), client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

2024-09-27T11:33:13.859545312Z 2024/09/27 13:33:13 [error] 32855#32855: *10529 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: broken pipe, client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

2024-09-27T11:33:13.859553917Z 2024/09/27 13:33:13 [error] 32855#32855: *10529 [lua] crowdsec.lua:651: Allow(): AppSec check: broken pipe, client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

2024-09-27T11:33:13.859564759Z 2024/09/27 13:33:13 [alert] 32855#32855: *10529 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied 'MY_IPADDRESS' with 'ban' (by appsec), client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

The issue was initially reported at ZoeyVid/NPMplus#1123.

Hey 👋🏻

I have setup a similar setup environment and don't see to be experiencing the same issue. I am uploading a 3mb file as per your reddit thread I dont see any errors.

Is CrowdSec / AppSec running locally to NPMPlus (on same host to reduce latency)?

Could you provide the full nginx configuration that is generated by NPMplus as I am using nginx (since its the same code I dont want to spend time configuring NPMPlus since its the same lua code)

server {
    server_name _;

    listen *:80;

    # allow large file uploads
    client_max_body_size 50000M;

    # Set headers
    proxy_set_header Host              $http_host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    # enable websockets: http://nginx.org/en/docs/http/websocket.html
    proxy_http_version 1.1;
    proxy_set_header   Upgrade    $http_upgrade;
    proxy_set_header   Connection "upgrade";
    proxy_redirect     off;

    # set timeout
    proxy_read_timeout 600s;
    proxy_send_timeout 600s;
    send_timeout       600s;

    location / {
        proxy_pass http://127.0.0.1:2283;
    }
}

everything seems to be getting processed:

root@bookworm:/etc/nginx/sites-enabled# cscli metrics show appsec
Appsec Metrics:
╭─────────────────┬───────────┬─────────╮
│ Appsec Engine   │ Processed │ Blocked │
├─────────────────┼───────────┼─────────┤
│ 127.0.0.1:7422/ │ 296       │ -       │
╰─────────────────┴───────────┴─────────

[Matthias-vdE]

Is CrowdSec / AppSec running locally to NPMPlus (on same host to reduce latency)?

Yup, both Crowdsec and NPMPlus are running in docker containers on the same host. Part of the same docker-compose:

services:
  npmplus:
    container_name: npmplus
    image: zoeyvid/npmplus:latest
    restart: always
    network_mode: host
    volumes:
      - "/opt/npm:/data"
    environment:
      - "TZ=Europe/Brussels" 
      - "NGINX_LOG_NOT_FOUND=true"
      - "LOGROTATE=true" 
      - "LOGROTATIONS=7" 
      - "GOA=true"

  crowdsec:
    container_name: crowdsec
    image: crowdsecurity/crowdsec:latest
    restart: always
    network_mode: bridge
    ports:
      - "127.0.0.1:7422:7422"
      - "127.0.0.1:8080:8080"
    environment:
      - "TZ=Europe/Brussels"
      - "COLLECTIONS=ZoeyVid/npmplus"
      - "LEVEL_FATAL=true"
      - "LEVEL_ERROR=true"
      - "LEVEL_WARN=true"
      - "LEVEL_INFO=false"
      - "LEVEL_DEBUG=false"
      - "LEVEL_TRACE=false"
    volumes:
      - "/opt/crowdsec/conf:/etc/crowdsec"
      - "/opt/crowdsec/data:/var/lib/crowdsec/data"
      - "/opt/npm/nginx:/opt/npm/nginx:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  geoipupdate:
    container_name: geoipupdate
    image: maxmindinc/geoipupdate:latest
    restart: always
    network_mode: bridge
    environment:
      - "TZ=Europe/Brussels"
      - "GEOIPUPDATE_EDITION_IDS=GeoLite2-Country GeoLite2-City GeoLite2-ASN"
      - "GEOIPUPDATE_ACCOUNT_ID=my_account_id"
      - "GEOIPUPDATE_LICENSE_KEY=my_license_key"
      - "GEOIPUPDATE_FREQUENCY=24"
    volumes:
      - "/opt/npm/etc/goaccess/geoip:/usr/share/GeoIP"

Immich is running on the same host as well, as a docker container. The config for that one is very standard with little to no customization.

My proxy config in NPMPlus looks like this:

Nothing in custom locations, and a regular certbot certificate to force enable HTTPS.

My crowdsec.conf file is this:

ENABLED=true
API_URL=http://127.0.0.1:8080
API_KEY=my_API_key
CACHE_EXPIRATION=1
# bounce for all type of remediation that the bouncer can receive from the local API
BOUNCING_ON_TYPE=ban
FALLBACK_REMEDIATION=ban
REQUEST_TIMEOUT=3000
UPDATE_FREQUENCY=10
# live or stream
MODE=live
# exclude the bouncing on those location
EXCLUDE_LOCATION=
#those apply for "ban" action
# /!\ REDIRECT_LOCATION and RET_CODE can't be used together. REDIRECT_LOCATION take priority over RET_CODE
BAN_TEMPLATE_PATH=/data/etc/crowdsec/ban.html
REDIRECT_LOCATION=
RET_CODE=
#those apply for "captcha" action
#valid providers are recaptcha, hcaptcha, turnstile
CAPTCHA_PROVIDER=
# Captcha Secret Key
SECRET_KEY=
# Captcha Site key
SITE_KEY=
CAPTCHA_TEMPLATE_PATH=/data/etc/crowdsec/captcha.html
CAPTCHA_EXPIRATION=3600
#APPSEC_URL=http://127.0.0.1:7422
#APPSEC_FAILURE_ACTION=deny

The two bottom lines are currently commented out to make it work. If I uncomment them, it breaks with the broken pipe error message when uploading anything.