Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Containers using publish all occasionally use the wrong port mapping for IPv6 binding #2639

Open
tomelliff opened this issue May 18, 2021 · 0 comments

Comments

@tomelliff
Copy link

tomelliff commented May 18, 2021

I've spent a chunk of time trying to chase down an issue we're seeing on some of our AWS ECS hosts where healthchecks were spuriously failing and causing containers to exit. I noticed from looking at the task state change events that somehow we had containers binding to different ports between IPv4 and IPv6 and all of our issues were caused by a mismatch here.

Rolling back to Docker 20.10.5 fixed this by not publishing IPv6 bindings for our containers at all (essentially undoing the work done in https://github.com/moby/libnetwork/pull/2608/files) but while we probably can avoid using IPv6 bindings either in Docker or in AWS' ECS Agent (According to https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-networking.html#task-networking-vpc-dual-stack IPv6 shouldn't be happening unless opted into anyway so I think this is a surprise caused by Docker 20.10.6 now showing IPv6 bindings in the API when it didn't previously) but that doesn't escape the fact that there's a bug somewhere in libnetwork (I think, I'm a bit hazy on where responsibilities lie now with the splitting of the projects) that is causing Docker to publish the wrong ports on IPv6.

Running the following command:

xargs -P 0 -I {} docker run --detach --rm --name test-{} -P nginx<<<$(seq 1 100)

attempts to launch 100 NGINX containers in parallel which makes my laptop completely fall over and most of the container starts to fail. However when I run this I reliably get port mismatches. Sorting the docker ps output by the IPv4 address shows this well:

CONTAINER ID IMAGE COMMAND PORTS NAMES
bb9c4984b395 nginx /docker-entrypoint.… 0.0.0.0:49164->80/tcp, :::49164->80/tcp test-2
571041910b22 nginx /docker-entrypoint.… 0.0.0.0:49165->80/tcp, :::49165->80/tcp test-6
6d6379bf998f nginx /docker-entrypoint.… 0.0.0.0:49166->80/tcp, :::49166->80/tcp test-4
c6fa6a39c31d nginx /docker-entrypoint.… 0.0.0.0:49167->80/tcp, :::49167->80/tcp test-15
82e49c9bbb0f nginx /docker-entrypoint.… 0.0.0.0:49168->80/tcp, :::49168->80/tcp test-11
b903859bdb9c nginx /docker-entrypoint.… 0.0.0.0:49169->80/tcp, :::49169->80/tcp test-8
c568a230b5b9 nginx /docker-entrypoint.… 0.0.0.0:49170->80/tcp, :::49170->80/tcp test-12
78d13f523722 nginx /docker-entrypoint.… 0.0.0.0:49171->80/tcp, :::49171->80/tcp test-3
aeef1fa41148 nginx /docker-entrypoint.… 0.0.0.0:49172->80/tcp, :::49172->80/tcp test-10
2b9cfea3221c nginx /docker-entrypoint.… 0.0.0.0:49173->80/tcp, :::49173->80/tcp test-16
1174325c7394 nginx /docker-entrypoint.… 0.0.0.0:49174->80/tcp, :::49174->80/tcp test-5
7258d99985b6 nginx /docker-entrypoint.… 0.0.0.0:49175->80/tcp, :::49175->80/tcp test-14
470391662adb nginx /docker-entrypoint.… 0.0.0.0:49176->80/tcp, :::49176->80/tcp test-9
3aa55932f786 nginx /docker-entrypoint.… 0.0.0.0:49177->80/tcp, :::49177->80/tcp test-17
fdad706caea0 nginx /docker-entrypoint.… 0.0.0.0:49178->80/tcp, :::49178->80/tcp test-13
ac893be13721 nginx /docker-entrypoint.… 0.0.0.0:49179->80/tcp, :::49179->80/tcp test-1
d16c3bc3db80 nginx /docker-entrypoint.… 0.0.0.0:49180->80/tcp, :::49180->80/tcp test-20
c159beb98784 nginx /docker-entrypoint.… 0.0.0.0:49181->80/tcp, :::49181->80/tcp test-19
0d449991367b nginx /docker-entrypoint.… 0.0.0.0:49182->80/tcp, :::49182->80/tcp test-7
6a99f572659a nginx /docker-entrypoint.… 0.0.0.0:49183->80/tcp, :::49183->80/tcp test-18
6cdc6c1aee92 nginx /docker-entrypoint.… 0.0.0.0:49184->80/tcp, :::49184->80/tcp test-21
9ac258de0f00 nginx /docker-entrypoint.… 0.0.0.0:49185->80/tcp, :::49185->80/tcp test-22
a753a81defcc nginx /docker-entrypoint.… 0.0.0.0:49186->80/tcp, :::49186->80/tcp test-25
2f8543fbc1ae nginx /docker-entrypoint.… 0.0.0.0:49187->80/tcp, :::49187->80/tcp test-27
bb1a37f274e0 nginx /docker-entrypoint.… 0.0.0.0:49188->80/tcp, :::49188->80/tcp test-30
787d4286597d nginx /docker-entrypoint.… 0.0.0.0:49189->80/tcp, :::49189->80/tcp test-24
cfca7dbe2815 nginx /docker-entrypoint.… 0.0.0.0:49191->80/tcp, :::49191->80/tcp test-31
93e001248f3a nginx /docker-entrypoint.… 0.0.0.0:49192->80/tcp, :::49192->80/tcp test-28
723055c393d6 nginx /docker-entrypoint.… 0.0.0.0:49193->80/tcp, :::49193->80/tcp test-29
336a0e93ca3b nginx /docker-entrypoint.… 0.0.0.0:49194->80/tcp, :::49194->80/tcp test-26
6f404ff119b9 nginx /docker-entrypoint.… 0.0.0.0:49195->80/tcp, :::49195->80/tcp test-33
61efecef0103 nginx /docker-entrypoint.… 0.0.0.0:49196->80/tcp, :::49196->80/tcp test-35
1a1633e3563c nginx /docker-entrypoint.… 0.0.0.0:49197->80/tcp, :::49197->80/tcp test-32
f00b4f4b3e43 nginx /docker-entrypoint.… 0.0.0.0:49198->80/tcp, :::49198->80/tcp test-39
b703b502cb0d nginx /docker-entrypoint.… 0.0.0.0:49199->80/tcp, :::49199->80/tcp test-65
95cd8136cb78 nginx /docker-entrypoint.… 0.0.0.0:49200->80/tcp, :::49200->80/tcp test-66
025aa7cd9e81 nginx /docker-entrypoint.… 0.0.0.0:49201->80/tcp, :::49201->80/tcp test-37
8067402d8030 nginx /docker-entrypoint.… 0.0.0.0:49202->80/tcp, :::49202->80/tcp test-41
1.79769313486232E+308 nginx /docker-entrypoint.… 0.0.0.0:49203->80/tcp, :::49203->80/tcp test-40
8dc601214115 nginx /docker-entrypoint.… 0.0.0.0:49204->80/tcp, :::49204->80/tcp test-42
b32b03f7d1e8 nginx /docker-entrypoint.… 0.0.0.0:49205->80/tcp, :::49205->80/tcp test-49
e5733efe3ac5 nginx /docker-entrypoint.… 0.0.0.0:49206->80/tcp, :::49206->80/tcp test-44
f433ea6d4085 nginx /docker-entrypoint.… 0.0.0.0:49208->80/tcp, :::49207->80/tcp test-46
364e5ecbd186 nginx /docker-entrypoint.… 0.0.0.0:49209->80/tcp, :::49208->80/tcp test-43
2276f9cada2e nginx /docker-entrypoint.… 0.0.0.0:49210->80/tcp, :::49209->80/tcp test-36
9ee5b8acc20b nginx /docker-entrypoint.… 0.0.0.0:49211->80/tcp, :::49210->80/tcp test-38
1cec14c1e094 nginx /docker-entrypoint.… 0.0.0.0:49212->80/tcp, :::49211->80/tcp test-50
aa3d7b553cd3 nginx /docker-entrypoint.… 0.0.0.0:49215->80/tcp, :::49214->80/tcp test-55
4593e5efdc49 nginx /docker-entrypoint.… 0.0.0.0:49216->80/tcp, :::49215->80/tcp test-56
5f37baa5f433 nginx /docker-entrypoint.… 0.0.0.0:49217->80/tcp, :::49216->80/tcp test-67
5ade3066ffef nginx /docker-entrypoint.… 0.0.0.0:49218->80/tcp, :::49217->80/tcp test-53
4610422e9af1 nginx /docker-entrypoint.… 0.0.0.0:49220->80/tcp, :::49219->80/tcp test-99
0a2f89796ce0 nginx /docker-entrypoint.… 0.0.0.0:49222->80/tcp, :::49221->80/tcp test-51

Container f433ea6d4085 is mapped to host port 49208 on IPv4 but mapped to host port 49207 for IPv6. This isn't an issue by itself as the host is essentially listening on two ports for the same container port but if you then look at the next container 364e5ecbd186 which is mapped to host port 49209 for IPv4 and 49208 for IPv6. So requests to this container on IPv6 will instead hit the other container. In this case with the containers all running the same image that's fine but where you have different containers then the request will fail as it will be mapped to the wrong container. The same repeats down the list so that all the next containers are then off by one throughout.

@tomelliff tomelliff changed the title Containers occasionally use the wrong port mapping for IPv6 binding Containers using publish all occasionally use the wrong port mapping for IPv6 binding May 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant