Socket Leak on Linux

[jchannon]

We've spotted a possible socket leak on linux v3.18.6 and mono 3.12.0

The source for a demo app is at : https://github.com/VQComms/SelfHostWithOwin

To reproduce run:

mono SelfHostWithOwin.exe
ps
Make note of process id
Use curl/wget to hit http://localhost:1999
Run ls /proc/{PID}/fd -l
Make note of number of socket handles present
Use curl/wget to hit http://localhost:1999 again
Run ls /proc/{PID}/fd -l
Make note of number of socket handles present and notice it has increased

It appears to increase each socket handler per request

[jchannon]

Using the new code that you have pushed and using the sample app I linked to and executing this:

!/bin/ash

while true; do
curl http://localhost:1999/ -s
done

The webserver locks up after 5 secs or so.

Any ideas?

[jchannon]

I ran the ab load test tool and here's the results. I told it to do 1000 request with 5 concurrent requests, as you see it only managed 254 requests. Very odd.

[Bobris]

Does it at least not leak sockets? Don't have idea, but I am afraid that Saea implementation could have more problems than just does socket leak...
By default I create 256 "connection listeners" so if they will not correctly reset and reuse after disconnect. It could do something like this...

[jchannon]

The socket leak seems to be fixed so thats a good thing.

Interesting that it gives up after 254 requests and you say you have 256 listeners so I suspect something going on there.

[Bobris]

You have -c 5, so +- 5 difference probably don't mean anything.

[jchannon]

ok, i'm a bit in the dark regarding this so if you could point us in the right direction it would be greatly appreciated 😄

[jchannon]

I've submitted a fix to the 256 listeners with #31 however perf is pretty bad

[Bobris]

Already installing Linux to try myself

But you can place this into .config inside element:

  <system.diagnostics>
    <trace autoflush="true" indentsize="0">
    </trace>
    <sources>
      <source name="Nowin.Core">
        <listeners>
          <remove name="Default"/>
          <add name="Nowin.Core" />
        </listeners>
      </source>
      <source name="Nowin.Core.Debug">
        <listeners>
          <remove name="Default"/>
          <add name="Nowin.Core" />
        </listeners>
      </source>
    </sources>
    <switches>
      <add name="Nowin.Core" value="Verbose" />
      <add name="Nowin.Core.Debug" value="Verbose" />
    </switches>
    <sharedListeners>
      <add name="Nowin.Core"
           type="System.Diagnostics.TextWriterTraceListener" 
           initializeData="Nowin.Core.log" />
    </sharedListeners>
  </system.diagnostics>

And send me Nowin.Core.log

Try something minimum what will hang it.
You can reduce number of connections by calling this on ServerBuilder:
.SetConnectionAllocationStrategy(new ConnectionAllocationStrategy(1, 0, 1, 0))

[Bobris]

Aha you found it in mean time. Perfect 👍

[jchannon]

Perf is still bad though so not sure what we can do to improve that. Do you fancy trying it out on your new Linux install? 😄

[Bobris]

Now that you solve it I could return to JS UI Framework I am developing. .Net + Linux is not critical right now for me, I am hoping it will be solved by MS till end of this year anyway :-)
Nowin solved for me problem of needing Admin rights to start HttpListener and much easier deployment.

[jchannon]

OK cool. I've ran some very basic load test tools comparing latest nowin release and owin.httplistener from MS on Linux and nowin's performance is pretty bad in comparison, it fails to even run so I'm not sure what's going on underneath, its as if it can't take many concurrent connections. If I drop it from 100 to 10 it at least runs the tests but the perf is still bad, even comparing 1 connection vs 2 concurrent connections you notice a big difference

Here's the stats:
https://gist.github.com/jchannon/45c0bbb0e10db56c7ea2

The endpoint is a Nancy endpoint setup to just return "Hi"

 public class HomeModule : NancyModule
    {
        public HomeModule()
        {
            Get["/"] = _ => "Hi";
        }
    }

[Bobris]

Mono HttpListener does not use Saea (and they probably know why :-)). Mono Socket implementation does not use epool (at least last time I looked) which is critical for speed of 1k+ connections. I don't have time to implement it by native calls for Linux (eventhough it would be interesting). So really hoping than someone will re-implement that completely from scratch. BTW you cannot use Windows implementation on Linux because OS API is completely different (IOCP vs epool). Also for me even higher priority would to use http://stackoverflow.com/questions/18419117/rio-registered-i-o-sockets-in-net in Windows Nowin, but again in this I hope that "cloud optimized" CoreCLR will eventually use that as Saea backend.
I am sorry, I cannot help you much more right now.

[borgdylan]

I have a related issue #52