fsnotify watcher: too many open files

[fosterseth]

receptor is not bubbling up this error message in some situations.

receptor/pkg/workceptor/kubernetes.go

Line 242 in 4addde8

if err == io.EOF {

On a system where fsnotify cannot create a watcher, ReadString returns EOF, and line has this content

failed to create fsnotify watcher: too many open files

however, we are leveraging EOF to also mean "we are done reading from the pod, all is good"

I think if err is EOF and line is not empty, then we may need to write bubble up line as an error.

We should make sure line is expected to be empty in successfully cases before doing that, however.

[fosterseth]

setting these values (or lower) on your system/VM might help replicate the issue

sysctl fs.inotify.max_user_instances=128
sysctl fs.inotify.max_user_watches=65536