Plugin fails on Graylog 2.4.6 with the following error - noexec on /tmp results in fail to start Graylog (linux/unix only)

[skuzbucket1]

plugin fails when assigned to a stream output on 2.4.6

com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: Could not initialize class org.graylog2.syslog4j.Syslog at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2216) ~[graylog.jar:?]

We are using the jar: graylog-output-syslog-2.4.5.jar

openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

Graylog 2.4.6

Any help is greatly appreciated

[huksley]

Please send more detailed logs and how this plugin is configured.
I can`t reproduce this error.

What I have done

• Download graylog-2.4.6.tar.gz
• Download plugin from releases graylog-output-syslog-2.4.5.jar
• Enabled in Linux UDP/TCP transports for rsyslog
• Installed Mongo 3.2 and Elastic 2.3.5 in docker
• Started graylog (no errors on start)
• Created global output syslog, transport = udp, localhost
• Attached syslog output to All messages stream in Graylog
• No errors in graylog console
• Linux log received message

[skuzbucket1]

I placed the plugin in the requisite plugin directory "/usr/share/graylog/plugin/"
we are selecting TCP
remote syslog
port 514 on said syslog

errors on all attempts to insert are as below:

com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: Could not initialize class org.graylog2.syslog4j.Syslog
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2216) ~[graylog.jar:?]
at com.google.common.cache.LocalCache.get(LocalCache.java:4147) ~[graylog.jar:?]
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:5053) ~[graylog.jar:?]
at org.graylog2.outputs.OutputRegistry.getOutputForIdAndStream(OutputRegistry.java:102) ~[graylog.jar:?]
at org.graylog2.outputs.OutputRouter.getMessageOutputsForStream(OutputRouter.java:42) ~[graylog.jar:?]
at org.graylog2.outputs.OutputRouter.getStreamOutputsForMessage(OutputRouter.java:62) ~[graylog.jar:?]
at org.graylog2.buffers.processors.OutputBufferProcessor.onEvent(OutputBufferProcessor.java:132) ~[graylog.jar:?]
at org.graylog2.buffers.processors.OutputBufferProcessor.onEvent(OutputBufferProcessor.java:51) ~[graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.graylog2.syslog4j.Syslog
at com.wizecore.graylog2.plugin.SyslogOutput.(SyslogOutput.java:136) ~[?:?]
at com.wizecore.graylog2.plugin.SyslogOutput$$FastClassByGuice$$8143f87b.newInstance() ~[?:?]
at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) ~[graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:111) ~[graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[graylog.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[graylog.jar:?]
at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) ~[graylog.jar:?]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) ~[graylog.jar:?]
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) ~[graylog.jar:?]
at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:776) ~[graylog.jar:?]

[skuzbucket1]

On start of graylog the only entry related to this plugin is logged as such

2018-11-15T15:56:35.460Z INFO [CmdLineTool] Loaded plugin: SyslogOutputPlugin 1.0.0 [com.wizecore.graylog2.plugin.SyslogOutput]

[skuzbucket1]

Tried UDP and IP of the host as well - no change in behavior

We will try a non-privileged high port and report back

[skuzbucket1]

no change with above 1024 port. Is there a class needed in the Java jar?

NoClassDefFoundError

[huksley]

Possibly, Could not initialize class org.graylog2.syslog4j.Syslog this is a key error - need to understand why - possibly missing some other class or can`t run static code of this class

[skuzbucket1]

ok - any idea on how to proceed?

graylog was installed via yum from the graylog repo

Name : graylog-server
Arch : noarch
Version : 2.4.6
Release : 1
Size : 127 M
Repo : installed
From repo : graylog
Summary : Graylog server
URL : https://www.graylog.org/
License : GPLv3
Description : Graylog server

[graylog]
name=graylog
baseurl=https://packages.graylog2.org/repo/el/stable/2.4/$basearch/

these are the files in the graylog.jar replated to syslog4j

./org/graylog2/syslog4j/impl/backlog/Syslog4jBackLogHandler.class
./org/graylog2/syslog4j/impl/log4j/Syslog4jAppender.class
./org/graylog2/syslog4j/impl/log4j/Syslog4jAppenderSkeleton.class
./org/graylog2/syslog4j/impl/message/processor/SyslogMessageProcessor.class
./org/graylog2/syslog4j/server/impl/event/SyslogServerEvent.class
./org/graylog2/syslog4j/server/SyslogServer.class
./org/graylog2/syslog4j/server/SyslogServerConfigIF.class
./org/graylog2/syslog4j/server/SyslogServerEventHandlerIF.class
./org/graylog2/syslog4j/server/SyslogServerEventIF.class
./org/graylog2/syslog4j/server/SyslogServerIF.class
./org/graylog2/syslog4j/server/SyslogServerMain$Options.class
./org/graylog2/syslog4j/server/SyslogServerMain.class
./org/graylog2/syslog4j/server/SyslogServerSessionEventHandlerIF.class
./org/graylog2/syslog4j/server/SyslogServerSessionlessEventHandlerIF.class
./org/graylog2/syslog4j/Syslog.class
./org/graylog2/syslog4j/Syslog4jVersion.class
./org/graylog2/syslog4j/SyslogBackLogHandlerIF.class
./org/graylog2/syslog4j/SyslogCharSetIF.class
./org/graylog2/syslog4j/SyslogConfigIF.class
./org/graylog2/syslog4j/SyslogConstants.class
./org/graylog2/syslog4j/SyslogIF.class
./org/graylog2/syslog4j/SyslogMain$Options.class
./org/graylog2/syslog4j/SyslogMain.class
./org/graylog2/syslog4j/SyslogMessageIF.class
./org/graylog2/syslog4j/SyslogMessageModifierConfigIF.class
./org/graylog2/syslog4j/SyslogMessageModifierIF.class
./org/graylog2/syslog4j/SyslogMessageProcessorIF.class
./org/graylog2/syslog4j/SyslogPoolConfigIF.class
./org/graylog2/syslog4j/SyslogRuntimeException.class

[skuzbucket1]

Switch of Java didn't work

Now: Oracle Corporation 1.8.0_191 on Linux 3.10.0-862.14.4.el7.x86_64

[huksley]

• Which Linux OS + version?
• Pleasecheck your log for more entries referencing org.graylog2.syslog4j.Syslog, particularly for any ExceptionInInitializerError messages

https://stackoverflow.com/a/1416543

[skuzbucket1]

Centos 7.4
no errors with ExceptionInInitializerError
the only error found is the one shown above

we may just have to do a full reinstall of everything :(

[jalogisch]

what other plugins did you have installed?

[skuzbucket1]

none - only the factory provided

-rw-r--r-- 1 root root 20654 Jun 13 19:39 graylog-output-syslog-2.4.5.jar
-rw-r--r-- 1 root root 15185446 Jul 16 19:53 graylog-plugin-aws-2.4.6.jar
-rw-r--r-- 1 root root 27035 Jul 16 19:53 graylog-plugin-beats-2.4.6.jar
-rw-r--r-- 1 root root 60155 Jul 16 19:53 graylog-plugin-cef-2.4.6.jar
-rw-r--r-- 1 root root 2971716 Jul 16 19:53 graylog-plugin-collector-2.4.6.jar
-rw-r--r-- 1 root root 4297633 Jul 16 19:53 graylog-plugin-enterprise-integration-2.4.6.jar
-rw-r--r-- 1 root root 6617237 Jul 16 19:53 graylog-plugin-map-widget-2.4.6.jar
-rw-r--r-- 1 root root 705989 Jul 16 19:53 graylog-plugin-netflow-2.4.6.jar
-rw-r--r-- 1 root root 5596198 Jul 16 19:53 graylog-plugin-pipeline-processor-2.4.6.jar
-rw-r--r-- 1 root root 4574608 Jul 16 19:53 graylog-plugin-threatintel-2.4.6.jar

[skuzbucket1]

at a standstill as of now. Only option we have is reinstall everything from scratch and try again but it seems as if the tar package install versus the yum package install differs greatly in behavior once it is running

I can turn on debug if it helps and see if anything else is created log wise but we are out of ideas

[skuzbucket1]

the only deviation is the following which we thought to be benign that "could" be an issue

rpm -Uvh https://s3.amazonaws.com/aaronsilber/public/authbind-2.1.1-0.1.x86_64.rpm
touch /etc/authbind/byport/514
chown graylog:graylog /etc/authbind/byport/514
chmod 755 /etc/authbind/byport/514
touch '/etc/authbind/byport/!514'
chown graylog:graylog '/etc/authbind/byport/!514'
chmod 755 '/etc/authbind/byport/!514'

https://www.google.com/search?q=authbind+centos&sa=X&ved=2ahUKEwjK557Vq-HeAhUSzlMKHTfwBCYQ1QIoAXoECAMQAg&biw=1680&bih=899

[skuzbucket1]

deep debug for the Win. The issue is related to system hardening and /tmp set to noexec

Once this was backed off, the plugin started as expected. Maybe a note to add to the deployment as a caveat.

[huksley]

Hmm, I wonder why system hardening might affect Syslog client initialization? Should not be the case as it might affect deployments to public clouds, etc.

Thanks for reporting this anyway!

[jandrusk]

I would suggest an option for the plugin that allows a custom 'tmp' directory to be configured so as not weaken the hardening config. The default drop point on UNIX for a lot of malware is /tmp and allow exec is bad.

[huksley]

Completely agree that allowing exec is bad, but the plugin itself does not write nor exec some files.

It seems like syslog4j does some initialization which involves /tmp folder access? Not sure.

Could you please help me setting up environment so I reproduce it? Thanks!

[huksley]

Closing this due to inactivity. It would be great to try to reproduce this problem with latest 3.3.x graylog.

[achevalet]

just reproduced on a fresh install with:

• graylog-server 3.3.16-1
• graylog-output-syslog 4.0.8

Elasticsearch runs fine with -Djna.tmpdir, Graylog runs fine with default jvm options, but this plugin requires the exec flag on /tmp... please help!

[huksley]

Unfortunately, because of org.graylog2.syslog4j package dependency on
https://github.com/java-native-access/jna/blob/master/src/com/sun/jna/Native.java
this library also needs -Djna.tmpdir setting configured.