Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cookie not writeable when readOnlyRootFilesystem: true #243

Open
alwinmark opened this issue Sep 28, 2020 · 2 comments · May be fixed by #246
Open

Cookie not writeable when readOnlyRootFilesystem: true #243

alwinmark opened this issue Sep 28, 2020 · 2 comments · May be fixed by #246

Comments

@alwinmark
Copy link

Background

In order to be able to install vernemq on a production grade security enhanced Kubernetes Cluster with Pod Security Policies enabled, VerneMQ should run with the least privileges possible.

Especially when VerneMQ is public available which might be case in most cases when it comes down to IoT Devices.

Setup:

As as a security aware Developer/Operation Engineer you want a default PSP like:

restricted-psp                     false                 RunAsAny   MustRunAsNonRoot   MustRunAs   MustRunAs   true             configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim

When one installs now vernemq:

helm install vernemq vernemq/vernemq

Expected Behaviour

Everything should come up and:

kubectl exec vernemq-0 /vernemq/bin/vmq-admin cluster show

should print everything is fine.

Actual Behaviour

=ERROR REPORT==== 28-Sep-2020::13:17:14.103869 ===                                                         
Failed to create cookie file '/vernemq/.erlang.cookie': erofs
=SUPERVISOR REPORT==== 28-Sep-2020::13:17:14.104081 ===
    supervisor: {local,net_sup}                                                                            
    errorContext: start_error                                                                              
    reason: {"Failed to create cookie file '/vernemq/.erlang.cookie': erofs",
             [{auth,init_cookie,0,[{file,"auth.erl"},{line,290}]},
              {auth,init,1,[{file,"auth.erl"},{line,144}]},
              {gen_server,init_it,2,[{file,"gen_server.erl"},{line,417}]},
              {gen_server,init_it,6,[{file,"gen_server.erl"},{line,385}]},
              {proc_lib,init_p_do_apply,3,                                                                 
                        [{file,"proc_lib.erl"},{line,226}]}]}
    offender: [{pid,undefined},                                                                            
               {id,auth},                                                                                  
               {mfargs,{auth,start_link,[]}},                                                              
               {restart_type,permanent},                                                                   
               {shutdown,2000},                                                                            
               {child_type,worker}]                                                                        

=CRASH REPORT==== 28-Sep-2020::13:17:14.104357 ===                                                         
  crasher:                                                                                                 
    initial call: auth:init/1                                                                              
    pid: <0.82.0>                                                                                          
    registered_name: []                                                                                    
    exception error: "Failed to create cookie file '/vernemq/.erlang.cookie': erofs"
      in function  auth:init_cookie/0 (auth.erl, line 290)
      in call from auth:init/1 (auth.erl, line 144)                                                        
      in call from gen_server:init_it/2 (gen_server.erl, line 417)
      in call from gen_server:init_it/6 (gen_server.erl, line 385)
    ancestors: [net_sup,kernel_sup,<0.47.0>]                                                               
    message_queue_len: 0                                                                                   
    messages: []                                                                                           
    links: [<0.80.0>]                                                                                      
    dictionary: []                                                                                         
    trap_exit: true                                                                                        
    status: running                                                                                        
    heap_size: 987                                                                                         
    stack_size: 28                                                                                         
    reductions: 2934                                                                                       
  neighbours:                                                                                              

escript: exception error: no match of right hand side value 
                 {error,                                                                                   
                     {{shutdown,                                                                           
                          {failed_to_start_child,auth,
                              {"Failed to create cookie file '/vernemq/.erlang.cookie': erofs",
                               [{auth,init_cookie,0,                                                       
                                    [{file,"auth.erl"},{line,290}]},
                                {auth,init,1,[{file,"auth.erl"},{line,144}]},
                                {gen_server,init_it,2,
                                    [{file,"gen_server.erl"},{line,417}]},
                                {gen_server,init_it,6,
                                    [{file,"gen_server.erl"},{line,385}]},
                                {proc_lib,init_p_do_apply,3,
                                    [{file,"proc_lib.erl"},{line,226}]}]}}},
                      {child,undefined,net_sup_dynamic,
                          {erl_distribution,start_link,
                              [['[email protected]',
                                longnames],                                                                
                               false,net_sup_dynamic]},
                          permanent,1000,supervisor,                                                       
                          [erl_distribution]}}}                                                            
command terminated with exit code 127

Proposal

In order to fix that, data generated or changed during Runtime of a Container should not be placed into the same folder as the static compiled/backed stuff.

One possible Solution could be moving the Erlang Cookie into the /vernemq/data directory.
Another one would be moving it into an own Volume and add that at least to the Helm Chart as emptyDir.
@alwinmark
Copy link
Author

I'm not into Erlang unfortunately but as far as I read, I guess that the cookie is always written to the home folder of the current user.
So this line seems to be the one, that needs to be changed:
https://github.com/vernemq/docker-vernemq/blob/master/Dockerfile#L7

@alwinmark
Copy link
Author

alwinmark commented Sep 28, 2020 •
edited
Loading

Ok I guess I could fix that on my site with:

Dockerfile

-    useradd --uid 10000 --system --gid vernemq --home-dir /vernemq vernemq
+    useradd --uid 10000 --system --gid vernemq --home-dir /vernemq/data/home vernemq

Entrypoint script

+mkdir -p /vernemq/data/home
+chmod g-rwx /vernemq/data/home/.erlang.cookie

alwinmark pushed a commit to alwinmark/docker-vernemq that referenced this issue Sep 29, 2020
@crosscan-ama
Enable Container run with readonly root filesystem
2ab48a0 
Therefore moved home directory into data volume and fix right of Erlang Cookie in case of reclaiming a volume on kubernetes.

fixes: vernemq#243
@alwinmark alwinmark linked a pull request Sep 29, 2020 that will close this issue
Open
hsudbrock pushed a commit to hsudbrock/docker-vernemq that referenced this issue Feb 26, 2024
@crosscan-ama @hsudbrock
Enable Container run with readonly root filesystem
1130ba2 
Therefore moved home directory into data volume and fix right of Erlang Cookie in case of reclaiming a volume on kubernetes.

fixes: vernemq#243
hsudbrock pushed a commit to hsudbrock/docker-vernemq that referenced this issue Feb 26, 2024
@crosscan-ama @hsudbrock
Enable Container run with readonly root filesystem
157a318 
Therefore moved home directory into data volume and fix right of Erlang Cookie in case of reclaiming a volume on kubernetes.

fixes: vernemq#243
ioolkos pushed a commit to alwinmark/docker-vernemq that referenced this issue Apr 28, 2024
@crosscan-ama @ioolkos
Enable Container run with readonly root filesystem
1183fee 
Therefore moved home directory into data volume and fix right of Erlang Cookie in case of reclaiming a volume on kubernetes.

fixes: vernemq#243
ioolkos pushed a commit to alwinmark/docker-vernemq that referenced this issue Apr 28, 2024
@crosscan-ama @ioolkos
Enable Container run with readonly root filesystem
cb0ff80 
Therefore moved home directory into data volume and fix right of Erlang Cookie in case of reclaiming a volume on kubernetes.

fixes: vernemq#243
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Enable Container run with readonly root filesystem alwinmark/docker-vernemq
1 participant
@alwinmark