Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change FragPipe temp directory on Linux/Unix #755

Closed
GregStacey opened this issue Jul 8, 2022 · 7 comments
Closed

Change FragPipe temp directory on Linux/Unix #755

GregStacey opened this issue Jul 8, 2022 · 7 comments

Comments

@GregStacey
Copy link

I'm successfully running the headless version of fragpipe, but I'm encountering a problem when submitting it via scheduler on a cluster. When submitted, the cache directory is read-only (~/.config/FragPipe/fragpipe/) and the run fails. The full error log is below.

I suspect this can be solved on the cluster side, but I'm also curious if there's a way to change the cache path in headless mode. Thanks!

2022-07-08 11:19:36,512 WARN - The output directory is not empty. Some files might be overwritten in: /scratch/st-ljfoster-1/staceyri/isabela_fragpipe/experiments/fullsearch
2022-07-08 11:19:36,580 ERROR - Error saving ui cache
java.nio.file.FileSystemException: /home/staceyri/.config/FragPipe/fragpipe/fragpipe-ui.cache: Read-only file system
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:478)
at java.base/java.nio.file.Files.newOutputStream(Files.java:219)
at com.dmtavt.fragpipe.api.PropsFile.save(PropsFile.java:98)
at com.dmtavt.fragpipe.Fragpipe.saveCache(Fragpipe.java:261)
at com.dmtavt.fragpipe.Fragpipe.on(Fragpipe.java:675)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:517)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:511)
at org.greenrobot.eventbus.AsyncPoster.run(AsyncPoster.java:46)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2022-07-08 11:19:36,594 ERROR - Error saving runtime cache
java.nio.file.FileSystemException: /home/staceyri/.config/FragPipe/fragpipe/fragpipe-runtime.cache: Read-only file system
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:478)
at java.base/java.nio.file.Files.newOutputStream(Files.java:219)
at com.dmtavt.fragpipe.api.PropsFile.save(PropsFile.java:98)
at com.dmtavt.fragpipe.Fragpipe.saveCache(Fragpipe.java:269)
at com.dmtavt.fragpipe.Fragpipe.on(Fragpipe.java:675)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:517)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:511)
at org.greenrobot.eventbus.AsyncPoster.run(AsyncPoster.java:46)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
[SEVERE] Could not dispatch event: class com.dmtavt.fragpipe.messages.MessageSaveCache to subscribing class class com.dmtavt.fragpipe.Fragpipe
java.lang.IllegalArgumentException: File parameter 'destDir is not writable: '/home/staceyri/.config/FragPipe/fragpipe/workflows'
at org.apache.commons.io.FileUtils.requireCanWrite(FileUtils.java:2619)
at org.apache.commons.io.FileUtils.doCopyDirectory(FileUtils.java:1305)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:699)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:630)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:531)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:502)
at com.dmtavt.fragpipe.Fragpipe.saveCache(Fragpipe.java:279)
at com.dmtavt.fragpipe.Fragpipe.on(Fragpipe.java:675)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:517)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:511)
at org.greenrobot.eventbus.AsyncPoster.run(AsyncPoster.java:46)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2022-07-08 11:19:36,629 ERROR - Error delivering events through the bus
java.lang.IllegalArgumentException: File parameter 'destDir is not writable: '/home/staceyri/.config/FragPipe/fragpipe/workflows'
at org.apache.commons.io.FileUtils.requireCanWrite(FileUtils.java:2619)
at org.apache.commons.io.FileUtils.doCopyDirectory(FileUtils.java:1305)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:699)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:630)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:531)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:502)
at com.dmtavt.fragpipe.Fragpipe.saveCache(Fragpipe.java:279)
at com.dmtavt.fragpipe.Fragpipe.on(Fragpipe.java:675)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:517)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:511)
at org.greenrobot.eventbus.AsyncPoster.run(AsyncPoster.java:46)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2022-07-08 11:19:36,633 ERROR - java.lang.IllegalArgumentException: File parameter 'destDir is not writable: '/home/staceyri/.config/FragPipe/fragpipe/workflows'
at org.apache.commons.io.FileUtils.requireCanWrite(FileUtils.java:2619)
at org.apache.commons.io.FileUtils.doCopyDirectory(FileUtils.java:1305)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:699)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:630)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:531)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:502)
at com.dmtavt.fragpipe.Fragpipe.saveCache(Fragpipe.java:279)
at com.dmtavt.fragpipe.Fragpipe.on(Fragpipe.java:675)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:517)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:511)
at org.greenrobot.eventbus.AsyncPoster.run(AsyncPoster.java:46)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

2022-07-08 11:19:37,308 ERROR - Could not save fragger_*.params file to working dir.

@fcyu
Copy link
Member

fcyu commented Jul 8, 2022

I think so. Can you try the approach in #306 (comment) to see if it works for you?

Best,

Fengchao

@GregStacey
Copy link
Author

Thanks for the quick response. Unfortunately I get the following error about the X11 display

(base) [staceyri@login02 isabela_fragpipe]$ JAVA_OPTS=-Djava.io.tmpdir=/scratch/st-ljfoster-1/staceyri/isabela_fragpipe/tmp /scratch/st-ljfoster-1/staceyri/fragpipe/bin/fragpipe

Exception in thread "main" java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
at java.desktop/java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:208)
at java.desktop/java.awt.Window.(Window.java:548)
at java.desktop/java.awt.Frame.(Frame.java:423)
at java.desktop/java.awt.Frame.(Frame.java:388)
at java.desktop/javax.swing.JFrame.(JFrame.java:180)
at com.dmtavt.fragpipe.FragpipeLoader.(FragpipeLoader.java:61)
at com.dmtavt.fragpipe.Fragpipe.main0(Fragpipe.java:358)
at com.dmtavt.fragpipe.FragPipeMain.main(FragPipeMain.java:40)

@fcyu
Copy link
Member

fcyu commented Jul 8, 2022

You need to add the --headless flag (and other commands to set the workflow and manifest) so that FragPipe will run in headless mode. Without any flags, FragPipe runs in GUI mode.

Best,

Fengchao

@GregStacey
Copy link
Author

GregStacey commented Jul 8, 2022

Thanks again. I see my mistake!

Unfortunately export JAVA_OPTS=-Djava.io.tmpdir=/scratch/st-ljfoster-1/staceyri/isabela_fragpipe/tmp/ doesn't fix the error for me. I can see empty folders being made in the new tmp folder:

$ ls /scratch/st-ljfoster-1/staceyri/isabela_fragpipe/tmp/
af6fdd7f-da18-4dd1-b374-2cb499fafd15  d1d6cf4a-2265-4e16-bbaa-d6af9041dfab  e109884b-958b-468a-b85c-4f73c44d0e59  f06836aa-9c9f-4e8c-bb02-77224a71b88a
$ ls /scratch/st-ljfoster-1/staceyri/isabela_fragpipe/tmp/af6fdd7f-da18-4dd1-b374-2cb499fafd15/
$ 

But the same error persists where fragpipe tries to save files in the /home/ directory:

2022-07-08 15:11:15,601 ERROR - Error saving ui cache
java.nio.file.FileSystemException: /home/staceyri/.config/FragPipe/fragpipe/fragpipe-ui.cache: Read-only file system
...

I heard back from the cluster administrators and the problem is that /home/ doesn't exist on the compute nodes. I tried naively redefining $HOME but that also doesn't solve the issue for me. The submission script is below in case that's helpful. Again I don't see the error when just running /path/to/fragpipe on the head node.

# change temp folder
export JAVA_OPTS=-Djava.io.tmpdir=/scratch/st-ljfoster-1/staceyri/isabela_fragpipe/tmp/
# confirm temp folder is changed
echo $JAVA_OPTS

# change home folder
HOME=/scratch/st-ljfoster-1/staceyri/
echo ~

# run fragpipe
/scratch/st-ljfoster-1/staceyri/fragpipe/bin/fragpipe       --headless       --workflow /scratch/st-ljfoster-1/staceyri/isabela_fragpipe/experiments/fullsearch/fullsearch.workflow       --manifest /scratch/st-ljfoster-1/staceyri/isabela_fragpipe/experiments/fullsearch/fullsearch.fp-manifest       --workdir /scratch/st-ljfoster-1/staceyri/isabela_fragpipe//experiments/fullsearch       --ram 0       --config-msfragger /scratch/st-ljfoster-1/staceyri/fragpipe/tools/MSFragger-3.5/MSFragger-3.5.jar       --config-philosopher /arc/project/st-ljfoster-1/philosopher/v4.2.2/philosopher       --config-python /arc/software/spack/opt/spack/linux-centos7-x86_64/gcc-9.1.0/miniconda3-4.6.14-wsswwucnigbnge5edropjstrirpjx7je/bin/python  

@fcyu
Copy link
Member

fcyu commented Jul 8, 2022

Thanks for your feedback.

I think I found the cause of the issue. In Linux and Unix, FragPipe always save cache to HOME:

final String dir = OsUtils.isUnix() ? XDG_CONFIG_HOME + "/FragPipe" : SYS_TEMP_DIR;

For your case, can you try setting a XDG_CONFIG_HOME=<temp path> environment variable? Details of the code can be found from

return System.getenv("XDG_CONFIG_HOME");

Best,

Fengchao

@GregStacey
Copy link
Author

Thanks, Fengchao. Adding the following lines to the submission script fixed the issue for me.

XDG_CONFIG_HOME="/path/to/working/directory/"
export XDG_CONFIG_HOME

@fcyu
Copy link
Member

fcyu commented Jul 11, 2022

Thanks for the positive feedback.

Best,

Fengchao

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants