Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot execute program. fork/exec /tmp/[...]/InteractParser: exec format error #634

Closed
GKok opened this issue Apr 3, 2022 · 25 comments
Closed
Assignees

Comments

@GKok
Copy link

GKok commented Apr 3, 2022

Hi!

I've used FragPipe on a local Windows machine for optimization of our workflow, but want to run it on a high performance cluster with X11 forwarding for the actual analyses.

Unfortunately, it crashes after a while, with the following set of errors repeated many times:

time="11:21:34" level=error msg="Cannot deploy asset. RefreshParser"
time="11:21:34" level=error msg="Cannot deploy asset. InteractParser"
time="11:21:34" level=error msg="Cannot deploy asset. PeptideProphetParser"
time="11:21:34" level=fatal msg="Cannot execute program. fork/exec /tmp/58f42d88-e192-4a0c-bc17-27c425b38e9a/InteractParser: exec format error
Process 'PeptideProphet' finished, exit code: 1
Process returned non-zero exit code, stopping
Processing interrupted, stopping PeptideProphet

Machine:

CentOS Linux 7
java version "1.7.0_261"
OpenJDK Runtime Environment (rhel-2.6.22.2.el7_8-x86_64 u261-b02)
OpenJDK 64-Bit Server VM (build 24.261-b02, mixed mode)

With latests versions of FragPipe, MSFragger and Philosopher.

Any ideas on what's going wrong?

@GKok
Copy link
Author

GKok commented Apr 4, 2022

Nevermind, I've noticed the Java version is not supported (1.8+). Apologies, seems like they're running multiple versions.

@GKok GKok closed this as completed Apr 4, 2022
@GKok
Copy link
Author

GKok commented Apr 4, 2022

Reopened: the issue persists with Java 1.8.

java version "1.8.0_321"
Java(TM) SE Runtime Environment (build 1.8.0_321-b07)
Java HotSpot(TM) 64-Bit Server VM (build 25.321-b07, mixed mode)

@GKok GKok reopened this Apr 4, 2022
@prvst
Copy link

prvst commented Apr 4, 2022

@GKok I don't think your error has anything to do with your Java version, InteractParser is written in C++. I don't know what the exec format error error means, but I presume it is related to the fact you're trying to run it on a HPC. I'm moving this to the FragPipe tracker so other can add their opinion too.

@prvst prvst transferred this issue from Nesvilab/philosopher Apr 4, 2022
@prvst
Copy link

prvst commented Apr 4, 2022

@GKok is having some issues running some of the TPP tools via X11 forwarding from a HPC, I don't know the source of the error.

@prvst
Copy link

prvst commented Apr 4, 2022

@guoci do you think it could be a problem with the CentOS binary deploy?

@GKok
Copy link
Author

GKok commented Apr 4, 2022

I've asked the HPC administrators their opinion, and they told me that /tmp only has 100 MB of space on the compute nodes. Instead, $TMPDIR should be used. Could that be the problem, and if so, is it possible to specify this anywhere?

@prvst
Copy link

prvst commented Apr 4, 2022

That could be the reason. Unfortunately, I don't know if you can change the temporary folder location using FragPipe, you can do that in Philosopher when you run it manually though.

@GKok
Copy link
Author

GKok commented Apr 4, 2022

Thanks! I've found this issue, and will try their solution: #306

> export JAVA_OPTS=-Djava.io.tmpdir=$TMPDIR
> ./fragpipe

@fcyu
Copy link
Member

fcyu commented Apr 4, 2022

That is changing the temp directory of FragPipe. You need to change the temp directory of Philosopher. But, unfortunately, you cannot do that in FragPipe.

Best,

Fengchao

@GKok
Copy link
Author

GKok commented Apr 4, 2022

That's unfortunate. Would it help to re-compile philosopher on the HPC? I guess this would needed to be done prior to every run, as the tmpdir changes every session.

@prvst
Copy link

prvst commented Apr 4, 2022

Philosopher is not the issue, the issue is that the FragPipe GUI does not provide users with the option to change the temporary files location. The program has an option that allows you to change the $temp, but the GUI does not expose that for you.

@guoci
Copy link
Member

guoci commented Apr 4, 2022

@GKok you can try the following updated version, which will propagate java.io.tmpdir to Philosopher.
https://drive.google.com/drive/folders/1g1Z_v_eWKhIuuC2wTqu0WahkIAgEBxzi?usp=sharing

@GKok
Copy link
Author

GKok commented Apr 5, 2022

@guoci Many thanks for your prompt update! This indeed passes the correct tmp directory to Philosopher, and fixes the errors above. However, it now crashes at the same step with a different error.

time="10:59:24" level=info msg="Executing PeptideProphet  v4.1.1"
time="10:59:25" level=fatal msg="Cannot execute program. fork/exec /scratch/1938675/InteractParser: text file busy"
Process 'PeptideProphet' finished, exit code: 1
Process returned non-zero exit code, stopping

@guoci
Copy link
Member

guoci commented Apr 5, 2022

@GKok does the file /scratch/1938675/InteractParser exists? Try executing it on the command line and show me the output.

@GKok
Copy link
Author

GKok commented Apr 5, 2022

@guoci Yes, it exists and works.
(the batch number is different, because I had to re-start it)

[gkok@n0082 ~]$ cd /scratch/1966425/
[gkok@n0082 1966425]$ ls
InteractParser  PeptideProphetParser  RefreshParser
[gkok@n0082 1966425]$ ./InteractParser 
./InteractParser (TPP v5.2.1-dev Flammagenitus, Build 201906251008-exported (Linux-x86_64))
usage  : InteractParser <outputfile> <file1.pep.xml> <file2.pep.xml>... <options>
options: 
	-a<path_to_data_dir> : path to the data directory
	-D<path_to_database> : path to the database
	-T<database_type>    : database type
	-X<experiment_label> : assign experiment_label to each spectrum_query
	-L<min_peptide_len>  : minimum peptide length not rejected (default 7)
	-R<max_rank>         : maximum peptide rank in output (default 1)
	-A : allow (don't reject) X containing peptides
	-S : record retention_time_sec in pepXML
	-i : record ion_injection_time in pepXML
	-I : record precursor intensity in pepXML
	-B : record base peak intensity in pepXML
	-O : record total ion current in pepXML 
	-2 : record signal to noise pepXML 
	-G : record collision energy in pepXML
	-V : record compensation voltage in pepXML
	-P : correct protein names (applied automatically for OMSSA results)
	-C : correct charge states (applied automatically for OMSSA results)
	-M : correct Pyro-Glu and Ammonia Loss Modifications E[111], Q[111] (applied automatically for OMSSA results)
	-N : don't write the input file reference to the output pepXML file (Use only when one input file is given)
	-W : only warn, don't skip files containing PeptideProphet results
	-E<enzyme>: enzyme used in sample, one of:   (semi-cleavage specification: -Esemi<enyzme>)
	   trypsin : cut(KR) nocuts(P) sense(C)
	   stricttrypsin : cut(KR) sense(C)
	   argc : cut(R) nocuts(P) sense(C)
	   aspn : cut(D) sense(N)
	   chymotrypsin : cut(FLWY) nocuts(P) sense(C)
	   clostripain : cut(R) nocuts(-) sense(C)
	   cnbr : cut(M) sense(C)
	   elastase : cut(AGILV) nocuts(P) sense(C)
	   formicacid : cut(D) nocuts(P) sense(C)
	   gluc : cut(DE) nocuts(P) sense(C)
	   gluc_bicarb : cut(E) nocuts(P) sense(C)
	   iodosobenzoate : cut(W) nocuts(-) sense(C)
	   lysc : cut(K) nocuts(P) sense(C)
	   lysc-p : cut(K) sense(C)
	   lysn : cut(K) sense(N)
	   lysn_promisc : cut(KR) sense(N)
	   ralphtrypsin : cut(KRST) nocuts(P) sense(C)
	   nonspecific : cut() sense(C)
	   pepsina : cut(FL) nocuts(-) sense(C)
	   proline_endopeptidase : cut(P) nocuts(-) sense(C)
	   trypsin/chymotrypsin : cut(FYWLKR) nocuts(P) sense(C)
	   staph_protease : cut(E) nocuts(-) sense(C)
	   tca : cut(KR) nocuts(P) sense(C)
	       : cut(YWFM) nocuts(P) sense(C)
	       : cut(D) sense(N)
	   trypsin/cnbr : cut(KR) nocuts(P) sense(C)
	                : cut(M) nocuts(P) sense(C)
	   trypsin_gluc : cut(DEKR) nocuts(P) sense(C)
	   trypsin_k : cut(K) nocuts(P) sense(C)
	   trypsin_r : cut(R) nocuts(P) sense(C)
	   thermolysin : cut(ALIVFM) nocuts(DE) sense(N)

InteractParser will also correct some other common issues found in pepXML files written by various search engines.

[gkok@n0082 1966425]$ ./PeptideProphetParser 
[gkok@n0082 1966425]$ ./RefreshParser 
 ./RefreshParser(TPP v5.2.1-dev Flammagenitus, Build 201906251008-exported (Linux-x86_64))
 usage: RefreshParser <pepxmlfile> <database> (<min ntt>(defaults to 0,unless specified in pepXML file with min_number_termini in enzymatic_search_constraint) (DEGEN) (PROT_MW) (PREV_AA_LEN=<length(default=1)>) (NEXT_AA_LEN=<length(default=1)>) (RESTORE_NONEXISTENT_IF_PREFIX=str)
[gkok@n0082 1966425]$ 

@guoci
Copy link
Member

guoci commented Apr 5, 2022

@GKok I don't know what is the cause then. Perphaps you can set the tmpdir to a normal location (outside of /scratch) and retry.

@GKok
Copy link
Author

GKok commented Apr 6, 2022

@guoci Thank you for the suggestion. I've tried that, in combination with updating to Philosopher 4.2.1, but unfortunately it throws the exact same error. Googling tells me the error is caused an attempt to modify an executable (InteractParser) while it's running. I'll check with our HPC administrators if they can think of something, and I'll keep trying :-)

Once again, thank you all for your assistance.

@guoci
Copy link
Member

guoci commented Apr 11, 2022

@GKok I think I might have fixed the issue, please try the following:
https://drive.google.com/drive/folders/1g1Z_v_eWKhIuuC2wTqu0WahkIAgEBxzi?usp=sharing

@GKok
Copy link
Author

GKok commented Apr 11, 2022

@guoci Thanks for the update! This one crashes immediately after initializing the run.

CheckCentroid
java -Xmx297G -cp /home/xxx/gkok/fragpipe/lib/fragpipe-17.2-build36.jar:/home/xxx/gkok/fragpipe/tools/batmass-io-1.25.2.jar com.dmtavt.fragpipe.util.CheckCentroid /hpc/shared/xxx/gkok/MS/mzml/01CPTAC_LSCC_W_BI_20190615_KL/01CPTAC_LSCC_W_BI_20190615_KL_f05.mzML 47
Done in 1695 ms.
Process 'CheckCentroid' finished, exit code: 0
WorkspaceCleanInit [Work dir: /hpc/xxx/users/gkok/MS/FPOut/CPTAC01_02_WT_17.2b36/CPTAC02]
/home/xxx/gkok/fragpipe-17.2-build33/tools/philosopher_v4.2.1_linux_amd64/philosopher workspace --clean --nocheck
time="11:43:19" level=info msg="Executing Workspace  v4.2.1"
time="11:43:19" level=info msg="Removing workspace"
time="11:43:19" level=warning msg="Cannot read file. open .meta/meta.bin: no such file or directory"
time="11:43:19" level=info msg=Done
Process 'WorkspaceCleanInit' finished, exit code: 0
WorkspaceCleanInit [Work dir: /hpc/xxx/users/gkok/MS/FPOut/CPTAC01_02_WT_17.2b36/CPTAC02]
/home/xxx/gkok/fragpipe-17.2-build33/tools/philosopher_v4.2.1_linux_amd64/philosopher workspace --init --nocheck --temp /scratch/2101170/0a47dc89-0cf9-405a-b7d9-1b5c2586f879
time="11:43:19" level=info msg="Executing Workspace  v4.2.1"
time="11:43:19" level=info msg="Creating workspace"
time="11:43:19" level=fatal msg="cannot find the custom temporary folder"
Process 'WorkspaceCleanInit' finished, exit code: 1
Process returned non-zero exit code, stopping

~~~~~~~~~~~~~~~~~~~~
Cancelling 273 remaining tasks

I did, however, manage to workaround the error by using the FragPipe 17.2-build 33 you sent me, with Philosopher 4.1.1 (instead of 4.2.1) and deselecting 'generate peptide-level summary' and 'generate protein-level summary'. This way, I get through to the end and get the reports I need.

Then, I noticed that the philosopher steps are quite time-consuming. While I'm running on a HPC with many cores available, FragPipe runs every philosopher step sequentially instead of in parallel, ignoring the 'parallelism' setting in the 'Workflow' tab. Do you know if there's another setting that I'm missing, to use the full CPU power available? It would save an enormous amount of time if FragPipe could run the philosoper steps (database/filter/etc.) in e.g., 128 parallel threads.

Thanks again :-)

@guoci
Copy link
Member

guoci commented Apr 11, 2022

@GKok Please, try a new update again:
https://drive.google.com/drive/folders/1g1Z_v_eWKhIuuC2wTqu0WahkIAgEBxzi?usp=sharing
Some parts of Philosopher (eg Proteinprophet) cannot be run in parallel, it is an external tool. The ProteinProphet step is especially slow in 4.1.1.
But in 4.2.1, the slow ProteinProphet step has been resolved and other parts of Philosopher should also run much faster.

@GKok
Copy link
Author

GKok commented Apr 12, 2022

@guoci Almost there, but it crashed at the end.

Process 'PhilosopherFilter' finished, exit code: 0
FreeQuant [Work dir: /hpc/xxx/users/gkok/MS/FPOut/CPTAC01_02_WT_17.2b36-2/CPTAC01]
/home/xxx/gkok/fragpipe-17.2-build33/tools/philosopher_v4.2.1_linux_amd64/philosopher freequant --ptw 0.4 --tol 10 --dir /hpc/shared/xxx/gkok/MS/mzml/01CPTAC_LSCC_W_BI_20190615_KL
time="09:18:32" level=info msg="Executing Label-free quantification  v4.2.1"
panic: reflect: reflect.flag.mustBeAssignable using unaddressable value

goroutine 1 [running]:
reflect.flag.mustBeAssignableSlow(0x16)
	/usr/lib/go-1.13/src/reflect/value.go:247 +0x138
reflect.flag.mustBeAssignable(...)
	/usr/lib/go-1.13/src/reflect/value.go:234
reflect.Value.Set(0xb9de40, 0xc0001e6520, 0x16, 0xb9de40, 0x0, 0x16)
	/usr/lib/go-1.13/src/reflect/value.go:1531 +0x3b
github.com/vmihailenco/msgpack/v5.ptrValueDecoder.func1(0xc059ae9710, 0xb9de40, 0xc0001e6520, 0x16, 0xb9de40, 0xc0001e6520)
	/home/prvst/go/pkg/mod/github.com/vmihailenco/msgpack/[email protected]/decode_value.go:130 +0xfc
github.com/vmihailenco/msgpack/v5.(*Decoder).DecodeValue(0xc059ae9710, 0xb9de40, 0xc0001e6520, 0x16, 0xc0001e6520, 0x16)
	/home/prvst/go/pkg/mod/github.com/vmihailenco/msgpack/[email protected]/decode.go:309 +0x8c
github.com/vmihailenco/msgpack/v5.(*Decoder).Decode(0xc059ae9710, 0xb00520, 0xc059adbe30, 0xc0001c2000, 0xc00000e168)
	/home/prvst/go/pkg/mod/github.com/vmihailenco/msgpack/[email protected]/decode.go:288 +0x1de
philosopher/lib/sys.Restore(0xb9de40, 0xc0001e6520, 0xc059b16140, 0xd, 0x0)
	/workspace/philosopher/lib/sys/sys.go:269 +0x22e
philosopher/lib/rep.RestoreProtein(0xc0001e6520)
	/workspace/philosopher/lib/rep/io.go:79 +0xd6
philosopher/lib/rep.(*Evidence).RestoreGranular(0xc0001e6000)
	/workspace/philosopher/lib/rep/io.go:59 +0x72
philosopher/lib/qua.RunLabelFreeQuantification(0x0, 0x0, 0x0, 0x0, 0xc3f1c5, 0x4, 0x7ffde9f79a52, 0x3d, 0x0, 0x0, ...)
	/workspace/philosopher/lib/qua/qua.go:44 +0x76
philosopher/cmd.glob..func6(0x1f2cf80, 0xc0000f69c0, 0x0, 0x6)
	/workspace/philosopher/cmd/freequant.go:46 +0x14f
github.com/spf13/cobra.(*Command).execute(0x1f2cf80, 0xc0000f6900, 0x6, 0x6, 0x1f2cf80, 0xc0000f6900)
	/home/prvst/go/pkg/mod/github.com/spf13/[email protected]/command.go:844 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0x1f2ec60, 0x455140, 0xc0001a5f48, 0xc0001a5f40)
	/home/prvst/go/pkg/mod/github.com/spf13/[email protected]/command.go:945 +0x317
github.com/spf13/cobra.(*Command).Execute(...)
	/home/prvst/go/pkg/mod/github.com/spf13/[email protected]/command.go:885
philosopher/cmd.Execute()
	/workspace/philosopher/cmd/root.go:35 +0x2d
main.main()
	/workspace/philosopher/main.go:25 +0x7c
Process 'FreeQuant' finished, exit code: 2
Process returned non-zero exit code, stopping

~~~~~~~~~~~~~~~~~~~~
Cancelling 12 remaining tasks

@guoci
Copy link
Member

guoci commented Apr 12, 2022

Please try an updated Philosopher here:
https://drive.google.com/drive/folders/1g1Z_v_eWKhIuuC2wTqu0WahkIAgEBxzi?usp=sharing

@GKok
Copy link
Author

GKok commented Apr 12, 2022

@guoci This version, in combination with your latest FP build works! And indeed approx. 25% faster than the previous version. Many thanks! :)

@GKok GKok closed this as completed Apr 13, 2022
@GKok
Copy link
Author

GKok commented Apr 13, 2022

Well... That's when using the first 50 files from 2 experiments. I now tried with 250 files from 10 experiments. That crashes again, now at Rewrite pepxml. Any ideas?

PeptideProphet: Delete temp
java -cp /home/xxx/gkok/fragpipe/lib/fragpipe-17.2-build36.jar com.github.chhh.utils.FileDelete /hpc/xxx/users/gkok/MS/FPOut/CPTAC01_10_WT/CPTAC10/fragpipe-10CPTAC_LSCC_W_BI_20190726_KL_f25.pepXML-temp
Process 'PeptideProphet: Delete temp' finished, exit code: 0
Rewrite pepxml [Work dir: /hpc/xxx/users/gkok/MS/FPOut/CPTAC01_10_WT/CPTAC07]
java -cp /home/xxx/gkok/fragpipe/lib/* com.dmtavt.fragpipe.util.RewritePepxml /hpc/xxx/users/gkok/MS/FPOut/CPTAC01_10_WT/CPTAC07/interact-07CPTAC_LSCC_W_BI_20190714_KL_f23.pep.xml /hpc/shared/xxx/gkok/MS/mzml/07CPTAC_LSCC_W_BI_20190714_KL/07CPTAC_LSCC_W_BI_20190714_KL_f23.mzML
Not all given paths exist: Optional[/hpc/xxx/users/gkok/MS/FPOut/CPTAC01_10_WT/CPTAC07/interact-07CPTAC_LSCC_W_BI_20190714_KL_f23.pep.xml]
Process 'Rewrite pepxml' finished, exit code: 1
Process returned non-zero exit code, stopping

~~~~~~~~~~~~~~~~~~~~
Cancelling 315 remaining tasks

Interstingly, the file it's referring to seems to exist as a tmp-file:

[gkok@hpcs03 CPTAC07]$ ls interact-07CPTAC_LSCC_W_BI_20190714_KL_f23*
interact-07CPTAC_LSCC_W_BI_20190714_KL_f23.pep.xml.tmp.Q1wuu9

And a separate InteractParser error in another run:

*** Error in `/scratch/2376728/f2d5a453-59ad-48d8-ac7c-d9a30423b473/InteractParser': realloc(): invalid old size: 0x0000000001bf8420 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7f474)[0x2b12ed47e474]
/lib64/libc.so.6(+0x84861)[0x2b12ed483861]
/lib64/libc.so.6(realloc+0x1d2)[0x2b12ed484e12]
/scratch/2376728/f2d5a453-59ad-48d8-ac7c-d9a30423b473/InteractParser[0x417557]
/scratch/2376728/f2d5a453-59ad-48d8-ac7c-d9a30423b473/InteractParser[0x416e65]
/scratch/2376728/f2d5a453-59ad-48d8-ac7c-d9a30423b473/InteractParser[0x4153b6]
/scratch/2376728/f2d5a453-59ad-48d8-ac7c-d9a30423b473/InteractParser[0x432f52]
/scratch/2376728/f2d5a453-59ad-48d8-ac7c-d9a30423b473/InteractParser[0x40d96a]
/scratch/2376728/f2d5a453-59ad-48d8-ac7c-d9a30423b473/InteractParser[0x40cdd3]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x2b12ed421555]
/scratch/2376728/f2d5a453-59ad-48d8-ac7c-d9a30423b473/InteractParser[0x40bea2]
======= Memory map: ========
00400000-00778000 r-xp 00000000 fd:04 74                                 /scratch/2376728/f2d5a453-59ad-48d8-ac7c-d9a30423b473/InteractParser
00977000-009b0000 r--p 00377000 fd:04 74                                 /scratch/2376728/f2d5a453-59ad-48d8-ac7c-d9a30423b473/InteractParser
009b0000-00a0d000 rw-p 003b0000 fd:04 74                                 /scratch/2376728/f2d5a453-59ad-48d8-ac7c-d9a30423b473/InteractParser
00a0d000-00a13000 rw-p 00000000 00:00 0 
01bee000-01c0f000 rw-p 00000000 00:00 0                                  [heap]
2b12ec589000-2b12ec5ab000 r-xp 00000000 fd:00 12583550                   /usr/lib64/ld-2.17.so
2b12ec5ab000-2b12ec5ad000 rw-p 00000000 00:00 0 
2b12ec5cd000-2b12ec5ce000 rw-p 00000000 00:00 0 
2b12ec5ce000-2b12ec6c9000 rw-p 00000000 00:00 0 
2b12ec7aa000-2b12ec7ab000 r--p 00021000 fd:00 12583550                   /usr/lib64/ld-2.17.so
2b12ec7ab000-2b12ec7ac000 rw-p 00022000 fd:00 12583550                   /usr/lib64/ld-2.17.so
2b12ec7ac000-2b12ec7ad000 rw-p 00000000 00:00 0 
2b12ec7ad000-2b12ec7c2000 r-xp 00000000 fd:00 12584009                   /usr/lib64/libz.so.1.2.7
2b12ec7c2000-2b12ec9c1000 ---p 00015000 fd:00 12584009                   /usr/lib64/libz.so.1.2.7
2b12ec9c1000-2b12ec9c2000 r--p 00014000 fd:00 12584009                   /usr/lib64/libz.so.1.2.7
2b12ec9c2000-2b12ec9c3000 rw-p 00015000 fd:00 12584009                   /usr/lib64/libz.so.1.2.7
2b12ec9c3000-2b12ec9da000 r-xp 00000000 fd:00 12692285                   /usr/lib64/libpthread-2.17.so
2b12ec9da000-2b12ecbd9000 ---p 00017000 fd:00 12692285                   /usr/lib64/libpthread-2.17.so
2b12ecbd9000-2b12ecbda000 r--p 00016000 fd:00 12692285                   /usr/lib64/libpthread-2.17.so
2b12ecbda000-2b12ecbdb000 rw-p 00017000 fd:00 12692285                   /usr/lib64/libpthread-2.17.so
2b12ecbdb000-2b12ecbdf000 rw-p 00000000 00:00 0 
2b12ecbdf000-2b12eccc8000 r-xp 00000000 fd:00 12763778                   /usr/lib64/libstdc++.so.6.0.19
2b12eccc8000-2b12ecec8000 ---p 000e9000 fd:00 12763778                   /usr/lib64/libstdc++.so.6.0.19
2b12ecec8000-2b12eced0000 r--p 000e9000 fd:00 12763778                   /usr/lib64/libstdc++.so.6.0.19
2b12eced0000-2b12eced2000 rw-p 000f1000 fd:00 12763778                   /usr/lib64/libstdc++.so.6.0.19
2b12eced2000-2b12ecee7000 rw-p 00000000 00:00 0 
2b12ecee7000-2b12ecfe8000 r-xp 00000000 fd:00 12584690                   /usr/lib64/libm-2.17.so
2b12ecfe8000-2b12ed1e7000 ---p 00101000 fd:00 12584690                   /usr/lib64/libm-2.17.so
2b12ed1e7000-2b12ed1e8000 r--p 00100000 fd:00 12584690                   /usr/lib64/libm-2.17.so
2b12ed1e8000-2b12ed1e9000 rw-p 00101000 fd:00 12584690                   /usr/lib64/libm-2.17.so
2b12ed1e9000-2b12ed1fe000 r-xp 00000000 fd:00 12676549                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
2b12ed1fe000-2b12ed3fd000 ---p 00015000 fd:00 12676549                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
2b12ed3fd000-2b12ed3fe000 r--p 00014000 fd:00 12676549                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
2b12ed3fe000-2b12ed3ff000 rw-p 00015000 fd:00 12676549                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
2b12ed3ff000-2b12ed5c3000 r-xp 00000000 fd:00 12583955                   /usr/lib64/libc-2.17.so
2b12ed5c3000-2b12ed7c2000 ---p 001c4000 fd:00 12583955                   /usr/lib64/libc-2.17.so
2b12ed7c2000-2b12ed7c6000 r--p 001c3000 fd:00 12583955                   /usr/lib64/libc-2.17.so
2b12ed7c6000-2b12ed7c8000 rw-p 001c7000 fd:00 12583955                   /usr/lib64/libc-2.17.so
2b12ed7c8000-2b12ed7cd000 rw-p 00000000 00:00 0 
2b12ed8c2000-2b12ed8c3000 ---p 00000000 00:00 0 
2b12ed8c3000-2b12edac3000 rw-p 00000000 00:00 0 
2b12f0000000-2b12f0021000 rw-p 00000000 00:00 0 
2b12f0021000-2b12f4000000 ---p 00000000 00:00 0 
7fff11ab6000-7fff11ad8000 rw-p 00000000 00:00 0                          [stack]
7fff11b0c000-7fff11b0e000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
*** Error in `/scratch/2376728/7087c18d-ba4a-41e0-9ea4-8c0e356e941d/InteractParser': malloc(): memory corruption (fast): 0x0000000001c20770 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7f474)[0x2b906a421474]
/lib64/libc.so.6(+0x82bb0)[0x2b906a424bb0]
/lib64/libc.so.6(__libc_malloc+0x4c)[0x2b906a42778c]
/lib64/libstdc++.so.6(_Znwm+0x1d)[0x2b9069be118d]
/lib64/libstdc++.so.6(_Znam+0x9)[0x2b9069be1289]
/scratch/2376728/7087c18d-ba4a-41e0-9ea4-8c0e356e941d/InteractParser[0x43531a]
/scratch/2376728/7087c18d-ba4a-41e0-9ea4-8c0e356e941d/InteractParser[0x4344ef]
/scratch/2376728/7087c18d-ba4a-41e0-9ea4-8c0e356e941d/InteractParser[0x41477e]
/scratch/2376728/7087c18d-ba4a-41e0-9ea4-8c0e356e941d/InteractParser[0x432f52]
/scratch/2376728/7087c18d-ba4a-41e0-9ea4-8c0e356e941d/InteractParser[0x40d96a]
/scratch/2376728/7087c18d-ba4a-41e0-9ea4-8c0e356e941d/InteractParser[0x40cdd3]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x2b906a3c4555]
/scratch/2376728/7087c18d-ba4a-41e0-9ea4-8c0e356e941d/InteractParser[0x40bea2]
======= Memory map: ========
00400000-00778000 r-xp 00000000 fd:04 134320218                          /scratch/2376728/7087c18d-ba4a-41e0-9ea4-8c0e356e941d/InteractParser
00977000-009b0000 r--p 00377000 fd:04 134320218                          /scratch/2376728/7087c18d-ba4a-41e0-9ea4-8c0e356e941d/InteractParser
009b0000-00a0d000 rw-p 003b0000 fd:04 134320218                          /scratch/2376728/7087c18d-ba4a-41e0-9ea4-8c0e356e941d/InteractParser
00a0d000-00a13000 rw-p 00000000 00:00 0 
01c16000-01c37000 rw-p 00000000 00:00 0                                  [heap]
2b906952c000-2b906954e000 r-xp 00000000 fd:00 12583550                   /usr/lib64/ld-2.17.so
2b906954e000-2b9069550000 rw-p 00000000 00:00 0 
2b9069570000-2b9069571000 rw-p 00000000 00:00 0 
2b9069571000-2b906966c000 rw-p 00000000 00:00 0 
2b906974d000-2b906974e000 r--p 00021000 fd:00 12583550                   /usr/lib64/ld-2.17.so
2b906974e000-2b906974f000 rw-p 00022000 fd:00 12583550                   /usr/lib64/ld-2.17.so
2b906974f000-2b9069750000 rw-p 00000000 00:00 0 
2b9069750000-2b9069765000 r-xp 00000000 fd:00 12584009                   /usr/lib64/libz.so.1.2.7
2b9069765000-2b9069964000 ---p 00015000 fd:00 12584009                   /usr/lib64/libz.so.1.2.7
2b9069964000-2b9069965000 r--p 00014000 fd:00 12584009                   /usr/lib64/libz.so.1.2.7
2b9069965000-2b9069966000 rw-p 00015000 fd:00 12584009                   /usr/lib64/libz.so.1.2.7
2b9069966000-2b906997d000 r-xp 00000000 fd:00 12692285                   /usr/lib64/libpthread-2.17.so
2b906997d000-2b9069b7c000 ---p 00017000 fd:00 12692285                   /usr/lib64/libpthread-2.17.so
2b9069b7c000-2b9069b7d000 r--p 00016000 fd:00 12692285                   /usr/lib64/libpthread-2.17.so
2b9069b7d000-2b9069b7e000 rw-p 00017000 fd:00 12692285                   /usr/lib64/libpthread-2.17.so
2b9069b7e000-2b9069b82000 rw-p 00000000 00:00 0 
2b9069b82000-2b9069c6b000 r-xp 00000000 fd:00 12763778                   /usr/lib64/libstdc++.so.6.0.19
2b9069c6b000-2b9069e6b000 ---p 000e9000 fd:00 12763778                   /usr/lib64/libstdc++.so.6.0.19
2b9069e6b000-2b9069e73000 r--p 000e9000 fd:00 12763778                   /usr/lib64/libstdc++.so.6.0.19
2b9069e73000-2b9069e75000 rw-p 000f1000 fd:00 12763778                   /usr/lib64/libstdc++.so.6.0.19
2b9069e75000-2b9069e8a000 rw-p 00000000 00:00 0 
2b9069e8a000-2b9069f8b000 r-xp 00000000 fd:00 12584690                   /usr/lib64/libm-2.17.so
2b9069f8b000-2b906a18a000 ---p 00101000 fd:00 12584690                   /usr/lib64/libm-2.17.so
2b906a18a000-2b906a18b000 r--p 00100000 fd:00 12584690                   /usr/lib64/libm-2.17.so
2b906a18b000-2b906a18c000 rw-p 00101000 fd:00 12584690                   /usr/lib64/libm-2.17.so
2b906a18c000-2b906a1a1000 r-xp 00000000 fd:00 12676549                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
2b906a1a1000-2b906a3a0000 ---p 00015000 fd:00 12676549                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
2b906a3a0000-2b906a3a1000 r--p 00014000 fd:00 12676549                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
2b906a3a1000-2b906a3a2000 rw-p 00015000 fd:00 12676549                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
2b906a3a2000-2b906a566000 r-xp 00000000 fd:00 12583955                   /usr/lib64/libc-2.17.so
2b906a566000-2b906a765000 ---p 001c4000 fd:00 12583955                   /usr/lib64/libc-2.17.so
2b906a765000-2b906a769000 r--p 001c3000 fd:00 12583955                   /usr/lib64/libc-2.17.so
2b906a769000-2b906a76b000 rw-p 001c7000 fd:00 12583955                   /usr/lib64/libc-2.17.so
2b906a76b000-2b906a770000 rw-p 00000000 00:00 0 
2b906a865000-2b906a866000 ---p 00000000 00:00 0 
2b906a866000-2b906aa66000 rw-p 00000000 00:00 0 
2b906c000000-2b906c021000 rw-p 00000000 00:00 0 
2b906c021000-2b9070000000 ---p 00000000 00:00 0 
7fff7fe83000-7fff7fea5000 rw-p 00000000 00:00 0                          [stack]
7fff7fffe000-7fff80000000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

And, another one in a third run:

************MASS CALIBRATION AND PARAMETER OPTIMIZATION DONE IN 17.358 MIN*********

************************************MAIN SEARCH************************************
Checking database...
Unknown parameters:
	remainder_fragment_masses = 
Parameters:
num_threads = 47
[.........]
Operating on slice 1 of 1: 
	Fragment index slice generated in 13.24 s
	001. 01CPTAC_LSCC_W_BI_20190615_KL_f01.mzBIN_calibrated 1.1 s
		[progress: 41318/41318 (100%) - 17949 spectra/s] 2.3s | postprocessing 4.6 s
	002. 01CPTAC_LSCC_W_BI_20190615_KL_f02.mzBIN_calibrated 1.2 s
		[progress: 42338/42338 (100%) - 17597 spectra/s] 2.4s | postprocessing 5.1 s
	003. 01CPTAC_LSCC_W_BI_20190615_KL_f03.mzBIN_calibrated 5.1 s
		[progress: 44366/44366 (100%) - 17044 spectra/s] 2.6s | postprocessing 6.5 s
	004. 01CPTAC_LSCC_W_BI_20190615_KL_f04.mzBIN_calibrated 1.1 s
		[progress: 44307/44307 (100%) - 18438 spectra/s] 2.4s | postprocessing 5.0 s
	005. 01CPTAC_LSCC_W_BI_20190615_KL_f05.mzBIN_calibrated 1.3 s
		[progress: 42390/42390 (100%) - 18414 spectra/s] 2.3s | postprocessing 3.0 s
	006. 01CPTAC_LSCC_W_BI_20190615_KL_f06.mzBIN_calibrated 1.3 s
		[progress: 43436/43436 (100%) - 16693 spectra/s] 2.6s | postprocessing 6.4 s
	007. 01CPTAC_LSCC_W_BI_20190615_KL_f07.mzBIN_calibrated 1.5 s
		[progress: 41185/41185 (100%) - 17891 spectra/s] 2.3s | postprocessing 6.7 s
	008. 01CPTAC_LSCC_W_BI_20190615_KL_f08.mzBIN_calibrated 1.4 s
		[progress: 40605/40605 (100%) - 16229 spectra/s] 2.5s | postprocessing 4.9 s
	009. 01CPTAC_LSCC_W_BI_20190615_KL_f09.mzBIN_calibrated 1.3 s
		[progress: 41554/41554 (100%) - 16608 spectra/s] 2.5s | postprocessing 4.1 s
	010. 01CPTAC_LSCC_W_BI_20190615_KL_f10.mzBIN_calibrated 1.5 s
		[progress: 42250/42250 (100%) - 16880 spectra/s] 2.5s | postprocessing 5.5 s
	011. 01CPTAC_LSCC_W_BI_20190615_KL_f11.mzBIN_calibrated 1.5 s
		[progress: 40406/40406 (100%) - 16815 spectra/s] 2.4s | postprocessing 3.2 s
	012. 01CPTAC_LSCC_W_BI_20190615_KL_f12.mzBIN_calibrated 1.2 s
		[progress: 40386/40386 (100%) - 9035 spectra/s] 4.5s | postprocessing 4.8 s
	013. 01CPTAC_LSCC_W_BI_20190615_KL_f13.mzBIN_calibrated 1.4 s
		[progress: 39774/39774 (100%) - 17278 spectra/s] 2.3s | postprocessing 4.7 s
	014. 01CPTAC_LSCC_W_BI_20190615_KL_f14.mzBIN_calibrated 2.1 s
		[progress: 40440/40440 (100%) - 17567 spectra/s] 2.3s | postprocessing 4.1 s
	015. 01CPTAC_LSCC_W_BI_20190615_KL_f15.mzBIN_calibrated 1.7 s
		[progress: 41380/41380 (100%) - 17227 spectra/s] 2.4s | postprocessing 3.9 s
	016. 01CPTAC_LSCC_W_BI_20190615_KL_f16.mzBIN_calibrated 1.3 s
		[progress: 40969/40969 (100%) - 16375 spectra/s] 2.5s | postprocessing 5.5 s
	017. 01CPTAC_LSCC_W_BI_20190615_KL_f17.mzBIN_calibrated 1.5 s
		[progress: 40511/40511 (100%) - 16191 spectra/s] 2.5s | postprocessing 3.0 s
	018. 01CPTAC_LSCC_W_BI_20190615_KL_f18.mzBIN_calibrated 1.0 s
		[progress: 39093/39093 (100%) - 16975 spectra/s] 2.3s | postprocessing 3.5 s
	019. 01CPTAC_LSCC_W_BI_20190615_KL_f19.mzBIN_calibrated 1.2 s
		[progress: 40574/40574 (100%) - 16210 spectra/s] 2.5sjava.io.FileNotFoundException: /hpc/shared/xxx/gkok/MS/mzml/01CPTAC_LSCC_W_BI_20190615_KL/01CPTAC_LSCC_W_BI_20190615_KL_f20.mzBIN_calibrated (No such file or directory)
	at java.io.RandomAccessFile.open0(Native Method)
	at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
	at umich.ms.fileio.filetypes.mzbin.MZBINFile.readMZBINStructure(MZBINFile.java:441)
	at umich.ms.fileio.filetypes.mzbin.MZBINFile.readMZBIN(MZBINFile.java:390)
	at umich.ms.fileio.filetypes.mzbin.MZBINFile.<init>(MZBINFile.java:62)
	at l.b(Unknown Source)
	at l.a(Unknown Source)
	at k.<init>(Unknown Source)
	at edu.umich.andykong.msfragger.MSFragger.b(Unknown Source)
	at edu.umich.andykong.msfragger.MSFragger.main(Unknown Source)
 | postprocessing 2.8 s
	020. 01CPTAC_LSCC_W_BI_20190615_KL_f20.mzBIN_calibrated Process 'MSFragger' finished, exit code: 1
Process returned non-zero exit code, stopping

~~~~~~~~~~~~~~~~~~~~
Cancelling 1316 remaining tasks

Update: reading the errors more carefully got me thinking, could this be because I was running FragPipe simultaneously in multiple instances on different HPC nodes? Looking at the files above, it seems to be writing some of the temporary files in the mzml directories. If different runs create and delete the same temporary files, this naturally would cause problems. Using the $TMPDIR, all three instances would have different tmp-directories (/scratch/unique-number), but that doesn't help if it's writing in the mzml source folder.

@GKok GKok reopened this Apr 13, 2022
@guoci
Copy link
Member

guoci commented Apr 14, 2022

@GKok can you post the full log?

@GKok GKok closed this as completed Apr 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants