Repository navigation
Closed
Labels
bugSomething isn't workingSomething isn't working
Description
Description of the bug
This command is failing:
#!/usr/bin/env bash
set -e # Exit if a tool returns a non-zero status/exit code
set -u # Treat unset variables and parameters as an error
set -o pipefail # Returns the status of the last command to exit with a non-zero status or zero if all successfully execute
set -C # No clobber - prevent output redirection from overwriting files.
quantmsutilsc ms2rescore \
--psm_file f07531_Prot_02_R2_F08_msgf.idXML \
--spectrum_path . \
--ms2_tolerance 0.4 \
--output_path f07531_Prot_02_R2_F08_msgf_ms2rescore.idXML \
--ms2pip_model_dir null \
--processes 372 \
--id_decoy_pattern ^DECOY_ \
--ms2pip_model TMT --rescoring_engine percolator --calibration_set_size 0.15 --test_fdr 0.05 --feature_generators deeplc,ms2pip \
2>&1 | tee f07531_Prot_02_R2_F08_ms2rescore.log
cat <<-END_VERSIONS > versions.yml
"NFCORE_QUANTMS:QUANTMS:TMT:ID:PSMRESCORING:MS2RESCORE":
MS2Rescore: $(echo $(ms2rescore --version 2>&1) | grep -oP 'MS²Rescore \(v\K[^\)]+' )
END_VERSIONS
Error here:
Error: 2024-11-25 15:33:11,160 [<module>] - Creating converter from 7 to 5
2024-11-25 15:33:11,160 [<module>] - Creating converter from 5 to 7
2024-11-25 15:33:11,160 [<module>] - Creating converter from 7 to 5
2024-11-25 15:33:11,160 [<module>] - Creating converter from 5 to 7
2024-11-25 15:33:11,207 [register_filter] - Registered filter: bshuf (/usr/local/lib/python3.10/site-packages/hdf5plugin/plugins/libh5bshuf.so)
2024-11-25 15:33:11,210 [register_filter] - Registered filter: blosc (/usr/local/lib/python3.10/site-packages/hdf5plugin/plugins/libh5blosc.so)
2024-11-25 15:33:11,212 [register_filter] - Registered filter: blosc2 (/usr/local/lib/python3.10/site-packages/hdf5plugin/plugins/libh5blosc2.so)
2024-11-25 15:33:11,213 [register_filter] - Registered filter: bzip2 (/usr/local/lib/python3.10/site-packages/hdf5plugin/plugins/libh5bzip2.so)
2024-11-25 15:33:11,216 [register_filter] - Registered filter: fcidecomp (/usr/local/lib/python3.10/site-packages/hdf5plugin/plugins/libh5fcidecomp.so)
2024-11-25 15:33:11,217 [register_filter] - Registered filter: lz4 (/usr/local/lib/python3.10/site-packages/hdf5plugin/plugins/libh5lz4.so)
2024-11-25 15:33:11,220 [register_filter] - Registered filter: sz (/usr/local/lib/python3.10/site-packages/hdf5plugin/plugins/libh5sz.so)
2024-11-25 15:33:11,232 [register_filter] - Registered filter: sz3 (/usr/local/lib/python3.10/site-packages/hdf5plugin/plugins/libh5sz3.so)
2024-11-25 15:33:11,233 [register_filter] - Registered filter: zfp (/usr/local/lib/python3.10/site-packages/hdf5plugin/plugins/libh5zfp.so)
2024-11-25 15:33:11,234 [register_filter] - init_filter not found for filter zstd: Init phase skipped.
2024-11-25 15:33:11,234 [register_filter] - Registered filter: zstd (/usr/local/lib/python3.10/site-packages/hdf5plugin/plugins/libh5zstd.so)
2024-11-25 15:33:12,843 [<module>] - Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client.
2024-11-25 15:33:15,677 [wrapper] - matplotlib data path: /usr/local/lib/python3.10/site-packages/matplotlib/mpl-data
2024-11-25 15:33:15,683 [wrapper] - CONFIGDIR=/homes/pst_prd/.config/matplotlib
2024-11-25 15:33:15,684 [<module>] - interactive is False
2024-11-25 15:33:15,684 [<module>] - platform is linux
2024-11-25 15:33:15,815 [wrapper] - CACHEDIR=/homes/pst_prd/.cache/matplotlib
2024-11-25 15:33:15,817 [_load_fontmanager] - Using fontManager instance from /homes/pst_prd/.cache/matplotlib/fontlist-v390.json
2024-11-25 15:33:16,706 [parse_cli_arguments_to_config] - Percolator rescoring engine has been specified. Use the idXML containing rescoring features and run Percolator in a separate step.
2024-11-25 15:33:16,706 [ms2rescore] - MS²Rescore config:
2024-11-25 15:33:16,706 [ms2rescore] - {'$schema': './config_schema.json', 'ms2rescore': {'feature_generators': {'ms2pip': {'model_dir': 'null', 'model': 'TMT', 'ms2_tolerance': 0.4}, 'deeplc': {'deeplc_retrain': False, 'calibration_set_size': 0.15}}, 'rescoring_engine': {}, 'co
nfig_file': None, 'psm_file': None, 'psm_file_type': 'infer', 'psm_reader_kwargs': {}, 'spectrum_path': '.', 'output_path': 'f07531_Prot_02_R2_F08_msgf_ms2rescore.idXML', 'log_level': 'info', 'id_decoy_pattern': '^DECOY_', 'psm_id_pattern': '(.*)', 'spectrum_id_pattern': '(.*)',
'lower_score_is_better': True, 'modification_mapping': {}, 'fixed_modifications': {}, 'processes': 48, 'rename_to_usi': False, 'fasta_file': None, 'write_report': True, 'ms2pip_model_dir': 'null', 'ms2pip_model': 'TMT', 'ms2_tolerance': 0.4, 'calibration_set_size': 0.15, 'rng':
4711}}
Warning: OPENMS_DATA_PATH environment variable not found and no share directory was installed. Some functionality might not work as expected.
Traceback (most recent call last):
File "/usr/local/bin/quantmsutilsc", line 10, in <module>
sys.exit(main())
File "/usr/local/lib/python3.10/site-packages/quantmsutils/quantmsutilsc.py", line 38, in main
cli()
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/quantmsutils/rescoring/ms2rescore.py", line 375, in ms2rescore
rescore_idxml(psm_file, output_path, config)
File "/usr/local/lib/python3.10/site-packages/quantmsutils/rescoring/ms2rescore.py", line 123, in rescore_idxml
psm_list = reader.read_file()
File "/usr/local/lib/python3.10/site-packages/psm_utils/io/_base_classes.py", line 46, in read_file
return PSMList(psm_list=[psm for psm in self.__iter__()])
File "/usr/local/lib/python3.10/site-packages/psm_utils/io/_base_classes.py", line 46, in <listcomp>
return PSMList(psm_list=[psm for psm in self.__iter__()])
File "/usr/local/lib/python3.10/site-packages/psm_utils/io/idxml.py", line 112, in __iter__
yield self._parse_psm(self.protein_ids, peptide_id, peptide_hit)
File "/usr/local/lib/python3.10/site-packages/psm_utils/io/idxml.py", line 201, in _parse_psm
rescoring_features={
File "/usr/local/lib/python3.10/site-packages/psm_utils/io/idxml.py", line 202, in <dictcomp>
key: float(peptide_hit.getMetaValue(key)) for key in self.rescoring_features
TypeError: float() argument must be a string or a real number, not 'NoneType'
Command used and terminal output
No response
Relevant files
Dataset here: https://ftp.pride.ebi.ac.uk/pub/databases/pride/resources/proteomes/quantms-benchmark/94508cae253c72810c2dc0200e2c3e/
System information
No response
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Activity
daichengxin commentedon Nov 27, 2024
There is a missing search engines feature that is causing the psm utils parse bug. A example

I don't know why is this the only PSM identification that doesn't have a search engine features? it's probably an internal bug in openms, export search engines features?
ypriverol commentedon Nov 27, 2024
@timosachsenberg @jpfeuffer looks like some of the search engine scores in MSGFAdapter for some of the psms are not written properly in the idXML.
Two things:
jpfeuffer commentedon Nov 27, 2024
Which feature is missing?
daichengxin commentedon Nov 27, 2024
These features

ypriverol commentedon Nov 27, 2024
All search engine scores.
jpfeuffer commentedon Nov 27, 2024
This is right out of msgfplusadapter?
Any logs?
ypriverol commentedon Nov 27, 2024
Here the log from MSGFAdapter.
timosachsenberg commentedon Nov 27, 2024
which OpenMS version is this?
ypriverol commentedon Nov 27, 2024
The nightly version.
timosachsenberg commentedon Nov 27, 2024
if you use the nightly version, then you might want to remove the PSMFeatureExtractor from the workflow.
I now integrate the annotation into MSGFPlusAdapter, Comet etc.
This was the relevant PR:
OpenMS/OpenMS#7323
Maybe try again without PSMFeatureExtractor? Then we can narrow it down (could also be a MSGFPlus error that we "fixed" by calling the PSMFeatureExtractor).
ypriverol commentedon Nov 27, 2024
This can't be the source of the error because, we have other files that pass.
jpfeuffer commentedon Nov 27, 2024
<MS-GF+ PSM with missing NumMatchedMainIons skipped.> occurred 2805 timesThis does not sound good.
timosachsenberg commentedon Nov 27, 2024
Apparently, we handle this here as some known MSGFPlus issue:
https://github.com/OpenMS/OpenMS/blob/develop/src/openms/source/ANALYSIS/ID/PercolatorFeatureSetHelper.cpp#L21-L101
Could also be that we calculated these generic meta values for every search engine result in the past... need to investigate a bi more.
timosachsenberg commentedon Nov 27, 2024
Is it possible to get the mzid file written by MSGF+? I want to directly check if the output by MSGF+
looking at some non OpenMS code: https://github.com/percolator/percolator/blob/master/src/converters/MsgfplusReader.cpp#L310-L333
it indeed seems as if these meta values should be written by MSGF+ to mzid.
15 remaining items
timosachsenberg commentedon Nov 28, 2024
ok let's write them out for now. They can be easily filtered with IDFilter if needed and this let's us evaluate if we loose some good ids. If not I can just skip them
ypriverol commentedon Nov 28, 2024
We will not introduce a new step IDFilter. But I guess when we pass it to
PercolatorAdapterthey will removed.jpfeuffer commentedon Nov 28, 2024
You can just have the filter in the same step. No need for additional steps.
ypriverol commentedon Nov 28, 2024
@jpfeuffer, what do you mean? Do you mean for msgf+ add an extra command rigth after MSGFAdapter to filter out the nonsense psms?
If we decide to implement this solution what means, in the commandline, this is the help page of IDFILTER:
timosachsenberg commentedon Nov 28, 2024
Probably something like:
right @jpfeuffer
ypriverol commentedon Nov 28, 2024
@timosachsenberg @jpfeuffer can you take a look at this PR, we are also validating it in the ms2rescore, just in case in the future something happen similarly: bigbio/quantms-utils#36
timosachsenberg commentedon Nov 28, 2024
I merged the fix for MSGF+ here OpenMS/OpenMS#7713
ypriverol commentedon Nov 28, 2024
@timosachsenberg what will be the parameter for idfilter, if we added it in the msgf process step.
ypriverol commentedon Nov 29, 2024
@timosachsenberg @jpfeuffer I have re-run the dataset with the latest version of MSGF+ 202403.. Here the results:
Current version in OpenMS 2023: https://ftp.pride.ebi.ac.uk/pub/databases/pride/resources/proteomes/quantms-benchmark/94508cae253c72810c2dc0200e2c3e/msgfplus_output-2023.mzid.gz
Latest version MSGF+ 2024: https://ftp.pride.ebi.ac.uk/pub/databases/pride/resources/proteomes/quantms-benchmark/94508cae253c72810c2dc0200e2c3e/msgfplus_output-2024.mzid.gz
Can you double check if for the mzId from the latest version the problems remains?
timosachsenberg commentedon Nov 29, 2024
both have the issue. I am creating the container with the fix on OpenMS side.
Update:
container should be updated now
ypriverol commentedon Nov 29, 2024
Ok, Do you want to report the issue to MSGF+ or should I do it? The conda package is still not available, my point is the deployment of the conda nightly.
timosachsenberg commentedon Nov 29, 2024
yes please report. If you are familiar with the code you might also give it a quick look why these values are not written out
ypriverol commentedon Dec 4, 2024
Here the issue in msgf+ repo: MSGFPlus/msgfplus#157
ypriverol commentedon Dec 4, 2024
I will close this issue because we have solved it "in principle".