Hey Jens,
I have tried a few different things and I have been unable to replicate the behavior locally.
Is this tool specific or configuration specific - i.e. do you see this behavior only with a specific tool or do the concatenate datasets tool experience this as well say?
If it is tool specific - you (or the underlying application) may be deleting metadata_results_XXX files in the working directory as part of the job? If you are convinced the tool is not deleting these files but it is a tool-specific problem can you pass along the tool you are using (or better a minimal version of the tool that produces this behavior.)
If it is configuration specific and you can get it to happen with many different tools - can you try to do it against a clean copy of galaxy-dist or galaxy-central and pass along the exact updates (universe_wsgi.ini, job_conf.xml, etc...) that you used to configure Galaxy to cause it to produce this error.
Couple more things - it hasn't gone into the if in the jobs/__init__.py statement - it is doing that first check external_metadata_set_successfully when the error occurs - so I don't think it is a problem with the retry_metadata_internally configuration option.
Additionally, you are using the local job runner so Galaxy will always "retry_metadata_internally" - the local job runner doesn't try to embed the metadata calculation into the job like the cluster job runners (so retry_metadata_internally doesn't really matter with the local job runner... right now anyway). If you don't want the metadata calculation to be embedded into the local job runner job the way cluster job runners do it (so retry_metadata_internally does in fact matter) there was an option added to the local job runner last release that I realized I hadn't documented - you can add <param id="embed_metadata_in_job">True</param> to the local job destination in job_conf.xml.
More information about this new option here: https://bitbucket.org/galaxy/galaxy-central/commits/75c63b579ccdd63e0558dd9a...
-John
On Wed, Jun 25, 2014 at 7:41 AM, Preussner, Jens Jens.Preussner@mpi-bn.mpg.de wrote:
Dear all,
I noticed a strange behaviour in our local galaxy installation. First of all, my universe_wsgi.ini contains “retry_metadata_internally = False” and “cleanup_job = always”. The tool writes its output simply into the job_working_directory and we move it via && mv static_filename.txt $output in the <command>-tag. This works fine.
When restarting the galaxy server and executing the tool after a fresh restart, everything is ok, there are no errors.
When executing the same tool a second time, galaxy brings a “tool error” stating that it was unable to finish the job. Nevertheless, the output files are all correct (but marked as red or failed).
The error report states:
Traceback (most recent call last):
File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/local.py", line 129, in queue_job
job_wrapper.finish( stdout, stderr, exit_code )File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 997, in finish
if ( notself.external_output_metadata.external_metadata_set_successfully( dataset, self.sa_session ) and self.app.config.retry_metadata_internally ):
File "/home/galaxy/galaxy-dist/lib/galaxy/datatypes/metadata.py", line 731, in external_metadata_set_successfully
rval, rstring = json.load( open( metadata_files.filename_results_code ))
IOError: [Errno 2] No such file or directory: u'/home/galaxy/galaxy-dist/database/job_working_directory/000/59/metadata_results_HistoryDatasetAssociation_281_oHFjx0'
And in the logfile you can find multiple entries like that:
galaxy.datatypes.metadata DEBUG 2014-06-25 14:29:35,466 Failed to cleanup external metadata file (filename_results_code) for HistoryDatasetAssociation_281: [Errno 2] No such file or directory: '/home/galaxy/galaxy-dist/database/job_working_directory/000/59/metadata_results_HistoryDatasetAssociation_281_oHFjx0'
The if-statement in /home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 997 should evaluate to False, since self.app.config.retry_metadata_internally is set to False in the universe_wsgi.ini but it seems it doesn’t in this case?
Anyone having experienced such a behavior? Any suggestions how to go on and solve the issue?
Many thanks!
Jens
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/