- galaxy-dev - lists.galaxyproject.org

Error cleaning up Condor jobs
by Branden Timm 07 May '13

07 May '13

Hi All, I've been working to configure a new Galaxy instance to run jobs under Condor. Things are 99% working at this point, but what seems to be happening is after the Condor job finishes Galaxy tries to clean up a cluster file that isn't there, namely the .ec (exit code) file. Relevant log info: galaxy.jobs DEBUG 2013-05-07 15:02:49,364 (1985) Working directory for job is: /home/GLBRCORG/galaxy/database/job_working_directory/001/1985 galaxy.jobs.handler DEBUG 2013-05-07 15:02:49,387 (1985) Dispatching to condor runner galaxy.jobs DEBUG 2013-05-07 15:02:49,720 (1985) Persisting job destination (destination id: condor) galaxy.jobs.handler INFO 2013-05-07 15:02:49,761 (1985) Job dispatched galaxy.jobs.runners.condor DEBUG 2013-05-07 15:02:56,368 (1985) submitting file /home/GLBRCORG/galaxy/database/condor/galaxy_1985.sh galaxy.jobs.runners.condor DEBUG 2013-05-07 15:02:56,369 (1985) command is: python /home/GLBRCORG/galaxy/galaxy-central/tools/fastq/fastq_to_fasta.py '/home/GLBRCORG/galaxy/database/files/000/dataset_3.dat' '/home/GLBRCORG/galaxy/database/files/002/dataset_2842.dat' ''; cd /home/GLBRCORG/galaxy/galaxy-central; /home/GLBRCORG/galaxy/galaxy-central/set_metadata.sh /home/GLBRCORG/galaxy/database/files /home/GLBRCORG/galaxy/database/job_working_directory/001/1985 . /home/GLBRCORG/galaxy/galaxy-central/universe_wsgi.ini /home/GLBRCORG/galaxy/database/tmp/tmpGe1JZJ /home/GLBRCORG/galaxy/database/job_working_directory/001/1985/galaxy.json /home/GLBRCORG/galaxy/database/job_working_directory/001/1985/metadata_in_HistoryDatasetAssociation_3161_are5Bg,/home/GLBRCORG/galaxy/database/job_working_directory/001/1985/metadata_kwds_HistoryDatasetAssociation_3161_p73Yus,/home/GLBRCORG/galaxy/database/job_working_directory/001/1985/metadata_out_HistoryDatasetAssociation_3161_tLqep6,/home/GLBRCORG/galaxy/database/job_working_directory/001/1985/metadata_results_HistoryDatasetAssociation_3161_3QSW5X,,/home/GLBRCORG/galaxy/database/job_working_directory/001/1985/metadata_override_HistoryDatasetAssociation_3161_JUFvmk galaxy.jobs.runners.condor INFO 2013-05-07 15:02:58,960 (1985) queued as 15 galaxy.jobs DEBUG 2013-05-07 15:02:59,110 (1985) Persisting job destination (destination id: condor) galaxy.jobs.runners.condor DEBUG 2013-05-07 15:02:59,536 (1985/15) job is now running galaxy.jobs.runners.condor DEBUG 2013-05-07 15:07:16,966 (1985/15) job is now running galaxy.jobs.runners.condor DEBUG 2013-05-07 15:07:17,279 (1985/15) job has completed galaxy.jobs.runners DEBUG 2013-05-07 15:07:17,417 (1985/15) Unable to cleanup /home/GLBRCORG/galaxy/database/condor/galaxy_1985.ec: [Errno 2] No such file or directory: '/home/GLBRCORG/galaxy/database/condor/galaxy_1985.ec' galaxy.jobs DEBUG 2013-05-07 15:07:17,560 setting dataset state to ERROR galaxy.jobs DEBUG 2013-05-07 15:07:17,961 job 1985 ended galaxy.datatypes.metadata DEBUG 2013-05-07 15:07:17,961 Cleaning up external metadata files I've done a watch on the condor job directory, and as far as I can tell galaxy_1985.ec never gets created. From a cursory look at lib/galaxy/jobs/runners/__init__.py and condor.py, it looks like the cleanup is happening in the AsynchronousJobState::cleanup method, which iterates on the cleanup_file_attributes list. I naively tried to override cleanup_file_attributes in CondorJobState to disinclude 'exit_code_file', to no avail. I'm hoping somebody can spot where the hiccup is here. Another question that is on my mind is should a failure to cleanup cluster files set the dataset state to ERROR? An inspection of the output file from my job leads me to believe it finished just fine, and indicating failure to the user because Galaxy couldn't cleanup a 1b error code file seems a little extreme to me. Thanks! -- Branden Timm btimm(a)energy.wisc.edu

1 0

Galaxy Keeps Freezing
by Kim Spradling 07 May '13

07 May '13

I have been trying to join the intervals of two datasets side-by-side using the public Main instance, but the job never finishes. I've gotten a message stating that Galaxy is unavailable a couple of times, but I have tried running this job multiple times with no success. I also tried to upload a text file to see if it was an issue with using that specific tool, but it won't complete either. Any help will be greatly appreciated. Thanks, Kim

2 1

Galaxy Jobs Waiting to Run for Several Days
by Elnagheeb, Marwa 07 May '13

07 May '13

Hello, I am a student, and I am using the psu galaxy server for an assignment. My workflow for each of my 5 data sets has been waiting to run 3 jobs (bwa alignment, sam to bam, and bam statistics). One of my datasets reached completion two days ago, but it has been 5 days for the others. Several others in my class have had the same issue. Is there something going on with the server this week? I really need to meet a deadline. Any chance I can get my jobs to run quicker without cloud? Thanks, Marwa

2 1

Strange issue where the dataset reported in the bug report doesn't correspond to the current dataset
by Jean-Francois Payotte 07 May '13

07 May '13

Dear Galaxy mailing-list, We are currently facing a strange issue with our local Galaxy installation (distribution from Jan. 11 2013). I'm going to try to describe the problem as much as I can, and feel free to ask for more details if it can help in solving this issue. So basically, we have a situation where (about half the time), when a job ends in error, the error message (shown when clicking on the bug icon), doesn't seem to be related to the dataset in error. As this particular problem seem to be present only when datasets ends in error, I created a simple python script which simply does 2 things: It outputs the hostname to stdout (in order to investigate if the problem is node-dependant) It outputs the following message "This is an error message printed to stderr", to stderr. (This is done in order to get only error datasets and to investigate the before-mentioned issue.) Here is the code of my python script: import sys import socket def main(): print socket.gethostname() sys.stderr.write('This is an error message printed to stderr\n') if __name__ == '__main__': main() I ran this tool about 50 times and I looked at the error messages displayed both in the preview section of the dataset (in the history), and the error message displayed when clicking on the bug icon "view or report this error". About half the time, the results were as expected: the dataset stdout would read the name of the node where the job was executed the dataset stderr would read: "This is an error message printed to stderr" and the error message displayed when clicking on the bug icon would be: Dataset generation errors Dataset 54: Test error Tool execution generated the following error message: This is an error message printed to stderr The tool produced the following additional output: some.server.name However, the other half of the time, the error message shown in stderr doesn't correspond to the error message displayed when clicking on the bug icon: the dataset stdout is still reading the name of the node the dataset stderr is still reading "This is an error message printed to stderr" but the error message displayed when clicking on the bug icon would be something like: Dataset generation errors Dataset 3: chrM.bed Tool execution did not generate any error messages. We can clearly see a discrepancy between the error message in stderr and the error message of the bug report. Actually, the bug report is saying that there is not any error and it makes reference to some "Dataset 3: chrM.bed", even when the actual dataset is "Dataset 53: Test error". There is absolutely no .bed file in my history and the dataset 3 actually reads "Dataset 3: Test error". Some of the datasets mentioned in the faulty bug reports seems to be really old datasets (like one year old). So my question to you is, could this be related to some mix-up between datasets ID? And how can I look this up. I must say that at the moment, I have absolutely no idea how to solve this issue. Many thanks for your help! Best regards, Jean-François

3 9

conditions within workflows
by Olivia Doppelt 07 May '13

07 May '13

Dear developpers, I have a question concerning the insertion of conditions inside workflows ? Here, at the pasteur Institute, we would like to use galaxy to create a microbiome analysis workflow. However, it requires at least 10 different tools which may be or may not be used in several cases. For the moment, all I see is to create a giant wrapper for this workflow but it would be loosing the great interest of galaxy. I see on bitbucket that there is a branch on galaxy-central named "galaxy-central-conditional-toolbox-elements" are there new elements since february ? Best regards, -- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur

2 1

tool shed installations and grouping in categories
by Björn Grüning 07 May '13

07 May '13

Hello all, during the last weeks the tool shed team did a lot of awesome work and soon it should be possible to install meta-packages like Ira's proteomics pipeline or the ChemicalToolBoX with a few clicks. Installing all nested dependencies, datatypes and so on automatically. One last problem I'm seeing is the grouping of tools in categories. If you install all your tools via a meta package you don't have any choice to group single tools in single categories. The same problem applies when your repo have more than one tool that do not fit in one single category. Before we had the repository dependencies features in Galaxy I shipped example tool_conf.xml files and explained how to modify it manually. These example files contained the tool-path with tools ID and the categories with ID. What do you think about shipping such files in a repository and during installation you offer a third choice to the user (next to, choose category, enter new category name) ... something like "accept included suggestions" and display them. Also for the installation via the API it would make installing a completely new Galaxy flavour easy. Any comments or ideas? Cheers, Björn

2 1

List of genomes
by Lee Katz 06 May '13

06 May '13

Hi, I am wondering what I can do to add more genomes to this list (hopefully this image will make it through the mailing list). I am importing fastq files from a MiSeq run. We are using a modified version of Galaxy called Tron. Everything I see on the wiki is about adding custom reference genome assemblies, but I don't understand what to do about adding a species without the reference assembly. Without the correct species, I am forced to use the option "unspecified (?)" Thanks for your help. [image: Inline image 3] -- Lee Katz, Ph.D.

2 1

Improving the change revision view on the Tool Shed
by Peter Cock 06 May '13

06 May '13

Hello all, I've noticed a couple of weaknesses in the current revision view for the Tool Shed. First, the commit comment is not shown - this would make sense next to the revision number and above the diffs. Second, there is no easy way to jump to the next/prev revision, you must first go back to the "View change log" page. Perhaps simple prev/next links would work well, or as used elsewhere in the Tool Shed a revision drop down control [*]? Regards, Peter [*] Use with caution, I've found this focus issue quite infuriating: https://trello.com/card/-/506338ce32ae458f6d15e4b3/821

2 1

user password different type encoding
by Vipin TS 06 May '13

06 May '13

Hello dev-team, I would like to add the different type of password encryption to the users in my galaxy instance. I started working with the current password encoding script: /home/apps/galaxy-dist/lib/galaxy/util/hash_util.py I will keep the current sha1 and add another layer of encryption to the sha1 hash, otherwise I need to force all my users to change the password and follow the new hashing method. Can anyone please point me any other place/script which I missed regarding the encryption/decryption of user authentication. thanks in advance, --/Vipin

3 9

Usability bug/inconsistency with workflows versus tools
by Peter Cock 06 May '13

06 May '13

I seem to be in feedback mode: another little rough edge in Galaxy: When using a tool in Galaxy, at the end of the parameters there is a nice eye catching blue button labelled "execute". When running a workflow in Galaxy, the equivalent button is plain grey and reads "Run workflow". I think it would be clearer if this was the same blue style, and perhaps also labelled "execute". Regards, Peter

2 1