- galaxy-dev - lists.galaxyproject.org

Uploading large files as Admin Still slow!
by Jackie Lighten 28 Jan '13

28 Jan '13

Hi, I am trying to upload 2x 2.5GB files to my data libraries (in Admin on a local instance) but it has so far been running over night and not completed. How can I speed this up? Can I upload compressed files? Can I not just copy these files directly in to $galaxy-dist/database/files/000 ? Thanks, Jack

2 3

Selecting Intermediary Files in a Workflow
by Kolby Chien 28 Jan '13

28 Jan '13

Hello, I've been working with a local installation of Galaxy and my question is about how Galaxy chooses which file to use in the next step of the workflow. I have created a custom file type called "cub" which is required for my custom tools and I have added this custom file type into the extension list. When I run the first step of my workflow, Galaxy will create an intermediate cub file called dataset_51.dat.cub containing my processed data, but also an empty intermediate dat file as well of the same name sans cub extension (dataset_51.dat) . The next tool in the workflow requires a cub file type input, but instead of using the cub file, it chooses to process the empty .dat file instead, causing the tool to process incorrectly. I've looked over some settings and double checked that my tool xml files specify cub as the output file type and as the input file type for the next tool, but now I am at a loss. Is there anyway to specify to Galaxy that the cub file should be used in the next step of the workflow instead of the dat file? Thank you very much for your help, Kolby

3 2

lwr job runner error.
by Luobin Yang 28 Jan '13

28 Jan '13

Hi, I checked out a refresh copy of latest galaxy-dist, but I found the following error in the log file after I ran Galaxy: galaxy.jobs.handler ERROR 2013-01-27 10:10:19,168 Job runner is not loadable: galaxy.jobs.runners.lwr Traceback (most recent call last): File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/handler.py", line 447, in _load_plugin module = __import__( module_name ) File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/lwr.py", line 232 worker = threading.Thread( ( name="LwrJobRunner.thread-%d" % i ), target=self.run_next ) ^ SyntaxError: invalid syntax Luobin

2 1

galaxy.json?
by Luobin Yang 27 Jan '13

27 Jan '13

Hi, When I run a job in my locally installed Galaxy, there is always an error message showing in the log file: Error opening galaxy.json file: [Errno 2] No such file or directory: '/home/galaxy/galaxy-dist/database/job_working_directory/005/5797/galaxy.json Any clue how to get rid of it? Thanks, Luobin

2 1

ce10
by Alfonso Garrido-Lecca 26 Jan '13

26 Jan '13

Hello, Could somebody please help me uploading the C. elegans genome (Ce10) into galaxy on the cloud? thanks alfonso

2 1

Tool metadata of Flexbar
by Johannes Röhr 25 Jan '13

25 Jan '13

Recently, I updated the Flexbar tool definition in the Galaxy Tool Shed repository flexbar to work with a new version of the program. I changed the version of the tool definition from 2.3 to 2.31, adjusted requirements and uploaded files. However, the Flexbar tool in the repo still shows the old tool version and requirements, whereas the tool command seems to be updated. Furthermore, the change log shows that new repository metadata is associated with the latest change set, but bizarrely the initial metadata is now associated with commit 11 instead of 12, which was the last bugfix commit before the update (commit 13). The update took place more than a week ago. Any suggestions why that is the case? Thanks!

2 2

Blank history panel / Error in history API at listing contents
by Peter Cock 25 Jan '13

25 Jan '13

Hello all, I've noticed on my test setup (tracking galaxy-central) a problem with some histories not displaying. If I click on saved histories, I am show a list. Some will load (slower than I recall), but others result in a blank right hand pane - with the following error from run.sh galaxy.webapps.galaxy.api.history_contents ERROR 2013-01-16 15:24:22,726 Error in history API at listing contents: <type 'exceptions.TypeError'>, cannot concatenate 'str' and 'NoneType' objects The same error and blank pane happens if I click on the reload history icon. A screenshot is attached. This error is coming from HistoryContentsController in file lib/galaxy/webapps/galaxy/api/history_contents.py - specifically in the call to get_hda_dict. I tried modifying this for loop so that any individual transaction with an error is ignored (but a debug message logged), and I could then load the rest of the problem histories. The error from get_hda_dict is from the first line, which takes the galaxy.model.HistoryDatasetAssociation object and asks: hda_dict = hda.get_api_value( view='element' ) That takes us to lib/galaxy/model/__init__.py and the root of the problem: I am seeing entries where both name and val are None, meaning this line fails with the TypeError above: rval['metadata_' + name] = val Clearly some of my histories now have some 'bad' data in them, and currently Galaxy is failing to cope. Based on the ID numbers this has been happening for a while, so not a recent regression. Here is a patch which seems to help. Peter $ hg diff lib/galaxy/model/__init__.py diff -r 81c3b8a6a621 lib/galaxy/model/__init__.py --- a/lib/galaxy/model/__init__.py Wed Jan 16 15:14:04 2013 +0000 +++ b/lib/galaxy/model/__init__.py Wed Jan 16 16:26:28 2013 +0000 @@ -1530,7 +1530,11 @@ val = val.file_name elif isinstance( val, list ): val = ', '.join( [str(v) for v in val] ) - rval['metadata_' + name] = val + if name is None: + log.debug("get_api_value(view=%r) : name %r, val %r" % (view, name, val)) + assert val is None, "get_api_value(view=%r) : name %r, val %r" % (view, name, val) + else: + rval['metadata_' + name] = val return rval class HistoryDatasetAssociationDisplayAtAuthorization( object ):

2 7

Galaxy is hiring Software Engineers and Post-Docs
by Dave Clements 25 Jan '13

25 Jan '13

Hello all, The Galaxy Project is once again expanding and we are looking for highly-motivated, seeking-a-challenge, computationally-savvy, biologically-aware software engineers and post-docs to work on the project. See http://bit.ly/gxyhiring for details. Really, it's a great project with a great community. If you or someone you know would be interested in working with this thriving community on a rapidly evolving high performance platform that is used by thousands of researchers around the world every day, then please consider joining the team. As always, please let me, or any of us ( http://wiki.galaxyproject.org/GalaxyTeam) know if you have any questions. Thanks, Dave Clements Galaxy Training and Outreach -- <http://galaxyproject.org/wiki/GCC2012>http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ http://wiki.galaxyproject.org/

1 0

Galaxy with Univa Grid Engine (UGE) instead of SGE?
by Peter Cock 25 Jan '13

25 Jan '13

Hello all, Our local Galaxy server had been running happily under SGE, using one of the last free releases (not sure exactly which - I could ask). Due to concerns about long term maintenance, the SysAdmin has moved us to an SGE compatible setup - Univa Grid Engine (UGE). However, in at least one respect this is not a drop in replacement, while other cluster usage appears to be working fine our Galaxy installation is not, e.g. galaxy.jobs.runners.drmaa DEBUG 2013-01-15 17:14:33,660 (331:842) submitting file /mnt/galaxy/galaxy-central/database/pbs/galaxy_331:842.sh galaxy.jobs.runners.drmaa DEBUG 2013-01-15 17:14:33,661 (331:842) command is: /mnt/galaxy/galaxy-central/extract_dataset_parts.sh /mnt/galaxy/galaxy-central/database/job_working_directory/000/331/task_0; blastp -query "/mnt/galaxy/galaxy-central/database/job_working_directory/000/331/task_0/dataset_344.dat" -db "/var/local/blast/ncbi/nr" -task blastp -evalue 0.001 -out /mnt/galaxy/galaxy-central/database/job_working_directory/000/331/task_0/dataset_373.dat -outfmt 5 -num_threads 8 galaxy.jobs.runners.drmaa ERROR 2013-01-15 17:14:33,666 Uncaught exception queueing job Traceback (most recent call last): File "/mnt/galaxy/galaxy-central/lib/galaxy/jobs/runners/drmaa.py", line 146, in run_next self.queue_job( obj ) File "/mnt/galaxy/galaxy-central/lib/galaxy/jobs/runners/drmaa.py", line 234, in queue_job job_id = self.ds.runJob(jt) File "/mnt/galaxy/galaxy-central/eggs/drmaa-0.4b3-py2.6.egg/drmaa/__init__.py", line 331, in runJob _h.c(_w.drmaa_run_job, jid, _ct.sizeof(jid), jobTemplate) File "/mnt/galaxy/galaxy-central/eggs/drmaa-0.4b3-py2.6.egg/drmaa/helpers.py", line 213, in c return f(*(args + (error_buffer, sizeof(error_buffer)))) File "/mnt/galaxy/galaxy-central/eggs/drmaa-0.4b3-py2.6.egg/drmaa/errors.py", line 90, in error_check raise _ERRORS[code-1]("code %s: %s" % (code, error_buffer.value)) DeniedByDrmException: code 17: error: no suitable queues Debugging this by attempting a manual submission, $ qsub /mnt/galaxy/galaxy-central/database/pbs/galaxy_331:842.sh Unable to run job: Colon (':') not allowed in objectname. Exiting. Renaming the file to replace the colon with (say) an underscore allows a manual qsub to work fine with UGE. I've edited Galaxy to avoid the colons (patch below) but the submission still fails. Additionally removing the SGE specific settings in universe_wsgi.ini did allow the job to be submitted I am still having problems. Perhaps I need to fix all the other filenames too (e.g. stdout, stderr, error code), or do that in one go by removing the colon in the job name? Has anyone else tried Galaxy under UGE, and do you have any advice? Thanks, Peter -- Quick filename hack to avoid colons in job script filenames - might be better to avoid this in the job name itself? $ hg diff diff -r 1bfe2768026a lib/galaxy/jobs/runners/drmaa.py --- a/lib/galaxy/jobs/runners/drmaa.py Mon Jan 14 17:21:25 2013 +0000 +++ b/lib/galaxy/jobs/runners/drmaa.py Tue Jan 15 18:44:31 2013 +0000 @@ -191,7 +191,7 @@ job_name = ''.join( map( lambda x: x if x in ( string.letters + string.digits + '_' ) else '_', job_name ) ) jt = self.ds.createJobTemplate() - jt.remoteCommand = "%s/galaxy_%s.sh" % (self.app.config.cluster_files_directory, job_wrapper.get_id_tag()) + jt.remoteCommand = ("%s/galaxy_%s.sh" % (self.app.config.cluster_files_directory, job_wrapper.get_id_tag())).replace(":", "_") jt.jobName = job_name jt.outputPath = ":%s" % ofile jt.errorPath = ":%s" % efile @@ -229,6 +229,7 @@ log.debug("(%s) submitting file %s" % ( galaxy_id_tag, jt.remoteCommand ) ) log.debug("(%s) command is: %s" % ( galaxy_id_tag, command_line ) ) + log.debug("(%s) spec: %s" % ( galaxy_id_tag, native_spec)) # runJob will raise if there's a submit problem if self.external_runJob_script is None: job_id = self.ds.runJob(jt) @@ -423,7 +424,7 @@ drm_job_state.ofile = "%s.drmout" % os.path.join(os.getcwd(), job_wrapper.working_directory, job_wrapper.get_id_tag()) drm_job_state.efile = "%s.drmerr" % os.path.join(os.getcwd(), job_wrapper.working_directory, job_wrapper.get_id_tag()) drm_job_state.ecfile = "%s.drmec" % os.path.join(os.getcwd(), job_wrapper.working_directory, job_wrapper.get_id_tag()) - drm_job_state.job_file = "%s/galaxy_%s.sh" % (self.app.config.cluster_files_directory, job.get_id()) + drm_job_state.job_file = ("%s/galaxy_%s.sh" % (self.app.config.cluster_files_directory, job.get_id())).replace(":", "_") drm_job_state.job_id = str( job_id ) drm_job_state.runner_url = job_wrapper.get_job_runner_url() job_wrapper.command_line = job.get_command_line()

3 7

Error opening galaxy.json file
by Peter Cock 25 Jan '13

25 Jan '13

Hi all, Could someone explain to me what the "Error opening galaxy.json file" logging means (set at DEBUG level so perhaps not really an error). I've seen this when my development Galaxy is running normally (here I have the logging set to show the DEBUG output) but also for the Galaxy unit tests. e.g. galaxy.jobs.manager DEBUG 2013-01-25 10:37:45,645 (2) Job assigned to handler 'main' galaxy.jobs DEBUG 2013-01-25 10:37:50,899 (2) Working directory for job is: /mnt/galaxy/galaxy-central/database/job_working_directory/000/2 galaxy.jobs.handler DEBUG 2013-01-25 10:37:50,900 dispatching job 2 to local runner galaxy.jobs.handler INFO 2013-01-25 10:37:51,175 (2) Job dispatched galaxy.jobs.runners.local DEBUG 2013-01-25 10:37:51,286 Local runner: starting job 2 galaxy.jobs.runners.local DEBUG 2013-01-25 10:37:51,820 executing: python /mnt/galaxy/galaxy-central/tools/protein_analysis/tmhmm2.py "$NSLOTS" /tmp/tmp6hm9Td/database/files/000/dataset_1.dat /tmp/tmp6hm9Td/database/files/000/dataset_2.dat galaxy.jobs.runners.local DEBUG 2013-01-25 10:38:11,967 execution finished: python /mnt/galaxy/galaxy-central/tools/protein_analysis/tmhmm2.py "$NSLOTS" /tmp/tmp6hm9Td/database/files/000/dataset_1.dat /tmp/tmp6hm9Td/database/files/000/dataset_2.dat galaxy.tools DEBUG 2013-01-25 10:38:12,265 Error opening galaxy.json file: [Errno 2] No such file or directory: '/mnt/galaxy/galaxy-central/database/job_working_directory/000/2/galaxy.json' galaxy.jobs DEBUG 2013-01-25 10:38:12,411 job 2 ended Thanks, Peter

2 2