May 2013 - galaxy-dev - lists.galaxyproject.org

Galaxy lists empty file for composite datatypes
by Peter Cock 17 May '13

17 May '13

Hello all, In replying to someone having trouble with the BLAST+ tools, I found a new bug (and was reminded of an old bug I'd not yet reported) in the handling of composite datatypes without a primary file (e.g. BLAST databases). Consider the screenshot, where the history text for my BLAST database says "empty" (above the line saying "format: blastdbn") and the "full path" (highlighted) shown via the "information" icon points at an empty file, in this case: $ ls /mnt/galaxy/galaxy-dist/database/files/009/dataset_9085.dat -l -rw-r--r-- 1 galaxy galaxy 0 May 17 14:40 /mnt/galaxy/galaxy-dist/database/files/009/dataset_9085.dat The real data is here: $ ls /mnt/galaxy/galaxy-dist/database/files/009/dataset_9085_files/ blastdb.nhd blastdb.nhi blastdb.nhr blastdb.nin blastdb.nog blastdb.nsd blastdb.nsi blastdb.nsq I am guessing the code assumes a composite datatypes like HTML with a primary file, where it makes some sense to measure the size of the primary file (although this omits the child files like images for an HTML file) and use its location on disk. In the case of basic composite datatypes like BLAST databases, the "full path" should be the folder name, and the size should count all the child files within that folder (which is NOT empty). Looking at the code, $ grep "Full Path:" */* templates/show_params.mako: <tr><td>Full Path:</td><td>${hda.file_name | h}</td></tr> This suggests that the hda.file_name attribute might be assigned the folder name in this case? Or might that have repercussions? Thanks, Peter

1 0

Galaxy group at Google+
by Samuel Lampa 17 May '13

17 May '13

Hi, I realized there was no Galaxy group on G+, although I find G+ is a really nice medium for some kind of posts, especially for things like showing off some work you have done, without disturbing others on the mailing lists. Feel free to chime in: http://gplus.to/galaxyportal Cheers, // Samuel -- Developer at SNIC-UPPMAX www.uppmax.uu.se Developer at Dept of Pharm Biosciences www.farmbio.uu.se

1 0

Test Tool Shed uploads broken
by Peter Cock 17 May '13

17 May '13

Hi Greg & Dave, I just tried to upload an update to my Blast2GO wrapper on the Test Tool Shed, http://testtoolshed.g2.bx.psu.edu/view/peterjc/blast2go This gave the following error, also note typo sucessfully [sic] -> successfully. Internal Server Error Galaxy was unable to sucessfully complete your request URL: http://testtoolshed.g2.bx.psu.edu/upload/upload?repository_id=e4de88c47079d… Module galaxy.web.framework.middleware.error:149 in __call__ >> app_iter = self.application(environ, sr_checker) Module paste.debug.prints:106 in __call__ >> environ, self.app) Module paste.wsgilib:543 in intercept_output >> app_iter = application(environ, replacement_start_response) Module paste.recursive:84 in __call__ >> return self.application(environ, start_response) Module paste.httpexceptions:633 in __call__ >> return self.application(environ, start_response) Module galaxy.web.framework.base:132 in __call__ >> return self.handle_request( environ, start_response ) Module galaxy.web.framework.base:190 in handle_request >> body = method( trans, **kwargs ) Module galaxy.web.framework:98 in decorator >> return func( self, trans, *args, **kwargs ) Module galaxy.webapps.tool_shed.controllers.upload:122 in upload >> self.upload_tar( trans, repository, tar, uploaded_file, upload_point, remove_repo_files_not_in_tar, commit_message, new_repo_alert ) Module galaxy.webapps.tool_shed.controllers.upload:326 in upload_tar >> undesirable_files_removed ) Module tool_shed.util.commit_util:116 in handle_directory_changes >> for undesirable_dir in undesirable_dirs: NameError: global name 'undesirable_dirs' is not defined Peter

2 2

Add a download process to a job handler in history queue
by Zinonas Antoniou 17 May '13

17 May '13

Hello, I developed a script to download file to Galaxy from other platform. If the size of the file is large, the process may take several minutes and I can't to anything else in Galaxy. Is there a way to add the download process in a job handler and add it to history queue? Any help is more than welcome! Thank you in advance!

1 0

Workflow: multiple input errors
by Joachim Jacob | VIB | 17 May '13

17 May '13

Hi all, I have fired up an cloud instance of Galaxy, 2 persistent nodes, scalable up to 10. I have updated the code to the latest stable, after issues with not being able to select multiple input for workflows (more background: http://dev.list.galaxyproject.org/selecting-multiple-inputs-for-workflows-n…) Running a workflow on multiple inputs (#=58) gives me - after a very long page loading - this error: ************************************************ Internal Server Error Galaxy was unable to sucessfully complete your request URL: http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b… Module galaxy.web.framework.middleware.error:*149* in |__call__| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b…>app_iter *=* self*.*application*(*environ*,* sr_checker*)*| Module paste.recursive:*84* in |__call__| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b…>*return* self*.*application*(*environ*,* start_response*)*| Module paste.httpexceptions:*633* in |__call__| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b…>*return* self*.*application*(*environ*,* start_response*)*| Module galaxy.web.framework.base:*128* in |__call__| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b…>*return* self*.*handle_request*(* environ*,* start_response *)*| Module galaxy.web.framework.base:*184* in |handle_request| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b…>body *=* method*(* trans*,* ****kwargs *)*| Module galaxy.webapps.galaxy.controllers.workflow:*1443* in |run| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b…>job*,* out_data *=* tool*.*execute*(* trans*,* step*.*state*.*inputs*,* history*=*target_history*)*| Module galaxy.tools:*2342* in |execute| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b…>*return* self*.*tool_action*.*execute*(* self*,* trans*,* incoming*=*incoming*,* set_output_hid*=*set_output_hid*,* history*=*history*,* ****kwargs *)*| Module galaxy.tools.actions:*397* in |execute| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b…>job*.*add_input_dataset*(* name*,* dataset *)*| Module galaxy.model:*247* in |add_input_dataset| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b…>self*.*input_datasets*.*append*(* JobToInputDatasetAssociation*(* name*,* dataset *)* *)*| Module ?:*4* in |__init__| Module sqlalchemy.orm.state:*82* in |initialize_instance| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b…>*return* manager*.*events*.*original_init*(****mixed*[**1**:**]**,* ****kwargs*)*| Module galaxy.model:*450* in |__init__| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b…>self*.*dataset *=* dataset| Module sqlalchemy.orm.attributes:*150* in |__set__| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b…>self*.*impl*.*set*(*instance_state*(*instance*)**,* instance_dict*(*instance*)**,* value*,* None*)*| Module sqlalchemy.orm.attributes:*590* in |set| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b…>value *=* self*.*fire_replace_event*(*state*,* dict_*,* value*,* old*,* initiator*)*| Module sqlalchemy.orm.attributes:*610* in |fire_replace_event| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b…>value *=* ext*.*set*(*state*,* value*,* previous*,* initiator *or* self*)*| Module sqlalchemy.orm.unitofwork:*69* in |set| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b…>sess*.*add*(*newvalue*)*| Module sqlalchemy.orm.session:*1091* in |add| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b…>self*.*_save_or_update_state*(*state*)*| Module sqlalchemy.orm.session:*1100* in |_save_or_update_state| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b…>self*.*_save_or_update_impl*(*state*)*| Module sqlalchemy.orm.session:*1267* in |_save_or_update_impl| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b…>self*.*_update_impl*(*state*)*| Module sqlalchemy.orm.session:*1259* in |_update_impl| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b…>self*.*_attach*(*state*)*| Module sqlalchemy.orm.session:*1286* in |_attach| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b…>*(*mapperutil*.*state_str*(*state*)**,* state*.*key*)*| *InvalidRequestError: Can't attach instance <HistoryDatasetAssociation at 0x8591c50>; another instance with key (<class 'galaxy.model.HistoryDatasetAssociation'>, (200,)) is already present in this session.* extra data <http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b…> full traceback <http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b…> text version <http://ec2-54-226-11-122.compute-1.amazonaws.com/workflow/run?id=e967e0737b…> This may be an intermittent problem due to load or other unpredictable factors, reloading the page may address the problem. ************************************************ After clicking history refresh, the workflow is actually in progress! It completes EXCEPT for the last dataset, which stays gray. Clicking on rerun on this last dataset, gives me the error: ************************************************ Internal Server Error Galaxy was unable to sucessfully complete your request URL: http://ec2-54-226-11-122.compute-1.amazonaws.com/tool_runner/rerun?id=091bc… Module galaxy.web.framework.middleware.error:*149* in |__call__| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/tool_runner/rerun?id=091bc…>app_iter *=* self*.*application*(*environ*,* sr_checker*)*| Module paste.recursive:*84* in |__call__| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/tool_runner/rerun?id=091bc…>*return* self*.*application*(*environ*,* start_response*)*| Module paste.httpexceptions:*633* in |__call__| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/tool_runner/rerun?id=091bc…>*return* self*.*application*(*environ*,* start_response*)*| Module galaxy.web.framework.base:*128* in |__call__| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/tool_runner/rerun?id=091bc…>*return* self*.*handle_request*(* environ*,* start_response *)*| Module galaxy.web.framework.base:*184* in |handle_request| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/tool_runner/rerun?id=091bc…>body *=* method*(* trans*,* ****kwargs *)*| Module galaxy.webapps.galaxy.controllers.tool_runner:*142* in |rerun| |>> <http://ec2-54-226-11-122.compute-1.amazonaws.com/tool_runner/rerun?id=091bc…>*raise* Exception*(*"Failed to get job information for dataset hid %d" *%* data*.*hid*)*| *Exception: Failed to get job information for dataset hid 112* extra data <http://ec2-54-226-11-122.compute-1.amazonaws.com/tool_runner/rerun?id=091bc…> full traceback <http://ec2-54-226-11-122.compute-1.amazonaws.com/tool_runner/rerun?id=091bc…> text version <http://ec2-54-226-11-122.compute-1.amazonaws.com/tool_runner/rerun?id=091bc…> This may be an intermittent problem due to load or other unpredictable factors, reloading the page may address the problem. ******************************************************** I have to manually rerun the last job with the corresponding workflow. Thanks, Joachim -- Joachim Jacob Contact details: http://www.bits.vib.be/index.php/about/80-team

1 1

Handling of tool_dependencies.xml errors in Tool Shed testing
by Peter Cock 17 May '13

17 May '13

Hi all, For defining the 'install script' for some of my tool wrappers, I have mainly following http://wiki.galaxyproject.org/ToolShedToolFeatures and looking at one or two examples. I've got some of my tools to work, but not all. For instance, this one appears not to be downloading or moving JAR files correctly: http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/5644c28cf965 <?xml version="1.0"?> <tool_dependency> <package name="effectiveT3" version="1.0.1"> <install version="1.0"> <actions>  <action type="set_environment"> <environment_variable name="EFFECTIVET3" action="set_to">$INSTALL_DIR</environment_variable> </action>  <action type="download_by_url">http://effectors.org/download/version/TTSS_GUI-1.0.1.jar</action> <action type="move_file"><source>TTSS_GUI-1.0.1.jar</source><destination>$INSTALL_DIR/</destination></action>  <action type="make_directory">$INSTALL_DIR/module</action> <action type="download_by_url">http://effectors.org/download/module/TTSS_ANIMAL-1.0.1.jar</action> <action type="move_file"><source>TTSS_ANIMAL-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action> <action type="download_by_url">http://effectors.org/download/module/TTSS_PLANT-1.0.1.jar</action> <action type="move_file"><source>TTSS_PLANT-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action> <action type="download_by_url">http://effectors.org/download/module/TTSS_STD-1.0.1.jar</action> <action type="move_file"><source>TTSS_STD-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action> </actions> </install> <readme> Downloads effectiveT3 v1.0.1 and the three models from http://effectors.org/ </readme> </package> </tool_dependency> Here my Python wrapper script has evidently read the $EFFECTIVET3 environment variable (defined in the tool_dependencies.xml to point to the tool's $INSTALL_DIR), but the JAR file isn't there: Effective T3 JAR file not found: /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmpYR9FXD/effectiveT3/1.0.1/peterjc/effectivet3/5644c28cf965/TTSS_GUI-1.0.1.jar My previous revision set the $EFFECTIVET3 environment variable at the end of the tool_dependencies.xml file, and from the test failure it appeared not to be getting set. My hunch is something breaks part way through the installation but the Tool Shed isn't reporting this - or am I just not looking in the right place? It would make sense to show any installation issues under the "Tool Dependencies" entry on the main tool page. Related to this, I'd like to suggest another couple of assertion action types, used for checking if directories or files exist, which would abort the installation with a clear error if the check fails: <action type="assert_dir_exists">$INSTALL_DIR/module/</action> <action type="assert_file_exists">$INSTALL_DIR/module/TTSS_STD-1.0.1.jar</action> An obvious application of these would be immediately after an <action type="download_by_url"> line to sanity test the download (and decompression) worked as expected - and didn't for example just download an HTML error page for instance. (In this case I am worried that perhaps the download_by_url action has been overly enthusiastic and unzipped the JAR file, perhaps based on the mime-type?). Thanks, Peter

4 19

selecting multiple inputs for workflows not possible anymore
by Joachim Jacob | VIB | 17 May '13

17 May '13

Hi all, I just found out that I cannot select multiple inputs in workflows anymore in our local Galaxy. This option used to work on these workflows in the past. I only get now a dropdown box, without the little icon at the top of the input list to select multiple inputs. I am a little puzzled of what's wrong. Any light in the dark? I have repeated these steps in a freshly started cloud galaxy instance, updated to the latest release, and imported that workflow. In that instance, I can select multiple files: but it is not anymore with the multiple selection, but in a 'dropdown-checkbox-list' (hope this is clear). The problem is here that only one input gets processed, even the input list contains multiple items. Every item that is checked gets into the inputbox. A second issue is that this entry (already in the input list) is not removed from the checkbox-list: so you can select multiple times the same input. (although this might be wanted in some cases?) Thanks, Joachim -- Joachim Jacob Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib

2 6

Bug in history when using Admin account ??
by Neil.Burdett＠csiro.au 17 May '13

17 May '13

Hi, I believe I have found a bug? Or maybe I'm not using Galaxy correctly? These are the steps I used: * Downloaded the latest version of Galaxy * Created a user me(a)me.com<mailto:me@me.com> * Uploaded several files * Logged out * Create a user you(a)you.com<mailto:you@you.com> * Uploaded several files * Logged out and then logged in as an Admin user (specified in universe_wsgi.ini file) * Uploaded a file. Now when I log into me(a)me.com<mailto:me@me.com> or you(a)you.com<mailto:you@you.com> the history shows only the uploaded file that I uploaded as an Admin. Even logging in as Admin and using the "impersonate" feature doesn't allow me to view the history of me(a)me.com<mailto:me@me.com> or you(a)you.com<mailto:you@you.com>. The files still exist as I can see them in database/files/000 Is this a bug? Or are Admins not supposed to upload files? Also I tried: * Deleted the above galaxy-dist * Downloaded the latest version of Galaxy * Created a user me(a)me.com<mailto:me@me.com> * Uploaded several files * Logged out * Create a user you(a)you.com<mailto:you@you.com> * Uploaded several files * Logged out and then logged in as an Admin user (specified in universe_wsgi.ini file) * Didn't upload a file * Logged out of Admin Now when I log into me(a)me.com<mailto:me@me.com> or you(a)you.com<mailto:you@you.com> the history is empty. This can't be correct is it? Neil

2 2

Resizing galaxy cloudman /mnt/galaxyData partition
by Iry Witham 16 May '13

16 May '13

Hi Team, I urgently need to resize the data partition on my galaxy cloudman instance without loosing anything. I notice that there a gui on the cloudman admin page that will allow me to increase it to 1000GB. If I use that will it restart all of the processes? I need to maintain this as a functional instance since it needs to be available for use tomorrow at 8:00 AM and I do not have the time to rebuild it. The gui says it will create a snapshot. The following is what it says: "Expand Disk Space Through this form you may increase the disk space available to Galaxy. All of the cluster services (but not the cluster) WILL BE STOPPED until the new disk is ready, at which point they will all be restarted. This may result in Galaxy jobs that are currently running to fail. Note that the new disk size must be larger than the current disk size. During this process, a snapshot of your data volume will be created, which can optionally be left in your account. If you decide to leave the snapshot for reference, you may also provide a brief note that will later be visible in the snapshot's description. New Disk Size (minimum 100GB, maximum 1000GB): OK Note (optional): or delete the created snapshot after filesystem resizing? If checked, the created snapshot will not be kept" What is the likelihood that this will be successful? Has anyone utilized it? Is there another way to resize the /mnt/galaxyData/ and /mnt/galaxyIndices partitions that will allow me to do what I need to? Thanks, Iry Witham The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.

3 16

Recall: [External] Re: [galaxy-user] Question on galaxy data visualization
by Sun, Wenping [USA] 16 May '13

16 May '13

Sun, Wenping [USA] would like to recall the message, "[External] Re: [galaxy-user] Question on galaxy data visualization".

2 1