- galaxy-dev - lists.galaxyproject.org

handling multiple output files on galaxy.
by Vipin TS 04 Jul '10

04 Jul '10

Hi Galaxy team, I would like to create output files of my tool based on the input request. I referred to the contents at http://bitbucket.org/galaxy/galaxy-central/wiki/ToolsMultipleOutput. Here it doesn't mention how I can handle if I have more conditional statements. <tool id="example_tool" name="Multiple output" description="example"> <command>example_tool.sh </command> <inputs> ... <conditional name="condition1"> ... <conditional name="condition2"> ..... <param name="format" type="select" label="Output format"> <option value="bed">BED format</option> <option value="sam">SAM format</option> </param> </condition> ... </condition> ... </inputs> ... <outputs> <data format="bed" name="output1" > <filter>condition1 condition2['format'] == "bed"</filter> ## How I can give multiple conditions. </data> <data format="sam" name="output2" > <filter>condition1 condition2['format'] == "sam"</filter> </data> </outputs> </tool> Many thanks in advance, Vipin T S

1 0

Recently used tools options
by Assaf Gordon 03 Jul '10

03 Jul '10

Again, very nice feature. And of course, some comments: 1. The search is too general (IMHO). If I search for "rna", I get (among other tools) the "lastz" tool, because it has the following line in the help section: === 3. RNAME Reference sequence NAME === (Note the "RNA" in "RNAME"...) To be useful, the basic search has to include the tool name and possibly the description, and preferably some keywords (specified manually in the XML). An advanced search might offer a global text search. 2. The "recently used tools" is not updated after running a new tool. One has to reload the page (and then the list is hidden by default). 3. a small bug: 1. reload galaxy (for a clean start) 2. click "options->Search Tools" (search box is shown) 3. click "options->Show recently used" (recently-used menu is shown) 4. Search for a tool which is NOT in the recently used menu (e.g. search for "foobar") 5. The "recently used" menu is forever gone, no way to get it back: the "options" menu still says "hide", but even clicking multiple times on hide/show does not bring it back. 4. not necessarily a bug: All workflows in the "workflow" menu (i.e. workflows that were configured to be on the user's tools menu) are always displayed, regardless of the active search. This is probably by design, but it's conceptually confusing (because you encourage users to use workflows as regular tools in that menu). -gordon

2 1

New Galaxy Community tool sharing site now available
by Greg Von Kuster 02 Jul '10

02 Jul '10

Hello Galaxy community, We have introduced a new Galaxy tool sharing community site at http://usegalaxy.org/community. The site currently allows you to upload an archive consisting of a single tool, but will soon allow you to upload an archive consisting of a suite of tools. We hope that you will find this site useful for hosting tools that you would like to contribute and share. We'll keep you informed as the site's features evolve. Here is the current information about uploading and downloading tools from the site. Uploading Tools Tool File Format A tool file is a tar-format (bzipped or gzipped tar are valid) archive containing all the files necessary to load the tool in Galaxy. At the very least, it must contain a Tool XML File, and will probably also include a tool script. If any steps are necessary to install your tool beyond the basic instructions below, please include a README file which details these steps. If the tool (or parts of it) are written in C, the source code can be included (or put links to the source in the README). Please do not include precompiled binaries without source, since Galaxy is run on a wide variety of platforms. Also, if you are only wrapping or providing a Galaxy config for a tool that is not your own, please be sure the license allows for redistribution before including any part of that tool in the tar archive! There are no requirements about the directory structure inside the tar archive, but for ease of use, it's generally a good idea to put everything inside of a subdirectory, instead of directly at the top level. Tool File Example To package up the LASTZ tool's config file, Galaxy wrapper, and the C source: user@host:~% tar jcvf ~/Desktop/galaxy_lastz_tool.tar.bz2 lastz lastz/ lastz/README lastz/lastz_wrapper.py lastz/lastz_wrapper.xml lastz/lastz-distrib-1.02.00/ lastz/lastz-distrib-1.02.00/src/ lastz/lastz-distrib-1.02.00/src/Makefile lastz/lastz-distrib-1.02.00/src/version.mak lastz/lastz-distrib-1.02.00/src/lastz.c lastz/lastz-distrib-1.02.00/src/lastz.h ... ~/Desktop/galaxy_lastz_tool.tar.bz2 is now ready to be uploaded. Editing Information, Categories, and Submitting For Approval Simply uploading a tool to the Community will not allow other users to find and download your tool. It will need to be approved by an administrator before it appears in the tool list. After the tool has successfully uploaded, you will be redirected to the Edit Tool page. Please provide a detailed description of what the tool does - this will be used by administrators to understand the tool before approving it for display on the site. Once approved, this information will be displayed to users who view your tool. In addition, the site administrators will have configured a number of categories with which you can associate your tool to make it easily findable by users looking to solve specific problems. Please associate as many categories as are relevant to your tool. You may change the description and associated categories as often as you'd like until you click the "Submit for approval" button. Once submitted, the tool will be approved or rejected by an administrator. Once approved, it will be visible to everyone. At that point, the description and associated categories can only be changed by an administrator. Once the tool has been approved or rejected, you may upload a new version by browsing to the tool's "View Tool" page, clicking the context menu to the right of the tool's name, and selecting "Upload a new version." Downloading and Installing Tools A tool's download link will send you the tool tar archive. Once downloaded, unpack the tool on your local Galaxy instance's server: user@host:~% tar xvf galaxy_tool.tar ... user@host:~% tar zxvf galaxy_tool.tar.gz ... user@host:~% tar jxvf galaxy_tool.tar.bz2 ... If the tar archive includes a README file, consult it for installation instructions. If not, follow these basic steps: Create a directory under galaxy_dist/tools/ to house downloaded tool(s). In the new directory, place the XML and any script file(s) which were contained in the tar archive. If the tool includes binaries, you'll need to copy them to a directory on your $PATH. If the tool depends on C binaries but does not come with them (only source), you'll need to compile the source first. Add the tool to galaxy_dist/tool_conf.xml. Restart the Galaxy server process. We plan to implement a more direct method to install tools via the Galaxy administrator user interface instead of placing files on the filesystem and managing the tool_conf.xml file by hand. In the meantime, this is the process. Greg Von Kuster Galaxy Development Team greg(a)bx.psu.edu

2 1

sge & pbs job dependency
by Andrey Tovchigrechko 01 Jul '10

01 Jul '10

I am looking at the code inside lib/galaxy/jobs. It appears that the dependency tracking between jobs is done entirely by Galaxy. In other words, the option of e.g. SGE to submit a job with a dependency on previously submitted jobs is not used. Galaxy simply waits until the dependency jobs have finished, and then it submits the dependent job. It seems like inefficient to me on a busy cluster. E.g. if a user has one "reducer" job in a workflow that depends on a 100 "mapper" jobs, and all are submitted at once with an SGE dependency of the mapper on the reducer, then the mapper can be at the top of the SGE queue by the time all mappers have finished, and run immediately. In the current Galaxy implementation, the reducer will only be submitted to SGE queue after all mappers have finished, and have to wait for all the jobs in front of it (assuming a simple FIFO policy on the SGE side). Why is such design necessary?

2 1

TopHat with FASTA and Illumina FASTQ
by Assaf Gordon 01 Jul '10

01 Jul '10

Hi all, Similar to the previous Bowtie patch, this one allows the TopHat wrapper to use FASTA and Illumina FASTQ files. Also available here: http://cancan.cshl.edu/labmembers/gordon/files/galaxy_tophat_wrapper_fasta_… Comments are welcomed, -gordon

1 0

Problem with history association after upgrade to 4cdf4cca0f31
by Dennis Gascoigne 01 Jul '10

01 Jul '10

I think there may be a bug with regards the history/job associations introduced in the upgrade. Many of the users have histories where the associated jobs are no longer shown. The histories themselves still exit. The problem seems to be mainly workflows with many jobs. The HistoryID/job association is still in the database, though there is no history/dataset association for these Histories (I havent investigated enough to know if this is expected). This is definitely a result of the upgrade and is reported by a number of users. It does not occur for all histories - only some. The latest upgrade does not help. Cheers Dennis

2 1

Issue with manage_db.py
by Dennis Gascoigne 01 Jul '10

01 Jul '10

It appears that the ini file is hardcoded into manage_db.py. You may want to make this an argument to the script as we have two different databases from two different configs and were wondering why the wrong one was updated. Cheers Dennis

2 1

User file access without uploading?
by Michael Siebauer 01 Jul '10

01 Jul '10

Hi, is there a way/module or data Library that allows users to access there files WITHOUT the need of uploading them first? Cause our Galaxy instance is running within the intranet, all files could be accessed straight via NFS. Thx, Michael ;-)

4 9

Samtools and BAM
by Davide Cittaro 01 Jul '10

01 Jul '10

Hi there, I've realized that SAM tools in galaxy do not operate on BAM files... is there a reason for this? d /* Davide Cittaro Cogentech - Consortium for Genomic Technologies via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro(a)ifom-ieo-campus.it */

1 0

Workflow post-actions
by Assaf Gordon 30 Jun '10

30 Jun '10

AWESOME! Truly amazing. And yet, I have some comments: Notification emails =================== 1. It seems the notification email is sent immediately when the workflow is submitted, before the job is completed. I've only tested it a few times, but a for a workflow with 5 steps (each is a shell scripts that does "sleep 30") and the last step is configured with EmailAction - I get the email immediately, before the last step is completed. 2. users don't really need a 6-digits microsecond accuracy in the "completed time". 3. The subject line contains "%s", not replaced with a variable ( lib/galaxy/actions/post.py:77 ) 4. The email line says "your job "X"..." but "X" is the history name. This implicitly assumes that users run a single workflow in a history, which is not the case. A more common use case (in our local instance): users load 5 FASTQ files in a single history, and start 5 workflows on those files (all in the same history). A friendlier message would be "You workflow 'X' on dataset 'Y' in history 'Z' is completed", with X being the workflow name, and "Y" being the first dataset used as input in the workflow (if multiple files were used, take just one). Also, remember that many (most? in my case) users still call their histories "unnamed history". So the history name alone isn't helping. 5. The time reported in the emails (for me) is not the local server time. This might be an indication for a deeper problem (e.g. my postgres time zone is wrong). 6. Link to the galaxy server: it's a good idea to say "at Galaxy instance XXX", but if I may ask for more: instead of just the host name, use the complete "url_for()" link, so that mirrors that use the "prefix" option will contain the full link. If you add "http://" prefix, most email clients automatically convert this to a clickable link (even in textual, non-html emails), so users will have easier time getting to galaxy. Example: === Your job 'Unnamed history' at Galaxy instance rave.cshl.edu is complete as of 2010-06-29 23:18:53.271963. === Will be more helpful as: === Your job 'Unnamed history' at Galaxy instance http://rave.cshl.edu/galaxy is complete as of 2010-06-29 23:18:53.271963. === To ask for even more: Construct a link that automatically switches to the relevant history - so users will get to their completed jobs with a single click. Hidden Dataset ============== 1. There's no way to look at hidden datasets (you've mentioned that you're working on that). 2. Hidden datasets are still counted on the "history list" view - a bit confusing (but I'm not sure I have a good suggestion, because of the next item). 3. Two useability issues with hidden datasets: A long-running job whose dataset is hidden is confusing. Imagine a workflow with a paired-end mapping job that is hidden - it could take hours, but the history pane will show nothing: only green datasets (the previous workflow steps) and grey datasets (those that haven't run yet). A failed hidden job (didn't test it, just thought about it): If a hidden job fails, its following (unhidden) job will also fail, but it would not be immediately obvious how to get to the origin of the failure. Combining 2+3, It might be more useful if all datasets are displayed until the entire workflow is successfully completed, and only then datasets are hidden (some sort of an automatic post-action on the last workflow step). If any step in the workflow fails (or if the workflow isn't completed yet) - everything is visible (I must admit I don't have a good solution for a workflow with multiple final outputs). Dataset Rename ============== Would be great to be able to use templates/variables in the new name (e.g. "$input"), so that each step could contain parameters or the name of it's input. (Remember that my users run the same workflow multiple times in the same history, so a fixed name will create multiple datasets with the same name - a bit confusing). Metadata changes ================ Would be great to be able to change dbkey/organism as well (in addition/alternative to columns/filetype). Keep up the good work! Thanks, -gordon

3 3