- galaxy-dev - lists.galaxyproject.org

Gnu Parallel
by Jesse Erdmann 23 Jul '10

23 Jul '10

http://savannah.gnu.org/projects/parallel/ Has anyone considered using something like Parallel under the covers in Galaxy for simple parallel processing like Bowtie. Essentially it works like xargs but can execute across multiple hosts and cores. I thought I would check with the community before spending any serious time evaluating it. Thanks! -- Jesse Erdmann Bioinformatics Analyst Masonic Cancer Center University of Minnesota jerdmann(a)umn.edu 612-626-3123 jesse(a)jesseerdmann.com Twitter: http://twitter.com/jesseerdmann

1 0

Galaxy NGS Tools
by Jacob Biesinger 23 Jul '10

23 Jul '10

Hi! In our lab, we've worked on several ChipSeq projects and have developed dozens of scripts for analyzing the results, including running several peak finders, several motif discovery tools, various data munging techniques, some parameter optimization for the above programs, calculating the genomic distribution of peaks, generating several summary graphs, calculating motif distributions within peaks, performing gene ontology analysis, etc. We've been thinking about making all of this into a standalone tool, possibly a web service, and have been considering Galaxy as a vehicle for automating the entire process and opening up the tools to a biologist community. From what I've seen in Galaxy Main and the recent inclusion of e.g., the MACS wrapper, it seems like the things I've listed would be of interest to the galaxy community at large. So I'm looking for feedback and possibly advice. Ideally, we'd like to be able to run the entire pipeline, look at the results, possibly change a few parameters in some of the steps (e.g., minimum FDR cutoff) and rerun only what needs to be rerun. Galaxy workflows are easy to create, but don't seem to have the flexibility that we're looking for. Perhaps several workflows tied together would do the job (i.e., have separate workflows for the major parts of the analysis) which we could tie together (possible in galaxy?) into one uber-pipeline. Has work like this already been done? Are there sample workflows that go beyond just calling peaks? Would the community be interested in the code + wrappers? Thanks for the help! -- Jake Biesinger Graduate Student Xie Lab, UC Irvine (949) 231-7587

5 5

Galaxy Tool Shed now includes the ability to upload an archive consisting of a suite of tools in addition to a single tool archive
by Greg Von Kuster 22 Jul '10

22 Jul '10

Hello Galaxy Community, The Galaxy Community Space has been renamed to be the Galaxy Tool Shed ( contributed by Dave Clements ), the location is still http://usegalaxy.org/community. The site now includes the ability to upload an archive consisting of a suite of tools in addition to a single tool archive. Here is the information about uploading both of these types of archives. Please start sharing tools, and let us know if you encounter any problems. Also, let us know what new features you would like to see in the tool shed. Our next update will include the ability to rate tools. Creating an archive containing a tool or a suite of tools A tool or tool suite archive is a tar-format file (bzipped or gzipped tar are valid) containing all the files necessary to load the tool(s) into a Galaxy instance. Tool Suite Archive A tools suite must include a file named suite_config.xml which provides information about the id, name, version and description of the tool suite, as well as the id, name, version and description of each tool in the suite. Here is an example suite_config.xml file. <suite id="lastz_toolsuite" name="Suite of Lastz tools" version="1.0.0"> <description>This suite contains all of my Lastz tools for Galaxy</description> <tool id="lastz_wrapper_2" name="Lastz" version="1.1.0"> <description> map short reads against reference sequence</description> </tool> <tool id="lastz_paired_reads_wrapper" name="Lastz paired reads" version="1.0.0"> <description> map short paired reads against reference sequence</description> </tool> </suite> New versions of the suite can be uploaded, replacing an older version of the suite, but the version attribute of the tag must be altered the same way that the version attribute of a single tool config must be altered if uploading a new version of a tool. The id, name and version attributes of each tag in the suite_config.xml file must exactly match the same attributes in each associated tool config in the archive or you will not be allowed to upload the archive. In addition to the suite_config.xml file, the archive must include all tool config files, executables, functional test data (if your tool config includes functional tests) and other files needed for each of the tools in your suite to function within Galaxy. See the information about single tool archives below for additional hints to enable ease-of-use when others download your suite of tools. For example, to package the above Lastz suite of tools: user@host:~% tar jcvf ~/Desktop/galaxy_lastz_toolsuite.tar.bz2 lastzsuite lastzsuite/ lastzsuite/README lastzsuite/suite_config.xml lastzsuite/lastz_paired_reads_wrapper.py lastzsuite/lastz_paired_reads_wrapper.xml lastzsuite/lastz_wrapper.py lastzsuite/lastz_wrapper.xml lastzsuite/lastz-distrib-1.02.00/ lastzsuite/lastz-distrib-1.02.00/src/ lastzsuite/lastz-distrib-1.02.00/src/Makefile lastzsuite/lastz-distrib-1.02.00/src/version.mak lastzsuite/lastz-distrib-1.02.00/src/lastz.c lastzsuite/lastz-distrib-1.02.00/src/lastz.h ... ~/Desktop/galaxy_lastz_tool.tar.bz2 is now ready to be uploaded. Single Tool Archive A single tool archive must include a tool config file and will probably also include a tool script. If any steps are necessary to install your tool beyond the basic instructions below, include a README file to provide details. If the tool (or parts of it) are written in C, the source code can be included (or put links to the source in the README). Do not include pre-compiled binaries without source since Galaxy is run on a wide variety of platforms. Also, if you are only wrapping or providing a Galaxy config for a tool that is not your own, be sure the license allows for redistribution before including any part of that tool in the archive. There are no requirements about the directory structure inside the archive, but for ease of use it's generally a good idea to put everything inside a sub-directory, instead of directly at the top level. For example, to package the Lastz tool's config file, Galaxy wrapper, and the C source: user@host:~% tar jcvf ~/Desktop/galaxy_lastz_tool.tar.bz2 lastz lastz/ lastz/README lastz/lastz_wrapper.py lastz/lastz_wrapper.xml lastz/lastz-distrib-1.02.00/ lastz/lastz-distrib-1.02.00/src/ lastz/lastz-distrib-1.02.00/src/Makefile lastz/lastz-distrib-1.02.00/src/version.mak lastz/lastz-distrib-1.02.00/src/lastz.c lastz/lastz-distrib-1.02.00/src/lastz.h ... ~/Desktop/galaxy_lastz_tool.tar.bz2 is now ready to be uploaded. Editing Information, Categories, and Submitting For Approval Simply uploading a tool to the Galaxy too shed will not allow other users to find and download your tool. It will need to be approved by an administrator before it appears in the tool list. After your archive has successfully uploaded, you will be redirected to the Edit Tool page. Provide a detailed description of what the tool does - this will be used by administrators to understand the tool before approving it for display on the site. Once approved, this information will be displayed to users who view your tool. In addition, the site administrators will have configured a number of categories with which you can associate your tool to make it easy to find by users looking to solve specific problems. Associate as many categories as are relevant to your tool. You may change the description and associated categories as often as you'd like until you click the "Submit for approval" button. Once submitted, the tool will be approved or rejected by an administrator. If the tool is rejected, you will see information about why it was rejected, and you can make appropriate changes to the archive and re-submit it for approval. When it is approved, your archive will be visible to everyone. At that point, the description and associated categories can only be changed by an administrator. When the tool has been approved or rejected, you may upload a new version by browsing to the tool's "View Tool" page, clicking the "Tool actions" menu in the upper right corner of the page, and selecting "Upload a new version" from the menu. Downloading and Installing Tools A tool's download link will send you the tool archive. Once downloaded, unpack the tool on your local Galaxy instance's server: user@host:~% tar xvf galaxy_lastz_tool.tar ... user@host:~% tar zxvf galaxy_lastz_tool.tar.gz ... user@host:~% tar jxvf galaxy_lastz_tool.tar.bz2 ... If the archive includes a README file, consult it for installation instructions. If not, follow these basic steps: Create a directory under galaxy_dist/tools/ to house downloaded tool(s). In the new directory, place the XML and any script file(s) which were contained in the archive. If the tool includes binaries, you'll need to copy them to a directory on your $PATH. If the tool depends on C binaries but does not come with them (only source), you'll need to compile the source first. Add the tool to galaxy_dist/tool_conf.xml. Restart your Galaxy server process. In the near future, we plan to implement a more direct method to install tools via the Galaxy administrator user interface instead of placing files on the filesystem and manually managing the tool_conf.xml file. Greg Von Kuster Galaxy Development Team greg(a)bx.psu.edu

1 0

Show STDOUT in reports page
by Assaf Gordon 22 Jul '10

22 Jul '10

Hi, May I ask to add STDOUT to the job-info page of the very cool new reports webapp ? Attached a patch for that. Thanks, -gordon

2 3

Problem with format="input" when using multiple inputs
by Assaf Gordon 21 Jul '10

21 Jul '10

Hi, I came across a small problem in the tool definition parameters. It's rare, but it happens. The problem is that if the tool has more than one input file, AND uses variable output format based on the input file, AND the multiple input files are of different formats, then there's no way to properly set the output format based on the input format of a specific input. Example: Assume an imaginary tool that accepts 2 inputs: one input can be BED/GFF/BAM, the other input is tabular. The output should be the same as the interval file (BED/GFF/BAM). The XML definition would be: ========== <inputs> <param format="bed,gff,bam" name="input1" type="data" label="Intervals" /> <param format="tabular" name="input2" type="data" label="filter" /> </inputs> <outputs> <data format="input" name="output" metadata_source="input1" /> </outputs> =========== The problem is with 'format="input"' . Any other fixed format (e.g. "fasta" or "bam") would work fine. The special value "input" is hard-coded (lib/galaxy/tools/actions/__init__.py:226) to take the extension of "one" of the input files, but there's no way to specify which input (due to the loop in lib/galaxy/tools/actions/__init__.py:159) I guess it's not very a common scenario, because most tools that accept multiple inputs (e.g. concatenate queries) implicitly assume all inputs are the same format. I propose this ugly patch (attached), which is not the cleanest solution, but it allows specifying an input source, while not breaking any other tool definition. With it, I can specify the output definition as: === <outputs> <data format="input:input1" name="output" metadata_source="input1" /> </outputs> === And then the output dataset format will be copied from 'input1'. Comments are welcomed, -gordon

1 1

Error when attempting to run galaxy
by Roger Kapul 21 Jul '10

21 Jul '10

Hello, I'm running a fresh install of Ubuntu 10.04 LTS, python version 2.6 and numpy version 1.3.0 and I get this error when I try to run galaxy on my machine. I downloaded and installed galaxy by following the steps on the wiki and used the mercurial repository to download it. galaxy.web.framework.base DEBUG 2010-07-20 12:36:28,494 Enabling 'async' controller, class: ASync Traceback (most recent call last): File "./scripts/paster.py", line 34, in <module> command.run() File "/home/roger/galaxy-python/galaxy_dist/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/command.py", line 84, in run invoke(command, command_name, options, args[1:]) File "/home/roger/galaxy-python/galaxy_dist/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/command.py", line 123, in invoke exit_code = runner.run(args) File "/home/roger/galaxy-python/galaxy_dist/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/command.py", line 218, in run result = self.command() File "/home/roger/galaxy-python/galaxy_dist/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/serve.py", line 276, in command relative_to=base, global_conf=vars) File "/home/roger/galaxy-python/galaxy_dist/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/serve.py", line 313, in loadapp **kw) File "/home/roger/galaxy-python/galaxy_dist/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 204, in loadapp return loadobj(APP, uri, name=name, **kw) File "/home/roger/galaxy-python/galaxy_dist/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 225, in loadobj return context.create() File "/home/roger/galaxy-python/galaxy_dist/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 625, in create return self.object_type.invoke(self) File "/home/roger/galaxy-python/galaxy_dist/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 110, in invoke return fix_call(context.object, context.global_conf, **context.local_conf) File "/home/roger/galaxy-python/galaxy_dist/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/util/fixtypeerror.py", line 57, in fix_call val = callable(*args, **kw) File "/home/roger/galaxy-python/galaxy_dist/lib/galaxy/web/buildapp.py", line 90, in app_factory add_controllers( webapp, app ) File "/home/roger/galaxy-python/galaxy_dist/lib/galaxy/web/buildapp.py", line 39, in add_controllers module = __import__( module_name ) File "/home/roger/galaxy-python/galaxy_dist/lib/galaxy/web/controllers/tracks.py", line 27, in <module> from galaxy.visualization.tracks.data.array_tree import ArrayTreeDataProvider File "/home/roger/galaxy-python/galaxy_dist/lib/galaxy/visualization/tracks/data/array_tree.py", line 8, in <module> from bx.arrays.array_tree import FileArrayTreeDict File "numpy.pxd", line 119, in init bx.arrays.array_tree (lib/bx/arrays/array_tree.c:11323) ValueError: numpy.dtype does not appear to be the correct type object Any advice? Thank you Roger

3 4

command interpreter not using interpreter in my $PATH
by Leandro Hermida 21 Jul '10

21 Jul '10

Dear all, Sorry if I missed it, I looked in the list archives and couldn't find any mention of the topic. I am developing tools for Galaxy and thus running a Galaxy server from my within my home directory under my username. I have my own Perl installed in my home directory and this is in my $PATH, I don't use the system Perl, but when I run a Galaxy tool with <command interpreter="perl"> it is using the system Perl, as if my $PATH is not used at all. Since the Galaxy server and tool runner is running under my username and it looks like it just forks off a call to "perl ...." when it forks why doesn't it use my Perl? best, Leandro

4 10

Galaxy Developer Conference 2011
by Anton Nekrutenko 19 Jul '10

19 Jul '10

Dear members of Galaxy Community: We are starting to prepare for the Second Galaxy Developer Conference in 2011. This time we would like to hold it in Europe and we were asked to consider Uppsala and Heidelberg. However, Galaxy is a community project and we would like to ask you for input. If any of you knows good and reliable conference facility, please let us know. We will then research this and finalize the conference planning. So let us know... Thanks, anton + galaxy team Anton Nekrutenko http://nekrut.bx.psu.edu http://usegalaxy.org

1 0

<repeat> tag
by Pei, Bing 16 Jul '10

16 Jul '10

Hi, I was wondering if I could get some more information on how the <repeat> tag works. If I am repeating an argument, say "$samplename," would the xml file just pass extra arguments to my command line, or do I have to do some extra configuration? If it is passing extra arguments, are they lumped together? For example if I want to repeat both $samplename and $lane using the same tag, is it sending to the command line $samplename1 $samplename2 $samplename3 $lane1 $lane2 $lane3 or $samplename1 $lane1 $samplename2 $lane2 $samplename3 $lane3? What types are supported by the <repeat> tag? Currently I am using type="text" but this does not seem to be working. Thanks!

5 7

nginx configuration and upload size
by Brad Chapman 16 Jul '10

16 Jul '10

Hey all; Nate, many thanks for the updated wiki documentation on configuring nginx. Awesome stuff, with the help of this I've been able to get nginx handling uploads on our local galaxy server. I added some notes to the documentation about getting this running with a prefix, and also submitted a patch which sorts out some redirect issues with both nginx and a prefix: http://bitbucket.org/galaxy/galaxy-central/issue/366/uploads-from-tool-menu… There's one issue related to uploading through nginx that I could use some advice on. Uploading larger (>1M) files was giving a 413 error, which appeared to be related to the client_max_body_size parameter: http://forum.slicehost.com/comments.php?DiscussionID=1714 Resetting client_max_body_size to a value larger than the upload file resolved the problem. Is this how you handle large file uploads on Galaxy main or is there a better way? If that's the right solution I can add a note to the wiki docs for future nginx users, but am not confident enough in my nginx skillz to start passing out advice without a bit of confirmation. Thanks much, Brad

2 2