- galaxy-dev - lists.galaxyproject.org

syntax and #if and #for statements within the <command> tag
by Nikhil Joshi 09 Aug '11

09 Aug '11

Hi all, So I've gotten some simple custom tools to work on our local Galaxy, however, I am having trouble figuring out exactly how the syntax works for the commands and lines within the <command> tag. E.g., I've seen #if and #for statements and also lines using functions like "str", but I can't really find any documentation about exactly how the syntax works. It kinda seems like PHP syntax, but not quite? Anyways, any help would be highly appreciated. Thanks! - Nik. On Wed, Aug 3, 2011 at 6:20 AM, Greg Von Kuster <greg(a)bx.psu.edu> wrote: > Hello Nikhil, > > I am responsible for the tool shed ( both implementation as well as > documentation ), and finally got a chance to get the wiki started yesterday > (that's why it is currently lacking in information). I will be working on > the wiki as the top priority now, so it should be fairly useful within the > next day or so. In the meantime, please feel free to direct any questions > to the mail list, and I'll answer them in short order. > > Getting started with the tool shed is fairly straight-forward. You can > browse existing repositories to see how others have designed their file > hierarchies. Many very good examples of repositories have been created by > Peter Cock ( his public username on the tool shed is peterjc ), so browse > some of his repositories to get an idea. Here's a good example of a repo > that includes a single tool that uses a .loc file: > > Contents: > effectivet3<http://toolshed.g2.bx.psu.edu/repository/browse_repository?id=92c548528a9a6…> > tool-data/<http://toolshed.g2.bx.psu.edu/repository/browse_repository?id=92c548528a9a6…> > effectiveT3.loc.sample<http://toolshed.g2.bx.psu.edu/repository/browse_repository?id=92c548528a9a6…> > tools/<http://toolshed.g2.bx.psu.edu/repository/browse_repository?id=92c548528a9a6…> > protein_analysis/<http://toolshed.g2.bx.psu.edu/repository/browse_repository?id=92c548528a9a6…> > effectiveT3.py<http://toolshed.g2.bx.psu.edu/repository/browse_repository?id=92c548528a9a6…> > effectiveT3.txt<http://toolshed.g2.bx.psu.edu/repository/browse_repository?id=92c548528a9a6…> > effectiveT3.xml<http://toolshed.g2.bx.psu.edu/repository/browse_repository?id=92c548528a9a6…> > > > You can browse repos without authenticating, but in order to create a repo, > you'll need to authenticate, so create an account on the tool shed if you > don't yet have one. > > Regarding your questions about the "XML format for the XML file", the > repository is simply a container for files of any type ( tool configs, data, > binaries, python scripts, exported Galaxy workflows, etc ), so the > repository itself has no requirement for XML formats, etc. Generally, XML > files will simply be the Galaxy tool config, the syntax of which is defined > on our wiki at > http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax > > Thanks very much for you interest in the tool shed, > > Greg Von Kuster > > > On Aug 2, 2011, at 10:36 PM, Nikhil Joshi wrote: > > Hi all, > > I have been using Galaxy for a few years now and I am about to start > creating my own repositories for the new Tool Shed. However, I can't seem > to find any good documentation on how to do that exactly. Can someone point > me in the right direction? I need to figure out the best way to organize a > repository, the XML format for the XML file, and any other important things > that I might need to know? Is there a good resource for all that? The wiki > has a section entitled "Create a new repository", but there is nothing in > it. Any help would highly appreciated. Thanks! > > - Nik. > > -- > Nikhil Joshi > Bioinformatics Programmer > UC Davis Genome Center > University of California, Davis > Davis, CA > http://bioinformatics.ucdavis.edu > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ > > > Greg Von Kuster > Galaxy Development Team > greg(a)bx.psu.edu > > > > -- Nikhil Joshi Bioinformatics Programmer UC Davis Genome Center University of California, Davis Davis, CA http://bioinformatics.ucdavis.edu

2 1

Failure to prepare job error
by Faiaz Khan 09 Aug '11

09 Aug '11

Hello, I'm running a Python/R tool from my Amazon Galaxy instance. To use R, I import a package I've installed called PypeR. My tool works fine from the command line, but when I try to run it on my Amazon instance, it gives me this error: Traceback (most recent call last): File "/nfs/software/galaxy/lib/galaxy/jobs/runners/local.py", line 61, in run_job job_wrapper.prepare() File "/nfs/software/galaxy/lib/galaxy/jobs/__init__.py", line 371, in prepare self.command_line = self.tool.build_command_line( param_dict ) File "/nfs/software/galaxy/lib/galaxy/tools/__init__.py", line 1543, in build_command_line command_line = fill_template( self.command, context=param_dict ) File "/nfs/software/galaxy/lib/galaxy/util/template.py", line 9, in fill_template return str( Template( source=template_text, searchList=[context] ) ) File "/nfs/software/galaxy/eggs/Cheetah-2.2.2-py2.6-linux-i686-ucs4.egg/Cheetah/Template.py", line 1004, in __str__ return getattr(self, mainMethName)() File "cheetah_DynamicallyCompiledCheetahTemplate_1311891781_86_41894.py", line 83, in respond EOFError: EOF when reading a line With the additional caption "failure to prepare job". I've been struggling with this for days, and I'm pretty sure I don't have a typo in my .xml file. How can I solve this? Thanks, Rafi

2 1

Catch problems preparing cluster job scripts
by Peter Cock 09 Aug '11

09 Aug '11

Hi all, I've run into some file permissions problems as part of using the same mapped directory on both the Galaxy server and our cluster. In the process I wrote the following patch which fixes the following bug - where Galaxy seems to leave the job in the pending state: galaxy.jobs INFO 2011-07-29 14:06:46,170 job 30 dispatched galaxy.jobs.runners.drmaa ERROR 2011-07-29 14:06:46,582 Uncaught exception queueing job Traceback (most recent call last): File "/data/galaxy/galaxy-central/lib/galaxy/jobs/runners/drmaa.py", line 114, in run_next self.queue_job( obj ) File "/data/galaxy/galaxy-central/lib/galaxy/jobs/runners/drmaa.py", line 164, in queue_job os.chmod( jt.remoteCommand, 0750 ) OSError: [Errno 1] Operation not permitted: '/data/galaxy/galaxy-central/database/pbs/galaxy_30.sh' The jobs was left stuck in the grey pending state. It looks like this exception should have been caught and the job put into an error state as in this patch: https://bitbucket.org/peterjc/galaxy-central/changeset/c5fa48633c0b This is currently the one and only change on this branch: https://bitbucket.org/peterjc/galaxy-central/src/job_scripts Please could this be reviewed and applied to the trunk. Thanks, Peter

2 4

[API] About workflow parameters :)
by Louise-Amélie Schmitt 09 Aug '11

09 Aug '11

Hello, Just a quick message to ask if you had the time to work on passing parameters to workflows through the API. Just to know, since I'm using workflows with the API. Best, L-A

2 3

Re: [galaxy-dev] [galaxy-user] configuring apache proxy server issue
by Jennifer Jackson 08 Aug '11

08 Aug '11

Hello, I am going to move this over to the galaxy-dev list so that the larger development community can more easily access your question and provide some feedback. Thanks! Jen Galaxy team On 8/4/11 12:55 PM, Sridhar A Malkaram wrote: > Hi, > > I have been using galaxy succesfully for my work using its internal > web-server. Recently I wanted to use it over the internet and so, I > configured it with apache at /galaxy of apache's root. I followed all > the instructions in setting up the proxy redirection. But I think the > re-direction is not happening. I highly appreciate any suggestions to > correct the issue. > > I checked for the mod_proxy, mod_proxy_http etc modules and they are > present and switched on. > > Apache gives the following error in its error_log file: > > [Thu Aug 04 14:15:56 2011] [error] [client 127.0.0.1] File does not > exist: /usr/local/apache/htdocs/galaxy > > Galaxy was set up in the directory /home/galaxy/galaxy-dist > The lines specific for proxy setting in universe_wsgi.ini are as follows: > use = egg:Paste#http > port = 8080 > host = localhost > [filter:proxy-prefix] > use = egg:PasteDeploy#prefix > prefix = /galaxy > [app:main] > filter-with = proxy-prefix > cookie_path = /galaxy > use_remote_user = True > remote_user_maildomain = unl.edu <http://unl.edu> > > httpd.conf settings: > > RewriteEngine on > <Proxy http://localhost:8080> > Order deny,allow > Allow from all > </Proxy> > <Location "/galaxy"> > Order deny,allow > Allow from all > # Take the $REMOTE_USER environment variable and set it as a header > in the proxy request. > RewriteCond %{IS_SUBREQ} ^false$ > RewriteCond %{LA-U:REMOTE_USER} (.+) > RewriteRule . - [E=RU:%1] > RequestHeader set REMOTE_USER %{RU}e > RequestHeader set X-URL-SCHEME https > RewriteRule ^/galaxy$ /galaxy/ [R] > RewriteRule ^/galaxy/(.*) http://localhost:8080/$1 [P] > # Compress all uncompressed content. > SetOutputFilter DEFLATE > SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary > SetEnvIfNoCase Request_URI \.(?:t?gz|zip|bz2)$ no-gzip dont-vary > </Location> > > > > > Thanks, > > > > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ -- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/Support

3 2

Filtering Illumina CASAVA 1.8 FASTQ files
by Assaf Gordon 07 Aug '11

07 Aug '11

Hello, With the recent upgrade to CASAVA 1.8, Illumina's FASTQ files now contain both filtered and non-filtered reads (i.e. high and low quality reads, what used to be called "reads that passed filtering"). Here's a program (+galaxy wrapper) that can filter those FASTQ files, preparing them for the down-stream genome mapping programs (Probably a temporary fix until mapping programs add a feature to ignore those reads). http://cancan.cshl.edu/labmembers/gordon/fastq_illumina_filter/ Comments are welcomed, -gordon

1 0

Request for pulling the updated FASTQ (de-)interlacer tool
by Florent Angly 05 Aug '11

05 Aug '11

Dear Galaxy team, Please consider pulling these changes I made to the interlacer/de-interlacer tools into the main Galaxy repository: https://bitbucket.org/fangly/galaxy-central/changeset/7f7ec8f05d36 While the tools were handling mate pairs fine, I now added the ability to also keep single reads (reads that miss their mate) in a separate file. The functional tests were updated and pass. Thanks, Florent

3 4

BWA wrapper with Illumina-1.3+ FASTQ files
by Assaf Gordon 05 Aug '11

05 Aug '11

Hi, The attached patch enables the BWA wrapper to work with Illumina-1.3+ FASTQ files without Grooming (which goes well with the name of the tool: "Map with BWA for Illumina" ). Actually, Changing the XML and the python code, testing the tool and mapping to the mouse genome all together took less time than grooming one 4.2GB FASTQ file :) Comments are welcomed, -gordon

3 3

Optional column select arguments?
by Peter Cock 05 Aug '11

05 Aug '11

Dear all, I'm trying to define an *optional* column parameter as follows: <param name="start_col" type="data_column" data_ref="input_tabular" multiple="False" numerical="False" optional="True" label="Column containing start co-ordindate" help="Leave empty to include everything from the first base/residue"/> i.e. I want the user to either select ONE of the columns of the input tabular data file (in which case getting "1" or "c1" etc as the string is fine) OR pick no column at all (in which case I want an empty string or similar in the command string. Is this possible? The above parameter loads fine, but defaults to the first column with no apparent way to un-make this implicit selection. Thanks, Peter

2 2

Re: [galaxy-dev] [galaxy-user] galaxy cluster setup
by Jennifer Jackson 05 Aug '11

05 Aug '11

Hello Bernard, I am going to cross-post your email over to the galaxy-dev mailing list, where there are many users doing these same sort of configuration tasks. Perhaps some of them have will also have comments and advice. Best! Jen Galaxy team On 8/5/11 6:15 AM, Bernhard Gschloessl wrote: > Dear all, > > I wonder if someone of you has already tested Galaxy with the ProActive > scheduler (since it is not listed on the page > http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Cluster )? If yes, > could you give me a feedback? > > Thanks in advance, > Bernhard >> >> >> >> From: *Roman Valls* <brainstorm(a)nopcode.org >> <mailto:brainstorm@nopcode.org>> >> Date: Thu, Jun 16, 2011 at 4:41 PM >> Subject: Re: [galaxy-user] galaxy cluster setup >> To: galaxy-user(a)lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu> >> >> >> Hello Robert, >> >> You may have a look at the official documentation on the wiki: >> >> https://bitbucket.org/galaxy/galaxy-central/wiki/Config/Cluster >> https://bitbucket.org/galaxy/galaxy-central/wiki/Config/ProductionServer >> >> You may also have a look at the slides from the past Galaxy Community >> conference: >> >> http://wiki.g2.bx.psu.edu/GCC2011 >> >> You might be interested on the one Nate Coraor gave: >> >> "Galaxy Deployment and API" >> >> Cheers ! >> Roman >> >> On 2011-06-15 23:01, Robert Jackson wrote: >> > Does anyone have practical info on galaxy setup on a cluster? -- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/Support

1 1