September 2011 - galaxy-dev - lists.galaxyproject.org

passing environment to pbs runner
by Oleksandr Moskalenko 14 Sep '11

14 Sep '11

I need to get Galaxy to use tools that are not located in the system path. At the command line I can export the PATH either manually or with environment modules 'module load foo'. However, there doesn't seem to be a clear way to pass the $PATH and other environmental variables to the pbs runner. For instance, I use Torque with a dedicated "galaxy" queue. So, I configured the "default_cluster_job_runner = pbs:///galaxy/" in universe_wsgi.ini. The problem is that the pbs runner does not pass the environment to the scheduler. I sort of circumvented this by sourcing an environment setup file from 'pbs_template' in pbs.py, but I keep wondering about a better way to do it. I currently use tools set up using environment modules, so this is what I added the following to pbs_template: . /etc/profile.d/modules.sh module load bio Thanks, Alex

3 3

Re: [galaxy-dev] Contribution of tools to galaxy main server
by Greg Von Kuster 14 Sep '11

14 Sep '11

Hello Dongjun, Thanks very much for your interest in sharing this tool with the Galaxy community. At this time the process for sharing tools is to upload your tool to the Galaxy tool shed at http://toolshed.g2.bx.psu.edu/. Ideally your tool will include functional tests and your tool shed repository will include small input / output datasets for these tests. If your tool uses any 3rd party dependencies, the <requirements> tag set in the tool config should reflect them appropriately. For clarification on tool config syntax, see our wiki at http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax. In the past, the process used by the core Galaxy development team to share tools was to include them in the Galaxy distribution, but we are beginning to phase this process out, replacing it with the tool shed. Over time we will be eliminating tools from the distribution, and local Galaxy instances will install them from the Galaxy tool shed. This same process will be used for populating our main Galaxy instance here at Penn State with tools. So for now, please upload your tool to the tool shed. Thanks again, Greg Von Kuster On Sep 9, 2011, at 4:08 PM, Dongjun Chung wrote: > Dear Greg von Kuster, > > Hello, this is Dongjun Chung, a PhD student at the University of Wisconsin-Madison. > > We recently developed a tool for galaxy and hope to contribute our tool to galaxy main server. We have developed a tool for allocation of multi-reads (reads that map to multiple locations on the reference genome) in ChIP-seq. In contrast, basically all the currently available ChIP-seq peak callers utilize only uni-reads (reads the map to a unique location on the reference genome). This tool implements our paper recently published in PLoS computational biology: > > Chung D, Kuan PF, Li B, SanalKumar R, Liang K, Bresnick E, Dewey C, and Keles S (2011), “Discovering transcription factor binding sites in highly repetitive regions of genomes with multi-read analysis of ChIP-Seq data,” PLoS Computational Biology, 7(7): e1002111. > > http://www.ploscompbiol.org/article/info%3Adoi%2F10.1371%2Fjournal.pcbi.100… > > We believe that our tool could be appropriate for galaxy main server because there is no tool for this purpose in the server yet and this tool should be beneficial to general galaxy users, especially researchers studying eukaryotic genomes. We already completed developing corresponding wrapper and tool definition files for galaxy. Our tool can accept FASTA/FASTQ as input and its output can also work nicely with diverse ChIP-seq peak callers such as MACS peak caller which is already in the galaxy main server. > > If the galaxy development team can accept our tool as a part of galaxy in the main server, please let us know anything you need for us to contribute our tool to galaxy. We can prepare what is needed to meet the requirements. > > Sincerely, > Dongjun Chung > > PhD student > Department of Statistics > University of Wisconsin-Madison > > On 09/09/2011 12:59 PM, Dongjun Chung wrote: >> >> >> >> ---------- Forwarded message ---------- >> From: Greg Von Kuster <greg(a)bx.psu.edu> >> Date: Wed, Aug 31, 2011 at 2:38 PM >> Subject: Re: [galaxy-dev] contribute tools to galaxy? >> To: Dongjun Chung <dongjun.chung(a)gmail.com> >> Cc: galaxy-dev(a)lists.bx.psu.edu >> >> >> Hello Dongjun, see my answers to your question inline. >> >> On Aug 28, 2011, at 10:12 AM, Dongjun Chung wrote: >> >>> Hi All, >>> >>> I'm a newbie to galaxy and enjoying it a lot these days. Thanks for the great work. >>> >>> I have a question regarding contribution of software to galaxy. We developed a ChIP-seq peak calling algorithm and software (R package) and hope to contribute it to galaxy. I have read the wiki and prior mailing list about the contribution system but it is still somewhat confusing to me. >>> >>> 1. It seems that I can contribute our software to "tool shed" if I prepare appropriate code & definition files. Then, users can download and use it with their locally installed galaxy. However, these files committed to tool shed will not appear in galaxy main or test servers. Am I correct? >> >> Tools from the Galaxy tool shed may or may not be available in the Galaxy test / main servers hosted at Penn State. >> >>> >>> 2. What is clear relationship between main/test servers & tool shed? >> >> The Galaxy tool shed enables the Galaxy community to share tools. These tools are generally used in local Galaxy installations, but not available on the Penn State instances. >> >>> Can we contribute our software to main or test servers as well? >> >> The tools available on the Penn State servers are generally developed by the Galaxy core development team, and are available in the Galaxy distribution. If your tools complement the tools currently available in the distribution, the Galaxy core development team may agree to include them. However, see my comments 2 answers below regarding upcoming enhancements to Galaxy and the tool shed. >> >>> Or only galaxy core developers can add new tools to galaxy main or test servers? >> >> Yes. >> >>> If so, which software is considered to be added to servers? Are they chosen from tools contributed to tool shed? >> >> Currently, tools available in the distribution and on the Penn State instances are generally developed by the core Galaxy development team. However, upcoming enhancements to Galaxy and the Galaxy tool shed will enable automatic installation of tools from the tool shed into local Galaxy instances, eliminating the necessity to include tools in the distribution. This automation will be available fairly soon. >> >>> >>> 3. If our software is a R package, then users need to download and install it first in their R system before they use our software within their galaxy system even in the case they have appropriate definition files. Am I right? Or is there any better solution for this? >> >> If your tools require R, then those that use your tools will need to install R in the Galaxy environment path so the tools will be functional. Tools that have dependencies like this should include a <requirements> tag set in the tool config. >> >> See http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax#A.3Crequiremen… >> >>> >>> Thanks! >>> >>> Best, >>> Dongjun ___________________________________________________________ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>> >>> http://lists.bx.psu.edu/ >> >> Greg Von Kuster >> Galaxy Development Team >> greg(a)bx.psu.edu >> >> >> >> > > Greg Von Kuster Galaxy Development Team greg(a)bx.psu.edu

2 1

ProFTPd on Ubuntu system.
by Luobin Yang 13 Sep '11

13 Sep '11

Hi, Has anyone set up ProFTPd successfully on Ubuntu 10.04 to enable FTP upload on Galaxy? I followed the instructions on this link ( http://wiki.g2.bx.psu.edu/Admin/Config/Upload%20via%20FTP) but it doesn't work. Thanks, Luobin

2 3

Re: [galaxy-dev] cluster path question
by Ann Black 13 Sep '11

13 Sep '11

Hello everyone - I am also running into this issue trying to get galaxy integrated with our sun grid engine. My galaxy user's .bash_profile does not appear to get sourced when the jobs run. I augmented the sample sam_filter.py tutorial such that it output path and user info so I could see how the jobs were being run: out = open( sys.argv[2], "w" ) out2 = open("/data/galaxy-dist/ann.out", "w") out2.write(socket.gethostname()) out2.write("\n") out2.write(os.environ['PATH']) out2.write("\n") drmaa = os.environ.get('DRMAA_LIBRARY_PATH') if drmaa is None: out2.write("None") else: out2.write(os.environ.get('DRMAA_LIBRARY_PATH')) out2.write("\n") out2.write(str(os.geteuid())) out2.write("\n") out2.write(str(os.getegid())) shutil.copytree("/data/galaxy-dist/database/pbs","/data/galaxy-dist/ann") the job is being dispatched as my galaxy user, however the my augments to PATH and additional env vars that I have exported in our galaxy user's .bash_profile are not present when the script runs (ie, .bash_profile is not sourced). When I use qsub to manually run the galaxy script that gets generated under database/pbs, the output to ann.out reflects my PATH and exported env vars. Was there any other solution to this issue besides the drmaa.py script augment? Thanks much for your help, Ann

2 4

Upgrading postgres version
by James Vincent 13 Sep '11

13 Sep '11

Hello, We'd like to migrate from postgres 8.4 installed system wide through Ubunutu package manager to postres version 9 installed from source. Is there a smooth and easy route for this or is it the manual database dump/migrate pain that I think it will be? Thanks, Jim

1 0

barchart not working: GNUplot problem
by Joachim Jacob 12 Sep '11

12 Sep '11

Hi all, I have got numpy and gnuplot on speaking terms. But now, trying to use the barchart, I get following error: gnuplot> set style histogram clustered gap 5 title offset character 0, 0, 0 ^ line 0: expecting 'data', 'function', 'line', 'fill' or 'arrow' line 0: undefined variable: in line 0: undefined variable: invert gnuplot> set style data histograms ^ line 0: expecting 'lines', 'points', 'linespoints', 'dots', 'impulses', 'yerrorbars', 'xerrorbars', 'xyerrorbars', 'steps', 'fsteps', 'histeps', 'filledcurves', 'boxes', 'boxerrorbars', 'boxxyerrorbars', 'vectors', 'financebars', 'candlesticks', 'errorlines', 'xerrorlines', 'yerrorlines', 'xyerrorlines', 'pm3d' Gnuplot is working from commandline and has version 4.0. Thanks for any suggestion! Joachim

1 0

Installing tools with dependencies using mi-deployment scripts
by Mattias de Hollander 12 Sep '11

12 Sep '11

Hello, I am trying to add tools (PyCogent, Qiime) to a Cloudman image using the mi-deployment scripts. Qiime depends on PyCogent and it needs to be accessible on the python path, otherwise I get an error: PyCogent not installed but required. (Is it installed? Is it in the current user's $PYTHONPATH or site-packages?) Is there a way to add the paths in env.sh to the sys.path during installation using tools_fabfile.py? All the tools are in /mnt/galaxyTools/tools. I tried the DependencyManager from galaxy but then I only receive the path to the env.sh file: In [43]: dependency_manager.find_dep('blast') Out[43]: ('/mnt/galaxyTools/tools/blast/2.2.25/env.sh', '/mnt/galaxyTools/tools/blast/2.2.25', '2.2.25') Is there a smarter way to add files to the PYTHONPATH during the execution of tools_fabfile.py? Thanks, Mattias -- Bioinformatician Netherlands Institute of Ecology (NIOO-KNAW) Wageningen, the Netherlands

2 6

bam grooming
by Paul Igor Costea 12 Sep '11

12 Sep '11

Hello, I am new at using the galaxy server and i have come across an annoying restriction when it comes to bam files. There is an assumption build in, that a bam file needs to be coordinate sorted. (does hold for most tools). However, most of my tools are using the bam file coming off the mapper. (saves some time by not having to do sorting). Is there any way to get around this restriction? Right now i've hacked it to always return false for binary.py::dataset_content_needs_grooming. I do however think that this could be a property of the bam file. A classification of the sort "Unkown", "Name Sorted", "Position sorted" sounds like a good idea to me. Regards, -Paul

1 0

Command line option
by Timothy Wu 10 Sep '11

10 Sep '11

Hi, I'm putting my own tool into Galaxy. Suppose I have an command line option that is designated by a switch. Such as $executable.py -a option input output The "-a option" is an optional argument. I want to make sure when the option is not given, "-a option" is entirely taken out. I tried Cheetah script like this (this fails): <command interpreter="python">executable #if $option != None -a $option #end if $input $output</command> This $option is suppose to be an optional tabular file and is written within <param> as: <param name="option" type="data" format="tabular" label="XXX" optional="true"/> Somehow "-a None" is always printed (as seen from the server print-to-screen). But I thought I checked explicitly for it to be anything but None. Then I realized that type($option) <class 'galaxy.tools.DatasetFilenameWrapper'>. So I coerced both that and NoneType into string prior to checking for equality. Is this the proper way to do it? Timothy

2 1

Exporting Galaxy Pages to another Galaxy server
by Kempenaar, M (med) 10 Sep '11

10 Sep '11

Hi, I was wondering if it is possible to export a Galaxy Page to another Galaxy server. Currently I'm using a development server and a production server and I have a Galaxy Page containing a manual with example data etc. that I'd like to make available on the production server. Also, since I sometimes distribute a virtual machine running galaxy to users (with instructions / scripts to keep software etc. up-to-date) and like to be able to share a page with them on their own server. I expect a Page to be in the database instead of a simple file on disk, so probably it requires database work. Thanks for any help! ________________________________ De inhoud van dit bericht is vertrouwelijk en alleen bestemd voor de geadresseerde(n). Anderen dan de geadresseerde(n) mogen geen gebruik maken van dit bericht, het niet openbaar maken of op enige wijze verspreiden of vermenigvuldigen. Het UMCG kan niet aansprakelijk gesteld worden voor een incomplete aankomst of vertraging van dit verzonden bericht. The contents of this message are confidential and only intended for the eyes of the addressee(s). Others than the addressee(s) are not allowed to use this message, to make it public or to distribute or multiply this message in any way. The UMCG cannot be held responsible for incomplete reception or delay of this transferred message.

2 1