December 2011 - galaxy-dev - lists.galaxyproject.org

Galaxy administrators - admin_users setting
by Peter Cock 30 May '12

30 May '12

Hi all, To make someone an administrator on a local Galaxy install, do I just need to add their email (login) to the comma separated setting admin_users in universe_wsgi.ini? https://bitbucket.org/galaxy/galaxy-central/wiki/Admin/AdminInterface I have this working on one server, but it doesn't seem to have any effect on a second server. Both are now running the current release (changeset 50e249442c5a). Is there some other setting needed to enable the admin interface? Thanks, Peter

2 4

how is metadata generated
by KOH Jia Yu Jayce 27 May '12

27 May '12

Hi Just wondering how metadata files in galaxy-dist/database/files/_metadata_files are generated? Are there any configurations in the xmls that specify these to be generated? Thanks

7 17

debugging jobs in 'new' state
by Shantanu Pavgi 10 May '12

10 May '12

We experienced an issue where some of the galaxy jobs were sitting in the 'new' state for a quite long time. They were not waiting for cluster resources to become available, but haven't been even queued up through DRMAA. We are currently using non-debug mode and following were my observations: * No indication of new jobs in paster.log file * database/pbs script didn't contain any associated job scripts * in backend database - job table contained their galaxy job id but no command_line input was recorded Also, not all the jobs are waiting in the 'new' state. Many jobs submitted after above waiting jobs got completed successfully on the cluster. Is there any job submission logic within galaxy which is being used for submitting jobs? Any clues on how to debug this issue will be really helpful. -- Thanks, Shantanu.

3 4

Error on local galaxy using SAM-to-BAM tool on a cluster
by Carlos Borroto 12 Apr '12

12 Apr '12

Hi, I'm running into this error: "Error sorting alignments from (/tmp/5800600.1.all.q/tmpXOc5mD/tmpAZCzt_), " When using SAM-to-BAM tool on a locally install Galaxy using a SGE cluster. I'm using the last version of galaxy-dist. I'm guessing I have a problem with the configuration for the tmp folder. I have this on "universe_wsgi.ini": # Temporary files are stored in this directory. new_file_path = /home/cborroto/galaxy_dist/database/tmp But I don't see this directory being used and from the error looks like /tmp in the node is used. I wonder if this is the problem, as I don't know if there is enough space in the local /tmp directory at the nodes? I ran the same tool in a subset of the same SAM file and it ran fine. Also, I see this in the description of the tool: "This tool uses the SAMTools toolkit to produce an indexed BAM file based on a sorted input SAM file." But what I actually need is to sort a SAM file output from bwa, I haven't found any other way than to converting it to BAM. Looking at "sam_to_bam.py" I see the BAM file will also be sorted. Would it be wrong to feed an unsorted SAM file into this tool? Finally, just to be sure there is nothing wrong with the initial SAM file, I ran "samtools view ..." and "samtools sort ..." on this file manually outside of Galaxy and it ran fine. Thanks in advance, Carlos

4 5

Defining Job Runners Dynamically
by John Chilton 05 Mar '12

05 Mar '12

Hello All, I just issued a pull request that augments Galaxy to allow defining job runners dynamically at runtime (https://bitbucket.org/galaxy/galaxy-central/pull-request/12/dynamic-job-run…) Whether it makes the cut or not, I thought I would describe enhancements here in case anyone else would find it useful. There a couple use cases we hope this will help us address for our institution - one is dynamically switching queues based on user (we have a very nice shared memory resource that can only be used by researchers with NIH funding) and the other is inspecting input sizes to give more accurate max walltimes to pbs (a small number of cufflinks jobs for instance take over three days on our cluster but defining max walltimes in excess of that for all jobs could result in our queue sitting idle around our monthly downtimes). You might also imagine using this to dynamically switch queues entirely based on input sizes or parameters, or alter queue priorities based on the submitting user or input sizes/parameters. There are two steps to use this - you must add a line in universe.ini and define a function to compute the true job runner string in the new file lib/galaxy/jobs/rules.py. This first step is similar to what you would do to statically assign a tool to a particular job runner. If you would like to dynamically assign a job runner for cufflinks you would start by adding a line like one of the following to universe.ini cufflinks = dynamic:///python -or- cufflinks = dynamic:///python/compute_runner If you use the first form, a function called cufflinks must be defined in rules.py, adding the extra argument after python/ lets you specify a particular function by name (compute_runner in this example). This second option could let you assign job runners with the same function for multiple tools. The only other step is to define a python function in rules.py that produces a string corresponding to a valid job runner such as "local:///" or "pbs:///queue/-l walltime=48:00:00/". If the functions defined in this file take in arguments, these arguments should have names from the follow list: job_wrapper, user_email, app, job, tool, tool_id, job_id, user. The plumbing will map these arguments to the implied galaxy object. For instance, job_wrapper is the JobWrapper instance for the job that gets passed to the job runner, user_email is the user's email address or None, app is the main application configuration object used throughout the code base that can be used for instance to get values defined in universe.ini, job, tool, and user are model objects, and job_id and tool_id the relevant ids. If you are writing a function that routes a certain list of users to a particular queue or increases their priority, you will probably only need to take in one argument - user_email. However, if you are going to look at input file sizes you may want to take in an argument called job and use the following piece of code to find the input size for input named "input1" in the tool xml. inp_data = dict( [ ( da.name, da.dataset ) for da in job.input_datasets ] ) inp_data.update( [ ( da.name, da.dataset ) for da in job.input_library_datasets ] ) input1_file = inp_data[ "input1" ].file_name input1_size = os.path.getsize( input1_file ) This whole concept works for a couple of small tests on my local machine, but there are certain aspects of the job runner code that makes me feel there may be corner cases I am not seeing where this approach may not work - so your millage may vary. -John ------------------------------------------------ John Chilton Software Developer University of Minnesota Supercomputing Institute Office: 612-625-0917 Cell: 612-226-9223 E-Mail: chilton(a)msi.umn.edu

3 2

bug: unsorted bam files and import into data library
by Florian Wagner 15 Feb '12

15 Feb '12

Hi, when a tool outputs an unsorted bam file, the indexing fails (quietly) and its metadata variable "bam_index" points to an inexistent file. This causes a nasty bug when trying to import the dataset into a data library and actually makes the library unusable unless you delete the broken entry from the - in my case Postgresql - database by hand. Are you working on it? Thanks, Florian

2 1

Garli and Tandem Repeats Finder wrappers
by Oleksandr Moskalenko 08 Feb '12

08 Feb '12

I would like to avoid reinventing the wheel if possible, so I wonder if anyone has wrappers for Garli (https://www.nescent.org/wg_garli/Main_Page) and Tandem Repeats Finder (http://tandem.bu.edu/trf/trf.html) they could share. Thanks, Alex

2 2

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?
by Jim Johnson 07 Feb '12

07 Feb '12

It is easiest to generate tools for galaxy when the applications or scripts can take arbitrarily named input files and generate output to given path names. Input directories, output directories are very convenient on the command line, but more of a challenge when crafting a galaxy tool. That said, many applications require a wrapper script to work with in galaxy. Thank you for the consistent script_info[] help/usage syntax in the qiime scripts, which enabled me to generate a skeleton galaxy tool_config file for each qiime script. I had some time last spring to work on integrating qiime into galaxy. Unfortunately, I haven't had any time since to work on this. I put those partial results on the Galaxy Tool Shed: http://toolshed.g2.bx.psu.edu/ There's a continuing effort at George Mason University to incorporate qiime into galaxy tools, so you may want to ask them what they need. I started by generating galaxy tool_config files, e.g. align_seqs.xml, by using python to get the script_info[] from the qiime script: $ cat generate_tool_config.bash #!/usr/bin/env bash python $1 > ${1%.*}.help cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" | python -i $1 -h > ${1%.*}.log (I'll attach tool_template.txt ) This generated skeleton tool_config .xml files that I could then edit as needed. ( http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax ) I originally was calling all qiime scripts from a tool wrapper: qiime_wrapper.py But, if a script can be called with any input filepaths and write its results to any filepaths, and only writes to STDERR when it fails, then you could call that script directly. When should you use a tool_wrapper or call the qiime script directly? Many of the qiime scripts could probably be called directly, especially if it can be called with arbitary input/output file pathnames. The reasons for using a tool wrapper may be if input/output needs to be manipulated, moved, renamed in order to be used by the qiime script. You'll also need a tool wrapper if the names or number of the output files can not be determined from the parameter settings. ( http://wiki.g2.bx.psu.edu/Admin/Tools/Multiple%20Output%20Files ) If your tool relies on a file ext to determine a format, you'll have to rename the input. ( Galaxy dataset pathnames will look something like: /<your_galaxy_file_path>/072/dataset_72931.dat ) The format/type of a dataset is stored in its metadata, so the tool_config can use that information, especially if a script can take muliple alternative input formats. A tool_wrapper can also be used to manage the stdout or stderr from a tool. Galaxy currently interprets any output on stderr as a failure. A couple changes in galaxy should make somethings easier than when I first attempted this: - galaxy now accepts dataset requests with sub directories. ( https://bitbucket.org/galaxy/galaxy-central/issue/494/support-sub-dirs-in-e… ) That means that output HTML files with links into sub directories can be left intact, with the html copied to the output dataset and the linked files to its "extra_files_path". - if you know the pathname of an output relative to the working directory, galaxy can copy it automatically to the output dataset using the from_work_dir attribute. ( see example in: https://bitbucket.org/galaxy/galaxy-central/src/21b645303c02/tools/ngs_rna/… ) Datatypes You may want to create new datatypes to make it easier for the user to correctly select inputs to a tool from previous outputs. For example, the qiime mapping file is a tabular file with specific requirements. I put a 'qiimemapping' datatype in lib/galaxy/datatypes/metagenomics.py and datatypes_conf.xml so an input could generate a select list containing only qiimemapping datasets rather than all tabular ones. Generating a configfile You can generate configfiles in the galaxy tool_config .xml file. The configfile is generated by the Cheetah interpreter just as the commandline is. see: alpha_rarefaction.xml The qiime_wrapper.py was patterned after the mothur_wrapper.py with some of the same wrapper params to handle run time determined output (perhaps not needed): --galaxy_datasets a comma separated list of regex:output_dataset the wrapper searches the working_dir and copies the file that matches the regex to the outout dataset if the exact pathname is known, use the "from_work_dir" attribute instead --galaxy_datasetid would be an output dataset id that would be used to dynamically create additional new datasets at job termination ( http://wiki.g2.bx.psu.edu/Admin/Tools/Multiple%20Output%20Files "Number of Output datasets cannot be determined until tool run") --galaxy_new_datasets a comma separated list of regex:datatype used to dynamically create additional new datasets at job termination --galaxy_new_files_path the galaxy dir for dynamically generated output datasets On 12/14/11 12:11 PM, Jose Carlos Clemente wrote: > Hi Jim, > > we just had a meeting to discuss ideas for a QIIME GUI, and were wondering how far did you manage to get with your plan to develop a QIIME wrapper for Galaxy. Do you have something working that we could take a look at? Were there any particular issues with the integration? We might have some bandwidth to work on this over next year, but thought it would be good to check with you first. > > Thanks, > Jose > > > On Mon, Feb 7, 2011 at 20:24, Rob Knight <Rob.Knight(a)colorado.edu <mailto:Rob.Knight@colorado.edu>> wrote: > > Hi Jim, > > Just to echo Greg's comments; it would be excellent to connect QIIME to Galaxy: let me know what is necessary from my end to ensure that this happens. > > Thanks, > Rob > > On Feb 7, 2011, at 12:30 PM, Greg Caporaso wrote: > >> Hi Jim, >> This is great! We're very enthusiastic about hooking up QIIME with Galaxy, and definitely encourage you to work on it. It's something we've discussed doing, but no one in our lab is actively working on it. One particular use case that I've been thinking about would be to set up Galaxy/QIIME in our QIIME EC2 images so our EC2 users could access their images via the Galaxy web server. Are you planning to release all of the xml files under an open source/free distribution model? >> >> I'm definitely interested in progress updates. Let me know if there are ways that we can help out, for example by giving you a list of what we believe are the most commonly used QIIME scripts so you can focus your efforts on those. I'm also cc'ing some of the folks in the lab here who are interested in Galaxy integration. >> >> Greg >> >> On Mon, Feb 7, 2011 at 10:36 AM, Jim Johnson <johns198(a)umn.edu <mailto:johns198@umn.edu>> wrote: >> >> Greg, >> >> Tim te Beek let me know that you have an interest in incorporating Qiime into Galaxy. I'm currently working on that in support of a grant at the University of Minnesota. I'll keep you apprised of my progress if you are interested. >> >> Thanks, >> >> JJ >> >> James Johnson >> University of Minnesota Supercomputing Institute >> 117 Pleasant St. SE >> Minneapolis, MN 55455 >> Email: jj(a)umn.edu <mailto:jj@umn.edu> >> >> >> >> -------- Original Message -------- >> Subject: Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy? >> Date: Mon, 07 Feb 2011 11:15:38 -0600 >> From: Jim Johnson <johns198(a)umn.edu> <mailto:johns198@umn.edu> >> Reply-To: jj(a)umn.edu <mailto:jj@umn.edu> >> Organization: MSI University of Minnesota >> To: Tim te Beek <tim.te.beek(a)nbic.nl> <mailto:tim.te.beek@nbic.nl> >> >> >> >> Tim, >> >> Sorry to hear we won't be doing this together. I'm finding some sporadic time to work on this. I also put my Mothur metagenomics wrapper code on http://community.g2.bx.psu.edu/ >> >> I looked at the initial work you did, and combined that with the script design ideas I used with Mothur. I'm writing a single script to call all the Qiime scripts, and generating a tool config (.xml) for each qiime script. Since the qiime scripts all contain a rather consistent script_info dictionary in the code, I wrote a script to generate an initial tool config from that script_info. I'm now starting the tedious process of editing those for each script. I'll attach the few I've started on. If you have any users that would able to beta test this when I have this more complete, please let me know. >> >> Thanks, >> >> JJ >> >> >> On 2/7/11 10:15 AM, Tim te Beek wrote: >>> Hi Jim, >>> >>> Sorry for the lack of feedback / updates on my part in converting QIIME to Galaxy. Following the initial difficulties with the conversion, the support project was put on hold pending further discussion. This morning the clients decided to forgo the Galaxy implementation, as their implementation would only ever be used by a single (technical) user, reducing the benefit of having a user friendly graphical interface. >>> >>> So I'm sorry to say I won't be able to contribute to the conversion, although your wrapper script did look like a viable solution to work around the unknown output file problems. Should you be able to complete the conversion yourself and be willing to share your implementation, I would still very much like to know. >>> >>> Perhaps the following could still be of some use to you: I contacted the QIIME development group about adding output file name arguments to the scripts, as can be seen here: >>> http://sourceforge.net/tracker/?func=detail&atid=1157167&aid=3164813&group_… <http://sourceforge.net/tracker/?func=detail&atid=1157167&aid=3164813&group_…> >>> Although it appears they wont be able to add output file name script parameters, they did say they're interested in Galaxy, and would be willing to help if there's a need for that. >>> >>> Additionally, we at NBIC do have a team that's more experienced at using Galaxy and converting tools to work in Galaxy, and if you want I could put you in contact with them about supporting this effort. In informal discussions they did seem to look favorable upon your solution, so perhaps that could be extended into a collaboration. >>> >>> Best regards, >>> Tim >>> >>> On Fri, Jan 14, 2011 at 20:01, Tim te Beek <tim.te.beek(a)nbic.nl <mailto:tim.te.beek@nbic.nl>> wrote: >>> >>> Ah ok, that's still fine. Thanks for the archive! I'll see what I can learn from how you've handled running command and handling in- & output for the non-trivial scripts, and apply that to the QIIME scripts. >>> Should you wish so I could get back to you if there's any significant new developments on my part in moving the QIIME scripts to Galaxy, other than the Trac / SVN changes. >>> >>> Have a good weekend! >>> Tim >>> >>> >>> On Fri, Jan 14, 2011 at 19:12, Jim Johnson <johns198(a)umn.edu <mailto:johns198@umn.edu>> wrote: >>> >>> I should probably be more clear, in that mothur and qiime overlap in functionality and in the algorithms they use. >>> They do use different data formats. >>> >>> >>> >>> On 1/14/11 12:02 PM, Tim te Beek wrote: >>>> Hi Jim, >>>> >>>> If you could send me the Mothur package, that would be great. If there's indeed as much overlap then it could be a great way to kick-start the QIIME pipeline conversion development. I can look into it starting monday, with hopefully nothing else to distract me the whole week. >>>> >>>> Best regards, >>>> Tim >>>> >>>> >>>> On Fri, Jan 14, 2011 at 18:49, Jim Johnson <johns198(a)umn.edu <mailto:johns198@umn.edu>> wrote: >>>> >>>> Tim, >>>> >>>> I just started looking at qiime myself. We are hoping to release a galaxy server to the University of Minnesota researchers in the next month, >>>> so there has been much to do in preparation. I'll take a look at what you've done. Thanks. >>>> >>>> I finished a galaxy interface to the Mothur metagenomics package http://www.mothur.org/wiki/Main_Page. It appears to have a lot of overlap with Qiime. >>>> I generated a number of datatypes, and made use of many of the tool config concepts that are available in galaxy. >>>> If you are interested, I can send that to you: mothur.tar.gz (55108 bytes). I hope to submit it tohttp://community.g2.bx.psu.edu/ in the next month. >>>> >>>> Regards, >>>> >>>> JJ >>>> >>>> >>>> On 1/14/11 9:52 AM, Tim te Beek wrote: >>>>> Hi Jim, >>>>> >>>>> Sorry to only respond just now: I've been tied up in other projects, so have not been able to get any work done on this up to today. >>>>> >>>>> What I currently have is not much, since my progress has been slowed by my unfamiliarity with both Galaxy and the QIIME tools (I'm working on this as an outsider for a small support project). But what I do have I've made available through the following Trac / SVN instance: >>>>> _ >>>>> _ >>>>> >>>>> https://trac.nbic.nl/brs2010p26/browser/trunk/galaxy_dist/tools/qiime >>>>> >>>>> >>>>> These configuration files so far can run: >>>>> >>>>> * check_id_map >>>>> * split_libraries >>>>> * pick_otus_through_otu_table >>>>> >>>>> For the first two the output is correctly directed to the Galaxy history, the last one generates quite a large set of output files which might have to be made available otherwise (the documentation on this gives a few options: https://bitbucket.org/galaxy/galaxy-central/wiki/ToolsMultipleOutput)) >>>>> >>>>> I had some trouble working around the fact that most QIIME scripts do not allow one to specify output filenames on the command line, which seems to be a requirement for Galaxy to import files into the history pane. What I've done is written a bash file to accompany each XML file, that expects as first command line argument the command that should be evaluated, and as additional arguments the output filenames expected by Galaxy. After running a tool I then move whatever output files were created to the paths expected by Galaxy. I must say I'm not (at all) sure if this current approach is something to recommend, or if better alternatives are available. Any feedback on this would be highly appreciated. >>>>> An additional source of pain was that the split_libraries script requires specific file extensions for it's fasta & quality files, which required passing them separately and symbolic linking them as .fna & .qual files. >>>>> >>>>> So far I've not taken the time to introduce new QIIME specific data types, but this could be something to consider given the large amount of 'tabular' input and output files. >>>>> >>>>> I hope this email and the scripts linked above can still be of some use to you (or vice-versa), but in either case any feedback or collaboration would be much appreciated. >>>>> >>>>> Best regards, >>>>> Tim >>>>> >>>>> >>>>> On Wed, Nov 24, 2010 at 15:48, Jim Johnson <ctoftey(a)comcast.net <mailto:ctoftey@comcast.net>> wrote: >>>>> > Tim, >>>>> > >>>>> > I hope to also look at incorporating QIIME into our local Galaxy instance at >>>>> > the University of Minnesota, but probably won't be able to start for a >>>>> > couple weeks. It would be good to develop that in coordination with others. >>>>> > >>>>> > I just finished incorporating "Mothur" metagenomics suite >>>>> > http://www.mothur.org/ (Dr. Patrick Schloss, Department of Microbiology & >>>>> > Immunology at The University of Michigan) into our Galaxy server at the >>>>> > University of Minnesota. I hope to contribute that to >>>>> > http://community.g2.bx.psu.edu/ after some testing by our researchers. If >>>>> > the Galaxy wrappings for Mothur are of any interest to you, I can send you a >>>>> > copy any time. >>>>> > >>>>> > Thanks, >>>>> > >>>>> > JJ >>>>> > >>>>> > James E Johnson >>>>> > Minnesota Supercomputing Institute >>>>> > University of Minnesota >>>>> > >>>>> > >>>>> > On Nov 23, 2010 at 07:22 AM, Tim te Beek wrote: >>>>> > >>>>> >> Hello all, >>>>> >> >>>>> >> Is anyone aware of any existing efforts to port the QIIME sequencing >>>>> >> pipeline (http://qiime.sourceforge.net/) to Galaxy? I would like to run >>>>> >> QIIME analyses through Galaxy to get better control of >>>>> >> intermediate processing steps, but before I start to convert (a subset of) >>>>> >> some 90 scripts, I'd first like to make sure this has not been done before >>>>> >> by anyone willing to share their work. >>>>> >> >>>>> >> So: has anyone converted the QIIME pipeline to Galaxy before, and would >>>>> >> they >>>>> >> be willing to share their scripts? >>>>> >> >>>>> >> Best regards, >>>>> >> Tim >>>>> > >>>>> > _______________________________________________ >>>>> > galaxy-dev mailing list >>>>> > galaxy-dev(a)lists.bx.psu.edu <mailto:galaxy-dev@lists.bx.psu.edu> >>>>> > http://lists.bx.psu.edu/listinfo/galaxy-dev >>>>> > >>>>> >>>> >>> >>> >>> >> >> >> > >

8 13

Galaxy Wiki
by John Major 01 Feb '12

01 Feb '12

Galaxy team- I've tried a few times to add some content to the FAQ, but the captcha questions keep failing, or i'm submitting the wrong answer. I'd suggest a slightly less burdensome barrier to contributing- i'm certain this is dialing down your user contributions significantly. John

4 3

Moving Galaxy's installed folder
by Peter Cock 01 Feb '12

01 Feb '12

Hi all, Is there any official guidance on relocating a working Galaxy installation? Ours is currently up and running using PostreSQL, with everything installed under /opt/galaxy-dist I want to move this under /mnt/galaxy (a RAID partition which will be shared with a cluster). What needs to be done to make this move? In particular, I am worried there may be some absolute paths in the database which would need updating. In principle if only relative paths are used, everything will be fine... Any advice? Thanks, Peter

2 6