December 2009 - galaxy-dev - lists.galaxyproject.org

Fwd: how to add genome build?
by Yuri Bendana 09 Dec '09

09 Dec '09

---------- Forwarded message ---------- From: Yuri Bendana <yrbendana(a)gmail.com> Date: Wed, Dec 9, 2009 at 2:39 PM Subject: Re: [galaxy-dev] how to add genome build? To: Greg Von Kuster <greg(a)bx.psu.edu> Hi Greg, I modified builds.txt for now and added: mt3 Medicago truncatula v3 It now appears in the dropdown, great! I also have this in faseq.loc: mt3 /opt/local/var/genome/mt3 This is the directory where my genome fasta files are located. However when I try to get sequence from the genome using the Extract Genomic DNA tool, I get this error: "Sequences are not currently available for the specified build." No other errors appear in paster.log. What am I missing? thanks, yuri On Tue, Dec 8, 2009 at 10:40 AM, Greg Von Kuster <greg(a)bx.psu.edu> wrote: > Hello Yuri, > > You are correct, new genomes can be specified by modifing the > ${GALAXY_ROOT}/tool-data/shared/ucsc/builds.txt. This file is generated > using a script found at ${GALAXY_ROOT}/cron/updateucsc.sh. If this script > is executed on a normal basis, it will be useful to instead add the required > information to ${GALAXY_ROOT}/tool-data/shared/ucsc/manual_builds.txt before > running the script (otherwise each time the script is run, the changes to > build.txt will be lost). See ${GALAXY_ROOT}/cron/README.txt for more > information as well as the headers of each data file. > > > > On Dec 2, 2009, at 9:54 PM, Yuri Bendana wrote: > > I recently installed Galaxy and am pleasantly surprised how easy it is to >> get up and running. The screencasts and wiki are very helpful. One thing I >> can't figure out is how to add a Medicago genome to the Database/Build >> dropdown that appears when I try to edit my query attributes (it looks like >> the same builds appear in the Genome dropdown on the Get Data->Upload page). >> I thought adding the genome directory to the tools-data/faseq.loc file >> might work but doesn't seem to. I noticed that the build list displayed is >> stored in tool-data/shared/ucsc/builds.txt and it looks like it gets updated >> by a cron script. I would like to use the Fetch Sequences -> Extract Genomic >> DNA tool on my local genomes. >> >> thanks, >> >> yuri >> _______________________________________________ >> galaxy-dev mailing list >> >> galaxy-dev(a)lists.bx.psu.edu >> http://lists.bx.psu.edu/listinfo/galaxy-dev >> > > Greg Von Kuster > Galaxy Development Team > greg(a)bx.psu.edu > > > >

1 0

Re: [galaxy-dev] External data source config
by Greg Von Kuster 09 Dec '09

09 Dec '09

I'm not quite sure how useful it will be to connect your external data source to Galaxy sInce you state: "The external data source's URL requires no parameters, and when the data is returned, it's just a URL directly to a file." It would be better if it could stream the data over HTTP to Galaxy. In any case, the parameters I describe are not contained in a <param> tag at the Galaxy end, but the request parameters generated by your external data source application which are sent to Galaxy so that it can post the request back to your data source. Have you read our wiki at http://bitbucket.org/galaxy/galaxy- central/wiki/DataSources? On Dec 9, 2009, at 2:10 PM, Chris Zaleski wrote: > Greg, > > Thanks for the quick response. Forgive me for not (yet) having much > Galaxy experience. Unfortunately there's still a concept I'm > missing... For my simple example, could you please give an example > of what would be the "value" attribute of the <param> tag? > > Thanks again, > Chris > > > > On Wed, Dec 9, 2009 at 1:51 PM, Greg Von Kuster <greg(a)bx.psu.edu> > wrote: > Chris, you are correct in thinking that data_source.py is the only > executable needed for this. > > Your tool config ( the .xml file for your new tool ) will include > "name", "id" and "tool_type" attributes in the tool tag like the > following: > > <tool name="Flymine" id="flymine" tool_type="data_source"> > > The Toolbox class in ~/tools/__init__.py includes the following > method which parses this information when the tool is loaded: > > def load_tool( elem, panel_dict ): > try: > path = elem.get( "file" ) > tool = self.load_tool( os.path.join > ( self.tool_root_dir, path ) ) > self.tools_by_id[ tool.id ] = tool > key = 'tool_' + tool.id > panel_dict[ key ] = tool > log.debug( "Loaded tool: %s %s" % ( tool.id, > tool.version ) ) > except: > log.exception( "error reading tool from path: %s" % > path ) > > So when Galaxy receives the post from the external data source, it > will know which tool to execute with the param values received in > the request as long as the "/tool_runner?tool_id=XXX" is included > in the request. > > > On Dec 9, 2009, at 12:22 PM, Chris Zaleski wrote: > >> Greetings, >> >> I have a question about setting up an external data source. My >> case should be fairly simple - The external data source's URL >> requires no parameters, and when the data is returned, it's just a >> URL directly to a file. Example: >> http://myserver/data_search_page.html >> returns... >> http://myserver/repository/the_file.bed >> >> I've looked at multiple examples of the XML files & the wiki, and >> I think I've got it. However there's one part I'm still unclear >> about - the <param> tag inside the <inputs> tag. Examples: >> <param name="GALAXY_URL" type="baseurl" value="/tool_runner/ >> biomart" /> >> <param name="GALAXY_URL" type="baseurl" value="/tool_runner" /> >> <param name="GALAXY_URL" type="baseurl" value="/tool_runner? >> tool_id=flymine" /> >> >> It seems that the "value" string gets appended onto the GALAXY_URL >> (which would then be returned to Galaxy). Could you please explain >> what's happening here? Do I need to write a separate tool which >> would be executed by 'tool_runner'? >> I thought "data_source.py" *was* the tool, no? >> >> Thanks very much for your help, >> Chris Zaleski >> Gingeras Lab - CSHL >> _______________________________________________ >> galaxy-dev mailing list >> galaxy-dev(a)lists.bx.psu.edu >> http://lists.bx.psu.edu/listinfo/galaxy-dev > > Greg Von Kuster > Galaxy Development Team > greg(a)bx.psu.edu > > > > Greg Von Kuster Galaxy Development Team greg(a)bx.psu.edu

1 0

External data source config
by Chris Zaleski 09 Dec '09

09 Dec '09

Greetings, I have a question about setting up an external data source. My case should be fairly simple - The external data source's URL requires no parameters, and when the data is returned, it's just a URL directly to a file. Example: http://myserver/data_search_page.html returns... http://myserver/repository/the_file.bed I've looked at multiple examples of the XML files & the wiki, and I think I've got it. However there's one part I'm still unclear about - the <param> tag inside the <inputs> tag. Examples: <param name="GALAXY_URL" type="baseurl" value="/tool_runner/biomart" /> <param name="GALAXY_URL" type="baseurl" value="/tool_runner" /> <param name="GALAXY_URL" type="baseurl" value="/tool_runner?tool_id=flymine" /> It seems that the "value" string gets appended onto the GALAXY_URL (which would then be returned to Galaxy). Could you please explain what's happening here? Do I need to write a separate tool which would be executed by 'tool_runner'? I thought "data_source.py" *was* the tool, no? Thanks very much for your help, Chris Zaleski Gingeras Lab - CSHL

2 1

Re: [galaxy-dev] [galaxy-bugs] Galaxy Question
by Kanwei Li 08 Dec '09

08 Dec '09

Dear Jens, You may edit the header in the galaxy-central/templates/base_panels.mako file. You can use regular HTML or see the existing code for formatting. However, if this file is modified in the repository in the future, you will have to revert and reapply your changes. Thanks, Kanwei On Tue, Dec 8, 2009 at 4:17 PM, Nate Coraor <nate(a)bx.psu.edu> wrote: > Jens Lichtenberg wrote: >> >> Hello Nate, >> >> I have a question regarding Galaxy. We would like to add a publication >> section to our instance and I would like to add it into the top header bar >> of the interface (among Libraries, Workflow, User, Help, etc.). Am I allowed >> to change that behavior? Where would I need to start to examine the source >> to be able to add to that navigation? >> >> Thank you so much for your help, >> >> Jens Lichtenberg > > Hi Jens, > > I just noticed that you sent this directly to me. For the most > accurate/prompt responses it's usually a good idea to send these requests to > the mailing lists: galaxy-user for tool and interface usage questions, and > galaxy-dev for everything else (including this question). > > I don't believe this is easily customizable, but hopefully someone else on > the team can provide the details. > > --nate > _______________________________________________ > galaxy-bugs mailing list > galaxy-bugs(a)lists.bx.psu.edu > http://lists.bx.psu.edu/listinfo/galaxy-bugs >

1 0

How to extract Job List
by Raghav Bhatnagar 08 Dec '09

08 Dec '09

Hi Galaxy Team I have been working to extract the job list before it is executed into Galaxy. I came to knwo that when we press the Run Workflow button, the following code is executed. Can you please let me know which line inserts code into the jobs table and how do I find that? outputs = odict() for i, step in enumerate( workflow.steps ): if step.type == 'tool' or step.type is None: tool = trans.app.toolbox.tools_by_id[ step.tool_id ] input_values = step.state.inputs # Connect up def callback( input, value, prefixed_name, prefixed_label ): if isinstance( input, DataToolParameter ): if prefixed_name in step.input_connections_by_name: conn = step.input_connections_by_name[ prefixed_name ] return outputs[ conn.output_step.id ][ conn.output_name ] visit_input_values( tool.inputs, step.state.inputs, callback ) # Execute it outputs[ step.id ] = tool.execute( trans, step.state.inputs ) else: outputs[ step.id ] = step.module.execute( trans, step.state ) -- Raghav Bhatnagar Graduate Student (CS) Viterbi School of Engineering University of Southern California

2 2

Could use some help with adding own perl scripts
by Michel P. Villerius 08 Dec '09

08 Dec '09

Hi, I recently started with galaxy, I got the : Galaxy build: $Rev 1733:a4214de3752e$ Now I need to add my own perl scipts to galaxy. This all goes fine, but there are some problems with the galaxy output look at the attachment: job 21 has an output in the white field, unlike job 40, I just can't seem to find out why? here are the xml's #######21############# <tool id="fast q 2 fasta " name="fastq_2_fasta"> <description>for each sequence in a file</description> <command interpreter="perl">fq2fa.pl $input $output</command> <inputs> <param format="fastq" name="input" type="data" label="Source file"/> </inputs> <outputs> <data format="fasta" name="output" /> </outputs> <tests> <test> <param name="input" value="fa_gc_content_input.fa"/> <output name="out_file1" file="fa_gc_content_output.txt"/> </test> </tests> <help> This tool computes GC content from a FASTA file. </help> </tool> ############################# ###########40################ <tool id="GAPSS_FASTA_to_FASTQ" name="GAPSS - FASTA to FASTQ"> <description>converter</description> <command interpreter="perl">GAPSS_FASTA2FASTQ.v2.pl $input $score $output</command> <inputs> <param format="fasta" name="input" type="data" label="FASTA File to convert" /> <param name="score" type="text" value="a" label="score" help="Use a if unsure." /> </inputs> <outputs> <data name="output" format="fastq" type="data" /> </outputs> <help> **What it does** This tool converts data from FASTQ format to FASTA format. #run as: perl GAPSS_FASTA2FASTQ.pl "FASTA file" "score to use or blank" #this script converts a FASTA file over to FASTQ #quality scores are by default Sanger, but can be adjusted #notes for user: # best sanger score = # best Solexa score = #input: FASTA file, and quality score to use #output: FASTQ file This tool is based on `GAPSS` by Matt and Michiel. http://www.lgtc.nl/GAPSS/ </help> </tool> ################ The perl script That needs to be added is for job 40 and looks like: ########PERl script job 40 ########### #!/usr/bin/perl #run as: perl GAPSS_FASTA2FASTQ.pl "FASTA file" "score to use or blank" #this script converts a FASTA file over to FASTQ #quality scores are by default Sanger, but can be adjusted #notes for user: # best sanger score = # best Solexa score = #input: FASTA file, and quality score to use #output: FASTQ file my $file_to_convert = $ARGV[0]; my $fastq_output_file_name = $file_to_convert.".fastq"; my $value_for_quality_score = $ARGV[1]; #set QC value to Sanger best () if no value given $value_for_quality_score =~ s/\n//; if ($value_for_quality_score eq ""){ $value_for_quality_score="a"; } #check is fasta format my $line1to4_fasta = `head -n 4 $file_to_convert`; my @array_line1to4_fasta=split(/\n/,$line1to4_fasta); if ( (@array_line1to4_fasta[0]=~/^>/) && (@array_line1to4_fasta[1]=~/[TAGCtagcNn]*/) && (@array_line1to4_fasta[2]=~/^>/) && (@array_line1to4_fasta[3]=~/[TAGCtagcNn]*/) ){}else{die "$file_to_convert is not a proper fasta file, terminated analysis\n";} #open file, open outfile, convert and print output, close infile, close outfile open(fasta_file,$file_to_convert)||die "could not open fasta file $file_to_convert\n"; open(fastq_file,">$fastq_output_file_name")||die "could not open fastq file $fastq_output_file_name to print to\n"; while (<fasta_file>){ chop; $_ =~ s/\n//; if ($_ =~ /^>/){ $_ =~ s/^>/@/;print fastq_file "$_\n"; }else{ print fastq_file "$_\n"; print fastq_file "+\n"; my $quality_line = $_; $quality_line =~ s/[TAGCtagcNn]/$value_for_quality_score/g; print fastq_file "$quality_line\n"; } } close (fasta_file)||die "could not close fasta file $file_to_convert \n"; close (fastq_file)||die "could not close fastq file $fastq_output_file_name\n"; ############################################ It just reformats a file and writes direct a file with .fastq behind it. I know there are other tools in galaxy but I just want mine to be put in and I just cant seem to get it to work. Can you give me some advise on the input and output parameters and how I control the white screen in galaxy, I just want the file that is written to be accessed by galaxy not the data displayed in the white screen, it will be for next generation sequencing and there wille be lots of fastq lines there. Thanks in advance. //Michel Villerius Human and Clinical Genetics Leiden University Medical Center

3 6

how to add genome build?
by Yuri Bendana 08 Dec '09

08 Dec '09

I recently installed Galaxy and am pleasantly surprised how easy it is to get up and running. The screencasts and wiki are very helpful. One thing I can't figure out is how to add a Medicago genome to the Database/Build dropdown that appears when I try to edit my query attributes (it looks like the same builds appear in the Genome dropdown on the Get Data->Upload page). I thought adding the genome directory to the tools-data/faseq.loc file might work but doesn't seem to. I noticed that the build list displayed is stored in tool-data/shared/ucsc/builds.txt and it looks like it gets updated by a cron script. I would like to use the Fetch Sequences -> Extract Genomic DNA tool on my local genomes. thanks, yuri

2 2

cluster problems
by Juan Carlos Perin 08 Dec '09

08 Dec '09

I have my Galaxy instance configured using Torque PBS with a MySQL backend. Everything works pretty well, although I've recently been getting errors when I attempt to submit more than 2 workflows. These workflows are very simple. They go from a fastq file, to a converter, then get mapped and converted to bam format. Occasionally I'll have a job fail, and never at the same point in the process, that says" "An error occurred running this job: unable to run this job due to a cluster error" I'm not sure where to even begin looking for cluster problems, it seems most output gets wiped out pretty quickly within galaxy. Re-running the job typically works fine, but since this is occurring in the middle of a workflow its rather disruptive. Any suggestions on where to look for problems would be much appreciated. Also, I assume changing the log_level value in universe_wsgi.ini to something other than DEBUG might help? what are the level values galaxy uses? I can't seem to find any notes on this anywhere. Thanks! Juan Perin

2 1

does galaxy work with rpy2?
by Dean Snyder 08 Dec '09

08 Dec '09

Can the latest Galaxy use RPy2 to access R via Python? If not can I have both RPy and RPy2 installed on my Mac at the same time and Galaxy still do the right thing? Dean A. Snyder Senior Programmer/Analyst Center for Inherited Disease Research (CIDR) Johns Hopkins School of Medicine Bayview Research Campus 333 Cassell Dr, Triad Bldg, Suite 2000 Baltimore, MD 21224 www.cidr.jhmi.edu cell:717 668-3048 office:410-550-4629

2 1

Easier integration of fastx-toolkit
by Assaf Gordon 08 Dec '09

08 Dec '09

Hello all, I'm trying to make the integration of the fastx-tools easier for new installations of galaxy, I'd appreciate any comments or suggestions you might have. Steps so far: 1. new version that compiles cleanly on Ubuntu,CentOS,OpenSolaris,FreeBSD 2. detailed compilation instructions for each operating system 3. pre-compiled binaries for Linux, FreeBSD, OpenSolaris The above are available here: http://cancan.cshl.edu/labmembers/gordon/fastx_toolkit/download_new.html Remains to be done: 1. Do the same for MacOS-X 2. Update the XML files, probably remove "zcat" entirely. 3. rewrite the instructions for "galaxy integration" - this is not required any more, because the XML files are now part of the galaxy repository. Please let me know if you think of anything else. thanks, -gordon

3 2