May 2012 - galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] Galaxy 'environment' variables
by Joachim Jacob 24 May '12

24 May '12

Dear Jean-François, But thanks for asking again, because now I had to look things up again, and now I found it. :-) Helping other people is apparently a strong drive to learn. The only variables I have found, are following variables below (See alsohttp://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax?highlight=…) You can use them in the tool definition file as shown in example below. $GALAXY_HOME : /home/galaxy/galaxy-dist ${GALAXY_DATA_INDEX_DIR} :$GALAXY_HOME/tool-data $__new_file_path__ : universe_wsgi.ini's new_file_path --> temporary directory $__tool_data_path__ : tool_data_path value $__root_dir__ Top-level Galaxy source directory $__datatypes_config__ universe_wsgi.ini's datatypes_config value $__user_id__ Email's numeric ID $__user_email__ User's email address $__app__ (don't know about this one, seems to be deprecated) <http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax?highlight=%28m…> An example of what I like to do: I work with a 'configfile' to pass all parameters and galaxy settings to my tool/wrapper. *Tool definition file:* <tool ....> <command interpreter='perl'> my_program.pl $configfile </command> <inputs> ... configure here the inputs ... </inputs> <configfiles> <configfile name="$configfile"> param1==$param1 param2==$param2 ... ##Galaxy Environment Variables tempdir==${__new_file_path__}* *galaxyhome==${GALAXY_HOME} </configfile> </configfiles> <outputs> ... </outputs> <tests> </tests> </tool> *An my program 'my_program.pl' starts like:* #!/usr/bin/perl use strict; #------------ PARSE GALAXY ARGUMENTS # Get file with program parameters and Galaxy settings #---------------------------------------------------------------------------# my $configfile =$ARGV[0]; #------------- PARSE PARAMETERS # Parse parameters from config file #---------------------------------------------------------------------------## my (%para); open(CONFIG,"<$configfile"); while (<CONFIG>) { if (/(\S+)==(.+)$/){ $para{ $1 } = $2 ; } } close(CONFIG); exit 0; So now I have all parameters available in my program, in the hash %para, e.g. $para{'tempdir'}. Quick and easy! I hope this is clear, if not, just ask. Good luck with your implementation, Joachim Joachim Jacob, PhD Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib On 05/23/2012 08:26 PM, Jean-Francois Payotte wrote: > Dear Joachim, > > I am interested at the exact same thing, such as finding a list of > 'environment' variables in Galaxy. > Did you ever got your answer? > > Best regards, > Jean-François > > > > *[galaxy-dev] Galaxy 'environment' variables* > > Joachim Jacob > Wed, 11 Apr 2012 08:18:46 -0700 > Hello all, > > Where can I find a list of 'environment' variables in Galaxy, such as: > $GALAXY_HOME > ${GALAXY_DATA_INDEX_DIR} > > > More specifically, I am looking for the temporary directory setting of > a user's Galaxy (as set in the universe_wsgi.ini). > > > Kind regards, > Joachim > > -- > Joachim Jacob, PhD > > Rijvisschestraat 120, 9052 Zwijnaarde > Tel: +32 9 244.66.34 > Bioinformatics Training and Services (BITS) > _http://www.bits.vib.be_ <http://www.bits.vib.be/> > @bitsatvib >

1 0

Using workflows as components in workflows?
by Peter van Heusden 24 May '12

24 May '12

Hi there Is there a way in Galaxy to use workflows as components in workflows? I've got two datasets that start as Genbank files and are processed (features extracted, translated to AA) by a small workflow, and then BLASTed against one another. Ideally I'd like to modularise the initial steps as a workflow so that I can say input data goes through this workflow and then the result is fed into either the query or the subject slot of blastp. So is this possible? Thanks! Peter

4 6

upload from local filesysem
by J. Greenbaum 23 May '12

23 May '12

It seems that this topic has been covered at least several times and I've found this thread to be helpful: http://user.list.galaxyproject.org/Speed-up-uploading-into-local-Galaxy-ter… However, as far as I understand, this will only allow the Admin to do the upload from a local disk. Has anyone had success allowing a regular user to upload from a local disk? This seems like a common issue, particularly with large sequencing files. It would be great to be able to add this as the first step in a workflow. Any info would be appreciated. Thanks, Jason -- Jason Greenbaum, Ph.D. Manager, Bioinformatics Core | jgbaum(a)liai.org La Jolla Institute for Allergy and Immunology

3 3

Create Trackster visualizations via API
by Ayton Meintjes 23 May '12

23 May '12

Is it possible to create visualizations via the API yet? Can't seem to find an example. Ayton

2 1

specifying nodes/processors with PBS
by J. Greenbaum 23 May '12

23 May '12

Hi, I've successfully configured a local galaxy instance to submit jobs to our local cluster, but have a question regarding setting PBS options such as the number of nodes and processors to use. I've followed the guide here: http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Cluster?highlight=%28pbs… Is there any way to do this in a tool-specific manner. For example, for running tophat to do a mapping run, I would want it to use 24 CPUs, but for simply converting a file type I would want only 1 CPU requested. Is there a way to specify this somewhere? Thanks, Jason -- Jason Greenbaum, Ph.D. Manager, Bioinformatics Core | jgbaum(a)liai.org La Jolla Institute for Allergy and Immunology

2 2

cannot access conditional parameter in command
by Anne Pajon 23 May '12

23 May '12

Hi, I'm writing a galaxy tool xml file for an in house existing R script. I have a conditional parameter set for selecting the dataset to work with but I would like to access the path of the file from the data table I've created: <table name="cri_datasets" comment_char="#"> <columns>value, name, path</columns> <file path="tool-data/cri/datasets.loc" /> </table> Here is the beginning of the xml file: <command interpreter="R --vanilla --slave -f"> rp.R --args $dataset.name.fields.name $dataset.name.fields.path $input $column $output_txt $output_html $output_html.files_path $dataset.outcome $dataset.receptor $dataset.treatment $dataset.er $dataset.age $dataset.grade $dataset.signature $dataset.margins $dataset.groups $dataset.gleason $dataset.tstage 2>stderr || cat stderr 1>&2 </command> <inputs> <param name="input" type="data" format="tabular,txt" label="Source file" help="File containing a list of gene symbols"/> <param name="column" type="data_column" data_ref="input" numerical="False" label="Column" help="Column containing gene symbols (only available for tabular input file)" /> <conditional name="dataset"> <param name="name" type="select" label="Dataset"> <options from_data_table="cri_datasets" /> </param> ... but when running the tool, here is the error message I get: Traceback (most recent call last): File "/opt/local/home/webapp/galaxy/lib/galaxy/jobs/runners/local.py", line 59, in run_job job_wrapper.prepare() File "/opt/local/home/webapp/galaxy/lib/galaxy/jobs/__init__.py", line 429, in prepare self.command_line = self.tool.build_command_line( param_dict ) File "/opt/local/home/webapp/galaxy/lib/galaxy/tools/__init__.py", line 1971, in build_command_line command_line = fill_template( self.command, context=param_dict ) File "/opt/local/home/webapp/galaxy/lib/galaxy/util/template.py", line 9, in fill_template return str( Template( source=template_text, searchList=[context] ) ) File "/opt/local/home/webapp/galaxy/eggs/Cheetah-2.2.2-py2.7-linux-x86_64-ucs2.egg/Cheetah/Template.py", line 1004, in __str__ return getattr(self, mainMethName)() File "DynamicallyCompiledCheetahTemplate.py", line 83, in respond NotFound: cannot find 'fields' while searching for 'dataset.name.fields.name' Is it possible to access the other columns of a loc file? It seems possible based of what I read on the wiki http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax#A.3Coptions.3E… using from_data_table and syntax of ${param.fields.path} but I could not manage to make it work. Any help would be very much appreciated. Thanks! Kind regards, Anne. -- Anne Pajon, Ph.D. Cancer Research UK - Cambridge Research Institute Li Ka Shing Centre, Robinson Way, Cambridge CB2 0RE anne.pajon(a)cancer.org.uk | +44 (0)1223 404 334 NOTICE AND DISCLAIMER This e-mail (including any attachments) is intended for the above-named person(s). If you are not the intended recipient, notify the sender immediately, delete this email from your system and do not disclose or use for any purpose. We may monitor all incoming and outgoing emails in line with current legislation. We have taken steps to ensure that this email and attachments are free from any virus, but it remains your responsibility to ensure that viruses do not adversely affect you. Cancer Research UK Registered in England and Wales Company Registered Number: 4325234. Registered Charity Number: 1089464 and Scotland SC041666 Registered Office Address: Angel Building, 407 St John Street, London EC1V 4AD.

1 1

Pushing changes upstream to Galaxy
by Peter van Heusden 23 May '12

23 May '12

Hi there As I adapt Galaxy to the needs of the particular workflows I'm implementing, I invariably end up "tweaking" the existing code, especially the tool definitions and the datatypes config file. These tweaks are often minor - e.g. adding an extra parameter to Ross Lazarus' ClustalW's code, changing the way the binary is called (e.g. clustalw2 on ubuntu is installed as clustalw, BLAST+ is installed single threaded so the -num_threads option in the ncbi_blastn_wrapper.xml doesn't work), but a) Sometimes they make sense to share with the rest of the Galaxy community b) local changes make it more tricky to keep in sync with upstream changes to Galaxy What's the best procedure for proposing changes up upstream galaxy? Posting patches to galaxy-dev? Is there some mercurial magic that can help? Thanks, Peter

2 1

LateValidationError in Extract features
by Peter van Heusden 23 May '12

23 May '12

Hi there I've got a workflow that uses BioPerl's bp_genbank2gff3 to convert Genbank to GFF3, then hands the GFF3 to Extract features to filter to only genes, before moving on. The workflow JSON is at http://pastebin.com/zHWsC6YT. Step 4 - the Genbank2GFF runs fine, and if I view the output in the history I can see the resulting GFF, but step 8 - the Extract features, fails with a LateValidationError, as follows: Traceback (most recent call last): File "/net/datasrv3.sanbi.ac.za/datastore/cip0/software/galaxy-dev/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 144, in queue_job job_wrapper.prepare() File "/net/datasrv3.sanbi.ac.za/datastore/cip0/software/galaxy-dev/galaxy-dist/lib/galaxy/jobs/__init__.py", line 130, in prepare self.tool.handle_unvalidated_param_values( incoming, self.app ) File "/net/datasrv3.sanbi.ac.za/datastore/cip0/software/galaxy-dev/galaxy-dist/lib/galaxy/tools/__init__.py", line 1872, in handle_unvalidated_param_values self.handle_unvalidated_param_values_helper( self.inputs, input_values, app ) File "/net/datasrv3.sanbi.ac.za/datastore/cip0/software/galaxy-dev/galaxy-dist/lib/galaxy/tools/__init__.py", line 1890, in handle_unvalidated_param_values_helper self.handle_unvalidated_param_values_helper( input.cases[current].inputs, values, app, context, prefix ) File "/net/datasrv3.sanbi.ac.za/datastore/cip0/software/galaxy-dev/galaxy-dist/lib/galaxy/tools/__init__.py", line 1912, in handle_unvalidated_param_values_helper raise LateValidationError( message ) LateValidationError This Galaxy is Postgres and SGE, with galaxy-dist as of last night. Another strange thing is that while I select that the output of step 4 should be visible, it is, in fact, hidden in the new history. How to debug this? How can I see which input is not validating? Thanks, Peter

1 0

Uploading problem
by julie dubois 23 May '12

23 May '12

Hi, I test a production environnement of Galaxy with apache proxy and mysql database. It work, but when I upload a file (it's not very big : 460 Mo), Galaxy says that dataset is uploading and it has runned since yesterday and it's not finish! Where can the problem be ? Is it an installation problem or a material limitation : it's run in a machine with 8 Go RAM and 4 cores processor. Thank you Julie

4 5

Load Files in local instance Ubuntu 12.04 LTS /tmp as /tmpfs
by Luciano Cosme 22 May '12

22 May '12

Hi, I decided to format my PC and reinstall everything. I have Ubuntu 12.04LTS installed in one 1TB hard drive (/ and swap) and the /home folder installed in a 3TB hard drive. I created a /tmp partition of 110GB and a /tmp partion of 110GB, both in the 1TB HD. Then I mounted /tmp as /tmpfs using fstab. Here is the df -h output: Filesystem Size Used Avail Use% Mounted on /dev/sda2 817G 3.9G 771G 1% / udev 12G 4.0K 12G 1% /dev tmpfs 100G 16K 100G 1% /tmp tmpfs 4.8G 992K 4.8G 1% /run none 5.0M 0 5.0M 0% /run/lock none 12G 148K 12G 1% /run/shm /dev/sdb2 2.6T 99G 2.4T 4% /home /dev/sdc1 1.9T 1.6T 301G 84% /media/My Book Then I try to upload files as administrator on my local instance of galaxy and it fails. Since I even have a swap partition with 100GB and using the /tmp partition I don't know why it is not working. What I notice is that it is using the /tmpfs mounted on /run, which is only 4.8GB. I also tried to change universe.ini to use /tmp as temporary files storage: # Temporary files are stored in this directory. new_file_path = /tmp I can upload files that are smaller than 4.8GB, but not bigger than it. Previously it was working fine on the other Ubuntu version. I was using /tmp as tmpfs, but I don't know how to configure the Galaxy to use the the tmpfs mounted at /tmp and not the tmpfs mounted at /run. I asked similar question before and I was able to fix it on Ubuntu 10.04 LTS, but now Ubuntu have this tmpfs mounted on /run, which is not in fstab. Thank you. Luciano -- *Luciano Cosme* --------------------------------------------- PhD Candidate Texas A&M Entomology Vector Biology Research Group www.lcosme.com 979 845 1885 cosme(a)tamu.edu ---------------------------------------------

2 4