- galaxy-dev - lists.galaxyproject.org

KeyError ['tools'] on importing workflow from locally installed repository
by jorrit poelen 19 Feb '13

19 Feb '13

Hey y'all, I am testing how to best share a workflow through a tool shed. I stumbled across a possible bug and am hoping that this is simply a configuration issue. Steps to reproduce: 1. In local galaxy instance goto . . admin > search and browse tool sheds 2. locate select "Galaxy test tool shed" 3. search for "obo" 4. preview and install repository "obo_test_workflows" 5. in next screen click "install to Galaxy" 6. in next screen click "install" 7. in next screen click "obo_test_workflows" 8. in next screen, under workflows, click "TestOboWorkflow1" 9. in svg graphics page > repository actions > import workflow to galaxy Expected results: Workflow is installed and made available in workflow section in local Galaxy instance. Actual results: URL: http://galaxy:7474/admin_toolshed/import_workflow?repository_id=2d9035b3fc1… File '/home/ubuntu/galaxy-dist/eggs/WebError-0.8a-py2.7.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/home/ubuntu/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/home/ubuntu/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/home/ubuntu/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/home/ubuntu/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/home/ubuntu/galaxy-dist/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/home/ubuntu/galaxy-dist/lib/galaxy/web/framework/base.py', line 185 in handle_request body = method( trans, **kwargs ) File '/home/ubuntu/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 216 in decorator return func( self, trans, *args, **kwargs ) File '/home/ubuntu/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py', line 744 in import_workflow tools_metadata = metadata[ 'tools' ] KeyError: 'tools' Thanks in advance for suggestions on how to solve / work around this issue. -jorrit PS Note that the workflow can be installed using a url pointing to the workflow from a hg repository on bitbucket (ie. https://bitbucket.org/jhpoelen/obo_workflows/raw/25d386a50562d02e03fa821d96…)

2 1

Changing job command line.
by James Boocock 19 Feb '13

19 Feb '13

Hi All, I am currently working on a clustering interface for galaxy that will hopefully enable users to pick the grid in the tool form. With tools that need access to files within the galaxy directory, I have an idea to create a symlinked fake galaxy root with all the files the tools needs for galaxy. This is done currently in my own xml format for each grid. Problem is when parsing a tool_wrapper variable I need to prepare the job and add my additional commands to the command line so that the fake galaxy dir is created with the symlinked databases. Is it bad practice to disable preparation in each tool runner if the job has come from my clustering module or is there a better way to do this. Will the job runners wrapper.prepare() overwrite any of my changes to the command-line when the job makes its way to the runner say local.py. Cheers James.

3 3

Searching tool shed for file format definitions?
by Peter Cock 19 Feb '13

19 Feb '13

Hello all, I was wondering how people keep track of the available file format definitions in the tool shed(s)? For core file types defined in Galaxy, I can just view the datatypes_conf.xml file and/or the classes in lib/galaxy/datatypes/ - but how can I do something similar on a Galaxy Tool Shed? One solution might be automatic functional categories: tool shed repositories which define a format, or define a tool for the user, or define a dependency for use by other tools, etc Thanks, Peter

2 1

shall we need to download dataset when we install our local Galaxy?
by 沈维燕 19 Feb '13

19 Feb '13

hi: When we running a local installation of Galaxy,there are some reference dataset ,such as Human Feb. 2009 (GRCh37/hg19) (hg19),A. gambiae Feb. 2003 (IAGEC MOZ2/anoGam1) (anoGam1),and so on.Shall we download these datasets and save in our computer ? This is my first use Galaxy,it is very good and iteresting.Thank you very much for replying.

2 1

all the data files in .loc files needed to download ?
by shenwiyn 19 Feb '13

19 Feb '13

Hi all, Are all the data files in .loc files needed to download and put to the right directory when we install our local galaxy? thaks shenwiyn shenwiyn

1 0

changing datasets file_path storage location
by Shantanu Pavgi (Campus) 19 Feb '13

19 Feb '13

We are planning to change datasets directory location (file_path setting in universe_wsgi.ini) in our Galaxy installation. We will be moving existing datasets to the new directory location and updating XSendfile setting in Apache as well. I was wondering if there are any side effects that we should be considering before making this change. There are certain fields in the database (e.g. command_line in jobs table) which contain absolute path of datasets. Do we need to update any of these fields to make sure Galaxy is functional after file_path is changed? -- Thanks, Shantanu

3 4

Re: [galaxy-dev] [galaxy-user] Uploading large archive histories
by Jeremy Goecks 18 Feb '13

18 Feb '13

> After discussing this with Getiria, I think what is needed here is a way to use FTP to upload compressed histories from the import screen as is available from the upload data tool. The problem we're currently experiencing is that when we try to upload a compressed history where the compressed file is greater than 2GB, the process fails. This is preventing us from being able to effectively have users archive data from Galaxy to reduce our disk usage. The problem can't be coming from the 2GB browser upload limit because the import tool uses wget to fetch the archived history before importing it. The best way to debug failed history imports is to use SQL to look at the Galaxy jobs table for the error: select * from job where tool_id='__IMPORT_HISTORY__'; There is trello card to improve import/export functionality: https://trello.com/c/qCfAWeYU > Additionally, if someone could provide a better understanding of how the exported histories are treated on the Galaxy server that would be helpful. e.g. how long do the compressed histories stick around on disk, etc? We would like to have users archive their data on their own systems and permanently delete their Galaxy history when an analysis is complete, but if the compressed history persists on the Galaxy server the utility of this is somewhat reduced. Archived histories are Galaxy datasets so that they're easy to read and write. History archive datasets are connected to histories via the JobExportHistoryArchive table, not the HistoryDatasetAssociation table (in order to avoid recursive archiving). They can be cleaned up/removed by using the pgcleanup.py script. Finally, I've moved this thread to the galaxy-dev list because it concerns local instance configuration and usage. Best, J.

1 0

Galaxy - Bismark
by Charles Murphy 18 Feb '13

18 Feb '13

Hi, I' am running galaxy locally and I have installed Bismark and its dependencies from the Galaxy main tool shed. I receive multiple errors when using a small test dataset. (1) My first error occurs when I try running Bismark bisulfite mapper (bowtie2): Error indexing reference sequence /bin/sh: /Users/charlesmurphy/Desktop/Research/tools/galaxy/shed_tools/ toolshed.g2.bx.psu.edu/repos/bjoern-gruening/bismark/183de9d00131/bismark/b…: No such file or directory Which I can fix by editing the env.sh file in the dependencies directory. I change: SCRIPT_PATH=/Users/charlesmurphy/Desktop/Research/tools/galaxy/shed_tools/ toolshed.g2.bx.psu.edu/repos/bjoern-gruening/bismark/183de9d00131/bismark; To: SCRIPT_PATH=/Users/charlesmurphy/Desktop/Research/tools/galaxy/shed_tools/ toolshed.g2.bx.psu.edu/repos/bjoern-gruening/bismark/183 de9d00131/bismark/bismark_wrapper; (2) My second error occurs when I try running Bismark methylation extractor: usage: bismark_methylation_extractor.py [-h] [--infile INFILE] [--single-end] [--paired-end] [--report-file REPORT_FILE] [--comprehensive] [--merge-non-cpg] [--no-overlap] [--compress COMPRESS] [--ignore-bps IGNORE_BPS] [--cpg_ot CPG_OT] [--chg_ot CHG_OT] [--chh_ot CHH_OT] [--cpg_ctot CPG_CTOT] [--chg_ctot CHG_CTOT] [--chh_ctot CHH_CTOT] [--cpg_ob CPG_OB] [--chg_ob CHG_OB] [--chh_ob CHH_OB] [--cpg_ctob CPG_CTOB] [--chg_ctob CHG_CTOB] [--chh_ctob CHH_CTOB] [--cpg_context CPG_CONTEXT] [--chg_context CHG_CONTEXT] [--chh_context CHH_CONTEXT] [--non_cpg_context NON_CPG_CONTEXT] [--non_cpg_context_ot NON_CPG_CONTEXT_OT] [--non_cpg_context_ctot NON_CPG_CONTEXT_CTOT] [--non_cpg_context_ob NON_CPG_CONTEXT_OB] [--non_cpg_context_ctob NON_CPG_CONTEXT_CTOB] bismark_methylation_extractor.py: error: unrecognized arguments: --bismark_path /Users/charlesmurphy/Desktop/Research/tools/galaxy/shed_tools/ toolshed.g2.bx.psu.edu/repos/bjoern-gruening/bismark/183de9d00131/bismark Seems to me the python wrapper lacks the code to tell galaxy where the bismark_methylation_extractor executable is. I can go around the error by editing bismark_methylation_extractor.py. I add the following line: parser.add_argument( '--bismark_path' ) Then on line 72 I change: cmd = 'bismark_methylation_extractor --no_header -o %s %s %s' To: cmd = args.bismark_path + '/bismark_methylation_extractor --no_header -o %s %s %s' I suspect I' am not doing something correctly. Let me know if anything is unclear or if you need more information. Thanks for any help! Charlie

2 1

Preffered way of running a tool on multiple input files
by Hagai Cohen 18 Feb '13

18 Feb '13

Hi, I'm looking for a preferred way of running Bowtie (or any other tool) on multiple input files and run statistics on the Bowtie output afterwards. The input is a directory of files fastq1..fastq100 The bowtie output should be bed1...bed100 The statistics tool should run on bed1...bed100 and return xls1..xls100 Then I will write a tool which will get xls1..xls100 and merge them to one final output. I searched for a smiliar cases, and I couldn't figure anyone which had this problem before. Can't use the parallelism tag, because what will be the input for each tool? it should be a fastq file not a directory of fastq files. Neither I would like to run each fastq file in a different workflow - creating a mess. I thought only on two solutions: 1. Implement new datatypes: bed_dir & fastq_dir and implements new tool wrappers which will get a folder instead of a file. 2. merge the input files before sending to bowtie, and use parallelism tag to make them be splitted & merged again on each tool. Does anyone has any better suggestion? Thanks, Hagai

3 6

Re: [galaxy-dev] DustMasker tool for ncbi_blast_plus
by Nicola Soranzo 15 Feb '13

15 Feb '13

Adding galaxy-dev list in CC as suggested by Peter. Il giorno mer, 06/02/2013 alle 16.57 +0000, Peter Cock ha scritto: > On Tue, Feb 5, 2013 at 11:45 AM, Nicola Soranzo <soranzo(a)crs4.it> wrote: > > Dear Peter, > > I have created a simple Galaxy tool for DustMasker of the NCBI BLAST+ > > suite, which I think would be a useful addition to the ncbi_blast_plus > > repository you're maintaining in the Galaxy Tool Shed. > > > > You can find it and hopefully pull it from: > > > > https://bitbucket.org/nsoranzo/ncbi_blast_plus > > > > Kind regards, > > Nicola > > Hi Nicola, > > Thanks for getting involved - we can discuss this on the galaxy-dev > mailing list if you prefer? For now I have CC'd Edward Kirton as he > is/was working on masking in BLAST databases for Galaxy. > > I can see the new file > tools/ncbi_blast_plus/ncbi_dustmasker_wrapper.xml however it refers to > multiple new file formats - where are they defined? > > * acclist > * maskinfo_asn1_bin > * maskinfo_asn1_text > * seqloc_asn1_bin > * seqloc_asn1_text Hi Peter, I added these file formats mostly as placeholders for a future implementation. Now I have changed a bit the tool by removing acclist and seqloc_xml formats since they are not recognized by the last versions of dustmasker (I also sent an email to blast-help(a)ncbi.nlm.nih.gov to inform them of this bug). As before, you can find the new version at: https://bitbucket.org/nsoranzo/ncbi_blast_plus I stripped the old commit and did a new one, not a very good practice, sorry about that. > Have you looked at the (commented out) bits in the makeblastdb wrapper > which would perhaps be relevant? This is something Edward Kirton wrote > which I haven't integrated yet: > >  >  > > Perhaps all you need to offer in ncbi_dustmasker_wrapper.xml is > 'fasta' and 'asnb' (binary ASN) formats? Edward - did you have an > 'asnb' definition? 'fasta' and 'interval' are the ones I'm interested for my use case. 'maskinfo_asn1_bin' is probably the one referenced as 'asnb' in the cited code (ASN1 is a general data serialization format like XML). A file in this format can be given as input to "makeblastdb -mask_data". Nicola -- Nicola Soranzo, Ph.D. CRS4 Bioinformatics Program Loc. Piscina Manna 09010 Pula (CA), Italy http://www.bioinformatica.crs4.it/

2 4