- galaxy-dev - lists.galaxyproject.org

SnpEff_and_CloudMap_for_other_speices_other_than_worm (galaxy not local)
by Wang, Xiaofei 18 Feb '14

18 Feb '14

Dear, Is it possible to use SnpEff with genomes of other species (e.g. Drosophila) other than C.elegan on galaxy website (not local)? It seems the only option for the genome menu (Caenorhabditis elegants : WS220.64). In fact, I am trying to use CloudMap (the EMS Variant Density Mapping workflow) to analyze data for Drosophila. So, I have to edit the workflow based on my own data. For SnpEff on the workflow, I tried to change "Genome" to other species from Caenorhabditis elegants : WS. But, there is no other options. Then, I changed it "to be set at runtime". When I run the workflow, I still can not change it to other species. Then, I went to see the SnpEff tool on galaxy website and found that there is no other options for "Genome" option. So, could you help me about this? Could I use SnpEff and CloudMap (the EMS Variant Density Mapping workflow) other than worm? Or, I have to install galaxy locally for this objective? Thank you so much! Have a nice day!

3 4

Reference Genome in snpEff Tool
by Clint Christensen 18 Feb '14

18 Feb '14

Howdy! I only see the "Caenorhabditis elegans: WS220.64" reference genome listed in the "Genome" drop down menu in the snpEff tool. If possible, I would like to use the latest human genome hg19. Can it be added as an option? Thanks in advance. Not: I did follow the link to the developer's site, but would prefer to use the tool from within Galaxy. Clint Christensen Research Assistant, Department of Genetics Texas Biomedical Research Institute 7620 NW Loop 410 San Antonio, TX 78245 210-258-9779 clintc(a)txbiomedgenetics.org

3 2

Re: [galaxy-dev] space in text tool results in two arguments
by Ross 18 Feb '14

18 Feb '14

Hi Ketan - great news - all the best with your project... On Tue, Feb 18, 2014 at 3:58 PM, Ketan Maheshwari <ketan(a)mcs.anl.gov> wrote: > Ross, > > That's correct. It seems I tested on a cached copy of tool. Works now! > > Thanks, > Ketan > > > On Mon, Feb 17, 2014 at 10:55 PM, Ross <ross.lazarus(a)gmail.com> wrote: > >> Hi Ketan. >> We routinely recommend quotes as you've shown for this reason and it >> certainly works for me at least ! >> Did you reload the tool xml after editing (via the admin menu or via a >> galaxy restart) before testing? If not, you tested a cached copy of the >> tool! >> >> >> Ross Lazarus MBBS MPH; >> Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444 >> http://scholar.google.com/citations?hl=en&user=UCUuEM4AAAAJ >> >> >> On Tue, Feb 18, 2014 at 3:40 PM, Ketan Maheshwari < >> ketancmaheshwari(a)gmail.com> wrote: >> >>> Hi Ross, >>> >>> I did try to use double quotes and curly braces: "${outloc}" but it >>> does not seem to address the issue. >>> >>> Thanks, >>> Ketan >>> >>> >>> On Mon, Feb 17, 2014 at 10:37 PM, Ross <ross.lazarus(a)gmail.com> wrote: >>> >>>> Hi Ketan. >>>> Please try quotation marks to enclose any parameter containing spaces >>>> in the tool command template - eg something like: >>>> python myscript.py "$text_with_spaces" $param2 $param3.... >>>> >>>> Please confirm that this solves the problem? >>>> >>>> >>>> >>>> On Tue, Feb 18, 2014 at 12:47 PM, Ketan Maheshwari < >>>> ketancmaheshwari(a)gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> My tool in galaxy accepts text argument which can have zero or more >>>>> spaces depending on user requirements. When user inputs one word it is >>>>> parsed fine but in the case when user inputs more words separated by space >>>>> it results in multiple separate arguments which messes with the way I am >>>>> treating commandline args in my script. Is it possible to tell Galaxy to >>>>> treat textbox as a single argument irrespective of spaces in the values >>>>> provided? >>>>> >>>>> Thanks for any suggestions. >>>>> >>>>> Best, >>>>> -- >>>>> Ketan >>>>> >>>>> >>>>> ___________________________________________________________ >>>>> Please keep all replies on the list by using "reply all" >>>>> in your mail client. To manage your subscriptions to this >>>>> and other Galaxy lists, please use the interface at: >>>>> http://lists.bx.psu.edu/ >>>>> >>>>> To search Galaxy mailing lists use the unified search at: >>>>> http://galaxyproject.org/search/mailinglists/ >>>>> >>>> >>>> >>> >>> >>> -- >>> Ketan >>> >>> >> >

1 0

space in text tool results in two arguments
by Ketan Maheshwari 18 Feb '14

18 Feb '14

Hi, My tool in galaxy accepts text argument which can have zero or more spaces depending on user requirements. When user inputs one word it is parsed fine but in the case when user inputs more words separated by space it results in multiple separate arguments which messes with the way I am treating commandline args in my script. Is it possible to tell Galaxy to treat textbox as a single argument irrespective of spaces in the values provided? Thanks for any suggestions. Best, -- Ketan

2 2

Local Galaxy web layout
by shenwiyn＠gmail.com 18 Feb '14

18 Feb '14

Galaxy developers:I want to change my local �Galaxy�web layout,for example,delete the top button of "Visualization" and "Cloud" or change their positions.Could anyone show me some details about this?Thank you very much. shenwiyn(a)gmail.com

3 2

New install and handlers for tools
by Thomas Green 17 Feb '14

17 Feb '14

Galaxy devs, I have just setup a Galaxy instance here in Cardiff University. We have used a VM to host the web frontend and setup DRMAA to connect to our HPC resource. We are still in the process of testing all the functionality it but one aspect which feels a bit strange is how to tell Galaxy to use the web frontend to run the tools which download data. Our HPC resource does not have a connection to the outside world (only the login nodes). I am not sure on the latest version but it seems handlers for tools can only be specified by id not by tool_type or any other "class" descriptor so non-computational tools can be run elsewhere whilst the computational tasks are run on HPC - I am currently using the id for each tool to specify a destination in the job_conf.xml. Am I missing something obvious? Another aspect I found whilst installing this was the delicate nature of permissions when integrating it with the cluster. This is especially true if ACLs are also used on the filesystem. It seems the versions I installed (a version from November) could fail without much information why it failed in the logs. Also had occasions where Galaxy never found out a job failed even though the log file had an error and traceback (usually when the DRMAA submit failed). Any advice (or documentation) what the disk permissions should be? And if anyone else is using NFS then root_squash on NFS really does cause issues. So after a bit of tweaking and getting used to the structure of the code I got something which seemed to work - I have possible a few fixes which others might find useful mainly to give better feedback in the log. Any advice how best to integrate these back to you - is there a ticketing system to track changes? I must really upgrade Galaxy to a more recent version to see whether these have already been fixed. Many thanks for all the work that goes into Galaxy and look forward collaborating with the community. All the best, Tom -- Thomas Green Senior Programmer ARCCA, Redwood Building (Tower), King Edward VII Avenue, Cardiff, Wales, CF10 3NB, UK Tel: +44 (0)29 208 79269 Fax: +44 (0)29 208 70734 Email: greent10(a)cardiff.ac.uk Web: http://www.cardiff.ac.uk

2 1

Parsing input format in galaxy tools
by Nikos Sidiropoulos 15 Feb '14

15 Feb '14

Hi all, I'm writing a tool in Galaxy and I need to parse the input file's format (bam, bed, etc) to pass it as a variable to the command. Something like: command.sh $input ${input.format} $output Is there a way to get it? Bests, Nikos

2 2

Error in Galaxy Reports
by Nikos Sidiropoulos 15 Feb '14

15 Feb '14

Hi all, after the February 2014 galaxy update I'm experiencing errors in Galaxy Reports. It only occurs when I'm trying to see which jobs were ran at a specific date (e.g. I can see how many jobs were ran today, but when I click on the number to see which ones it returns the following error): URL: http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_… Module paste.exceptions.errormiddleware:*144* in __call__ >> <http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_…> app_iter *=* self*.*application*(*environ*,* sr_checker*)* Module paste.debug.prints:*106* in __call__ >> <http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_…> environ*,* self*.*app*)* Module paste.wsgilib:*543* in intercept_output >> <http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_…> app_iter *=* application*(*environ*,* replacement_start_response*)* Module paste.recursive:*84* in __call__ >> <http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_…> *return* self*.*application*(*environ*,* start_response*)* Module paste.httpexceptions:*633* in __call__ >> <http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_…> *return* self*.*application*(*environ*,* start_response*)* Module galaxy.web.framework.base:*132* in __call__ >> <http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_…> *return* self*.*handle_request*(* environ*,* start_response *)* Module galaxy.web.framework.base:*190* in handle_request >> <http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_…> body *=* method*(* trans*,* ****kwargs *)* Module galaxy.webapps.reports.controllers.jobs:*177* in specified_date_handler >> <http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_…> *return* self*.*specified_date_list_grid*(* trans*,* ****kwd *)* Module galaxy.web.framework.helpers.grids:*296* in __call__ >> <http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_…> kwargs*=*kwargs *)* Module galaxy.web.framework:*1184* in fill_template >> <http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_…> *return* self*.*fill_template_mako*(* filename*,* ****kwargs *)* Module galaxy.web.framework:*1199* in fill_template_mako >> <http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_…> *return* template*.*render*(* ****data *)* Module mako.template:*296* in render >> <http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_…> *return* runtime*.*_render*(*self*,* self*.*callable_*,* args*,* data*)* Module mako.runtime:*660* in _render >> <http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_…> ****_kwargs_for_callable*(*callable_*,* data*)**)* Module mako.runtime:*692* in _render_context >> <http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_…> _exec_template*(*inherit*,* lclcontext*,* args*=*args*,* kwargs*=*kwargs*)* Module mako.runtime:*718* in _exec_template >> <http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_…> callable_*(*context*,* ***args*,* ****kwargs*)* Module _base_mako:*45* in render_body >> <http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_…> __M_writer*(*unicode*(*next*.*body*(**)**)**)* Module _grid_base_mako:*134* in render_body >> <http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_…> __M_writer*(*unicode*(*self*.*load*(**)**)**)* Module _grid_base_mako:*108* in render_load >> <http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_…> self*.*grid_javascripts*(**)* Module _grid_base_mako:*517* in render_grid_javascripts >> <http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_…> __M_writer*(*unicode*(*h*.*to_json_string*(*self*.*grid_config*)**)**)* Module json:*231* in dumps >> <http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_…> *return* _default_encoder*.*encode*(*obj*)* Module json.encoder:*200* in encode >> <http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_…> chunks *=* self*.*iterencode*(*o*,* _one_shot*=*True*)* Module json.encoder:*263* in iterencode >> <http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_…> *return* _iterencode*(*o*,* *0**)* Module json.encoder:*177* in default >> <http://localhost:9001/jobs/specified_date_handler?webapp=reports&specified_…> *raise* TypeError*(*repr*(*o*)* *+* " is not JSON serializable"*)* *TypeError: datetime.datetime(2014, 2, 13, 12, 12, 24, 966618) is not JSON serializable* Bests, Nikos

3 4

WARNING:galaxy.datatypes.registry:Overriding conflicting datatype
by Jim Johnson 14 Feb '14

14 Feb '14

I now have 2 versions of snpeff installed from the toolshed on my galaxy server. Each snpeff version includes an identical datatypes_conf.xml The galaxy server is setting metadata externally. When any job runs, (in may case I was running a picard tool), the following message is written to the job stderr: WARNING:galaxy.datatypes.registry:Overriding conflicting datatype with extension 'snpeffdb', using datatype from /galaxy/database/tmp/tmpdKGVnQ. Without stdio tags or otherwise catching the stderr, the job state is set to error. I'm now assuming that best practice would be to put the datatypes in a separate toolshed repository that would be a dependency for the tools repository. Thus the datatypes would not loaded multiple times and avoid overrride conflicts. In the meantime, I have just commented out the warning message. Any other advice on this issue? Thanks, JJ -- James E. Johnson, Minnesota Supercomputing Institute, University of Minnesota

3 4

Twill compares wrong file; was: Twill comparisons ignore GALAXY_TEST_NO_CLEANUP
by Peter Cock 14 Feb '14

14 Feb '14

On Tue, Feb 11, 2014 at 3:41 PM, Dave Bouvier <dave(a)bx.psu.edu> wrote: > Peter, > > Thanks for pointing this out, I've applied your changes in > 12478:15fc8675064e on the default branch. > > --Dave B. Thanks Dave, RE: http://lists.bx.psu.edu/pipermail/galaxy-dev/2014-February/018154.html https://bitbucket.org/galaxy/galaxy-central/commits/15fc8675064ea46b7e081d9… I'm still digging into the underlying problem, by focussing on just one unit test from https://github.com/peterjc/pico_galaxy/blob/master/tools/mira4/mira4_de_nov… Namely: <test> <param name="job_type" value="genome" /> <param name="job_quality" value="accurate" /> <param name="type" value="none" /> <param name="filenames" value="ecoli.fastq" ftype="fastqsanger" /> <output name="out_fasta" file="ecoli.mira4_de_novo.fasta" ftype="fasta" /> </test> After the test runs, we have five files, which is expected as I have one input and four outputs: <outputs> <data name="out_fasta" format="fasta" label="MIRA de novo contigs (FASTA)" /> <data name="out_bam" format="bam" label="MIRA de novo assembly (BAM)" /> <data name="out_maf" format="mira" label="MIRA de novo assembly" /> <data name="out_log" format="txt" label="MIRA de novo log" /> </outputs> $ ls /tmp/tmp09TawY/tmpFOX9hS/database/files/000/ dataset_1.dat dataset_2.dat dataset_3.dat dataset_4.dat dataset_5.dat dataset_1.dat --> test-data/ecoli.fastq dataset_2.dat --> out_fasta dataset_3.dat --> out_bam dataset_4.dat --> out_maf dataset_5.dat --> out_log Of these, the test should only attempt to compare dataset_2.dat (out_fasta) to my sample output file test-data/ecoli.mira4_de_novo.fasta which it matches: $ diff test-data/ecoli.mira4_de_novo.fasta /tmp/tmp09TawY/tmpFOX9hS/database/files/000/dataset_2.dat However, for reasons unknown, it is comparing dataset_5.dat (out_log) to the sample output file test-data/ecoli.mira4_de_novo.fasta instead. A little more logging (e.g. [1]) shows the problem is when calling into the verify_dataset_correctness method (should use hid=2): base.interactor: DEBUG: About to call verify_dataset_correctness('ecoli.mira4_de_novo.fasta', '5', ...) base.interactor: DEBUG: About to call verify_dataset_correctness(...) for output_data 'state': 'queued' base.interactor: DEBUG: About to call verify_dataset_correctness(...) for output_data 'hid': '5' base.interactor: DEBUG: About to call verify_dataset_correctness(...) for output_data 'id': '5' base.interactor: DEBUG: About to call verify_dataset_correctness(...) for output_data 'dbkey': 'hg17' base.interactor: DEBUG: About to call verify_dataset_correctness(...) for output_data 'name': 'MIRA de novo log' base.twilltestcase: DEBUG: Verifying dataset correctness for 'ecoli.mira4_de_novo.fasta' (hid='5') Now what is strange is how can the call to get the hid for the current outfile can know if it should be 2, 3, 4, or 5? The bug appears to be in method _verify_outputs in test/functional/test_toolbox.py - which was last touched by John... output_data = data_list[ len(data_list) - len(testdef.outputs) + output_index ] In this example, we have data_list[5 - 1 + 0] = 4, meaning since data_list is essentially [dataset_1, ..., dataset_5] we get dataset_5 I believe the error is that rather than the number of defined test outputs (here 1), you should be subtracting the number of tool outputs (here 4). i.e. data_list[5 - 4 + 0] = data_list[1] = dataset_2 As a workaround, I could probably provide expected data for all four output files... Regards, Peter [1] Patch which adds more logging to test/base/interactor.py (similar debugging added to other files to trace this.) $ hg diff test/base/interactor.py diff -r f3dc213a5773 test/base/interactor.py --- a/test/base/interactor.py Mon Feb 10 22:13:35 2014 -0600 +++ b/test/base/interactor.py Tue Feb 11 16:19:51 2014 +0000 @@ -309,6 +309,9 @@ def verify_output( self, history, output_data, outfile, attributes, shed_tool_id, maxseconds ): hid = output_data.get( 'hid' ) + log.debug("About to call verify_dataset_correctness(%r, %r, ...)" % (outfile, hid)) + for key in output_data.keys(): + log.debug("About to call verify_dataset_correctness(...) for output_data %r: %r" % (key, output_data.get(key))) self.twill_test_case.verify_dataset_correctness( outfile, hid=hid, attributes=attributes, shed_tool_id=shed_tool_id, maxseconds=maxseconds ) def get_job_stream( self, history_id, output_data, stream ):

4 10