- galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] [galaxy-iuc] Fwd: [Samtools-devel] Samtools/BCFtools/HTSlib 1.3 released
by Peter Cock 16 Dec '15

16 Dec '15

Thanks Youri, That's the bit of code I was worrying about (BAM grooming). I've filed an issue for this: https://github.com/galaxyproject/galaxy/issues/1336 Peter On Wed, Dec 16, 2015 at 9:47 AM, Y. Hoogstrate <y.hoogstrate(a)erasmusmc.nl> wrote: > Great news Peter, > > This is how Galaxy seems to do it: > > command = "samtools sort %s %s" % ( file_name, > tmp_sorted_dataset_file_name_prefix ) > proc = subprocess.Popen( args=command, shell=True, cwd=tmp_dir, > stderr=open( stderr_name, 'wb' ) ) > > > https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/datatypes/binar… > > > Since galaxy ships with a builtin pysam (0.8.3) wheel, I think it would be > handier to use pysam for sorting (once updated to the latest HTSlib). > Pysam now also has all samtools c functions directly builtin: > > pysam.index( "arg1", "arg2" ) is the equivalent of the command line > argument *samtools index "arg1" "arg2"* > > ... > Youri >

1 0

samtools bam2fq --> samtools fastq / samtools fasta
by Peter Cock 16 Dec '15

16 Dec '15

Hello all, About a year ago I stated wrapping the "samtools bam2fq" command from samtools v1.1 for use in Galaxy: https://github.com/peterjc/pico_galaxy/tree/master/tools/samtools_bam2fq That uncovered a number of bugs in "samtools bam2fq" which have resulted in two new commands "samtools fastq" and "samtools fasta" for converting a SAM/BAM/CRAM file into FASTQ or FASTA reads with the release of samtools 1.3. When time permits, I had planned to update my wrapper to target samtools 1.3, probably as two separate tools which could potentially hook into the Galaxy datatype conversion framework? Where would be the best place for them? Under tools-devteam? https://github.com/galaxyproject/tools-devteam/tree/master/tool_collections… Peter

1 0

Fwd: [Samtools-devel] Samtools/BCFtools/HTSlib 1.3 released
by Peter Cock 16 Dec '15

16 Dec '15

Just a heads up, I'm sure we the IUC will have Tool Shed packages prepared for the samtools 1.3 etc release shortly. However, I'm less clear what Galaxy itself is doing for sorting BAM files - noting the original "samtools sort" API has been withdrawn. Peter ---------- Forwarded message ---------- From: John Marshall <jm18(a)sanger.ac.uk> Date: Wed, Dec 16, 2015 at 12:47 AM Subject: [Samtools-devel] Samtools/BCFtools/HTSlib 1.3 released To: samtools-announce(a)lists.sourceforge.net Cc: Samtools-help List <samtools-help(a)lists.sourceforge.net>, samtools-devel <samtools-devel(a)lists.sourceforge.net> New 1.3 versions of Samtools, BCFtools, and HTSlib have been released. These have numerous bug fixes, improvements, and new commands. The changes are too numerous to list here, but we would particularly like to draw your attention to these three items: * The old way of invoking the samtools sort subcommand has been superseded since samtools 1.0 and from this release no longer works: samtools sort [-f] [-o] in.bam out.prefix If you do use this obsolete way of specifying sort options, samtools will now in most cases fail and give you a hint about the new ways. If you are still using sort ‑f, ‑o, or out.prefix, convert to use -T PREFIX and/or -o FILE instead. See the manual page for details. * The samtools rmdup command has been restored to its basic functionality. It no longer immediately aborts, but it remains not recommended for most use. * When HTSlib has been built to use libcurl, most tools can now access remote files over HTTPS and via AWS S3 in addition to the previous HTTP, FTP, and iRODS. See HTSlib's INSTALL and Samtools's README for details. For further details of these and other changes in these releases, see the release notes for each project linked to at <http://www.htslib.org/download/>, or see all three together at <http://sourceforge.net/projects/samtools/files/samtools/1.3/>. These releases can be found at <http://www.htslib.org/>. Development and source code of these projects are linked from that web site and can be found in various Git repositories under <https://github.com/samtools>. Release archives are available at both GitHub and SourceForge. Installing the tools will also install manual pages for them. Once installed, you should be able to use "man samtools" etc at your command line to find details of new commands and functionality. Manual pages and other documentation are also available online at <http://www.htslib.org/doc/>. John -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. ------------------------------------------------------------------------------ _______________________________________________ Samtools-devel mailing list Samtools-devel(a)lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/samtools-devel

2 1

Apache proxy under Ubuntu
by Mic 16 Dec '15

16 Dec '15

Hello, I started to setup Apache Proxy under Ubuntu (14.04) in the following way: *sudo apt-get install apache2 libapache2-mod-proxy-html libapache2-mod-xsendfile * *sudo a2enmod rewrite deflate expires proxy proxy_html xsendfile* *vim /home/galaxy/galaxy/config/galaxy.ini* *apache_xsendfile = True* *Where do I insert the following settings?* *RewriteEngine on* *RewriteRule ^/static/style/(.*) /home/galaxy/galaxy/static/june_2007_style/blue/$1 [L]* *RewriteRule ^/static/scripts/(.*) /home/galaxy/galaxy/static/scripts/packed/$1 [L]* *RewriteRule ^/static/(.*) /home/galaxy/galaxy/static/$1 [L]* *RewriteRule ^/favicon.ico /home/galaxy/galaxy/static/favicon.ico [L]* *RewriteRule ^/robots.txt /home/galaxy/galaxy/static/robots.txt [L]* *RewriteRule ^(.*) http://localhost:8080$1 [P]* *<Location "/">* * # Compress all uncompressed content.* * SetOutputFilter DEFLATE* * SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary* * SetEnvIfNoCase Request_URI \.(?:t?gz|zip|bz2)$ no-gzip dont-vary* * SetEnvIfNoCase Request_URI /history/export_archive no-gzip dont-vary* * # Sending files using Apache* * XSendFile on* * XSendFilePath /* *</Location>* *<Location "/static">* * # Allow browsers to cache everything from /static for 6 hours* * ExpiresActive On* * ExpiresDefault "access plus 6 hours"* *</Location>* Thank you in advance, Mic

2 1

GATK issues
by Tiago Rodrigues Antao 15 Dec '15

15 Dec '15

Hi, I have a couple of GATK issues 1. First is there a data manager for it? I have tried Generate GATK-sorted Picard indexes (data_manager_gatk_picard_index_builder) - builder and got no entries on the cached picard genomes. 2. With problem 1, we are using genomes from the history. It so happens that some XML files have the filtering of forcing the dbkey of the from-history genome to be the same as the BAM files (reduced below for easy reading): <when value="history"> <param name="input_bam" type="data" format="bam"/> <param name="ref_file" type="data" format="fasta"> <options> <filter type="data_meta" key="dbkey" ref="input_bam" /> </options> </param> </when> For example, here: https://github.com/galaxyproject/tools-devteam/blob/master/tool_collections… Now, this does not seem to work. I.e. when I make the dbkey of the BAM file and the FASTA file the same, the ref_file entry is still empty. Also, some xml files have it, others do not. As a temporary measure, I removed the options/filter in my local installation, but might this be a bug? Or is it a bug in my head? Thanks, Tiago

1 0

(no subject)
by Justin Powell 15 Dec '15

15 Dec '15

1) Could it be made possible to load generalised tabular data from file with column headings being recognised as column headings rather than appearing as the first row of the table? It looks like this is in general possible as more specific file types (e.g. gtf) do get displayed with column headings, so the data structures obviously support that. 2) With data sets that do have column headings, is there any reason why the 'Join two Datasets' tool couldn't retain the column headings from both sets in the final output rather than just the first one? 3) Could the 'Join two Datasets' tool allow selection of the columns by name rather than by number? To me these three changes would make it much easier for our users to manipulate and join tables of genes and gene annotations, but I'm new to Galaxy and maybe there are reasons why it doesn't work this way? Justin

2 2

How to define test with multiple inputs from one parameter
by Ulf Schaefer 15 Dec '15

15 Dec '15

Hi all Unfortunately I was unable to find this in the documentation or find a working example. In my tool definition I have inputs and outputs like this: <param name="input_vcfs" type="data" multiple="true" format="vcf" label="Input VCF file(s)" /> <outputs> <data format="fasta" name="output" label="${tool.name} on ${on_string}: FASTA file" /> </outputs> I would like to define a test which takes two vcfs from the "input_vcfs" parameter. Is this correct? <tests> <test> <param name="input_vcfs" value="1_short.vcf" ftype="vcf" /> <param name="input_vcfs" value="2_short.vcf" ftype="vcf" /> <output name="output" file="testresult.fa" ftype="fasta" /> </test> </tests> or option 2? <tests> <test> <param name="input_vcfs" value="1_short.vcf,2_short.vcf" ftype="vcf" /> <output name="output" file="testresult.fa" ftype="fasta" /> </test> </tests> or something else entirely? Thanks for your help Ulf -- Ulf Schaefer, PhD Bioinformatics Scientist Bioinformatics Unit - Infectious Disease Informatics National Infection Service Public Health England ulf.schaefer(a)phe.gov.uk http://www.gov.uk/phe Protecting and improving the nation’s health ************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of Public Health England, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. http://www.gov.uk/PHE **************************************************************************

1 0

Incorrect User Disk Usage
by Ping Luo 14 Dec '15

14 Dec '15

Hello, We have a galaxy user who has only one history which occupied 239.9GB disk space. However, total disk usage, as showing at the top right corner, is 925.6GB. I have done dataset cleanup and run set_user_disk_usage, but the total is still the same. Actually, running set_user_disk_usage.py had added up about 90G to the somewhat 840G total disk space showning in his account. Anybody knows what could be wrong? Thanks, Ping

2 1

Visualization issue
by Scott Szakonyi 14 Dec '15

14 Dec '15

Hi all, If I try to build a new track browser from a custom build, I'm not able to see any of the datasets in my build after clicking add datasets and selecting my workflow from history. Am I doing something wrong, or is there a problem? I've been able to recreate this on both my local server and usegalaxy.org. Thanks! -- Scott B. Szakonyi Research Programmer *Center for Research Computing* 107 Information Technology Center Notre Dame, IN 46556 http://crc.nd.edu

2 1

workflow export name attribute changes
by Jorrit Boekel 14 Dec '15

14 Dec '15

Hi all, I noticed something strange in my (latest) galaxy-dist. I have around 10-20 workflows, which are in certain cases very similar for e.g. slightly different experimental setups with the same output. I then usually copy the first wf and adapt a few steps to fit the second one. Anyway, they are on a testserver and I export them to import in my production instance. There I noticed that sometimes in (always the same 3) workflows, the export to json changes the ‘name’ attribute of the workflow. So when importing them in production, the wf has a different name. I haven’t been able to pinpoint where the problem lies, but the new name is very similar to the old one. I therefore believe that it may be that I have copied the workflow and the name of the source-wf is hanging around in some db table somewhere. Unlikely as it sounds. To be more specific it doesn’t show up when I do bioblend: workflows.show_workflow(id=x), but only when export_workflow_json(). It only happens in 3 of my workflows and I have surely copied more of them. Easy enough to workaround though, and kind of hoping this is a glitch in my setup only but wanted to let you know. cheers, — Jorrit Boekel Proteomics systems developer BILS / Lehtiö lab Scilifelab Stockholm, Sweden

2 1