November 2014 - galaxy-dev - lists.galaxyproject.org

importing a large number of samples via a sample sheet
by Ryan G 04 Nov '14

04 Nov '14

Hi all - I've recently come back to Galaxy to see how its progressed over the last few years. The tools shed is completely new to me and will take some exploring on my part. My immediate question is if there is a way to import a large number of samples associated with a project via a sample sheet (txt) file? I have a text file with 3 columns: sample name, location of fastq files. I haven't seen a way to import this and haven't seen anything in the docs either. I suspect the answer is no, but I'm hoping the answer is actually yes....? Ryan

1 0

Can existing SAM/BAM filter tools give me mapped/unmapped pairs?
by Peter Cock 04 Nov '14

04 Nov '14

Hi all, I'm looking for a little advice on the pre-existing SAM/BAM filtering tools already in the Galaxy Tool Shed (to avoid reinventing the wheel). As I mentioned on another thread, I'm working on a wrapper for the "samtools bam2fq" command (targeting samtools 1.1 which fixed some bugs in this tool and added new functionality compared to samtools 0.1.19), see: https://github.com/peterjc/pico_galaxy/tree/master/tools/samtools_bam2fq https://toolshed.g2.bx.psu.edu/view/peterjc/samtools_bam2fq https://testtoolshed.g2.bx.psu.edu/view/peterjc/samtools_bam2fq One of my motivating use cases is a workflow like this: 1. Upload paired end FASTQ files. 2. Map them against a known contaminant genome giving a BAM file (note I need the mapper to report unmapped reads in the output). 3. Filter the BAM to get unmapped reads, plus reads whose partner is unmapped (conversely, remove reads where both partners are mapped). 4. Convert the filtered BAM back into FASTQ (with samtools bam2fq). 5. Proceed with analysis (e.g. de novo assembly). Assuming I have understood "samtools view", this filtering step has to be multiple parts: This would get the unmapped reads $ samtools view -f 0x4 ... This would get reads with an unmapped partner: $ samtools view -f 0x8 ... However this would only get unmapped reads with an unmapped partner: $ samtools view -f 0x12 ... i.e. samtools view allows logical AND, not logical OR, when combining flag filters. So, I believe using samtools directly, a two stage filter is needed followed by a merge (and sort), taking care not to duplicate reads, perhaps: $ samtools view -f 4 ... > unmapped.bam $ samtools view -f 8 -F 4 ... > mapped_with_partner_unmapped.bam $ samtools merge unmapped.bam mapped_with_partner_unmapped.bam > ... That could be repeated within Galaxy but is surprisingly complicated with multiple steps in the history - so I do not want to go that route. Have I overlooked a simple ToolShed solution using samtools? As far as I could tell, the only other option on the current Tool Shed is the Sambamba Filter tool (using "unmapped or mate_is_unmapped"), which has a very capable looking filter system: https://toolshed.g2.bx.psu.edu/view/lomereiter/sambamba_filter @Artem - have you explored updating your tool_dependencies.xml to download your pre-compiled binaries by default? That would make deployment far easier, since D compilers are still rare, and would mean we can see the test results on the Tool Shed :) Please ask if you'd like advice on Tool Shed packaging. Thanks, Peter

2 2

Galaxy's dependency on old samtools vs tools wrapping later versions?
by Peter Cock 04 Nov '14

04 Nov '14

Hello all, Galaxy currently requires samtools on the $PATH in order to sort and index BAM files automatically, and samtools 0.1.19 works fine. Unfortunately later versions of samtools index have a regression: https://github.com/samtools/samtools/issues/199 This has caught several people out already, e.g. https://biostar.usegalaxy.org/p/7928/ and https://biostar.usegalaxy.org/p/9335/ While eventually samtools will be fixed, right now this means we can't have samtools 1.1 as the first samtools on the $PATH used by Galaxy. I am working on a wrapper for "samtools bam2fq": https://github.com/peterjc/pico_galaxy/tree/master/tools/samtools_bam2fq https://testtoolshed.g2.bx.psu.edu/view/peterjc/samtools_bam2fq The bam2qf command in samtools 0.1.19 has a number of bugs, so I want to target samtools 1.1. However this has complicated my testing since for my BAM input files Galaxy will call "samtools index", and if it calls samtools 1.1 this will fail. I'm not using the tool shed dependencies during development so instead came up with the following hack: https://github.com/peterjc/picobio/blob/master/sambam/samtools_auto.py My question is, what is expected to happen with a Tool Shed installed wrapper for samtools 1.1 and Galaxy's attempts to automatically call samtools to index any BAM output file? Would the tool environment put samtools 1.1 on the (local) $PATH which would then break setting the metadata as part of the same job? Regards, Peter

2 3

(no subject)
by Dave Clements 03 Nov '14

03 Nov '14

1 0

job_working / tmp folders
by Shrum, Donald C 03 Nov '14

03 Nov '14

I changed the values for new_file_path and job_working_directory in my config file. After restarting galaxy I'm finding some of the tools still reference the old path which no longer exists. Do these end up hard coded in some place... I haven't done a find + grep to go looking yet but that may be the next step... Thanks for any help --Don FSU Research Computing Center

2 2

Login issue with a nginx proxy
by Alexandre Loywick 03 Nov '14

03 Nov '14

Hello galaxians, I'm currently trying to set up a production galaxy and make it available to the public but I face a weird issue, probably because of my nginx proxy configuration. Galaxy is on a kvm virtual machine running on a host using nginx as proxy. to make it a bit clearer: 172.30.24.35 (client) ---> http://172.30.24.63/machine1 (Host & nginx proxy) --- > http://192.168.122.254:8080/ (virtual machine) I can connect to galaxy but when logging in, the virtual machine redirects me on 192.168.122.254:8080, which is unreachable from the 172.30.24.0 network. If I log in from the Host, it's all fine. in the universe_wsgi.ini, i've set: # Define the proxy-prefix filter. [filter:proxy-prefix] use = egg:PasteDeploy#prefix prefix = /machine1 ... # [filter:proxy-prefix] section above. filter-with = proxy-prefix as described in the documentation. My virtual machine nginx conf file: user galaxy; worker_processes 2; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; #log_format main '$remote_addr - $remote_user [$time_local] "$request" ' # '$status $body_bytes_sent "$http_referer" ' # '"$http_user_agent" "$http_x_forwarded_for"'; #access_log logs/access.log main; sendfile on; #tcp_nopush on; #keepalive_timeout 0; keepalive_timeout 65; gzip on; gzip_http_version 1.1; gzip_vary on; gzip_comp_level 4; gzip_proxied any; gzip_types text/plain text/css application/x-javascript text/xml application/xml text/javascript application/json; gzip_buffers 16 8k; gzip_disable "MSIE [1-6].(?!.*SV1)"; upstream galaxy_app{ server localhost:8080; } server { client_max_body_size 10G; listen 80; server_name localhost; #charset koi8-r; #access_log logs/host.access.log main; location /machine1 { proxy_pass http://galaxy_app; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } location /machine1/static { alias /home/galaxy/galaxy-dist/static; } location /machine1/static/style { alias /home/galaxy/galaxy-dist/static/june_2007_style/blue; } location /machine1/static/scripts { alias /home/galaxy/galaxy-dist/static/scripts/packed; } location /machine1/favicon.ico { alias /home/galaxy/galaxy-dist/static/favicon.ico; } location /machine1/robots.txt { alias /home/galaxy/galaxy-dist/static/robots.txt; } location /_x_accel_redirect/ { internal; alias /machine1; } #error_page 404 /404.html; #redirect server error pages to the static page /50x.html # error_page 500 502 503 504 /50x.html; location = /50x.html { root html; } } } And my host configuration : server { listen 80 default_server; listen [::]:80 default_server ipv6only=on; root /usr/share/nginx/html; index index.html index.htm; # Make site accessible from http://localhost/ server_name localhost; location / { try_files $uri $uri/ =404; } location /machine1 { proxy_pass http://192.168.122.254:8080; } }

1 0

Problems with upload.py and changes to json.py
by Robert Davidson 03 Nov '14

03 Nov '14

Dear Galaxy-Dev, Last week I downloaded a fresh version of Galaxy with all stable updates. I've just come across a NameError in the upload.py tool. ---- Traceback (most recent call last): File "/home/rob/galaxy-dist/tools/data_source/upload.py", line 390, in <module> __main__() File "/home/rob/galaxy-dist/tools/data_source/upload.py", line 368, in __main__ dataset = from_json_string( line ) NameError: global name 'from_json_string' is not defined ---- A quick investigation led me to: from galaxy.util.json import * and from there to json.py. I compared this file with my previous version and note that there are quite a few changes. The problem seems to be that from_json_string() is not included in the __all__ list of public functions... can someone tell me if this is a bug/oversight or if I'm missing something? Cheers, Rob

1 1

error running cuffdiff
by Elizabeth Chia 02 Nov '14

02 Nov '14

Hi there I'm running into this error with cuffdiff: I'm running galaxy locally on a cluster (1 head+3 compute nodes) with MySQL. Here's my hg summary: Any clues? I also keep getting the a similar 149 Internal Server error (with different Key Error) thrown when trying to install some tools. But strangely, after leaving it for awhile, it went back to normal. So I'm not too sure what's causing it. Should I be updating galaxy to the latest stable one? Please let me know if you require any other info. Thanks! liz

3 4

Configuring r script galaxy tool to install on testtoolshed with dependencies in the same repository
by Jeremy Liu 02 Nov '14

02 Nov '14

Hi Galaxy Dev, Been having trouble getting my Galaxy tools to work with the testtoolshed. Here is the tools repository: https://jeremyjliu@testtoolshed.g2.bx.psu.edu/repos/jeremyjliu/region_motif… In particular, I've been focusing on getting the region_motif_intersect.r script to work properly when installed from the testtoolshed. intersect.r runs using files located in the same location as where the repository is installed, in the folder region_motif_db. To make sure that the tool knows where these files are, I've been trying to pass the repository install location as an argument in the r script invocation (see region_motif_intersect.xml and tool_dependencies.xml) . To set the repository install location in the environment, I've been basing off of this wiki entry: https://wiki.galaxyproject.org/ToolsWithDependenciesInSameRepository When I install the repository to my local Galaxy, it gives me the following warnings: galaxy.tools.deps DEBUG 2014-10-25 15:12:35,376 Building dependency shell command for dependency 'R_SCRIPT_PATH' galaxy.tools.deps WARNING 2014-10-25 15:12:35,377 Failed to resolve dependency on 'R_SCRIPT_PATH', ignoring Then when I run the tool from Galaxy, this is the run command: PACKAGE_BASE=/Users/jeremyliu1/galaxy-dist/tool-dependency/R/2.15.1/jeremyjliu/region_motif_enrichment/7a3bb055f3d0; export PACKAGE_BASE; . /Users/jeremyliu1/galaxy-dist/tool-dependency/R/2.15.1/jeremyjliu/region_motif_enrichment/7a3bb055f3d0/env.sh; Rscript /Users/jeremyliu1/shed_tools/testtoolshed.g2.bx.psu.edu/repos/jeremyjliu/region_motif_enrichment/7a3bb055f3d0/region_motif_enrichment/region_motif_intersect.r --args $R_SCRIPT_PATH p /Users/jeremyliu1/galaxy-dist/database/files/000/dataset_504.dat /Users/jeremyliu1/galaxy-dist/database/files/000/dataset_515.dat What exactly is going on here? How do I make sure my tool knows where the reference files it needs are, since it won't be in the job working directory? Note: the files that intersect.r needs are not in the toolshed repository, since they are a few GB. I was planning on providing that as a separate download or using the Data Managers to solve the problem. But as long as I can get the tool to look for it in that folder, that would be perfect. Thanks! Jeremy Liu jeremy.liu(a)yale.edu 352.871.1258

2 1

Reducing the size of a volume in Galaxy Cloud
by Denise Clark 02 Nov '14

02 Nov '14

Hi, I wish to remove most of the mnt/galaxyIndices/genomes files so I can have a smaller volume that contains those files and save money. I am new to this system. I have set up a Galaxy Cloudman account and used the tools on the web interface, and I can ssh the server using Ubuntu. So far, so good. I tried the first approach here twice https://wiki.galaxyproject.org/CloudMan/CustomizeGalaxyCloud but the system hangs (>24h). I can't follow the "manual" instructions on this page and I'm not sure it would result in a smaller volume anyway. So, I was trying to follow Enis' steps from this thread to reduce the size of the galaxyIndices volume: http://dev.list.galaxyproject.org/Reducing-costs-in-Cloud-Galaxy-td4483448.… His instructions are: "1. Start a new Galaxy cluster (ie, one you can easily delete later) 2. ssh into the master instance and delete whatever genomes you don't need/want (these are all located under /mnt/galaxyIndices) 3. Create a new EBS volume of size that'll fit whatever's left on the original volume, attach it and mount it 4. Copy over the data from the original volume to the new one while keeping the directory structure the same (rsync is probably the best tool for this) 5. Unmount & detach the new volume; create a snapshot from it 6. For the cluster you want to keep around (while it is terminated), edit persistent_data.yaml in it's bucket on S3 and replace the existing snap ID for the galaxyIndices with the snapshot ID you got in the previous step 7. Start that cluster and you should have a file system from the new snapshot mounted. 8. Terminate & delete the cluster you created in step 1" Here are my questions to help me to get through step 4: 1) Step 1: Is a "cluster" the same thing as an "instance"? 2) Step 2: I deleted the directories for individual genomes using rm -rf . Is that the correct approach? 3) Step 3: Do I add the newly created EBS volume to the same instance where I deleted the genomes? Or is it added the instance I want to keep? 4) Step 3: I can see how to attach this newly created volume using the AWS EC2 management console, but how do I mount it? (and unmount it in Step 5?) 5) Step 4: what is the syntax for the rsync (or cp) command to copy directories/files from one volume to another volume (within the same instance, or in if they are in different instances)? Any help would be gratefully received! Denise

2 1