- galaxy-dev - lists.galaxyproject.org

Basic setup question
by Malcolm Tobias 07 Jun '13

07 Jun '13

All, I manage a cluster and have been asked to set up a local instance of galaxy. I'm neither a galaxy user nor a biologist, so apologies in advance if my terminology is off. I have a basic setup running now, but have been asked to include the option to "display at UCSC Main" when a user uploads a bam and clicks on the dataset. Currently I have the options of "display in IGB Local Web". I've tried this at a few of the other Galaxy sites, and the behavior seems unpredictable. Uploading the same data set to some sites gives me this option, while others don't. I'm assuming this is determined by the option in the universe_wsgi.ini file: # UCSC browsers: tool-data/shared/ucsc/ucsc_build_sites.txt ucsc_display_sites = main,test,archaea,ucla but uncommenting this option and restarting the server doesn't help. Increasing the log_level to DEBUG doesn't provide any insight. Any suggestions about how to proceed? Could there be any firewall-related issues? I ask because of this in the configuration file: # If use_remote_user = True, display application servers will be denied access # to Galaxy and so displaying datasets in these sites will fail. # display_servers contains a list of hostnames which should be allowed to # bypass security to display datasets. Please be aware that there are security # implications if this is allowed. More details (including required changes to # the proxy server config) are available in the Apache proxy documentation on # the wiki. Thanks in advance, Malcolm -- Malcolm Tobias 314.362.1594

2 2

galaxy cloud ftp
by Mallon, Eamonn B. (Dr.) 07 Jun '13

07 Jun '13

Hi I started a cloud instance using cloud man (very easy by the way). The ftp instructions say use localhost. I'm assuming that's the webpage http://184.73.86.130/ (in my case). When I try and connect using cyberduck it says connection reset. Any ideas? Thanks Eamonn Dr Eamonn Mallon Lecturer in Evolutionary Biology Adrian 220 Biology Department University of Leicester http://www2.le.ac.uk/departments/biology/people/mallon

1 0

Problems running functional tests
by Ira Cooke 06 Jun '13

06 Jun '13

Hi All, I'm starting to add functional tests to my tools but I've found that I can't get the test framework to work well with tool dependencies. The tool I'm considering is xtandem http://testtoolshed.g2.bx.psu.edu/view/iracooke/xtandem When I install this tool on my local galaxy instance (on a clean install) the tool runs fine and if I look in the handler logs I see executing: PACKAGE_BASE=/home/iracooke/tool_dependencies/galaxy_protk/1.2.0/iracooke/xtandem/0940ff5d28ab; export PACKAGE_BASE; . /home/iracooke/tool_dependencies/galaxy_protk/1.2.0/iracooke/xtandem/0940ff5d28ab/env.sh; PACKAGE_BASE=/home/iracooke/tool_dependencies/trans_proteomic_pipeline/4.6.1/iracooke/xtandem/0940ff5d28ab; export PACKAGE_BASE; . /home/iracooke/tool_dependencies/trans_proteomic_pipeline/4.6.1/iracooke/xtandem/0940ff5d28ab/env.sh; rvm 1.9.3(a)protk-1.2.0 do tandem_search.rb -d /home/iracooke/galaxy-central/database/files/000/dataset_1.dat --var-mods=' 57.021464__at__C ' --fix-mods=' 15.994915__at__M ' /home/iracooke/galaxy-central/database/files/000/dataset_2.dat -o /home/iracooke/galaxy-central/database/files/000/dataset_4.dat -r --enzyme=Trypsin --precursor-ion-tol-units=ppm -v 0 -f 0.65 -p 100.0 --strict-monoisotopic-mass --keep-params-files galaxy.jobs DEBUG 2013-04-27 06:53:35,586 (4) Persisting job destination (destination id: local:///) Note that in this case the tool environments from my tool dependencies are loaded ... and the tool runs fine. When running functional tests however the tests are executed without first loading the environments from tool dependencies ... and naturally the tool fails. On my local machine I get the following when running the tests (Note that no loading of dependency shel environments occurs). (This seems to be the same issue on the testtoolshed.g2.bx.psu.edu ). Job dispatched galaxy.jobs.runners.local DEBUG 2013-04-27 06:55:19,564 (3) executing: rvm 1.9.3(a)protk-1.2.0 do tandem_search.rb -d /tmp/tmp3I8GgN/database/files/000/dataset_1.dat --var-mods=' 15.994915__at__M , ' --fix-mods=' 57.021464__at__C ' /tmp/tmp3I8GgN/database/files/000/dataset_2.dat -o /tmp/tmp3I8GgN/database/files/000/dataset_3.dat -r --enzyme=Trypsin --precursor-ion-tol-units=ppm -v 0 -f 0.65 -p 100.0 --strict-monoisotopic-mass --keep-params-files galaxy.jobs DEBUG 2013-04-27 06:55:19,730 (3) Persisting job destination (destination id: local:///) galaxy.jobs.runners.local DEBUG 2013-04-27 06:55:19,889 execution finished: rvm 1.9.3(a)protk-1.2.0 do tandem_search.rb -d /tmp/tmp3I8GgN/database/files/000/dataset_1.dat --var-mods=' 15.994915__at__M ,' --fix-mods=' 57.021464__at__C ' /tmp/tmp3I8GgN/database/files/000/dataset_2.dat -o /tmp/tmp3I8GgN/database/files/000/dataset_3.dat -r --enzyme=Trypsin --precursor-ion-tol-units=ppm -v 0 -f 0.65 -p 100.0 --strict-monoisotopic-mass --keep-params-files ... galaxy.jobs DEBUG 2013-04-27 06:55:24,277 setting dataset state to ERROR Is there something wrong with the way I've defined my tool dependencies for this tool? Or perhaps it's a bug in the testing framework. Cheers Ira

4 7

Bypassing the proxy server when using the API, without breaking the automatic update of the history in the web interface
by Anthonius deBoer 06 Jun '13

06 Jun '13

Hi, I am trying desperately to setup my GALAXY system so that if you enter the website, you need to authenticate yourself using the proxy setup... I also would like to be able to BYPASS the proxy server when an API call is made... In the example proxy setting I have setup in the following way: ReWriteRule ^(/api/histories/.*) http://localhost:8080$1 [P,L] ReWriteRule ^(/api/.*) http://localhost:8081$1 [P,L] RewriteRule ^(.*) http://localhost:8080$1 [P] I then setup a <PROXY setting in the apache galaxy.d in /etc/httpd/conf.d/galaxy.d to require authentication when a user is re-directed to the localhost:8080 and allow the calls to localhost:8081 to pass unhindered... This works brilliantly EXCEPT for when you want to make an API call that involves some histories.... The call fails, since /api/histories calls are routed through the proxy server and require authentication, which I want to avoid... It seems that histories are "special" so they have their own line in the configuration file above... I tried to change ReWriteRule ^(/api/histories/.*) http://localhost:8080$1 [P,L] to ReWriteRule ^(/api/histories/.*) http://localhost:8081$1 [L] Which DOES work, but it also breaks the automatic update of the history in the webbrowser... I have to reload the page manually to update the history... Is there anyway I can get the automatic update to work again, but don't require autentication? Thanks Thon

3 3

local install problem
by Andrew Krohn 06 Jun '13

06 Jun '13

I looked through others' problems, but this seems unique. I just did a clean install of Ubuntu 12.04 and wanted to add Galaxy to integrate our various pipelines (16S rRNA via qiime, RADseq data, small genome assemblies). However, I seem to be stuck at the beginning of the install. Followed instructions from this page explicitly ( http://wiki.galaxyproject.org/Admin/Get%20Galaxy) When I run sh run.sh, I get the below output, and the server cannot be accessed via web browser. I am using python 2.7.3. I opened a python instance and ran import bz2, but then I get the same error (ImportError: No module named bz2). enggen@OptimusPrime:~/galaxy-dist$ sh run.sh Traceback (most recent call last): File "./scripts/paster.py", line 33, in <module> serve.run() File "/home/enggen/galaxy-dist/lib/galaxy/util/pastescript/serve.py", line 1049, in run invoke(command, command_name, options, args[1:]) File "/home/enggen/galaxy-dist/lib/galaxy/util/pastescript/serve.py", line 1055, in invoke exit_code = runner.run(args) File "/home/enggen/galaxy-dist/lib/galaxy/util/pastescript/serve.py", line 220, in run result = self.command() File "/home/enggen/galaxy-dist/lib/galaxy/util/pastescript/serve.py", line 643, in command app = loadapp( app_spec, name=app_name, relative_to=base, global_conf=vars) File "/home/enggen/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py", line 350, in loadapp return loadobj(APP, uri, name=name, **kw) File "/home/enggen/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py", line 374, in loadobj global_conf=global_conf) File "/home/enggen/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py", line 399, in loadcontext global_conf=global_conf) File "/home/enggen/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py", line 423, in _loadconfig return loader.get_context(object_type, name, global_conf) File "/home/enggen/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py", line 561, in get_context section) File "/home/enggen/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py", line 620, in _context_from_explicit value = import_string(found_expr) File "/home/enggen/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py", line 125, in import_string return pkg_resources.EntryPoint.parse("x=" + s).load(False) File "/home/enggen/galaxy-dist/lib/pkg_resources.py", line 1954, in load entry = __import__(self.module_name, globals(),globals(), ['__name__']) File "/home/enggen/galaxy-dist/lib/galaxy/web/buildapp.py", line 3, in <module> from galaxy.webapps.galaxy.buildapp import app_factory File "/home/enggen/galaxy-dist/lib/galaxy/webapps/galaxy/buildapp.py", line 17, in <module> import galaxy.model File "/home/enggen/galaxy-dist/lib/galaxy/model/__init__.py", line 23, in <module> import galaxy.datatypes.registry File "/home/enggen/galaxy-dist/lib/galaxy/datatypes/registry.py", line 5, in <module> import data, tabular, interval, images, sequence, qualityscore, genetics, xml, coverage, tracks, chrominfo, binary, assembly, ngsindex File "/home/enggen/galaxy-dist/lib/galaxy/datatypes/tabular.py", line 14, in <module> from galaxy.datatypes.checkers import is_gzip File "/home/enggen/galaxy-dist/lib/galaxy/datatypes/checkers.py", line 1, in <module> import os, gzip, re, gzip, zipfile, binascii, bz2, imghdr ImportError: No module named bz2 Andrew Krohn PhD Candidate, NAU Biological Sciences Research Technician, EnGGen

2 3

Issue with set_user_disk_usage.py and Postgres 8.x
by Lance Parsons 06 Jun '13

06 Jun '13

The recent updates to set_user_disk_usage.py for Postgres users have an issue with Postgres 8.x. The SQL in the pgcalc method (line 51) leads to the following error: sqlalchemy.exc.ProgrammingError: (ProgrammingError) column "d.total_size" must appear in the GROUP BY clause or be used in an aggregate function LINE 4: FROM ( SELECT d.total_siz... ^ The problem is that version of Postgres before 9.x were a bit more restrictive in the use of GROUP BY. This can be fixed using DISTINCT ON instead. See this StackOverflow post for more info: http://stackoverflow.com/questions/1769361/postgresql-group-by-different-fr… I've included a patch below. Let me know if a pull request would be preferred. --- a/scripts/set_user_disk_usage.py +++ b/scripts/set_user_disk_usage.py @@ -52,7 +52,7 @@ sql = """ UPDATE galaxy_user SET disk_usage = (SELECT COALESCE(SUM(total_size), 0) - FROM ( SELECT d.total_size + FROM ( SELECT DISTINCT ON (d.id) d.total_size, d.id FROM history_dataset_association hda JOIN history h ON h.id = hda.history_id JOIN dataset d ON hda.dataset_id = d.id @@ -62,7 +62,7 @@ AND d.purged = false AND d.id NOT IN (SELECT dataset_id FROM library_dataset_dataset_association) - GROUP BY d.id) sizes) + ) sizes) WHERE id = :id RETURNING disk_usage; """ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University

3 4

Installation Instructions slightly wrong
by Michael E. Cotterell 06 Jun '13

06 Jun '13

Just wanted to mention that the "Get the latest copy [..]" section at http://wiki.galaxyproject.org/Admin/Get%20Galaxy should have the user "cd" into galaxy-dist before performing "hg update stable". Sincerely, Michael E. Cotterell Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia mepcotterell(a)gmail.com (mailto:mepcotterell@gmail.com) mepcott(a)uga.edu (mailto:mepcott@uga.edu) mec(a)cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/

2 1

nested repositories and tools
by Simon Gladman 06 Jun '13

06 Jun '13

Hi all, I've been trying to get my velvet optimiser wrapper to automatically install the tool and its dependencies to no avail. Here is my problem in detail. I have what I thought would be a simple wrapper and tool to pack into the new toolshed. I have a tool_wrapper which points to a perl tool (VelvetOptimiser.pl) which in turn depends on some binaries being installed (velveth and velvetg). When I try and load these with tool dependencies and repository dependencies, galaxy cannot find the VelvetOptimiser.pl script nor the velvet binaries after they have been installed. (They're there, I've checked.) It comprises of a repository (velvetoptimiser_gvl) with: velvet_optimiser_vlsci.xml which points to a more complex velvet_optimiser_vlsci_wrapper.pl to create the actual command line. (This works when the dependencies are installed on the machine and available in the galaxy users path). A repository_dependencies.xml file which points to an orphan repository (velvetoptimiser_orphan_225) velvetoptimiser_orphan_225 has a tool_dependencies.xml to install the VelvetOptimiser.pl and associated modules and a repository_dependencies.xml file to point to the velvet orphan repository (velvet_1208). velvet_1208 has a tool_dependencies.xml file which clones, installs and makes the velvet binaries and puts them in the right place. All of the installations work when you install the tool from the toolshed. However, when I go to run the tool, it kept coming back saying it couldn’t find the VelvetOptimser.pl file. Then after the documentation was updated, I thought I may have to include the path as an environment like in the R_SCRIPT_PATH example in the wiki. So I set that up as well to no avail. I’ve left it in the files attached.. Both the toolshed that these repos are in and the galaxy being installed into are the latest versions as of 05/06/2013. Please help! I’ve attached all of the relevant files below. Cheers, Simon Gladman. Victorian Bioinformatics Consortium Monash University, Australia. *Respository: velvetoptimiser_gvl -- the main repo.* *File: velvet_optimiser_vlsci.xml (salient parts)* <tool id="velvetoptimiser_vlsci" name="Velvet Optimiser vlsci" version="1.0.0"> <description>Automatically optimise a de-novo assembly using Velvet.</description> <command interpreter="perl"> velvet_optimiser_wrapper_vlsci.pl \$VOPT_SCRIPT_PATH '$start_kmer' '$end_kmer' '$kmer_step' #for i in $inputs: '$i.__index__' '${i.filetype}' … … </outputs> <requirments> <requirement type="set_environment">VOPT_SCRIPT_PATH</requirement> <requirement type="package" version="2.2.5">velvetoptimiser_orphan_225</requirement> </requirments> … *File: repository_dependencies.xml* <?xml version="1.0"?> <repositories description="Velvet Optimiser requires access to the VelvetOptimiser package."> <repository toolshed="http://galaxy-dev.genome.edu.au:9009/" name=" velvetoptimiser_orphan_225" owner="slugger" changeset_revision="fc5a642b8741" /> </repositories> *Repository: velvetoptimiser_orphan_225* * * *File: tool_dependencies.xml* <?xml version="1.0"?> <tool_dependency> <set_environment version="1.0"> <environment_variable name="VOPT_SCRIPT_PATH" action="set_to">$REPOSITORY_INSTALL_DIR</environment_variable> </set_environment> <package name="VelvetOptimiser-2.2.5" version="2.2.5"> <install version="1.0"> <actions> <action type="download_by_url"> http://www.vicbioinformatics.com/VelvetOptimiser-2.2.5.tar.gz</action> <action type="move_directory_files"> <source_directory>.</source_directory> <destination_directory>$INSTALL_DIR/bin</destination_directory> </action> <action type="set_environment"> <environment_variable name="PATH" action="prepend_to">$INSTALL_DIR/bin</environment_variable> </action> </actions> </install> </package> </tool_dependency> *File: repository_dependencies.xml* <?xml version="1.0"?> <repositories description="Velvet Optimiser requires access to the velvet binaries."> <repository toolshed="http://galaxy-dev.genome.edu.au:9009/" name="velvet_1208" owner="slugger" changeset_revision="2c95298637c9" /> </repositories> *Repository: velvet_1208* * * *File: tool_dependencies.xml* <?xml version="1.0"?> <tool_dependency> <package name="velvet" version="1.2.08"> <install version="1.0"> <actions> <action type="shell_command">git clone git:// github.com/dzerbino/velvet.git</action> <action type="shell_command">make -i CATEGORIES=4 MAXKMERLENGTH=191 OPENMP=1</action> <action type="move_file"> <source>velvetg</source> <destination>$INSTALL_DIR/bin</destination> </action> <action type="move_file"> <source>velveth</source> <destination>$INSTALL_DIR/bin</destination> </action> <action type="set_environment"> <environment_variable name="PATH" action="prepend_to">$INSTALL_DIR/bin</environment_variable> </action> </actions> </install> <readme> Installs velvet latest version. (called 1.2.08 for Galaxy purposes.) </readme> </package> </tool_dependency>

2 1

Re: [galaxy-dev] none mapping is not currently available!
by Sameer Dixit 05 Jun '13

05 Jun '13

Sorry Ross, didn't realise the image wouldn't come up. Please find attached a screen shot of the workflow. I tried to run a small workflow on Galaxy, which basically attempts to establish a correlation between the Chip-Chip, Polymerase binding and Dnase Hypersensitivity data. The input would be these 3 files, I am initially performing a liftover on the Chip-Chip data to convert from hg18 to hg19 and then using the output to establish the correlation. As I am running this workflow, the desired output isn't obtained and the following error: "none mapping is not currently available" is obtained. Can someone please explain what this means, and what am I to do after this! On Wed, Jun 5, 2013 at 5:31 PM, Ross <ross.lazarus(a)gmail.com> wrote: > Sameer, there has to be a better way of communicating - the email list is > not a good way to share images it seems. Perhaps a note to the list with a > url to a blog or other more flexible resource? > > > On Thu, Jun 6, 2013 at 2:49 AM, Sameer Dixit <samwise.techno(a)gmail.com>wrote: > >> Hi all, >> >> I tried to run a small workflow on Galaxy, which basically attempts to >> establish a correlation between the Chip-Chip, Polymerase binding and Dnase >> Hypersensitivity data. The input would be these 3 files, I am initially >> performing a liftover on the Chip-Chip data to convert from hg18 to hg19 >> and then using the output to establish the correlation. Here, you can see a >> screenshot of the pipeline here: >> >> As I am running this workflow, the desired output isn't obtained and the >> following error: "none mapping is not currently available" is obtained. >> Can someone please explain what this means, and what am I to do after >> this! >> >> Thanks a lot people!! >> -Sameer >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ >> > > > > -- > Ross Lazarus MBBS MPH; > Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444 > http://scholar.google.com/citations?hl=en&user=UCUuEM4AAAAJ >

1 0

Galaxy Config Questions
by Nicholas Kline 05 Jun '13

05 Jun '13

Hi, Our lab is in the process of installing a local version of Galaxy on a computer with the following specs: - Mac Pro mid-2012 - OS X 10.8.3 - 2 2.4 GHz Intel Xeon Processors - 6 cores per processor, 12 cores total - 64 GB RAM This installation is meant to be used by a single user. We followed the instructions at http://wiki.galaxyproject.org/Admin/Get%20Galaxy and now have a working version of Galaxy available at localhost:8080, but we have a few questions before using it: 1. We want to use the NGS:Mapping tool 'Map with Bowtie for Illumina' which requires 'bowtie (0.12.7) with bowtie indexes'. Do we simply download the bowtie software, place it into /Users/administrator/galaxy-dist/tools, restart Galaxy, and the tool is available under NGS:Mapping ? 2. Continuing from question #1, does a type of 'package manager' exist for automating the installation of tools and dependencies? Can mercurial accomplish this or is this the purpose of the Tool Shed? 3. What is the purpose of the Galaxy-specific environment file and how is it used? 4. We have two hard drives: drive1 and drive2. Galaxy is installed on drive1 and our data files are on drive2. Will Galaxy know how to find the location of our data files on drive2? Also, will Galaxy know how to write the result of a job to drive2? 5. Does Galaxy come configured with an admin account and, if so, how is it accessed? 6. Does Galaxy use as many hardware resources as possible to execute each job (for example, all 12 cores)? If not, can it be configured to do so? Thank you for your help!

2 1