- galaxy-dev - lists.galaxyproject.org

Reducing the size of a volume in Galaxy Cloud
by Denise Clark 02 Nov '14

02 Nov '14

Hi, I wish to remove most of the mnt/galaxyIndices/genomes files so I can have a smaller volume that contains those files and save money. I am new to this system. I have set up a Galaxy Cloudman account and used the tools on the web interface, and I can ssh the server using Ubuntu. So far, so good. I tried the first approach here twice https://wiki.galaxyproject.org/CloudMan/CustomizeGalaxyCloud but the system hangs (>24h). I can't follow the "manual" instructions on this page and I'm not sure it would result in a smaller volume anyway. So, I was trying to follow Enis' steps from this thread to reduce the size of the galaxyIndices volume: http://dev.list.galaxyproject.org/Reducing-costs-in-Cloud-Galaxy-td4483448.… His instructions are: "1. Start a new Galaxy cluster (ie, one you can easily delete later) 2. ssh into the master instance and delete whatever genomes you don't need/want (these are all located under /mnt/galaxyIndices) 3. Create a new EBS volume of size that'll fit whatever's left on the original volume, attach it and mount it 4. Copy over the data from the original volume to the new one while keeping the directory structure the same (rsync is probably the best tool for this) 5. Unmount & detach the new volume; create a snapshot from it 6. For the cluster you want to keep around (while it is terminated), edit persistent_data.yaml in it's bucket on S3 and replace the existing snap ID for the galaxyIndices with the snapshot ID you got in the previous step 7. Start that cluster and you should have a file system from the new snapshot mounted. 8. Terminate & delete the cluster you created in step 1" Here are my questions to help me to get through step 4: 1) Step 1: Is a "cluster" the same thing as an "instance"? 2) Step 2: I deleted the directories for individual genomes using rm -rf . Is that the correct approach? 3) Step 3: Do I add the newly created EBS volume to the same instance where I deleted the genomes? Or is it added the instance I want to keep? 4) Step 3: I can see how to attach this newly created volume using the AWS EC2 management console, but how do I mount it? (and unmount it in Step 5?) 5) Step 4: what is the syntax for the rsync (or cp) command to copy directories/files from one volume to another volume (within the same instance, or in if they are in different instances)? Any help would be gratefully received! Denise

2 1

Unable to add R package dependency to tool_dependencies.xml
by Bruno Grande 01 Nov '14

01 Nov '14

I'm attempting to create a tool dependency definition file for ExomeCNV, which notably depends on R (I'm trying to use the package_r_3_1_1 repository). However, I can't seem to set up the dependency properly. See attached for my tool_dependencies.xml. Also, see here <http://cl.ly/YK3C> for the error message when I attempt to upload the tool_dependencies.xml file to the Tool Shed. One of the possible issues is that the toolshed repository tag attribute for specifying an external Tool Shed isn't being considered. This is being done in a local Tool Shed installation that doesn't include the package_r_3_1_1 repository. Therefore, I must refer to the Test Tool Shed. This is inspired by a tool definition <https://github.com/bgruening/galaxytools/blob/78f35bc97160bdfe4783c9ac2e017…> written by Björn and the <repository> tag description <https://wiki.galaxyproject.org/ToolDependenciesTagSets>. Best regards, Bruno

4 3

Test failure, JSONDecodeError: Unpaired high surrogate
by Peter Cock 31 Oct '14

31 Oct '14

Hi all, I'm getting the following exception in a failing unit test: https://travis-ci.org/peterjc/pico_galaxy/builds/39398677 Testing this tool (where two of the three near identical tests passed): https://github.com/peterjc/pico_galaxy/blob/dd03346710e6a46cb6ec9dda1eed23d… ``` Traceback (most recent call last): File "/home/travis/build/peterjc/pico_galaxy/galaxy-central-master/test/functional/test_toolbox.py", line 116, in test_tool self.do_it( td ) File "/home/travis/build/peterjc/pico_galaxy/galaxy-central-master/test/functional/test_toolbox.py", line 35, in do_it self._verify_outputs( testdef, test_history, jobs, shed_tool_id, data_list, galaxy_interactor ) File "/home/travis/build/peterjc/pico_galaxy/galaxy-central-master/test/functional/test_toolbox.py", line 75, in _verify_outputs galaxy_interactor.verify_output( history, jobs, output_data, output_testdef=output_testdef, shed_tool_id=shed_tool_id, maxseconds=maxseconds ) File "/home/travis/build/peterjc/pico_galaxy/galaxy-central-master/test/base/interactor.py", line 89, in verify_output self._verify_metadata( history_id, hid, attributes ) File "/home/travis/build/peterjc/pico_galaxy/galaxy-central-master/test/base/interactor.py", line 102, in _verify_metadata dataset = self._get( "histories/%s/contents/%s" % ( history_id, hid ) ).json() File "/home/travis/build/peterjc/pico_galaxy/galaxy-central-master/eggs/requests-2.2.1-py2.7.egg/requests/models.py", line 740, in json return json.loads(self.content.decode(encoding), **kwargs) File "/usr/lib/python2.7/dist-packages/simplejson/__init__.py", line 413, in loads return _default_decoder.decode(s) File "/usr/lib/python2.7/dist-packages/simplejson/decoder.py", line 402, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python2.7/dist-packages/simplejson/decoder.py", line 418, in raw_decode obj, end = self.scan_once(s, idx) JSONDecodeError: Unpaired high surrogate: line 1 column 785 (char 785) ``` Probably relevant: - https://github.com/simplejson/simplejson/issues/62 - http://bugs.python.org/issue11489 Any thoughts? What does Galaxy write to these job-associated JSON metadata files? Thanks, Peter

2 2

Editing admin rights on an (empty) ToolShed repo
by Peter Cock 30 Oct '14

30 Oct '14

Hi all, It seems the ToolShed now uses roles for granting admin rights... but still has the old "Grant authority to make changes" feature? I just hit a possible glitch here - I wanted to create a new repo under the "iuc" user, edit the admin settings, then log in as my normal personal account and upload the first version of the tool. So: 1. Log into the Tool Shed as the "iuc" user. 2. Created https://toolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_30 3. Attempted to add other administrators, e.g. IUC group or myself, but the top right menu only offered "upload", and the old panel to do this was also missing: "Grant authority to make changes" If I do the first upload as "iuc", then the menu changes to include "Manage Repository Administrators", plus the panel on the main page appears "Grant authority to make changes" (which is what we used to use). Is this a transition stage, or are the change rights a subset of the admin role? Peter

1 0

ebio_deseq tool error
by Sergei Manakov 29 Oct '14

29 Oct '14

Hello, I installed 'ebio_deseq' tool via Admin/Tool sheds, and I am getting this error (looks like someting is making R plotting code unhappy). Error running DESeq analyse: Error in if (!ok) warning("some notches went outside hinges ('box'): maybe set notch=FALSE") : missing value where TRUE/FALSE needed Calls: PlotViolin ... boxplot.matrix -> boxplot -> boxplot.default -> do.call -> bxp Execution halted I would be grateful for help or suggestions for addressing this issue. thanks, Sergei

1 0

Galaxy with Rocks Cluster / Submitting Job As Real User
by Tim Gustafson 29 Oct '14

29 Oct '14

Hi, I've been trying to get Galaxy to submit jobs to my cluster as the "real user" rather than the "galaxy" user. In my galaxy.ini, I have: outputs_to_working_directory = True set_metadata_externally = True drmaa_external_runjob_script = /galaxy/scripts/drmaa_external_runner.py drmaa_external_killjob_script = /galaxy/scripts/drmaa_external_killer.py drmaa_external_chown_script = /galaxy/scripts/external_chown_script.py In my sudoers file I have: galaxy ALL= (root) NOPASSWD: SETENV: /galaxy/scripts/drmaa_external_runner.py galaxy ALL= (root) NOPASSWD: SETENV: /galaxy/scripts/drmaa_external_killer.py galaxy ALL= (root) NOPASSWD: SETENV: /galaxy/scripts/external_chown_script.py In my log file, I'm seeing: galaxy.jobs.runners.drmaa DEBUG 2014-10-29 14:45:54,777 (15) submitting file /galaxy/database/job_working_directory/000/15/galaxy_15.sh galaxy.jobs.runners.drmaa DEBUG 2014-10-29 14:45:54,783 (15) submitting with credentials: tjg [uid: 28369] galaxy.jobs.runners ERROR 2014-10-29 14:45:54,784 (15) Unhandled exception calling queue_job Traceback (most recent call last): File "/galaxy/lib/galaxy/jobs/runners/__init__.py", line 96, in run_next method(arg) File "/galaxy/lib/galaxy/jobs/runners/drmaa.py", line 189, in queue_job filename = self.store_jobtemplate(job_wrapper, jt) File "/galaxy/lib/galaxy/jobs/runners/drmaa.py", line 348, in store_jobtemplate f = open(filename,'w') IOError: [Errno 2] No such file or directory: '/galaxy/database/pbs/15.jt_json' My /var/log/messages file is not reporting any attempt to run "sudo", so I think the process is failing before it gets to that point in the Galaxy code. If I comment out the "drmaa_external_*" lines, the jobs are submitted successfully and run as the "galaxy" user. What am I missing here? :) -- Tim Gustafson tjg(a)ucsc.edu 831-459-5354 Baskin Engineering, Room 313A

1 0

Role of suite_config.xml in current Tool Shed?
by Peter Cock 28 Oct '14

28 Oct '14

Hi all, I have a suite_config.xml file in one of my Tool Shed packages, but I am unclear if this is still used, or simply a legacy from the old pre-hg-based Tool Shed? e.g. https://github.com/peterjc/pico_galaxy/blob/master/tools/protein_analysis/s… My understanding was this allowed the tool author some control over the appearance/order that the tools will be shown in the Galaxy left hand pane. Thanks, Peter

1 0

Database Error When Bootstrapping Tool Shed
by Bruno Grande 27 Oct '14

27 Oct '14

I initially posted this question on Galaxy Biostars <https://biostar.usegalaxy.org/p/9389/>, but realized that galaxy-dev is probably more appropriate. I'm attempting to host a local instance of the Galaxy Tool Shed for development purposes. I've already gotten the Galaxy server up and running. I don't know where the Galaxy version number is located, but I have revision 78c9a97be4b6+ of galaxy-dist repository. I did manage to get the Tool Shed server working, but upon reading this blog post <http://gregvonkuster.org/galaxy-tool-shed-introduction/> by Greg Von Kuster (see section "Hosting Your Own Tool Shed for Developing Galaxy Tools and Utilities"), I wanted to bootstrap from the main Galaxy Tool Shed ( http://toolshed.g2.bx.psu.edu) First of all, I'm trying to make use of PostgreSQL as the database backend. I have enabled the database_connection option in my tool_shed.ini configuration file using the appropriate postgresql://connection string. However, the default SQLite database file in database/community.sqlite is still created, even with the database_file = database/community.sqlite line commented out in the tool_shed.ini config file. Is this normal? I can see a Galaxy-like schema in both the SQLite and PostgreSQL databases. However, it seems that the SQLite is the one being used. For instance, in order to re-run the bootstrapping command, the SQLite database file must be cleared. Otherwise, I get the error below. The same isn't true with the PostgreSQL database. Unable to bootstrap tool shed. The database configured for this Tool Shed is not new, so bootstrapping is not allowed. Create a new database that has not been migrated before attempting to bootstrap. Second of all, while running the bootstrapping command (command and output <https://gist.github.com/brunogrande/4277625d8be45e6a4840>), I run into a SQLAlchemy error claiming that the "repository" table doesn't exist. It's unclear which database it's referring to, but that table exists in both the SQLite and PostgreSQL databases. Therefore, I'm left wondering if it's referring to the public Tool Shed I'm bootstrapping from. Also, while the following steps say that they're "done", neither databases are populated with categories, nor with users. When I look into the tool_shed_bootstrap.log file (see attached), one thing I notice that that one of the GET HTTP requests fails with a 404 status ("GET /api/authenticate/baseauth/ HTTP/1.1" 404 - "-" "curl/7.35.0"), followed by many POST requests ("POST /api/users?key=a HTTP/1.1" 200 - "-" "Python-urllib/2.7"). Let me know if you could use any additional information to troubleshoot these issues. Best regards, Bruno

1 0

Dependencies not working on toolshed?
by Lukasse, Pieter 27 Oct '14

27 Oct '14

Hi , I am trying to get a dependency to work in practice (when running a tool after the installation). But I keep getting the following message when running the tool: [sshexec] galaxy.tools.deps DEBUG 2014-10-24 16:04:15,514 Building dependency shell command for dependency 'R_bioc_metams' [sshexec] galaxy.tools.deps WARNING 2014-10-24 16:04:15,514 Failed to resolve dependency on 'R_bioc_metams', ignoring This is the tool requirements section: <requirements> <requirement type="package" version="3.1.1">R_bioc_metams</requirement> </requirements> This is the repository_dependencies.xml : <repositories description="Required metaMS dependencies."> <repository toolshed="http://testtoolshed.g2.bx.psu.edu" name="prims_metabolomics_dependencies" owner="pieterlukasse" changeset_revision="71356c62e5cd" /> </repositories> This is the package name of the package that is being referred: <tool_dependency> <package name="R_bioc_metams" version="3.1.1"> <install version="1.0"> <actions_group> .... I can't find the problem. Any ideas? I have updated to the latest Galaxy version (today) and removed both items (with option to remove files) and installed them again. Am I facing a bug here? Thanks, Pieter Lukasse Wageningen UR, Plant Research International Department of Bioinformatics (Bioscience) Wageningen Campus, Building 107, Droevendaalsesteeg 1, 6708 PB, Wageningen, the Netherlands T: +31-317481122; M: +31-628189540; skype: pieter.lukasse.wur http://www.pri.wur.nl<http://www.pri.wur.nl/>

4 5

My CloudMan EC2 instance won't start Galaxy
by Jim White 27 Oct '14

27 Oct '14

When I start CloudMan following the instructions at (https://wiki.galaxyproject.org/CloudMan) (ami-a7dbf6ce, Galaxy CloudMan 2.3) I get pretty far in the process but then Galaxy won't start because the EBS volumes are never mounted. The Volume console shows that the volumes get created based on the public snapshots but they never go to "in use" which suggest that the call to mount them isn't getting done for some reason. I probably will end up building my own CloudMan AMI (I'm mostly interested in HTCondor DAGman rather than Galaxy) but I figured you'd want to know about this issue. Thanks, Jim Admin Console:https://dl.dropboxusercontent.com/u/73840666/CloudMan%20Trouble/Adm… EC2 Volumes:https://dl.dropboxusercontent.com/u/73840666/CloudMan%20Trouble/Vol… Logs: Cluster status log 13:22:17 - Master starting 13:22:18 - Successfully retrieved root user's public key from file. 13:22:18 - Completed the initial cluster startup process. This is a new cluster; waiting to configure the type. 13:22:22 - Migration service prerequisites OK; starting the service 13:22:22 - SGE service prerequisites OK; starting the service 13:22:28 - Setting up SGE... 13:22:36 - Initializing 'Galaxy' cluster type with storage type 'volume'. Please wait... 13:22:40 - For filesystem galaxy, unknown kind: galaxy 13:22:40 - For filesystem galaxyIndices, unknown kind: galaxyIndices 13:22:45 - PostgreSQL data directory '/mnt/galaxy/db' does not exist (yet?) 13:22:46 - Adding volume vol-9fe10c87 (FS object for galaxy)... 13:22:46 - Could not determine next available device from frozenset([]) 13:22:46 - Error adding file system service FS object for galaxy: 'NoneType' object is not iterable 13:22:46 - Adding volume vol-6ae10c72 (FS object for galaxyIndices)... 13:23:32 - Could not determine next available device from frozenset([]) 13:23:32 - Error adding file system service FS object for galaxyIndices: 'NoneType' object is not iterable 13:23:36 - Empty disk usage for FS transient_nfs 13:23:36 - STATUS CHECK: File system named 'galaxy' is not mounted. Error code 0 13:23:36 - STATUS CHECK: File system named 'galaxyIndices' is not mounted. Error code 0 13:23:36 - PostgreSQL data directory '/mnt/galaxy/db' does not exist (yet?) 13:23:49 - Empty disk usage for FS transient_nfs 13:23:49 - STATUS CHECK: File system named 'galaxy' is not mounted. Error code 0 13:23:49 - STATUS CHECK: File system named 'galaxyIndices' is not mounted. Error code 0 13:23:49 - PostgreSQL data directory '/mnt/galaxy/db' does not exist (yet?) 13:24:01 - Empty disk usage for FS transient_nfs 13:24:01 - STATUS CHECK: File system named 'galaxy' is not mounted. Error code 0 13:24:01 - STATUS CHECK: File system named 'galaxyIndices' is not mounted. Error code 0 13:24:01 - PostgreSQL data directory '/mnt/galaxy/db' does not exist (yet?) 13:24:14 - Empty disk usage for FS transient_nfs 13:24:14 - STATUS CHECK: File system named 'galaxy' is not mounted. Error code 0 13:24:14 - STATUS CHECK: File system named 'galaxyIndices' is not mounted. Error code 0 Admin Console log 2014-10-25 13:27:42,910 DEBUG master:2445 Monitor adding service 'Galaxy' 2014-10-25 13:27:42,910 DEBUG __init__:300 Galaxy service prerequisites are not yet satisfied, waiting for: [, , , , ]. Setting Galaxy service state to 'Unstarted' 2014-10-25 13:27:42,911 DEBUG master:2445 Monitor adding service 'GalaxyReports' 2014-10-25 13:27:42,911 DEBUG __init__:300 GalaxyReports service prerequisites are not yet satisfied, waiting for: []. Setting GalaxyReports service state to 'Unstarted' 2014-10-25 13:27:47,307 DEBUG sge:552 qstat: ['all.q(a)ip-10-158-92-92.ec2.inte BIP 0/0/1 0.33 lx24-amd64 '] 2014-10-25 13:27:47,344 WARNING filesystem:518 Empty disk usage for FS transient_nfs 2014-10-25 13:27:47,366 ERROR filesystem:582 STATUS CHECK: File system named 'galaxy' is not mounted. Error code 0 2014-10-25 13:27:47,390 ERROR filesystem:582 STATUS CHECK: File system named 'galaxyIndices' is not mounted. Error code 0 2014-10-25 13:27:47,391 WARNING postgres:199 PostgreSQL data directory '/mnt/galaxy/db' does not exist (yet?) 2014-10-25 13:27:47,404 DEBUG master:2534 S&S: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; PSS..Unstarted; FS object for galaxy..Error; FS object for galaxyIndices..Error; Postgres..Unstarted; ProFTPd..Unstarted; Galaxy..Unstarted; GalaxyReports..Unstarted; 2014-10-25 13:27:47,404 DEBUG master:2445 Monitor adding service 'PSS' 2014-10-25 13:27:47,404 DEBUG pss:87 FS object for galaxy not running (Error), PSS service prerequisites not met afterall,not starting the service yet 2014-10-25 13:27:47,404 DEBUG master:2445 Monitor adding service 'Postgres' 2014-10-25 13:27:47,405 DEBUG __init__:300 Postgres service prerequisites are not yet satisfied, waiting for: []. Setting Postgres service state to 'Unstarted' 2014-10-25 13:27:47,405 DEBUG master:2445 Monitor adding service 'ProFTPd' 2014-10-25 13:27:47,405 DEBUG __init__:300 ProFTPd service prerequisites are not yet satisfied, waiting for: [, ]. Setting ProFTPd service state to 'Unstarted' 2014-10-25 13:27:47,405 DEBUG master:2445 Monitor adding service 'Galaxy' 2014-10-25 13:27:47,405 DEBUG __init__:300 Galaxy service prerequisites are not yet satisfied, waiting for: [, , , , ]. Setting Galaxy service state to 'Unstarted' 2014-10-25 13:27:47,405 DEBUG master:2445 Monitor adding service 'GalaxyReports' 2014-10-25 13:27:47,406 DEBUG __init__:300 GalaxyReports service prerequisites are not yet satisfied, waiting for: []. Setting GalaxyReports service state to 'Unstarted' 2014-10-25 13:27:51,407 DEBUG master:2445 Monitor adding service 'PSS' 2014-10-25 13:27:51,408 DEBUG pss:87 FS object for galaxy not running (Error), PSS service prerequisites not met afterall,not starting the service yet

2 2