- galaxy-dev - lists.galaxyproject.org

Seeking Job Metrics Clarity
by Ziphozakhe Mashologu 26 Feb '16

26 Feb '16

Hi All I have enabled job metrics on a local galaxy install, with latest code from dev branch. I have "<collectl /> " uncommented in job_metrics_conf.xml file: Following are the errors in the logs: galaxy.jobs.metrics ERROR 2016-02-24 12:28:08,741 Failed to collect job properties for plugin <galaxy.jobs.metrics.instrumenters.collectl.CollectlPlugin object at 0x7ff0b6af3a50> Traceback (most recent call last): File "dev/galaxy/lib/galaxy/jobs/metrics/__init__.py", line 101, in collect_properties properties = plugin.job_properties( job_id, job_directory ) File "dev/galaxy/lib/galaxy/jobs/metrics/instrumenters/collectl.py", line 105, in job_properties raise Exception( message ) Exception: Failed to find collectl log in directory It seems like collectl is looking for the following files: ['__instrument_collectl_pid', '__instrument_env_vars', 'galaxy.json', 'metadata_kwds_HistoryDatasetAssociation_2_1rh1WC', 'set_metadata_N05Q24.py', '__instrument_core_galaxy_slots', 'metadata_results_HistoryDatasetAssociation_2_33gTDj', '__instrument_core_epoch_start', 'galaxy_2.sh', 'metadata_out_HistoryDatasetAssociation_2_G5bZYP', '__instrument_core_epoch_end', '__instrument_meminfo_meminfo', '__instrument_uname_uname', 'metadata_override_HistoryDatasetAssociation_2_krldDM', 'set_metadata_lpIXS5.py', 'metadata_in_HistoryDatasetAssociation_2_iiYW0t', '__instrument_cpuinfo_cpuinfo', 'tool_script.sh', 'galaxy_2.ec'] Which are suppose to exist under galaxy/database/job_working_directory/000 job directory. Has anybody faced anything similar issue or is it a configuration issue, or perhaps a known issue with job_metrics (collectl)? BTW: I installed collectl packages on the host. Regards Zipho

2 2

Re: [galaxy-dev] JBrowse on dev branch: progress
by Peter van Heusden 26 Feb '16

26 Feb '16

Thanks Peter I managed, on the second attempt, to get past the gnu coreutils problem. Zipho noticed a weird lockup though: conftest that seemed to be from the gnu coreutils install wanted to listen on port 4001, which is the same port our uwsgi server listens on. The install has now completed but I'll try and find time to go back and verify this. On the second attempt perl seems to be installing, so I'll see if it completes or hits the error you're mentioning. Peter On 22 February 2016 at 11:34, Peter Briggs <peter.briggs(a)manchester.ac.uk> wrote: > Hi Peter > > I think that package_perl_5_18 in the tool shed is broken, I came across > the same error last week on both our production instance and a local 'test' > instance (both v15.10). > > My suspicion is that the following lines are to blame: > > ... > <action > sha256sum="5c69adb47ab828aa3e8b5be89b88cd49c6a0d0dae2e8b3bca17a9ce699190e7b" > type="download_file"> > http://www.cpan.org/authors/id/A/AP/APEIRON/local-lib-1.008009.tar.gz > </action> > <action type="shell_command">tar xfvz > local-lib-1.008009.tar.gz</action> > ... > > Looking at the Galaxy internals I believe that the 'download_file' action > automatically unpacks the archive and cd's into the resulting directory, > hence the failure for the subsequent 'tar xfvz ...' command. But I haven't > had the time yet to verify this or create a patch to see if the removal of > this line addresses the problem. > > I don't think I've seen the problem with 'package_gnu_coreutils_8_22' that > you describe however so it may be that my hypothesis is not correct. > > HTH > > Best wishes > > Peter > > > On 22/02/16 05:10, Peter van Heusden wrote: > >> Hi there Eric and others >> >> I tried to install the JBrowse tool on a dev branch server running on an >> Ubuntu 14.04 VM. >> >> It got stuck at the package_gnu_coreutils_8_22 dependency (revision >> ac64dfe4b1fb) - this ended up in "Installing tool dependencies" state. >> >> If I go look at the tool details it highlights errors >> in package_perl_5_18 which in turn refers to errors in gnu_coreutils >> (just "Error" state) and then this more informative failure for perl: >> >> "tar (child): local-lib-1.008009.tar.gz: Cannot open: No such file or >> directory >> tar (child): Error is not recoverable: exiting now >> tar: Child returned status 2 >> tar: Error is not recoverable: exiting now" >> >> I wonder if this is a sign of a download error? >> >> Digging a bit more I find the error "Shutting down process id 21094 >> because it generated no output for the defined timeout period of 3600.0 >> seconds." for gnu_coreutils. >> >> I'm trying to do a reinstall of the gnu_coreutils and perl packages to >> see if I can get further. >> >> Peter >> >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> https://lists.galaxyproject.org/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ >> >> > -- > Peter Briggs peter.briggs(a)manchester.ac.uk > Bioinformatics Core Facility University of Manchester > B.1083 Michael Smith Bldg Tel: (0161) 2751482 >

3 5

Re: [galaxy-dev] correctly installing to toolshed
by Gildas Le Corguillé 26 Feb '16

26 Feb '16

Hi all, You will found in attachment the modification, I proposed in my first email. As I said when your script is near your xml, you just need to use the interpreter attribute in <command> Cheers Gildas ----------------------------------------------------------------- Gildas Le Corguillé - Bioinformatician/Bioanalyste Plateforme ABiMS (Analyses and Bioinformatics for Marine Science) Station Biologique de Roscoff - UPMC/CNRS - FR2424 Place Georges Teissier 29680 Roscoff FRANCE tel: +33 2 98 29 23 81 http://abims.sb-roscoff.fr ------------------------------------------------------------------ > Le 25 févr. 2016 à 18:36, Karen Reddy <kreddy4(a)jhmi.edu> a écrit : > > Dear Gildas And Bjorn, > > Thanks so much for working on this! We are looking to test these tools ASAP and provide them within a workflow in an AMI (for a manuscript we are submitting to Genes and Development). Obviously, we have these scripts working from command line but we would like to also provide this to a more general audience. > > We are hoping to have everything up and running by this weekend, so please let us know if you need any more information from us. As Xianrong (Jose) mentioned this all works in planemo, and the tool appears to be deposited and easily installed, but remains 'unfound'. Also, please let us know if we did something wrong! > > Thanks again. > > Karen > > (PS you guys are awesome--Bjorn, Mo and i were just reminiscing about Galaxy 2015--are you going to Indianapolis?) > From: Karen Reddy <anotherpostdoc(a)gmail.com <mailto:anotherpostdoc@gmail.com>> > Sent: Thursday, February 25, 2016 12:26 PM > To: Karen Reddy > Subject: Fwd: [galaxy-dev] correctly installing to toolshed > > > ---------- Forwarded message ---------- > From: Xianrong Wong <wongxr(a)gmail.com <mailto:wongxr@gmail.com>> > Date: Wed, Feb 24, 2016 at 2:26 PM > Subject: Fwd: [galaxy-dev] correctly installing to toolshed > To: Karen Reddy <anotherpostdoc(a)gmail.com <mailto:anotherpostdoc@gmail.com>> > > > I followed instructions from here for tool building. > > > ---------- Forwarded message ---------- > From: Xianrong Wong <wongxr(a)gmail.com <mailto:wongxr@gmail.com>> > Date: Tue, Feb 23, 2016 at 1:05 PM > Subject: Re: [galaxy-dev] correctly installing to toolshed > To: Gildas Le Corguillé <lecorguille(a)sb-roscoff.fr <mailto:lecorguille@sb-roscoff.fr>> > Cc: Björn Grüning <bjoern.gruening(a)gmail.com <mailto:bjoern.gruening@gmail.com>>, galaxy-dev(a)lists.bx.psu.edu <mailto:galaxy-dev@lists.bx.psu.edu> > > > Hi Bjoern, it's stored in the same folder as the .xml > > Hi Gildas, that is the tool. > > Thanks! > > On Tue, Feb 23, 2016 at 12:43 PM, Gildas Le Corguillé <lecorguille(a)sb-roscoff.fr <mailto:lecorguille@sb-roscoff.fr>> wrote: > Is it this tool : https://testtoolshed.g2.bx.psu.edu/view/xianrong/removing_consecutive_bins_… <https://testtoolshed.g2.bx.psu.edu/view/xianrong/removing_consecutive_bins_…> > > Gildas > > ----------------------------------------------------------------- > Gildas Le Corguillé - Bioinformatician/Bioanalyste > Plateforme ABiMS (Analyses and Bioinformatics for Marine Science) > > Station Biologique de Roscoff - UPMC/CNRS - FR2424 > Place Georges Teissier 29680 Roscoff FRANCE > tel: +33 2 98 29 23 81 <tel:%2B33%202%2098%2029%2023%2081> > http://abims.sb-roscoff.fr <http://abims.sb-roscoff.fr/> > ------------------------------------------------------------------ > > > >> Le 23 févr. 2016 à 18:35, Björn Grüning <bjoern.gruening(a)gmail.com <mailto:bjoern.gruening@gmail.com>> a écrit : >> >> Hi, >> >> is the script rm_consecutive_sameScore.pl part of your TS repo and next >> to your tool XML file or where is it stored? >> >> Thanks, >> Bjoern >> >> Am 23.02.2016 um 17:46 schrieb Xianrong Wong: >>> Hi, I'm new to publishing tools/repository on the toolshed. I have used >>> planemo and lint and tests as well as serve all worked. I was able to >>> use the tool. However, when I uploaded it onto the test tool shed and >>> imported the tool on a cloud instance, I keep getting this error when >>> running job: >>> >>> Fatal error: Exit code 2 () Can't open perl script >>> "rm_consecutive_sameScore.pl": No such file or directory >>> >>> >>> >>> Please advise! I've been downloading and looking at what others did but >>> I still can't figure this out. >>> >>> >>> >>> I've attached my .xml file and here are some of the commands I used to >>> upload the tool. >>> >>> planemo shed_create --shed_target testtoolshed >>> >>> >>> >>> ___________________________________________________________ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>> https://lists.galaxyproject.org/ <https://lists.galaxyproject.org/> >>> >>> To search Galaxy mailing lists use the unified search at: >>> http://galaxyproject.org/search/mailinglists/ <http://galaxyproject.org/search/mailinglists/> >>> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> https://lists.galaxyproject.org/ <https://lists.galaxyproject.org/> >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ <http://galaxyproject.org/search/mailinglists/> > > > > > > -- >

3 2

AssertionError: there is no script for 130 version
by William Ge 25 Feb '16

25 Feb '16

Hi, all, I am new to Galaxy. The following error was encountered when I tried to run Galaxy after getting PostgreSQL up: File "/home/galaxy/galaxy/eggs/sqlalchemy_migrate-0.9.6-py2.6.egg/migrate/versioning/version.py", line 204, in script "There is no script for %d version" % self.version AssertionError: There is no script for 130 version I searched the lists and forums, but have not found any message. Anyone can give some hits? I'm running this on CentOS 6.7, virtual environment, python2.6, PostgresQL 8.4.20. Thanks in advance! Bill

2 1

Has anyone started defining a DataSource that connect to Google Big Query outputs?
by rbrown1422＠comcast.net 25 Feb '16

25 Feb '16

Dear team, Has anyone started defining a DataSource that connects to Google Big Query outputs? Thanks Bob

1 0

Query
by 유창혁 25 Feb '16

25 Feb '16

Dear Sir, I used your "Convert Format" from gff file of NCBI. but I encounter the following message. What can I do? Best regards, Chang-Hyuk Yoo. Traceback (most recent call last): File "/home/galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/vipints/fml_gff3togtf/5c6f33e20fcc/fml_gff3togtf/gff_to_gtf.py", line 76, in <module> printGTF(Transcriptdb) File "/home/galaxy/shed_tools/toolshed.g2.

3 3

Galaxy 16.01 Release
by Nate Coraor 25 Feb '16

25 Feb '16

The Galaxy Committers team is pleased to announce the January 2016 (v16.01) release of Galaxy. Galaxy administrators should also be aware of the security announcements that I am posting simultaneously with this release announcement. The release notes follow. From: https://docs.galaxyproject.org/en/master/releases/16.01_announce.html January 2016 Galaxy Release (v 16.01) Highlights *Interactive Tours* The interactive tours framework allows developers and deployers to build interactive tutorials for users superimposed on the actual Galaxy web front end. Unlike video tutorials, these will not become stale and are truly interactive (allowing users to actually navigate and interact with Galaxy). Galaxy 16.01 ships with two example tours and new ones can easily be added by creating a small YAML file describing the tour. Try the Galaxy UI tour on Main. *Wheels* Galaxy’s Python dependencies have traditionally been distributed as eggs using custom dependency management code to enable Galaxy to distribute binary dependencies (enabling quick downloads and minimal system requirements). With this release all of that infrastructure has been replaced with a modern Python infrastructure based on pip and wheels. Work done as part of this to enable binary dependencies on Linux has been included with the recently released pip 8. Detailed documentation on these changes and their impact under a variety of Galaxy deployment scenarios can be found in the Galaxy Framework Dependencies section of the Admin documentation. *Nested Workflows* Workflows may now run other workflows as a single abstract step in the parent workflow. This allows for reusing or subworkflows in your analyses. Github *New* % git clone -b master https://github.com/galaxyproject/galaxy.git *Update to latest stable release* % git checkout master && pull --ff-only origin master *Update to exact version* % git checkout v16.01 BitBucket *Upgrade* % hg pull % hg update latest_16.01 See our wiki for additional details regarding the source code locations. Deprecation Notices Barring a strong outcry from deployers, 16.01 will be the last release of Galaxy to support Python 2.6. For more information, see Galaxy Github Issue #1596.

2 3

Multiple web servers and handlers Galaxy failed
by Makis Ladoukakis 25 Feb '16

25 Feb '16

Hello everyone, I am trying to setup a Galaxy instance for a multi-user production environment and I tried to follow the instructions here: https://wiki.galaxyproject.org/Admin/Config/Performance/Scaling https://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer https://wiki.galaxyproject.org/Admin/Config/ApacheProxy but when I tried to start my galaxy instance (GALAXY_RUN_ALL=1 sh run.sh --daemon) I got the following error in my browser:Bad Gateway The proxy server received an invalid response from an upstream server. I don't get any errors from the paster.log files so I am guessing the error is somewhere in my apache configuration although I can't see where it could be. Has anyone encountered this problem before? Kind regards, Makis P.S. My apache configuration is as follows: I added that to my apache httpd.conf file: #for galaxy #RewriteRule ^/galaxy(.*) http://localhost:8091$1 [P] RewriteRule ^/galaxy$ /galaxy/ [R] RewriteRule ^/galaxy/static/style/(.*) /home/galaxy/galaxy/static/june_2007_style/blue/$1 [L] RewriteRule ^/galaxy/static/scripts/(.*) /home/galaxy/galaxy/static/scripts/packed/$1 [L] RewriteRule ^/galaxy/static/(.*) /home/galaxy/galaxy/static/$1 [L] RewriteRule ^/galaxy/favicon.ico /home/galaxy/galaxy/static/favicon.ico [L] RewriteRule ^/galaxy/robots.txt /home/galaxy/galaxy/static/robots.txt [L] LoadModule uwsgi_module /etc/httpd/modules/mod_uwsgi.so <Location "/galaxy"> Sethandler uwsgi-handler uWSGISocket 127.0.0.1:4001 uWSGImaxVars 512 </Location> I am also attaching my job_conf.xml and galaxy.ini file.

5 15

Galaxy Security Vulnerability - Read/write arbitrary filesystem paths, arbitrary code execution
by Nate Coraor 24 Feb '16

24 Feb '16

*DESCRIPTION* Multiple security vulnerabilities were recently discovered in Galaxy that allow malicious actors to read and write files on the Galaxy server. Additionally, Galaxy servers on which a rarely used feature has been enabled are vulnerable to an arbitrary code execution exploit. 1. A write vulnerability exists in the history import mechanism. It is possible to create a history tar archive that contains files with parent directory components in the file path (e.g. "foo/../../bar" would extract to "../bar"), and these archive members would be written if the user running the Galaxy server had write permission to the given path. 2. A read vulnerability exists in the object store path composition code. Galaxy allows clients to add elements to the end of a path to "extra" files associated with a dataset (as is the case with composite datatypes). These elements were not being checked to ensure they did not contain relative parent references ('..') or did not start with an absolute path character ('/'). Because of this, the dataset display methods could be manipulated to return the contents of any files for which the Galaxy server user had read permission. 3. An arbitrary code execution vulnerability exists in the Galaxy sample tracking system. The sample tracking system included a feature which allowed administrators to browse remote "external services" (such as sequencers) to choose files to transfer to the Galaxy server. This browsing code used a shell invocation which did not sanitize user input. However, this code is only reachable if at least one external service has ever been defined. The Galaxy Committers would like to thank Youri Hoogstrate at the Erasmus MC, Rotterdam, who initially reported a similar vulnerability in the Tool Shed. Through additional auditing based on this attack vector, we discovered the other vulnerabilities. *AFFECTED VERSIONS* This issue affects all known releases of Galaxy in at least the last 3 years. *IMPACT* The read and write vulnerabilities can be exploited to write to any path on the Galaxy server writable by the Galaxy user, which can be potentially destructive to Galaxy configuration files and data. Additionally, they can be used to read any file available on the system that is readable by the user running the Galaxy server, including Galaxy config files, data, system config files, contents of /proc, etc. As such, Galaxy administrators are strongly encouraged to update immediately. The arbitrary code execution vulnerability can be exploited to execute any shell command as the Galaxy user, which is also very dangerous. If your Galaxy instance has had an external service configured at any point in the past, you are strongly encouraged to update immediately. *SOLUTION* Fixes for these vulnerabilities have been applied to the 14.10 and newer release branches in the public GitHub and Bitbucket repositories. Galaxy versions older than 14.10 should upgrade (preferably to 16.01). To apply the fix, first identify your current Galaxy release version using the `git branch` or `hg branch` commands. If you are on a 'release_YY.MM' branch, you can update with: % git pull or: % hg pull -u The process above can also be used to update to the 16.01 release if you are on the 'master' git branch or the 'stable' hg branch. If you are on the 'master'/'stable' branch and wish to remain on your current Galaxy major release, check the 'lib/galaxy/version.py' file to determine your major release version, then update to the appropriate branch: % git checkout -b release_YY.MM origin/release_YY.MM % git pull or: % hg pull % hg update release_YY.MM For the changes to take effect, *YOU MUST RESTART ALL GALAXY SERVER PROCESSES*. On behalf of the Galaxy Committers, --nate

1 0

Galaxy/Tool Shed Security Vulnerability - Read/write arbitrary filesystem paths
by Nate Coraor 24 Feb '16

24 Feb '16

*DESCRIPTION* Multiple security vulnerabilities were recently discovered in the Tool Shed that allow malicious actors to read and write files on the Tool Shed server outside of normal Tool Shed repository directories. 1. A write vulnerability exists in the Tool Shed tarball and capsule upload functionality. It is possible to create a tar archive that contains files with parent directory components in the file path (e.g. "foo/../../bar" would extract to "../bar"), and these archive members would be written if the user running the Tool Shed had write permission to the given path. The Tool Shed tarball handling code checked for invalid characters ('/' or '..') at the beginning of the path but not for '..' in the middle of a path. 2. A read vulnerability exists in multiple places. The first is in the (now deprecated) `hg push` functionality for updating Tool Shed repositories. This method allows malicious actors to push symlinks whose targets are outside the repository (either via an absolute or relative path). The contents of the targets would then be visible in the Tool Shed repository contents viewer, if the Tool Shed user has read permission on the target. 3. A second read vulnerability exists in the Tool Shed repository contents viewer. The viewer would allow a malicious actor to specify a path outside the repository, and if the Tool Shed system user had read permissions on that path, it would be displayed. The viewer also did not check to ensure that the targets of symlinks in a repository did not point outside the repository. The repository contents viewer read vulnerability also exists in Galaxy, but is only reachable/exploitable by admin users. The Galaxy Committers would like to thank Youri Hoogstrate at the Erasmus MC, Rotterdam, who initially reported the `hg push` vulnerability. Through additional auditing based on this attack vector, we discovered the other vulnerabilities. *AFFECTED VERSIONS* These issues affect all known releases of the Tool Shed and Galaxy in at least the last 3 years. *IMPACT* This vulnerability can be exploited to read any file available on the system that is readable by the user running the Tool Shed or Galaxy server, including Galaxy/Tool Shed config files, data, system config files, contents of /proc, etc. As such, Tool Shed administrators are strongly encouraged to update immediately. For Galaxy administrators, the impact is not as great, unless Galaxy administrators do not have access to the Galaxy server command line. *SOLUTION* Fixes for these vulnerabilities have been applied to the 15.01 and newer (14.10 in the case of the portion of the vulnerability affecting Galaxy) release branches in the public GitHub and Bitbucket repositories. Tool Shed versions older than 15.01 should upgrade (preferably to 16.01). To apply the fix, first identify your current Galaxy release version using the `git branch` or `hg branch` commands. If you are on a 'release_YY.MM' branch, you can update with: % git pull or: % hg pull -u The process above can also be used to update to the 16.01 release if you are on the 'master' git branch or the 'stable' hg branch. If you are on the 'master'/'stable' branch and wish to remain on your current Galaxy major release, check the 'lib/galaxy/version.py' file to determine your major release version, then update to the appropriate branch: % git checkout -b release_YY.MM origin/release_YY.MM % git pull or: % hg pull % hg update release_YY.MM For the changes to take effect, *YOU MUST RESTART ALL GALAXY SERVER PROCESSES*. On behalf of the Galaxy Committers, --nate

1 0