- galaxy-dev - lists.galaxyproject.org

Galaxy/Tool Shed Security Vulnerability - Read/write arbitrary filesystem paths
by Nate Coraor 24 Feb '16

24 Feb '16

*DESCRIPTION* Multiple security vulnerabilities were recently discovered in the Tool Shed that allow malicious actors to read and write files on the Tool Shed server outside of normal Tool Shed repository directories. 1. A write vulnerability exists in the Tool Shed tarball and capsule upload functionality. It is possible to create a tar archive that contains files with parent directory components in the file path (e.g. "foo/../../bar" would extract to "../bar"), and these archive members would be written if the user running the Tool Shed had write permission to the given path. The Tool Shed tarball handling code checked for invalid characters ('/' or '..') at the beginning of the path but not for '..' in the middle of a path. 2. A read vulnerability exists in multiple places. The first is in the (now deprecated) `hg push` functionality for updating Tool Shed repositories. This method allows malicious actors to push symlinks whose targets are outside the repository (either via an absolute or relative path). The contents of the targets would then be visible in the Tool Shed repository contents viewer, if the Tool Shed user has read permission on the target. 3. A second read vulnerability exists in the Tool Shed repository contents viewer. The viewer would allow a malicious actor to specify a path outside the repository, and if the Tool Shed system user had read permissions on that path, it would be displayed. The viewer also did not check to ensure that the targets of symlinks in a repository did not point outside the repository. The repository contents viewer read vulnerability also exists in Galaxy, but is only reachable/exploitable by admin users. The Galaxy Committers would like to thank Youri Hoogstrate at the Erasmus MC, Rotterdam, who initially reported the `hg push` vulnerability. Through additional auditing based on this attack vector, we discovered the other vulnerabilities. *AFFECTED VERSIONS* These issues affect all known releases of the Tool Shed and Galaxy in at least the last 3 years. *IMPACT* This vulnerability can be exploited to read any file available on the system that is readable by the user running the Tool Shed or Galaxy server, including Galaxy/Tool Shed config files, data, system config files, contents of /proc, etc. As such, Tool Shed administrators are strongly encouraged to update immediately. For Galaxy administrators, the impact is not as great, unless Galaxy administrators do not have access to the Galaxy server command line. *SOLUTION* Fixes for these vulnerabilities have been applied to the 15.01 and newer (14.10 in the case of the portion of the vulnerability affecting Galaxy) release branches in the public GitHub and Bitbucket repositories. Tool Shed versions older than 15.01 should upgrade (preferably to 16.01). To apply the fix, first identify your current Galaxy release version using the `git branch` or `hg branch` commands. If you are on a 'release_YY.MM' branch, you can update with: % git pull or: % hg pull -u The process above can also be used to update to the 16.01 release if you are on the 'master' git branch or the 'stable' hg branch. If you are on the 'master'/'stable' branch and wish to remain on your current Galaxy major release, check the 'lib/galaxy/version.py' file to determine your major release version, then update to the appropriate branch: % git checkout -b release_YY.MM origin/release_YY.MM % git pull or: % hg pull % hg update release_YY.MM For the changes to take effect, *YOU MUST RESTART ALL GALAXY SERVER PROCESSES*. On behalf of the Galaxy Committers, --nate

1 0

Error in plot phylogenetic tree
by Fan, Xiaozhou 24 Feb '16

24 Feb '16

Hi, I was trying to plot the attached tree by using galaxy GraPhlAn<javascript:void(0)>. However, I got the following error: Traceback (most recent call last): File "/net/rcstore02/ifs/rc_labs/galaxy/www/galaxy_hutlab14/galaxy-hutlab14-20150109/tools/graphlan/graphlan.py", line 64, in <module> out_pad = args['pad']) Could you please help me out with this issue? Thank you very much! Best, Jane, Xiaozhoiu ------------------------------------------------------------ This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain information that is proprietary, confidential, and exempt from disclosure under applicable law. Any unauthorized review, use, disclosure, or distribution is prohibited. If you have received this email in error please notify the sender by return email and delete the original message. Please note, the recipient should check this email and any attachments for the presence of viruses. The organization accepts no liability for any damage caused by any virus transmitted by this email. =================================

1 0

Custom tool with multiple inputs for different datasets. How to preferrentially select
by Ryan G 24 Feb '16

24 Feb '16

Next question - My custom tool take as input mulitple datasets from different tools and collates all the information. For instance, 1 input as tabular input, sampleManifest, another input is also tabular, expressionData. When the tool is display, both entries default to the most recent tabular dataset in the history. Is there a way to have Galaxy try to guess which dataset to default to as input instead of always choosing the latest input for all the fields?

2 1

Restricting tool input to 1 dataset instead of multiple
by Ryan G 24 Feb '16

24 Feb '16

Hi all - I'm creating a tool in Galaxy that takes as input several datasets for different variables. For each variable, I want to restrict the input to a single dataset. The docs, https://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax, indicate type="data" needs to be set for a dataset from the current history to be available for input. When I do this, Galaxy displays the icon for Single Dataset, Mulitple Datasets, and Dataset collection. I only want to give the option for Single Dataset. I tried include multiple="false", but that doesn't seem to have any effect. Is there a way to accomplish this?

2 1

Re: [galaxy-dev] cleanup
by Nicola Soranzo 23 Feb '16

23 Feb '16

Hi Philipp, please alway reply to the mailing list. I'm not using Docker, but if Galaxy is in the /galaxy-central/ directory and its version is 16.01, then: $ cd /galaxy-central/ $ . .venv/bin/activate $ python scripts/cleanup_datasets/cleanup_datasets.py config/galaxy.ini -d -30 -1 $ deactivate The command ". .venv/bin/activate" will activate the virtualenv installed in /galaxy-central/.venv/ , "deactivate" will deactivate it. Cheers, Nicola On 23/02/16 19:14, Rathert, Philipp, Dr. wrote: > Hi Nicola, > Could you be a bit more specific? > I guess i activate this in the specific container but how exactly? > > Thanks a lot for your help. > > > Cheers, > > Philipp > ---Philipp on the road--- > > Am 23.02.2016 um 16:20 schrieb Nicola Soranzo <nsoranzo(a)tiscali.it > <mailto:nsoranzo@tiscali.it>>: > >> Hi Philipp, >> if you are using Galaxy release 16.01, you should load the virtualenv >> before running the cleanup script, e.g.: >> >> $ cd GALAXY_HOME >> $ . .venv/bin/activate >> $ python scripts/cleanup_datasets/cleanup_datasets.py ... >> >> Cheers, >> Nicola >> >> On 23/02/16 15:07, Hans-Rudolf Hotz wrote: >> > Hi Philipp >> > >> > >> > point 1 : that's as expected - good. >> > >> > point 2 : looks like a docker specific issue? Maybe Bjoern can jump in? >> > >> > >> > Hans-Rudolf >> > >> > >> > >> > >> > On 02/23/2016 12:17 PM, Rathert, Philipp, Dr. wrote: >> >> Dear Hans-Rudolf, >> >> >> >> thanks a lot for your help. >> >> >> >> I did as you suggested: >> >> >> >> 1. when I go into the scripts folder I get the error message you >> >> guessed: >> >> root@galaxy:/galaxy-central/scripts# python >> >> cleanup_datasets/cleanup_datasets.py >> >> /export/galaxy-central/config/galaxy.ini -d -30 -1 >> >> Traceback (most recent call last): >> >> File "cleanup_datasets/cleanup_datasets.py", line 21, in <module> >> >> import galaxy.config >> >> ImportError: No module named galaxy.config >> >> >> >> >> >> 2. from the root directory I recieve the following error message: >> >> root@galaxy:/galaxy-central# python >> >> scripts/cleanup_datasets/cleanup_datasets.py >> >> /export/galaxy-central/config/galaxy.ini -d -60 -1 >> >> Traceback (most recent call last): >> >> File "scripts/cleanup_datasets/cleanup_datasets.py", line 21, in >> >> <module> >> >> import galaxy.config >> >> File "/galaxy-central/lib/galaxy/config.py", line 21, in <module> >> >> from galaxy.util.dbkeys import GenomeBuilds >> >> File "/galaxy-central/lib/galaxy/util/dbkeys.py", line 7, in >> >> <module> >> >> from galaxy.util.object_wrapper import sanitize_lists_to_string >> >> File "/galaxy-central/lib/galaxy/util/object_wrapper.py", line >> >> 45, in >> >> <module> >> >> from six.moves import UserDict >> >> ImportError: cannot import name UserDict >> >> >> >> Any idea what this means? I thought this should be automatically >> >> installed with python... >> >> >> >> Cheers, >> >> >> >> Philipp >> >> >> >> -----Original message----- >> >> *From:* Hans-Rudolf Hotz <hrh(a)fmi.ch <mailto:hrh@fmi.ch>> >> >> *Sent:* Monday 22nd February 2016 17:09 >> >> *To:* Rathert, Philipp, Dr. >> <philipp.rathert(a)ibc.uni-stuttgart.de >> <mailto:philipp.rathert@ibc.uni-stuttgart.de>>; >> >> galaxy-dev(a)lists.galaxyproject.org >> <mailto:galaxy-dev@lists.galaxyproject.org> >> >> *Subject:* Re: [galaxy-dev] cleanup >> >> >> >> Hi Philipp >> >> >> >> > Since the new galaxy version, I am unable to perform a cleanup >> >> anymore. >> >> > >> >> > From the galaxy root directory I want to start the script >> >> > cleanup_datasets.py with the following line: >> >> > >> >> > $ python cleanup_datasets.py >> >> /export/galaxy-central/config/galaxy.ini -d >> >> > -60 -1 >> >> > >> >> > I get the error message: >> >> > >> >> > Can’t open file ‘cleanup_datasets.py’: no such file or >> directory. >> >> >> >> Have you tried: >> >> >> >> $ python scripts/cleanup_datasets/cleanup_datasets.py >> >> /export/galaxy-central/config/galaxy.ini -d -60 -1 >> >> >> >> > But when I go into the folder in which the script is located >> >> it doesn’t >> >> > work as well. >> >> >> >> what is the error this time? Is it "ImportError: No module named >> >> galaxy"? >> >> >> >> >> >> Regards, Hans-Rudolf >> >> >> >> >> >> > For your information: we are running the latest >> >> docker-galaxy-stable >> >> > version. >> >> > >> >> > I hope somebody has an idea. >> >> > >> >> > Cheers, >> >> > >> >> > Philipp >> >> > >> >> > ---------------------------------------------- >> >> > >> >> > Philipp Rathert, PhD >> >> > >> >> > Institute of Biochemistry >> >> > >> >> > University Stuttgart >> >> > >> >> > Pfaffenwaldring 55 >> >> > >> >> > D-70569 Stuttgart >> >> > >> >> > Germany >> >> > >> >> > Tel: +49 711 685 64388 >> >> > >> >> > Fax: +49 711 685 64392 >> >> > >> >> > philipp.rathert(a)ibc.uni-stuttgart.de >> <mailto:philipp.rathert@ibc.uni-stuttgart.de> >> >> > <mailto:philipp.rathert@ibc.uni-stuttgart.de> >> >> > >> >> > http://www.ibc.uni-stuttgart.de/ >> >> > >> >> > >> http://www.ibc.uni-stuttgart.de/mitarbeiter/Rathert/?__locale=en >> >> > >> >> > >> >> > >> >> > ___________________________________________________________ >> >> > Please keep all replies on the list by using "reply all" >> >> > in your mail client. To manage your subscriptions to this >> >> > and other Galaxy lists, please use the interface at: >> >> > https://lists.galaxyproject.org/ >> >> > >> >> > To search Galaxy mailing lists use the unified search at: >> >> > http://galaxyproject.org/search/mailinglists/ >> >> > >> >> >> > >> > ___________________________________________________________ >> > Please keep all replies on the list by using "reply all" >> > in your mail client. To manage your subscriptions to this >> > and other Galaxy lists, please use the interface at: >> > https://lists.galaxyproject.org/ >> > >> > To search Galaxy mailing lists use the unified search at: >> > http://galaxyproject.org/search/mailinglists/ >>

2 1

correctly installing to toolshed
by Xianrong Wong 23 Feb '16

23 Feb '16

Hi, I'm new to publishing tools/repository on the toolshed. I have used planemo and lint and tests as well as serve all worked. I was able to use the tool. However, when I uploaded it onto the test tool shed and imported the tool on a cloud instance, I keep getting this error when running job: Fatal error: Exit code 2 () Can't open perl script "rm_consecutive_sameScore.pl": No such file or directory Please advise! I've been downloading and looking at what others did but I still can't figure this out. I've attached my .xml file and here are some of the commands I used to upload the tool. planemo shed_create --shed_target testtoolshed

3 5

Re: [galaxy-dev] cleanup
by Hans-Rudolf Hotz 23 Feb '16

23 Feb '16

Hi Philipp point 1 : that's as expected - good. point 2 : looks like a docker specific issue? Maybe Bjoern can jump in? Hans-Rudolf On 02/23/2016 12:17 PM, Rathert, Philipp, Dr. wrote: > Dear Hans-Rudolf, > > thanks a lot for your help. > > I did as you suggested: > > 1. when I go into the scripts folder I get the error message you guessed: > root@galaxy:/galaxy-central/scripts# python > cleanup_datasets/cleanup_datasets.py > /export/galaxy-central/config/galaxy.ini -d -30 -1 > Traceback (most recent call last): > File "cleanup_datasets/cleanup_datasets.py", line 21, in <module> > import galaxy.config > ImportError: No module named galaxy.config > > > 2. from the root directory I recieve the following error message: > root@galaxy:/galaxy-central# python > scripts/cleanup_datasets/cleanup_datasets.py > /export/galaxy-central/config/galaxy.ini -d -60 -1 > Traceback (most recent call last): > File "scripts/cleanup_datasets/cleanup_datasets.py", line 21, in <module> > import galaxy.config > File "/galaxy-central/lib/galaxy/config.py", line 21, in <module> > from galaxy.util.dbkeys import GenomeBuilds > File "/galaxy-central/lib/galaxy/util/dbkeys.py", line 7, in <module> > from galaxy.util.object_wrapper import sanitize_lists_to_string > File "/galaxy-central/lib/galaxy/util/object_wrapper.py", line 45, in > <module> > from six.moves import UserDict > ImportError: cannot import name UserDict > > Any idea what this means? I thought this should be automatically > installed with python... > > Cheers, > > Philipp > > -----Original message----- > *From:* Hans-Rudolf Hotz <hrh(a)fmi.ch> > *Sent:* Monday 22nd February 2016 17:09 > *To:* Rathert, Philipp, Dr. <philipp.rathert(a)ibc.uni-stuttgart.de>; > galaxy-dev(a)lists.galaxyproject.org > *Subject:* Re: [galaxy-dev] cleanup > > Hi Philipp > > > Since the new galaxy version, I am unable to perform a cleanup anymore. > > > > From the galaxy root directory I want to start the script > > cleanup_datasets.py with the following line: > > > > $ python cleanup_datasets.py /export/galaxy-central/config/galaxy.ini -d > > -60 -1 > > > > I get the error message: > > > > Can’t open file ‘cleanup_datasets.py’: no such file or directory. > > Have you tried: > > $ python scripts/cleanup_datasets/cleanup_datasets.py > /export/galaxy-central/config/galaxy.ini -d -60 -1 > > > But when I go into the folder in which the script is located it doesn’t > > work as well. > > what is the error this time? Is it "ImportError: No module named galaxy"? > > > Regards, Hans-Rudolf > > > > For your information: we are running the latest docker-galaxy-stable > > version. > > > > I hope somebody has an idea. > > > > Cheers, > > > > Philipp > > > > ---------------------------------------------- > > > > Philipp Rathert, PhD > > > > Institute of Biochemistry > > > > University Stuttgart > > > > Pfaffenwaldring 55 > > > > D-70569 Stuttgart > > > > Germany > > > > Tel: +49 711 685 64388 > > > > Fax: +49 711 685 64392 > > > > philipp.rathert(a)ibc.uni-stuttgart.de > > <mailto:philipp.rathert@ibc.uni-stuttgart.de> > > > > http://www.ibc.uni-stuttgart.de/ > > > > http://www.ibc.uni-stuttgart.de/mitarbeiter/Rathert/?__locale=en > > > > > > > > ___________________________________________________________ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > > https://lists.galaxyproject.org/ > > > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ > > >

2 1

Galaxy 16.01, eggs vs wheels, and running jobs as the user
by Peter Cock 23 Feb '16

23 Feb '16

Hi all, Last week at the Galaxy Admins video hangout Nate gave an overview of the way Python dependency handling is changing from using egg files to using wheel files instead - slide links at: https://wiki.galaxyproject.org/Community/GalaxyAdmins/Meetups/2016_02_18 I understand the Galaxy v16.01 (2016 January feature freeze) release which switches from eggs to wheels is due out shortly - hopefully this week? Recently we've been trying to setup a replacement Galaxy instance where jobs are submitted to our SGE cluster as the linux user requesting the job. This has been a bumpy road, e.g. http://dev.list.galaxyproject.org/Python-on-cluster-for-setting-metadata-af… Right now we've run into a problem with the metadata scripts trying to access ~/.python-eggs and going back over the archives it looks like we may need to set $PYTHON_EGG_CACHE to a temp folder on a per-node per-user basis, see e.g. http://dev.list.galaxyproject.org/python-egg-cache-exists-error-tt4656276.h… Sadly the wiki does not cover this aspect of running jobs as the real user (but the egg settings are about to be obsolete): https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#Submitting_… Clearly with Galaxy v16.01 the eggs and $PYTHON_EGG_CACHE will go away. What happens with wheels, Python virtual environments, and running jobs as the real user? Thanks, Peter

2 3

Docker and issue with shed_tool_conf.xml
by Rathert, Philipp, Dr. 23 Feb '16

23 Feb '16

Dear All, I have a serious problem with installing new tools to our local Docker-galaxy-stable instance (latest version). All tools seem to install without any issue from the tool shed. When I check under Manage installed tools they all show a green "installed" button. However, the local tool shed is not updated anymore. none of the newly installed tools show up. It seems the shed_tool_conf.xml file is not updated. the tools are all found in the folder shed_tools but not under tool-data, not sure if this means anything. A work-around could be to manually enter these tools to the shed_tool_conf.xml file but to do this I wasn't able to find the guid for each tool. All other parameters are easy. But this would be quite some effort and I am afraid that something is seriously wrong. It would be great to get your thoughts on this issue before I kill everything and start from scratch. Thanks a lot. Cheers, Philipp PS: I am a wet lab scientist please be patient if I don't get everything immediately.

1 0

Plans for workflow & parallelisation work?
by Peter van Heusden 22 Feb '16

22 Feb '16

Hi there I see from the PR landing in Galaxy and the comments on things like issue #1701 (https://github.com/galaxyproject/galaxy/issues/1701) that there's lots of work happening on the workflow side of Galaxy. This is an area of interest at SANBI too, so we'd like to coordinate development efforts as much as possible. To this end: 1) Are there forks to track so we can see what new code is landing? 2) Is there a roadmap for workflow work or perhaps can we have a Hangout to talk about this? 3) Specifically in terms of workflows and parallelisation: are there any plans to work on running workflows as opposed to just generating lots of jobs? I know this is a major change to how Galaxy works - it would mean something like submitting a workflow specification to a job runner that is located on the cluster, and then returning the results of workflow execution. 4) Currently parallelisation in Galaxy is supported using two mechanisms: collections and dataset splitters/tasks. Are there plans on extending and harmonising Galaxy's parallelisation capabilities? Thanks, Peter

3 2