- galaxy-dev - lists.galaxyproject.org

Writing Auth Layer as WSGI Middleware
by Stephen Rosen 23 Apr '15

23 Apr '15

Hi All, I have a need for authentication as a layer in front of Galaxy which is more specialized than the available options -- specifically 3-legged OAuth against site of my choice. After looking into writing this in PHP and having the webserver (nginx) set remote_user, I decided to nix that approach for a couple of reasons -- one of which is that I don't have PHP experience. After a few discussions with other devs, I've decided that there are two easy options available to me: - Write a WSGI app which does authentication, and *proxies* authenticated requests to Galaxy with remote_user set. Since that's a WSGI app doing proxying, obvious code smell there - Write a WSGI middleware that wraps the existing Galaxy WSGI app, and passes authenticated requests directly to the Galaxy app That second solution seems much better, but I'm now faced with the question of "How do I do it?" Looking over the sample config, I see these lines: # The factory for the WSGI application. This should not be changed. paste.app_factory = galaxy.web.buildapp:app_factory I'm thinking that I could change that to my middleware, which will turn to `galaxy.web.buildapp` when the time comes. One problem I'm seeing is that my middleware and galaxy both have to run in the same virtualenv, so there's potential for dependency conflicts. The lib I want to use for this does rely on PyYAML and a few other things which Galaxy also needs, so that possibility is very real. Other than that hurdle, are there any gotchas I should be aware of with this approach? Are there similarly simple alternatives to this which I am not seeing? Ultimately, if I have to write an app that does proxying, I'd prefer that to the wide variety of highly effortful solutions I have envisioned. Those include, but are not limited to, a PAM which does the OAuth and doing Basic Authentication against that, just to give a flavor. Thanks very much for your help, -Stephen

1 0

Galaxy Tool Shed security vulnerability
by Nate Coraor 23 Apr '15

23 Apr '15

*Please note: This notice affects Galaxy Tool Shed servers only. Galaxy servers are unaffected.* A security vulnerability was recently discovered by Daniel Blankenberg of the Galaxy Team that would allow a malicious person to execute arbitrary code on a Galaxy Tool Shed server. The vulnerability is due to reuse of tool loading code from Galaxy, which executes "code files" defined by Galaxy tool config files. Because the Tool Shed allows any user to create and "load" tools, any user could cause arbitrary code to be executed by the Tool Shed server. In Galaxy, administrators control which tools are loaded, which is why this vulnerability does not affect Galaxy itself. Although we recommend upgrading to the latest stable version (15.03.2), a fix for this issue has been committed to Galaxy versions from 14.08 and newer. If you are using Mercurial, you can update with (where YY.MM corresponds to the Galaxy release you are currently running): % hg pull % hg update release_YY.MM If you are using git, you can update with (assuming your remote upstream is set to https://github.com/galaxyproject/galaxy/) If you have not yet set up a remote tracking branch for the release you are using: % git fetch upstream % git checkout -b release_YY.MM upstream/release_YY.MM Otherwise: % git pull upstream release_YY.MM For the changes to take effect, *you must restart all Tool Shed server processes*. Credit for the arbitrary code execution fix also goes to my fellow Galaxy Team member Daniel Blankenberg. On behalf of the Galaxy Team, --nate

1 0

Galaxy on HPC and Bright Cluster Manager?
by Carlos Lijeron 22 Apr '15

22 Apr '15

Good day everyone, Has anyone of you been able to implement Galaxy on a HPC using Bright Cluster Manager as the main DRM? I noticed that only a few have been known to work with Galaxy, but the list does not include Bright. Any advice/ideas will be greatly appreciated. TORQUE Resource Manager PBS Professional Open Grid Engine Univa Grid Engine (previously known as Sun Grid Engine and Oracle Grid Engine) Platform LSF HTCondor Slurm Galaxy Pulsar (formerly LWR) Thanks. Carlo

3 2

Issue writing functional tool test with repeat tag in config
by Christy Shafer 21 Apr '15

21 Apr '15

Hello, I'm trying to format a test tag in a tool config for the public Tool Shed that includes a repeated input parameter. I followed the syntax suggested in the following archived galaxy-dev post (was unable to find much other guidance): http://git.net/galaxy-development-source-control/msg15241.html, but the config file must be invalid because my repository on the Tool Shed does not register as an installable Galaxy utility. Any guidance is much appreciated.. Below are the relevant config code sections: <inputs> <repeat name="Files" title="Join file"> <param name="joinMe" type="data" checked="yes" format="tabular" label="Join" /> <param name="joinCol" label="using column" type="data_column" data_ref="joinMe" /> </repeat> <param name="headerYes" type="select" label="Treat first line as header?"> <option value="yes" selected="true">Yes</option> <option value="no">No</option> </param> </inputs> <outputs> <data format="tabular" name="Joined_all" label="Multi-Join result"/> </outputs> <tests> <test> <param name="Files_0|joinMe" value="multi_join_serial_in1.tab" ftype="tabular"/> <param name="Files_0|joinCol" value="1"/> <param name="Files_1|joinMe" value="multi_join_serial_in2.tab" ftype="tabular"/> <param name="Files_1|joinCol" value="1"/> <param name="Files_2|joinMe" value="multi_join_serial_in3.tab" ftype="tabular"/> <param name="Files_2joinCol" value="2"/> <param name="headerYes" value="yes"/> <output name="Joined_all" value="multi_join_serial_out.tab" ftype="tabular"/> <test/> <tests/> Thanks for your help, CS

3 3

Problem killing jobs (via Admin>Manage Jobs)
by Poole, Richard 21 Apr '15

21 Apr '15

Hi all, I am having an issue killing jobs via Admin>Manage Jobs. When I try to submit kills I see this in the Galaxy log: galaxy.jobs.runners.local WARNING 2015-04-19 15:55:03,089 stop_job(): 10374: no PID in database for job, unable to stop I have updated my local Galaxy server (via hg) and am currently running 17052:28113eee212. I am scared the issue is a database issue, however everything else and communication with the PostgreSQL database seems functional. Any help greatly appreciated :) Richard Richard J Poole PhD Wellcome Trust Fellow Department of Cell and Developmental Biology University College London 21 University Street, London WC1E 6DE Office (518 Rockefeller): +44 20 7679 6577 (int. 46577) Lab (529 Rockefeller): +44 20 7679 6133 (int. 46133)

1 0

GCC2015 Registration and Lodging Scholarship for US-Based Post-Docs and Students
by Dave Clements 21 Apr '15

21 Apr '15

Hello all, *We are pleased to offer registration <http://gcc2015.tsl.ac.uk/registration> and lodging <http://gcc2015.tsl.ac.uk/logistics/#Lodging> scholarships for GCC2015 for students and post-docs who are based in the United States, and who are members of historically underrepresented groups in life science research.* These scholarships cover registration costs <http://gcc2015.tsl.ac.uk/registration> for any GCC2015 events <http://gcc2015.tsl.ac.uk/proigramme> the recipients sign up for, and also covers lodging <http://gcc2015.tsl.ac.uk/logistics/#Lodging> during those events in the official conference lodging <http://gcc2015.tsl.ac.uk/logistics/#Lodging> on the nearby UEA campus. *The scholarships do not cover travel or other expenses.* To apply for a scholarship qualifying applicants need to 1. Submit a poster abstract <http://bit.ly/gcc2015sub>. *The deadline is May 1* 2. Have a faculty member / PI send a letter of recommendation to the Galaxy Community Fund Board <community-fund(a)lists.galaxyproject.org> no later than May 15. 3. Submit an application form <http://bit.ly/gcc2015scholarship>, no later than May 1. 4. Have sufficient funds to cover your travel and other expenses. Applicants will be notified of their scholarship status no later than May 19. Scholarships are funded by the *Galaxy Community Fund*. The Galaxy Community Fund was established with the surplus funds from GCC2014, and is currently administered by the GCC2014 organizers. Please do let us know if you have any questions, and we hope to see you at GCC2015 <http://gcc2015.tsl.ac.uk/>! Galaxy Community Fund Board <community-fund(a)lists.galaxyproject.org> -- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ https://wiki.galaxyproject.org/

1 0

Adding a dataset to a dataset list breaks "Run this job again"
by Brian Claywell 21 Apr '15

21 Apr '15

While testing an upgrade from latest_2014.10.06 to GitHub release_15.03 branch (cloned today) I ran into an issue trying to use the "Run this job again" functionality on a dataset. I was playing with creating dataset lists using "Operations on multiple datasets" and noticed that after adding a dataset to a dataset list, trying to run the job again fails. This is the tool form that gets displayed: [image: Inline image 1] Additionally an error is logged: galaxy.tools ERROR 2015-04-15 21:58:53,258 'HistoryDatasetAssociation' object has no attribute 'collection' Deleting the dataset list makes no difference. However, if I copy this dataset to a new history and click "Run this job again", no error occurs and the proper tool form appears. If I then add the dataset in the new history to a dataset list, the error reappears. Cheers, Brian -- Brian Claywell | programmer/analyst Matsen Group | http://matsen.fredhutch.org <http://matsen.fhcrc.org/> Fred Hutchinson Cancer Research Center

2 1

metadata in parallelization
by Roberto Alonso 21 Apr '15

21 Apr '15

Hello, I am writing some code to enable parallelization for some tool wrappers. First, I did it for simple bwa wrapper, but now I am modifying toolshed.g2.bx.psu.edu/repos/devteam/bwa/c71dd035971e/bwa/bwa-mem.xml to check if the code would work with this wrapper. So, I wrote some code that I thing was necessary in order to merge some bam and I added the parallelism tag (in bold) to the config file: <tool id="bwa_mem" name="BWA-MEM" version="0.1"> <macros> <import>bwa_macros.xml</import> </macros> <requirements> <requirement type="package" version="0.7.10.039ea20639">bwa</requirement> <requirement type="package" version="1.1">samtools</requirement> </requirements> <description>- map medium and long reads (> 100 bp) against reference genome</description> *<parallelism method="multi" split_size="3" shared_inputs="ref_file" split_mode="number_of_parts" merge_outputs="bam_output" split_inputs="fastq_input1,fastq_input2" ></parallelism>* <command> ... So, everything works well, and the resulting bam from parallelization mode and without the parallelization mode is the same but the Galaxy log throws an error regarding metadata, it says something like this: galaxy.jobs.splitters.multi DEBUG 2015-04-17 09:54:58,335 merge finished: /home/ralonso/galaxy/database/files/000/dataset_198.dat galaxy.jobs.runners.tasks DEBUG 2015-04-17 09:54:58,473 executing external set_meta script for job 200: python /home/ralonso/galaxy/database/tmp/set_metadata_E5fGIE.py /home/ralonso/galaxy/database/tmp/tmpHS8Byo /home/ralonso/galaxy/database/job_working_directory/000/200/galaxy.json /home/ralonso/galaxy/database/tmp/metadata_in_HistoryDatasetAssociation_198_yOGiQG,/home/ralonso/galaxy/database/tmp/metadata_kwds_HistoryDatasetAssociation_198_nAsQoq,/home/ralonso/galaxy/database/tmp/metadata_out_HistoryDatasetAssociation_198_I_cLs4,/home/ralonso/galaxy/database/tmp/metadata_results_HistoryDatasetAssociation_198_qhjzoV,/home/ralonso/galaxy/database/files/000/dataset_198.dat,/home/ralonso/galaxy/database/tmp/metadata_override_HistoryDatasetAssociation_198_ScKLqH Traceback (most recent call last): File "/home/ralonso/galaxy/database/tmp/set_metadata_E5fGIE.py", line 1, in <module> from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata() ImportError: No module named galaxy_ext.metadata.set_metadata galaxy.jobs.runners.tasks DEBUG 2015-04-17 09:54:58,624 execution of external set_meta finished for job 200 *galaxy.datatypes.metadata DEBUG 2015-04-17 09:54:58,714 setting metadata externally failed for HistoryDatasetAssociation 198: External set_meta() not called* When using no parallelization mode, there is no problem, also because Galaxy doesn't go through this part of code, I mean it doesn't execute this. I see that Galaxy have to do something with metada attributes, but what is t trying to do? is there any way to solve this? Thank you very much Regards, Roberto

2 1

pbs/torque config
by Beginner TI 21 Apr '15

21 Apr '15

Hi, I'm trying to enable the capability to submit jobs to a pbs cluster, however I am quite confused by the documentation. I don't know where I could specify the pbs server. I saw the following on the page: https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster 1 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> <plugins> 2 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> <plugin id="pbs" type="runner" load="galaxy.jobs.runners.pbs:PBSJobRunner"/> 3 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> </plugins> 4 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> <destinations default="pbs_default"> 5 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> <destination id="pbs_default" runner="pbs"/> 6 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> <destination id="other_cluster" runner="pbs"> 7 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> <param id="destination">@other.cluster</param> 8 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> </destination> 9 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> <destination id="long_jobs" runner="pbs"> 10 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> <param id="Resource_List">walltime=72:00:00,nodes=1:ppn=8</param> 11 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> <param id="-p">128</param> 12 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> </destination> 13 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> </destinations> is this <param id="destination"> the place you specify where your pbs/torque server? and the syntax is to prepend a '@' to the server name? for example, should a most basic config look like: <plugins> <plugin id="pbs" type="runner" load="galaxy.jobs.runners.pbs:PBSJobRunner"> </plugins> <destinations default="pbs"> <destination id="pbs" runner="pbs" > <param id="destination">@pbs_server_full_name</param> <param id="Resource_List">walltime=72:00:00,nodes=1:ppn=8</param> <param id="-q">myqueue</param> </destination> <destinations> I searched online and to my surprise I didn't find any useful information, or any working sample...please give me a hand guys! Thanks! Rui

2 1

Re: [galaxy-dev] Tool param type "library_data" (Dooley, Damion)
by Dooley, Damion 20 Apr '15

20 Apr '15

I was successful getting library dataset selection and form submission to happen within tool form tool form input <param type="library_data" ...> - Yay! But it appears one last fix (besides the "url: "/visualization/list_libraries",' bug) is needed to enable history tool item to be RERUN. So if a galaxy dev can tackle this, great! Otherwise I'll have to drop this approach :<( ***************** For others who may need... tool form's param: <param name="library_datasets" type="library_data" label="Reference Bin file" help="Select one reference bin at a time from the popup window's Reference Bin library."/> <command> tag contents.... #if $library_datasets -B " #for $i, $dataset in enumerate($library_datasets) #$dataset.get_file_name() $dataset.id #end for " #end if Successfully submits either the full file path of selected dataset, or its id. ***************** However trying to rerun this history tool item gets us an error (not related to the above <command> contents): Error Traceback: View as: Interactive | Text | XML (full) ⇝ AttributeError: 'int' object has no attribute 'id' .... Module galaxy.tools.parameters.basic:146 in value_to_basic view >> return self.to_string( value, app ) Module galaxy.tools.parameters.basic:2406 in to_string app <galaxy.app.UniverseApplication object at 0x59a0710> ldda 331 self <galaxy.tools.parameters.basic.LibraryDatasetToolParameter object at 0x7fced01d2090> value [331] >> return [ldda.id for ldda in value] AttributeError: 'int' object has no attribute 'id' which is in /lib/galaxy/tools/parameters/basic.py: class LibraryDatasetToolParameter( ToolParameter ): """ Parameter that lets users select a LDDA from a modal window, then use it within the wrapper. """ ... def to_string( self, value, app ): if not value: return value return [ldda.id for ldda in value] "Value" is getting an array of dataset ids, whereas this method is expecting value to be a list of dataset objects. I tried changing ldda.id to just "id" but found other parts of LibraryDatasetTool code were broken as they too were expecting an object. Maybe this is related to the tool job's history detail link showing "Reference Bin file" as a string label of an object?: Input Parameter Value Note for rerun BLAST results as XML 240: megablast Pasted Entry vs 16SMicrobial.fasta Reference Bin file [<galaxy.model.LibraryDatasetDatasetAssociation object at 0x7fcee04d4350>] Tabular Report Column Labels Short name HTML Report template templates.html_report ******************************** The other bug: 2) Tool_form.mako has a hardcoded url that doesn't work for galaxy installs that have a prefix so that needs adjustment: $(".add-librarydataset").click(function() { var link = $(this); $.ajax({ url: "/visualization/list_libraries",

2 3