- galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] Galaxy on HPC and Bright Cluster Manager?
by David Trudgian 23 Apr '15

23 Apr '15

Hi Carlos, sorry for the slow reply. With a small 10 user setup (I assume just one group) your installation is probably going to be a lot less complex than ours. We are running services for users from multiple labs and departments, so a chief concern was making sure private datasets always stay private - which is relatively involved when using Galaxy to run jobs on a general-purpose shared cluster. I think if I had to recommend picking a job scheduler I would suggest SLURM since it's probably the most 'fashionable' choice at present. You also have the advantage that SLURM is used with Galaxy in the Galaxy docker image etc. which ensures people notice if the Galaxy->DRMAA->SLURM setup isn't working. I've also used Galaxy with GridEngine in the past, and that was fine - but is becoming a less common choice as a scheduler. Having said that, I don't think that the job scheduler needs to be your biggest concern. I would focus most on the file system and user account setup you are going to need. * How are you going to migrate from standalone Galaxy to a situation where your new cluster can see the Galaxy data files, tools etc. Are you purchasing storage with the cluster? If so do you move Galaxy onto that storage, or can you mount existing Galaxy data onto the cluster nodes? If you can do that, is your networking such that performance is sufficient for the type of analysis you are going to run? * Do you need to, or will you need to, keep track of per-user usage of the cluster for things that Galaxy will be running? If not then you can just have a galaxy user on your cluster and things are pretty easy for file permissions etc. If you need to track jobs per-user then it becomes more complex, and the solution depends on how much privacy you need for datasets, how your cluster will authenticate users etc. The filesystem and user accounts issues are, in my mind, the ones to focus on. You can always modify Galaxy's config to switch to a different job scheduler fairly easily. You cannot as easily move around large amounts of data, and reconcile local vs cluster user accounts, should that be necessary. Cheers, Dave Trudgian -----Original Message----- From: Carlos Lijeron [mailto:clijeron@hunter.cuny.edu] Sent: Wednesday, April 22, 2015 10:54 AM To: David Trudgian; John Chilton Cc: RODRIGO GONZALEZ SERRANO Subject: Re: [galaxy-dev] Galaxy on HPC and Bright Cluster Manager? Hello David, Thank you for the great feedback. We are at Hunter College in NYC, part of the City University of New York. We recently ordered the cluster which comes with Bright Cluster Management, and our PI wants to implement Galaxy for all the users (about 10) on the cluster and manage all job submissions through a job scheduler. So, to answer your question, we are not really using any scheduler at this point, but only a stand alone server with a local installation of Galaxy. Our Cluster should be assembled and installed by the end of May, so I¹m trying to gather as much information as possible in preparation for the deployment. Based on your experience, what do you think I should focus on to ensure we maximize outcome and reduce the possibility of mistakes? In other words, any lessons learned that you would like to share will be greatly appreciated. Thanks again, Carlos Lijeron. On 4/22/15, 10:34 AM, "David Trudgian" <David.Trudgian(a)UTSouthwestern.edu> wrote: >Carlo, > >We have Bright Cluster Manager in use on our cluster for node >provisioning etc. but the actual job scheduler in use in our case is >SLURM, which we use directly. > >Are you using one of the integrated workload managers such as SLURM / >SGE / TORQUE directly, or indirectly via cmsub? > >I guess the easiest way to come up with some kind of advice is if you >can provide an example of generic job script you are using on your system. >If you're using cmsub is it specifying a --wlmanager etc. > >DT > >-----Original Message----- >From: galaxy-dev [mailto:galaxy-dev-bounces@lists.galaxyproject.org] On >Behalf Of John Chilton >Sent: Wednesday, April 22, 2015 8:26 AM >To: Carlos Lijeron >Cc: galaxy-dev(a)lists.galaxyproject.org >Subject: Re: [galaxy-dev] Galaxy on HPC and Bright Cluster Manager? > >Hello Carlos, > > I have never heard of anyone running Galaxy with Bright Cluster >Manager (though hopefully someone will chime in if they have). If you >are interested in adding support it should be possible. One >complication is that Bright Cluster Manager doesn't appear to have a >DRMAA interface >(http://www.drmaa.org/) which is the most direct way to utilize new DRMs. >Without that my approach would be to build a new CLI runner: > >There are a few examples here that one can use as template: > >https://github.com/galaxyproject/galaxy/tree/dev/lib/galaxy/jobs/runner >s/u >til/cli/job > >I guess you would have to write a new one targeting cmsub I guess - you >also need to be able to parse a job status somehow - I haven't figured >out how to do that from the documentation - but I assume there is a way. > >I looks like Bright supports running SGE, SLURM, and Torque on the >cluster - doing this and interfacing with one of those more common >options directly might be a better approach for Galaxy (and other users >if your cluster has them). > >-John > > > > > > > >On Wed, Apr 22, 2015 at 8:56 AM, Carlos Lijeron ><clijeron(a)hunter.cuny.edu> wrote: >> Good day everyone, >> >> >> >> Has anyone of you been able to implement Galaxy on a HPC using Bright >> Cluster Manager as the main DRM? I noticed that only a few have been >> known to work with Galaxy, but the list does not include Bright. Any >> advice/ideas will be greatly appreciated. >> >> >> >> TORQUE Resource Manager >> >> PBS Professional >> >> Open Grid Engine >> >> Univa Grid Engine (previously known as Sun Grid Engine and Oracle >> Grid >> Engine) >> >> Platform LSF >> >> HTCondor >> >> Slurm >> >> Galaxy Pulsar (formerly LWR) >> >> >> >> >> >> Thanks. >> >> >> >> >> >> Carlo >> >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this and other >> Galaxy lists, please use the interface at: >> https://lists.galaxyproject.org/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ >___________________________________________________________ >Please keep all replies on the list by using "reply all" >in your mail client. To manage your subscriptions to this and other >Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > >To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > >________________________________ > >UT Southwestern > > >Medical Center > > > >The future of medicine, today. >

1 0

Writing Auth Layer as WSGI Middleware
by Stephen Rosen 23 Apr '15

23 Apr '15

Hi All, I have a need for authentication as a layer in front of Galaxy which is more specialized than the available options -- specifically 3-legged OAuth against site of my choice. After looking into writing this in PHP and having the webserver (nginx) set remote_user, I decided to nix that approach for a couple of reasons -- one of which is that I don't have PHP experience. After a few discussions with other devs, I've decided that there are two easy options available to me: - Write a WSGI app which does authentication, and *proxies* authenticated requests to Galaxy with remote_user set. Since that's a WSGI app doing proxying, obvious code smell there - Write a WSGI middleware that wraps the existing Galaxy WSGI app, and passes authenticated requests directly to the Galaxy app That second solution seems much better, but I'm now faced with the question of "How do I do it?" Looking over the sample config, I see these lines: # The factory for the WSGI application. This should not be changed. paste.app_factory = galaxy.web.buildapp:app_factory I'm thinking that I could change that to my middleware, which will turn to `galaxy.web.buildapp` when the time comes. One problem I'm seeing is that my middleware and galaxy both have to run in the same virtualenv, so there's potential for dependency conflicts. The lib I want to use for this does rely on PyYAML and a few other things which Galaxy also needs, so that possibility is very real. Other than that hurdle, are there any gotchas I should be aware of with this approach? Are there similarly simple alternatives to this which I am not seeing? Ultimately, if I have to write an app that does proxying, I'd prefer that to the wide variety of highly effortful solutions I have envisioned. Those include, but are not limited to, a PAM which does the OAuth and doing Basic Authentication against that, just to give a flavor. Thanks very much for your help, -Stephen

1 0

Galaxy Tool Shed security vulnerability
by Nate Coraor 23 Apr '15

23 Apr '15

*Please note: This notice affects Galaxy Tool Shed servers only. Galaxy servers are unaffected.* A security vulnerability was recently discovered by Daniel Blankenberg of the Galaxy Team that would allow a malicious person to execute arbitrary code on a Galaxy Tool Shed server. The vulnerability is due to reuse of tool loading code from Galaxy, which executes "code files" defined by Galaxy tool config files. Because the Tool Shed allows any user to create and "load" tools, any user could cause arbitrary code to be executed by the Tool Shed server. In Galaxy, administrators control which tools are loaded, which is why this vulnerability does not affect Galaxy itself. Although we recommend upgrading to the latest stable version (15.03.2), a fix for this issue has been committed to Galaxy versions from 14.08 and newer. If you are using Mercurial, you can update with (where YY.MM corresponds to the Galaxy release you are currently running): % hg pull % hg update release_YY.MM If you are using git, you can update with (assuming your remote upstream is set to https://github.com/galaxyproject/galaxy/) If you have not yet set up a remote tracking branch for the release you are using: % git fetch upstream % git checkout -b release_YY.MM upstream/release_YY.MM Otherwise: % git pull upstream release_YY.MM For the changes to take effect, *you must restart all Tool Shed server processes*. Credit for the arbitrary code execution fix also goes to my fellow Galaxy Team member Daniel Blankenberg. On behalf of the Galaxy Team, --nate

1 0

Galaxy on HPC and Bright Cluster Manager?
by Carlos Lijeron 22 Apr '15

22 Apr '15

Good day everyone, Has anyone of you been able to implement Galaxy on a HPC using Bright Cluster Manager as the main DRM? I noticed that only a few have been known to work with Galaxy, but the list does not include Bright. Any advice/ideas will be greatly appreciated. TORQUE Resource Manager PBS Professional Open Grid Engine Univa Grid Engine (previously known as Sun Grid Engine and Oracle Grid Engine) Platform LSF HTCondor Slurm Galaxy Pulsar (formerly LWR) Thanks. Carlo

3 2

Issue writing functional tool test with repeat tag in config
by Christy Shafer 21 Apr '15

21 Apr '15

Hello, I'm trying to format a test tag in a tool config for the public Tool Shed that includes a repeated input parameter. I followed the syntax suggested in the following archived galaxy-dev post (was unable to find much other guidance): http://git.net/galaxy-development-source-control/msg15241.html, but the config file must be invalid because my repository on the Tool Shed does not register as an installable Galaxy utility. Any guidance is much appreciated.. Below are the relevant config code sections: <inputs> <repeat name="Files" title="Join file"> <param name="joinMe" type="data" checked="yes" format="tabular" label="Join" /> <param name="joinCol" label="using column" type="data_column" data_ref="joinMe" /> </repeat> <param name="headerYes" type="select" label="Treat first line as header?"> <option value="yes" selected="true">Yes</option> <option value="no">No</option> </param> </inputs> <outputs> <data format="tabular" name="Joined_all" label="Multi-Join result"/> </outputs> <tests> <test> <param name="Files_0|joinMe" value="multi_join_serial_in1.tab" ftype="tabular"/> <param name="Files_0|joinCol" value="1"/> <param name="Files_1|joinMe" value="multi_join_serial_in2.tab" ftype="tabular"/> <param name="Files_1|joinCol" value="1"/> <param name="Files_2|joinMe" value="multi_join_serial_in3.tab" ftype="tabular"/> <param name="Files_2joinCol" value="2"/> <param name="headerYes" value="yes"/> <output name="Joined_all" value="multi_join_serial_out.tab" ftype="tabular"/> <test/> <tests/> Thanks for your help, CS

3 3

Problem killing jobs (via Admin>Manage Jobs)
by Poole, Richard 21 Apr '15

21 Apr '15

Hi all, I am having an issue killing jobs via Admin>Manage Jobs. When I try to submit kills I see this in the Galaxy log: galaxy.jobs.runners.local WARNING 2015-04-19 15:55:03,089 stop_job(): 10374: no PID in database for job, unable to stop I have updated my local Galaxy server (via hg) and am currently running 17052:28113eee212. I am scared the issue is a database issue, however everything else and communication with the PostgreSQL database seems functional. Any help greatly appreciated :) Richard Richard J Poole PhD Wellcome Trust Fellow Department of Cell and Developmental Biology University College London 21 University Street, London WC1E 6DE Office (518 Rockefeller): +44 20 7679 6577 (int. 46577) Lab (529 Rockefeller): +44 20 7679 6133 (int. 46133)

1 0

GCC2015 Registration and Lodging Scholarship for US-Based Post-Docs and Students
by Dave Clements 21 Apr '15

21 Apr '15

Hello all, *We are pleased to offer registration <http://gcc2015.tsl.ac.uk/registration> and lodging <http://gcc2015.tsl.ac.uk/logistics/#Lodging> scholarships for GCC2015 for students and post-docs who are based in the United States, and who are members of historically underrepresented groups in life science research.* These scholarships cover registration costs <http://gcc2015.tsl.ac.uk/registration> for any GCC2015 events <http://gcc2015.tsl.ac.uk/proigramme> the recipients sign up for, and also covers lodging <http://gcc2015.tsl.ac.uk/logistics/#Lodging> during those events in the official conference lodging <http://gcc2015.tsl.ac.uk/logistics/#Lodging> on the nearby UEA campus. *The scholarships do not cover travel or other expenses.* To apply for a scholarship qualifying applicants need to 1. Submit a poster abstract <http://bit.ly/gcc2015sub>. *The deadline is May 1* 2. Have a faculty member / PI send a letter of recommendation to the Galaxy Community Fund Board <community-fund(a)lists.galaxyproject.org> no later than May 15. 3. Submit an application form <http://bit.ly/gcc2015scholarship>, no later than May 1. 4. Have sufficient funds to cover your travel and other expenses. Applicants will be notified of their scholarship status no later than May 19. Scholarships are funded by the *Galaxy Community Fund*. The Galaxy Community Fund was established with the surplus funds from GCC2014, and is currently administered by the GCC2014 organizers. Please do let us know if you have any questions, and we hope to see you at GCC2015 <http://gcc2015.tsl.ac.uk/>! Galaxy Community Fund Board <community-fund(a)lists.galaxyproject.org> -- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ https://wiki.galaxyproject.org/

1 0

Adding a dataset to a dataset list breaks "Run this job again"
by Brian Claywell 21 Apr '15

21 Apr '15

While testing an upgrade from latest_2014.10.06 to GitHub release_15.03 branch (cloned today) I ran into an issue trying to use the "Run this job again" functionality on a dataset. I was playing with creating dataset lists using "Operations on multiple datasets" and noticed that after adding a dataset to a dataset list, trying to run the job again fails. This is the tool form that gets displayed: [image: Inline image 1] Additionally an error is logged: galaxy.tools ERROR 2015-04-15 21:58:53,258 'HistoryDatasetAssociation' object has no attribute 'collection' Deleting the dataset list makes no difference. However, if I copy this dataset to a new history and click "Run this job again", no error occurs and the proper tool form appears. If I then add the dataset in the new history to a dataset list, the error reappears. Cheers, Brian -- Brian Claywell | programmer/analyst Matsen Group | http://matsen.fredhutch.org <http://matsen.fhcrc.org/> Fred Hutchinson Cancer Research Center

2 1

metadata in parallelization
by Roberto Alonso 21 Apr '15

21 Apr '15

Hello, I am writing some code to enable parallelization for some tool wrappers. First, I did it for simple bwa wrapper, but now I am modifying toolshed.g2.bx.psu.edu/repos/devteam/bwa/c71dd035971e/bwa/bwa-mem.xml to check if the code would work with this wrapper. So, I wrote some code that I thing was necessary in order to merge some bam and I added the parallelism tag (in bold) to the config file: <tool id="bwa_mem" name="BWA-MEM" version="0.1"> <macros> <import>bwa_macros.xml</import> </macros> <requirements> <requirement type="package" version="0.7.10.039ea20639">bwa</requirement> <requirement type="package" version="1.1">samtools</requirement> </requirements> <description>- map medium and long reads (> 100 bp) against reference genome</description> *<parallelism method="multi" split_size="3" shared_inputs="ref_file" split_mode="number_of_parts" merge_outputs="bam_output" split_inputs="fastq_input1,fastq_input2" ></parallelism>* <command> ... So, everything works well, and the resulting bam from parallelization mode and without the parallelization mode is the same but the Galaxy log throws an error regarding metadata, it says something like this: galaxy.jobs.splitters.multi DEBUG 2015-04-17 09:54:58,335 merge finished: /home/ralonso/galaxy/database/files/000/dataset_198.dat galaxy.jobs.runners.tasks DEBUG 2015-04-17 09:54:58,473 executing external set_meta script for job 200: python /home/ralonso/galaxy/database/tmp/set_metadata_E5fGIE.py /home/ralonso/galaxy/database/tmp/tmpHS8Byo /home/ralonso/galaxy/database/job_working_directory/000/200/galaxy.json /home/ralonso/galaxy/database/tmp/metadata_in_HistoryDatasetAssociation_198_yOGiQG,/home/ralonso/galaxy/database/tmp/metadata_kwds_HistoryDatasetAssociation_198_nAsQoq,/home/ralonso/galaxy/database/tmp/metadata_out_HistoryDatasetAssociation_198_I_cLs4,/home/ralonso/galaxy/database/tmp/metadata_results_HistoryDatasetAssociation_198_qhjzoV,/home/ralonso/galaxy/database/files/000/dataset_198.dat,/home/ralonso/galaxy/database/tmp/metadata_override_HistoryDatasetAssociation_198_ScKLqH Traceback (most recent call last): File "/home/ralonso/galaxy/database/tmp/set_metadata_E5fGIE.py", line 1, in <module> from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata() ImportError: No module named galaxy_ext.metadata.set_metadata galaxy.jobs.runners.tasks DEBUG 2015-04-17 09:54:58,624 execution of external set_meta finished for job 200 *galaxy.datatypes.metadata DEBUG 2015-04-17 09:54:58,714 setting metadata externally failed for HistoryDatasetAssociation 198: External set_meta() not called* When using no parallelization mode, there is no problem, also because Galaxy doesn't go through this part of code, I mean it doesn't execute this. I see that Galaxy have to do something with metada attributes, but what is t trying to do? is there any way to solve this? Thank you very much Regards, Roberto

2 1

pbs/torque config
by Beginner TI 21 Apr '15

21 Apr '15

Hi, I'm trying to enable the capability to submit jobs to a pbs cluster, however I am quite confused by the documentation. I don't know where I could specify the pbs server. I saw the following on the page: https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster 1 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> <plugins> 2 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> <plugin id="pbs" type="runner" load="galaxy.jobs.runners.pbs:PBSJobRunner"/> 3 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> </plugins> 4 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> <destinations default="pbs_default"> 5 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> <destination id="pbs_default" runner="pbs"/> 6 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> <destination id="other_cluster" runner="pbs"> 7 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> <param id="destination">@other.cluster</param> 8 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> </destination> 9 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> <destination id="long_jobs" runner="pbs"> 10 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> <param id="Resource_List">walltime=72:00:00,nodes=1:ppn=8</param> 11 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> <param id="-p">128</param> 12 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> </destination> 13 <https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CA-e4c8fb8b…> </destinations> is this <param id="destination"> the place you specify where your pbs/torque server? and the syntax is to prepend a '@' to the server name? for example, should a most basic config look like: <plugins> <plugin id="pbs" type="runner" load="galaxy.jobs.runners.pbs:PBSJobRunner"> </plugins> <destinations default="pbs"> <destination id="pbs" runner="pbs" > <param id="destination">@pbs_server_full_name</param> <param id="Resource_List">walltime=72:00:00,nodes=1:ppn=8</param> <param id="-q">myqueue</param> </destination> <destinations> I searched online and to my surprise I didn't find any useful information, or any working sample...please give me a hand guys! Thanks! Rui

2 1