June 2012 - galaxy-dev - lists.galaxyproject.org

mapreduce approach in workflows
by Cittaro Davide 11 Jun '12

11 Jun '12

Hi all, We are trying to write our own GATK workflow (although I guess an "official" one will be released...) and I realized I have to parallelize execution as some steps are painfully slow. In order to run GATK quickly I need to split the whole process on specific genomic intervals (-L option) and merge results at the end (http://www.broadinstitute.org/gsa/wiki/index.php/Parallelism_and_the_GATK) hence I have to inform galaxy that it has to create parallel jobs, one for each interval, in a MapReduce way. I see from here http://gmod.org/mediawiki/images/5/50/April2012GalaxyUpdate.pdf that this feature will be somehow implemented, I also see from bitbucket commits that a BAM slicer has been added... Is there a release plan for this? Should I write my own wrapper/adapter? Thanks d /* Davide Cittaro, PhD Coordinator of Bioinformatics Core Center for Translational Genomics and Bioinformatics San Raffaele Scientific Institute Via Olgettina 58 20132 Milano Italy Office: +39 02 26439140 Mail: cittaro.davide(a)hsr.it Skype: daweonline */ -------------------------------------------------------------------------- LA TUA CURA E' SCRITTA NEL TUO DNA. AL SAN RAFFAELE LA STIAMO REALIZZANDO. AIUTA LA RICERCA, DAI IL TUO 5XMILLE - CF: 03 06 80 153 info:www.5xmille@hsr.it - www.5xmille.org Disclaimer added by CodeTwo Exchange Rules 2007 http://www.codetwo.com

1 0

API extension
by Frank Sørensen 11 Jun '12

11 Jun '12

Hi All, We have just released the first beta version of a Galaxy API extension, that greatly simplifies some of the typical jobs that are performed by external jobs, that uses the Galaxy API. Please note that this API extension project is *NOT a part of the Galaxy project*. The Galaxy developer team can not in any way be expected to support this Python module. Please also note that this project is still in beta. The extended API consists of a single Python source file. The download includes a Python script for easy installation. If you want to try it out, please visit our project wiki page for examples of usage and more info about download, installation and setup. http://moma.ki.au.dk/mediawiki/index.php/Galaxy_extended_API We hope you find this useful. Kind regards - Frank -- Frank Sørensen, B.Sc., Programmer Molecular Diagnostic Laboratory (MDL) Molekylær Medicinsk Afdeling (MOMA) Århus Universitetshospital Skejby, Brendstrupgårdsvej, 8200 Århus N Tlf. +45 7845 5363

1 0

GCC2012 Early Registration ENDS THIS MONDAY JUNE 11
by Dave Clements 10 Jun '12

10 Jun '12

Hello all, Just a *final* reminder that early registration for the 2012 Galaxy Community Conference (GCC2012)<http://galaxyproject.org/wiki/Events/GCC2012> *closes on Monday June 11 (*which is probably* today* when you read this*)*. Registering early saves 36 to 42% on registration costs, and allows you to sign up for the GCC2012 Training Day<http://galaxyproject.org/wiki/Events/GCC2012/TrainingDay>and book discounted conference lodging<http://wiki.g2.bx.psu.edu/Events/GCC2012/Logistics#Lodging> *before they fill up*. *Register today <http://wiki.g2.bx.psu.edu/Events/GCC2012/Register>. * GCC2012 <http://galaxyproject.org/wiki/Events/GCC2012> will be held July 25-27, in Chicago, Illinois, United States. This year GCC2012 features a full day of tutorial sessions<http://galaxyproject.org/wiki/Events/GCC2012/TrainingDay>with 3 parallel tracks, each featuring four, 90 minute workshops and covering 10 different topic, including the newly added Variant and SNP Analsys, RNA-Seq Analysis, and Galaxy Code Architecture sessions<http://wiki.g2.bx.psu.edu/Events/GCC2012/TrainingDay>. The two-day main meeting<http://wiki.g2.bx.psu.edu/Events/GCC2012/Program#Day_1:_July_26.2C_Thursday>includes over 25 talks by Galaxy community members and Galaxy developers addressing the challenges of integrating, analyzing, and sharing the diverse and very large datasets that are now typical in biomedical research. GCC2012 is an opportunity to share best practices with, and learn from, a large community of researchers and support staff who are facing the challenges of data-intensive biology. Galaxy <http://gmod.org/wiki/Galaxy>is an open web-based platform for data intensive biomedical research<http://galaxyproject.org>that is widely used and deployed at research organizations of all sizes around the world. See you in Chicago! Dave Clements, on behalf of the GCC2012 Organizing Committee<http://galaxyproject.org/wiki/Events/GCC2012/Organizing%20Committee> Links: http://galaxyproject.org/GCC2012 <http://galaxyproject.org/wiki/GCC2012> http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ http://galaxyproject.org/wiki/

1 0

Re: [galaxy-dev] Tool Shed Workflow
by John Chilton 09 Jun '12

09 Jun '12

On Fri, Jun 8, 2012 at 3:27 PM, Greg Von Kuster <greg(a)bx.psu.edu> wrote: > Hi John, > > On Jun 8, 2012, at 1:22 PM, John Chilton wrote: > >> Hello Greg, >> >> Thanks for the prompt and detailed response (though it did make me >> sad). I think deploying tested, static components and configurations >> to production environments and having production environments not >> depending on outside services (like the tool shed) should be >> considered best practices. > > I'm not sure I understand this issue. What processes are you using to upgrade your test and production servers with new Galaxy distributions? If you are pulling > new Galaxy distributions from our Galaxy dist repository in bitbucket, then pulling tools from the Galaxy tool shed is not much different - both are outside services. Updating your test environment, determining it is functionally correct, and then updating your production environment using the same approach would generally follow a best practice approach. This is the approach we are currently using for our public test and main Galaxy instances at Penn State. We don't pull down from bitbucket directly to our production environment, we pull galaxy-dist changes into our testing repository, merge (that can be quite complicated, sometimes a multihour process), auto-deploy to a testing server, and then finally we push the tested changes into a bare production repo. Our sys admins then pull in changes from that bare production repo in our production environment. We also prebuild eggs in our testing environment not live on our production system. Given the complicated merges we need to do and the configuration files that need to be updated each dist update it would seem making those changes on a live production system would be problematic. Even if one was pulling changes directly from bitbucket into a production codebase, I think the dependency on bitbucket would be very different than on N toolsheds. If our sys admin is going to update Galaxy and bitbucket is down, that is no problem he or she can just bring Galaxy back up and update later. Now lets imagine they shutdown our galaxy instance, updated the code base, did a database migration, and went to do a toolshed migration and that failed. In this case instead of just bringing Galaxy back up they will now need to restore the database from backup and pullout of the mercurial changes. Anyway all of that is a digression right, I understand that we will need to have the deploy-time dependencies on tool sheds and make these tool migration script calls part of our workflow. My lingering hope is for a way of programmatically importing and updating new tools that were never part of Galaxy (Qiime, upload_local_file, etc...) using tool sheds. My previous e-mail was proposing or positing a mechanism for doing that, but I think you read it like I was trying to describe a way to script the migrations of the existing official Galaxy tools (I definitely get that you have done that). Thanks again for your time and detailed responses, -John

2 1

Proftpd Problem
by CHEBBI Mohamed Amine 08 Jun '12

08 Jun '12

Hi, I'am trying to set up ProFTP on Debain to enable FTP upload on Galaxy installed locally? I followed the instructions on this link ( http://wiki.g2.bx.psu.edu/Admin/Config/Upload%20via%20FTP) but it doesn't work. When I restart proftpd i get this message : mod_tls_memcache/0.1: notice: unable to register 'memcache' SSL session cache: Memcache support not enabled Thanks, Amine

2 3

RNA-seq wig file from UCSC conversion to gene expression values, using Galaxy
by Karmella Haynes 08 Jun '12

08 Jun '12

Hello Galaxy list, I'm helping a student with a human bioinformatics project. We would like to generate a single relative gene expression value for each gene in the human genome (for different cell lines of interest) using the RNA-seq data from the UCSC archive. We can see raw signal values by displaying the following file as a track in the Genome Browser: Track name: BJ cell pA+ + 1 Table name: wgEncodeCshlLongRnaSeqBjCellPapMinusRawSigRep1 File name: /gbdb/hg19/bbi/wgEncodeCshlLongRnaSeqBjCellPapPlusRawSigRep1.bigWig But where we get stuck is trying to import the wig into Galaxy to generate expression values for each gene. Cufflinks seems like the right tool for the output we want, but we're not sure what the inputs are supposed to be. We'd appreciate any help you can provide. Best regards, ---Karmella

1 0

How does Galaxy access datasets?
by Dorset, Daniel C 08 Jun '12

08 Jun '12

I'm trying to troubleshoot why I can't retrieve output from my Galaxy cluster instance. I notice that when I click on any output, the URL is something like: http://[root galaxy address]/datasets/[some 16-character hash]/display/[file name] I'm not able to find the "datasets" directory on the local machine, and I couldn't figure anything out by searching the paster.log file and the apache access and error logs. Everytime I try to access output, it downloads a zero-byte file. The files that I want to download through Galaxy are in the correct subdirectory of /database/files/... If someone could explain to be what's going on "behind the scenes," it would help me quite a bit. I'm guessing that the absolute path is stored in a database, but beyond that I don't know any specifics. If that's the case, knowing the relevant database tables would be a huge hint. Thanks! Dan

2 2

Tool Shed Workflow
by John Chilton 08 Jun '12

08 Jun '12

I have read through the documentation a couple times, but I still have a few questions about the recent tool shed enhancements. At MSI we have a testing environment and a production environment and I want to make sure the tool versions and configurations don't get out of sync, I would also like to test everything in our testing environment before it reaches production. Is there a recommended way to accomplish this rather than just manually repeating the same set of UI interactions twice? Can I just import tools through the testing UI and run the ./scripts/migrate_tools/XXXX scripts on our testing repository and then move the resulting migrated_tools_conf.xml and integrated_tool_panel.xml files into production? I have follow up questions, but I will wait for a response on this point. Also as you are removing tools from Galaxy and placing them into our tool shed, what is the recommended course of actions for deployers that have made local minor tweaks to those tool configs and scripts and adapt them to our local environments? Along the same lines, what is the recommended course of action if we need to make minor tweaks to tools pulled into through the UI to adapt them to our institution. Thanks for your time, -John ------------------------------------------------ John Chilton Senior Software Developer University of Minnesota Supercomputing Institute Office: 612-625-0917 Cell: 612-226-9223

3 3

Remote Job Submission from Local Install?
by Gallant, Jason 08 Jun '12

08 Jun '12

Greetings Galaxy Devs! I've been doing a fair amount of poking around in the forums and on the wiki but haven't found a satisfying answer to my query... Has anyone come up with a workable solution to run jobs from a local installation of galaxy on a campus cluster or on XSEDE/TeraGrid? While it might be possible for us (though probably a headache) to install galaxy on one of the head nodes on campus, it seems very unlikely for me to be able to do this on something like this on a resource such as blacklight. It seems that over the years past there have been a few mentions different strategies taken to accomplish something like this, but the details on this seem scarce. If this is something that someone on the list has worked out, does anyone have time to describe their setup in more detail? More generally, are there plans in the Galaxy roadmap to include this as a potential feature in future releases? It seems that many small labs like ours would find this feature quite useful. Appreciative of any help you might be able to provide-- keep up the great work Galaxy developers! ----- Jason Gallant, Ph.D. Postdoctoral Research Associate Department of Biology BRB 224, 5 Cummington Street Boston University Boston, MA 02215 Lab: 617-358-4590 Office: 617-358-3291 www.jgallant.org

2 2

Unnamed histories
by Sarah Maman 08 Jun '12

08 Jun '12

Dear all, Each time a user connects to our local instance of Galaxy, a new history is automatically created. Therefore, the list of "Unnamed history" can become very long. Is there a way to reuse the same empty "Unnamed history" to avoid creating a new one every time? Thank you in advance, Sarah

3 2