- galaxy-dev - lists.galaxyproject.org

Keeping track of what you did.
by Anthony Ferrari 23 Dec '10

23 Dec '10

Hi all, I am currently setting up a Galaxy instance on a cluster. My team is intended to analyze data from NGS experiments. Our different pipelines/workflows will be integrated into Galaxy. It might be a simple question but I don't find useful information on this point. When looking back on a file result, a crucial need for us is to be able to know : - who ? (generated this file) - when ? (precise date and time) - what ? (applied workflow) - how ? (applied parameters in each workflow 'box') - and stuff like computational time & others if possible. Is there something allowing to retrieve these informations in galaxy ? Logfiles, a table in the database that we could interrogate ? As far as I currently understand, histories are managed on a per user basis, allowing one to save and share histories. Does it enable one to reproduce exactly the same pipeline without having to re-specify parameters used with new input data ('applying a workflow instance') ? Workflows may indeed be modified/improved frequently. We have to keep memory of which 'version' was used to generate our results. And where are stored the output files ? Is it possible to force galaxy to put a file result in pre-specified place ? It seems like once you have given galaxy the input files, you do not control where the output files will be stored (internal management) and all you have is a link to this output in your history panel. For the input files, I saw that there is a way to avoid Galaxy to duplicate input files when uploading them and so keep your tree structure, something which is especially useful when dealing with huge NGS data. Does exist this kind of flexibility for output files ? Best regards, A.

2 1

sample form creation, nglims, seq tracking
by Collins, Mike 21 Dec '10

21 Dec '10

I'm using the nglims build via "hg clone http://www.bx.psu.edu/hg/galaxy galaxy_dist" as mentioned in the wiki, Apache proxy and Postgres DB enabled. Please correct me if any of my initial "goals" below are unattainable or if I am misinterpreting Galaxy's capabilities. I'm still attempting to learn the overall system capabilities of Galaxy and its integration between nglims samples+projects and the native Sample/Seq tracking components. Goals I'm trying to accomplish: (1) Create a fresh custom sample form in the Admin tab's "Manage Form Definitions" to use in the Lab tab's (nglims section) "next gen sample submission" process, and assign this custom sample form to be used as the "default" form when creating a new sample in nglim section's "next gen sample submission", (2) Allow multiple/various defined samples created with this custom sample submission form to be submitted in the Lab tab's "Submit samples as a project" as a project for tracking in the Admin tab section, (3) Allow "Find Samples" in the Admin tab to potentially query for nglims samples created with this sample form. Uncertainties: (1) Can samples created in the nglims section currently be tracked in the Admin tab's "Sample Tracking" area, or does the sample tracking functionality only allow tracking of the samples created with the native form definitions (within the Admin section, outside of the Lab tab "next gen sample submission")? There was a prior posting a while back that mentioning the following (verbatim): "You probably don't want to mix the "nglims" approach with Galaxy's native sample tracking functionality. While I've build the nglims part to use the same database tables and be fairly interoperable, it does take some different data representation choices, especially with regards to the sample/request relationship." Does this mean we're limited to creating samples outside of the nglims section if we want to track them within the native sequencing section? Secondly, will there be future support for tracking nglims-created samples with the native sample/seq tracking? (2) How to change which sample form is used in the Lab tab's "Define samples and services". In fact, I cannot determine what form is actually used which corresponds to the form which appears when one clicks the Lab tab's "next gen sample submission". Additional concern: When creating and/or editing a form in the Admin tab's "manage form definitions", I've created a sample submission form with multiple grids, some grids which have multiple fields beneath. When editing this form and assigning fields to the respective grids, each time I save and re-edit the form, all of the fields get reset to whatever is listed as the 1st grid in the "Select the grid layout to place this field" dropdown menu, rendering all of my assigned fields to be reset to the first grid. Adding or removing a grid with the submit button also does the same thing. Any thoughts? Galaxy appears to be a very promising product and we're looking forward to having the researchers and sequence team use this environment, once we finalize the configuration and push into production. I'm sure I'll have additional questions once I work through some of these current issues. My apologies for the extended length of this post and if I've somehow misinterpreted some of the functionalities and compatibilities. Thanks in advance for your feedback. Best Regards, Mike

3 4

FormValues content upgrade script
by Brad Chapman 20 Dec '10

20 Dec '10

Greg and rc; I've been trying to keep up with the latest changes in the FormValues table, and am running into one issue with the upgrade script: https://bitbucket.org/galaxy/galaxy-central/src/714afada18de/lib/galaxy/mod… During the conversion of the field from a list to dictionary, this assumes all of the values are strings, and strips out any quote characters. I'm using this table to store more complex objects like dictionaries, and they are disrupted by that change. Is this step necessary? The to_json_string call should quote any of these characters appropriately. This patch gets rid of the quote stripping so should be safe to make the conversion without changing content: https://bitbucket.org/chapmanb/galaxy-central/changeset/920527658e04 Thanks, Brad

1 0

nextgen snp calling with dbSNP rs#s
by Mike Cariaso 18 Dec '10

18 Dec '10

Does your SNP pipeline recognize and assign dbSNP rs#### identifiers? I'm interested coding better integration between galaxy and SNPedia annotations of the human genome http://www.SNPedia.com Probably this should be done as a sort of https://bitbucket.org/galaxy/galaxy-central/wiki/DataSources I'm hoping to ask a few pipeline related questions to anyone who's pipelines assign rs#s to nextgen data. -- -- Mike Cariaso http://www.cariaso.com

2 2

how do I migrate accounts from one galaxy to another
by Dean Snyder 17 Dec '10

17 Dec '10

I'm sure this has been covered somewhere but I haven't been able to find it on the wiki or by searching the list archives. Can I, and if I can, how do I, migrate all user accounts from a previous local Galaxy development install to a new production one (a brand new install in a separate location on the same machine), preserving account info, workflows, histories, data, etc.? If this can be done, can it be done selectively, i.e., for only specified users? Thanks for any help offered, Dean A. Snyder Senior Programmer/Analyst Center for Inherited Disease Research (CIDR) Johns Hopkins School of Medicine Bayview Research Campus 333 Cassell Dr, Triad Bldg, Suite 2000 Baltimore, MD 21224 cell:717 668-3048 office:410-550-4629 www.cidr.jhmi.edu

2 3

Galaxy Roadshow | San Diego, CA | Jan 2011
by Anton Nekrutenko 17 Dec '10

17 Dec '10

Dear Galaxy Users and Developers: Dan Blankenberg (dan(a)bx.psu.edu) from the Galaxy Team will be in San Diego between 14 and 20 of January. If anyone in SD area runs local Galaxy installs and would like a bit face-to-face time with Dan (he is one of the senior developers on the project) e-mail him directly. Thanks! anton Anton Nekrutenko http://nekrut.bx.psu.edu http://usegalaxy.org

1 0

Re: [galaxy-dev] [galaxy-user] Read shuffler and code contributions
by Florent Angly 16 Dec '10

16 Dec '10

On 16/12/10 18:50, Peter wrote: >> Hi Peter, >> >>> Are you asking for a tool to interleave to FASTQ or FASTA files with >>> matching entries (with matching names in the same order) into one >>> file which alternates forward then reverse read? >> Yes, indeed, this is what I am proposing. >> >>> Would you prefer it with or without error checking? >> Error checking is best. > I'd agree. > >>> I think the scripts in velvet are fast but will fail horribly with >>> bad input... note there is a simple Biopython script to do this >>> included with velvet already (simple version with no error checking, >>> I have written a more robust version too - it looks like I haven't >>> sent it to Daniel to include in velvet though). >> I rolled my own FASTQ paired read interlacer and deinterlacer today, using >> the Galaxy Python modules in lib/galaxy_utils/. I must say these modules >> made it quite convenient and efficient to implement error-checking in the >> (de)interlacing. You can find the scripts here if you're interested: >> http://bitbucket.org/fangly/galaxy-central >> I'll make the XML wrappers tomorrow and test them. Hopefully after this is >> done, my changes can be pulled into the official Galaxy repository. > For the deinterlacer, I previously offered to write something like > that for Galaxy and was told to submit it to the Tool Shed initially > (although it may be merged into the official repository at some > point). See "Divide FASTQ file into paired and unpaired reads" > on http://community.g2.bx.psu.edu/ for my tool. > I also note you've changed the return behaviour of the Galaxy > FASTQ library method get_paired_identifier - that API change > could break other parts of Galaxy or 3rd party tools. Yes, you're right about breaking the API. I realized that and reverted my change. I am now running the Galaxy tests to make sure everything is alright. > Looking at that Galaxy lib, perhaps I can offer some of my > code for identifying Sanger read pairs and the .f .r suffices > to enhance the class fastqJoiner (look like it only does > Illumina /1 and /2 right now which I think is too narrow). I am not familiar with the nomenclature for Sanger mate pairs / paired-read, but that's a good point. Florent

1 0

rpy2 integration
by henry＠mpi-cbg.de 16 Dec '10

16 Dec '10

I have a local installation of galaxy running and I'm trying to run a custom python script "sequence_logo.py" from galaxy. We have many other custom scripts installed and running perfectly well, but this is the first to import rpy2 modules. If galaxy runs the code it gives the following error: Traceback (most recent call last): File "/home/galaxy/galaxy_dist/tools/MPItools/sequence_logo.py", line 3, in import rpy2.robjects as robjects File "/usr/local/lib/python2.6/dist-packages/rpy2-2.1.9_20101216-py2.6-linux-x86_64.egg/rpy2/robjects/__init__.py", line 14, in import rpy2.rinterface as rinterface File "/usr/local/lib/python2.6/dist-packages/rpy2-2.1.9_20101216-py2.6-linux-x86_64.egg/rpy2/rinterface/__init__.py", line 79, in from rpy2.rinterface.rinterface import * ImportError: libR.so: cannot open shared object file: No such file or directory However if I run the script from the command line on our galaxy machine the import statements work and the code runs. Originally when I installed rpy2 via easy_install the script ran neither from the command line nor galaxy. I read from Rpy help that this was because RPy could not find libR.so and I followed the following fix: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH\:R_HOME/bin As a result I can now run the script from the command line but still not from galaxy. The export is now in my .bashrc too. Does anyone have any ideas why the rpy2 import statements work at the command line and python console, but not from within Galaxy. Any help would be greatly apprecriated. The import line that fails is as follows: import rpy2.robjects as robjects Thanks, Ian Henry

2 2

Re: [galaxy-dev] [galaxy-user] Setting up a local Galaxy instance
by Kelly Vincent 16 Dec '10

16 Dec '10

Michael, (I'm CCing galaxy-dev because this has to do with a local installation, not our public server.) There is a wiki page discussing how to install several of the NGS tools, including Bowtie: https://bitbucket.org/galaxy/galaxy-central/wiki/NGSLocalSetup . Note that this page is slightly out-of-date regarding working with the loc files. We have introduced data tables since the page was written and it hasn't yet been updated. There is a basic page on data tables: https://bitbucket.org/galaxy/galaxy-central/wiki/DataTables. TopHat and Cufflinks are not detailed on that page, but the process should be very similar to Bowtie. Let us know if you have further questions. Regards, Kelly On Dec 16, 2010, at 4:12 PM, Weiner, Michael wrote: > I have been asked by a research fellow to setup a local instance of > Galaxy. In addition, he would like the latest versions of packages > like > bowtie, cufflinks, tophat, etc to be installed and available through > that instance. Being a newbie to Galaxy, and just a systems > administrator responsible for building this environment, I am unsure > how > exactly to go about doing this. > > I have an instance running, and worked my way through the fastx- > toolkit > installation/update but I cannot seem to find a similar how-to for the > other packages. Could someone point me in the right direction please? > > Thank you in advance > Michael Weiner > UNIX Systems Administrator > Lerner Research Institute > Cleveland Clinic > > =================================== > > P Please consider the environment before printing this e-mail > > Cleveland Clinic is ranked one of the top hospitals > in America by U.S.News & World Report (2009). > Visit us online at http://www.clevelandclinic.org for > a complete listing of our services, staff and > locations. > > > Confidentiality Note: This message is intended for use > only by the individual or entity to which it is addressed > and may contain information that is privileged, > confidential, and exempt from disclosure under applicable > law. If the reader of this message is not the intended > recipient or the employee or agent responsible for > delivering the message to the intended recipient, you are > hereby notified that any dissemination, distribution or > copying of this communication is strictly prohibited. If > you have received this communication in error, please > contact the sender immediately and destroy the material in > its entirety, whether electronic or hard copy. Thank you. > > > _______________________________________________ > galaxy-user mailing list > galaxy-user(a)lists.bx.psu.edu > http://lists.bx.psu.edu/listinfo/galaxy-user

2 1

boolean type param (for an input tickbox) does not work correctly
by Marina Gourtovaia 16 Dec '10

16 Dec '10

Hello The boolean widget in the following tool config always works as if not ticked while the select drop-down box in the same situation (teh commented out xml) works ok. What's wrong with my tickbox? <tool id="bam_to_fastq" name="BAM-to-FASTQ" version="1.0.0"> <requirements> <requirement type="package">picard</requirement> </requirements> <description>converts BAM format to FASTQ format</description> <command> java -jar ${GALAXY_DATA_INDEX_DIR}/shared/jars/SamToFastq.jar VALIDATION_STRINGENCY=SILENT QUIET=true INPUT=$bam_in FASTQ=$fastq1_out #if $sPaired == "paired": SECOND_END_FASTQ=$fastq2_out #end if </command> <inputs> <param name="sPaired" type="boolean" truevalue="paired" falsevalue="single" checked="yes" label="Reads are paired" />  <param name="bam_in" type="data" format="bam" label="BAM File to Convert" /> </inputs> <outputs> <data name="fastq1_out" format="fastqsanger" /> <data name="fastq2_out" format="fastqsanger" > <filter>sPaired == "paired"</filter> </data> </outputs> <tests> </tests> <help> </help> </tool> Regards Marina Gourtovaia -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

3 4