February 2011 - galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] NCBI BLAST+ wrappers in Galaxy?
by Peter 01 Feb '11

01 Feb '11

On Wed, Oct 13, 2010 at 10:22 AM, Peter <peter(a)maubp.freeserve.co.uk> wrote: > On Wed, Oct 13, 2010 at 12:58 AM, Kanwei Li <kanwei(a)gmail.com> wrote: >> >> Hi Peter, >> >> The BLAST+ branch has been transplanted into galaxy-central. Thank you! >> >> Kanwei > > Wow. Thanks :) There are another few change sets on this follow up branch, http://bitbucket.org/peterjc/galaxy-central/src/blastplus2 These are: Include BLAST-XML to tabular in tool_conf.xml.sample http://bitbucket.org/peterjc/galaxy-central/changeset/629448f1a17b Workaround Issue 397 using 'if' in Cheetah template to avoid hidden parameters http://bitbucket.org/peterjc/galaxy-central/changeset/02ce0b68936a Ignore Unix editor backup files ending with tilde (not BLAST+ specific): http://bitbucket.org/peterjc/galaxy-central/changeset/708816ffd451 Ignore # lines in BLAST tabular output in my filter script http://bitbucket.org/peterjc/galaxy-central/changeset/b2dcd6b8d434 And a capitalisation fix in my XML caption: http://bitbucket.org/peterjc/galaxy-central/changeset/6efadbea0cfc > > I mentioned it needs tests: > I've worked out how to add unit tests to the XML wrapper (that is well documented) and run them via run_functional_tests.sh - so adding some BLAST+ tests should be possible now. Peter

5 29

Problem upgrading DB from 67 to 68
by Assaf Gordon 01 Feb '11

01 Feb '11

Hi, Trying to upgrade DB schema ('sh manage_db.sh upgrade') from 67 to 68 fails with the following message: === sqlalchemy.exc.ProgrammingError: (ProgrammingError) syntax error at or near "RENAME" LINE 1: ALTER SEQUENCE sequencer_id_seq RENAME TO external_service_i... ^ 'ALTER SEQUENCE sequencer_id_seq RENAME TO external_service_id_seq' {} === Full back-trace attached at the end of this email. My postgres version is 8.1.18 on CentOS 5.4. I suspect the error is that "ALTER SEQUENCE RENAME" is new to Postgres 9: http://www.postgresql.org/docs/9.0/static/sql-altersequence.html Where as in Postgres 8.1, there"s no "ALTER SEQUENCE RENAME": http://www.postgresql.org/docs/8.1/interactive/sql-altersequence.html The bottom notes on the 8.1 help page say: " Some variants of ALTER TABLE can be used with sequences as well; for example, to rename a sequence use ALTER TABLE RENAME. ". Any recommendations on how to proceed will be appreciated, otherwise I'll just change: lib/galaxy/model/migrate/versions/0068_rename_sequencer_to_external_services.py Thanks, -gordon ---------------------- sh manage_db.sh upgrade 67 -> 68... This migration script renames the sequencer table to 'external_service' table and creates a association table, 'request_type_external_service_association' and populates it. The 'sequencer_id' foreign_key from the 'request_type' table is removed. The 'sequencer_type_id' column is renamed to 'external_service_type_id' in the renamed table 'external_service'. Finally, adds a foreign key to the external_service table in the sample_dataset table and populates it. Traceback (most recent call last): File "./scripts/manage_db.py", line 55, in <module> main( repository=repo, url=db_url ) File "/home/gordon/projects/galaxy_devel/eggs/sqlalchemy_migrate-0.5.4-py2.5.egg/migrate/versioning/shell.py", line 150, in main ret = command_func(**kwargs) File "/home/gordon/projects/galaxy_devel/eggs/sqlalchemy_migrate-0.5.4-py2.5.egg/migrate/versioning/api.py", line 221, in upgrade return _migrate(url, repository, version, upgrade=True, err=err, **opts) File "/home/gordon/projects/galaxy_devel/eggs/sqlalchemy_migrate-0.5.4-py2.5.egg/migrate/versioning/api.py", line 349, in _migrate schema.runchange(ver, change, changeset.step) File "/home/gordon/projects/galaxy_devel/eggs/sqlalchemy_migrate-0.5.4-py2.5.egg/migrate/versioning/schema.py", line 184, in runchange change.run(self.engine, step) File "/home/gordon/projects/galaxy_devel/eggs/sqlalchemy_migrate-0.5.4-py2.5.egg/migrate/versioning/script/py.py", line 101, in run func() File "lib/galaxy/model/migrate/versions/0068_rename_sequencer_to_external_services.py", line 106, in upgrade db_session.execute( cmd ) File "/home/gordon/projects/galaxy_devel/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/orm/scoping.py", line 127, in do return getattr(self.registry(), name)(*args, **kwargs) File "/home/gordon/projects/galaxy_devel/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/orm/session.py", line 755, in execute clause, params or {}) File "/home/gordon/projects/galaxy_devel/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py", line 824, in execute return Connection.executors[c](self, object, multiparams, params) File "/home/gordon/projects/galaxy_devel/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py", line 874, in _execute_clauseelement return self.__execute_context(context) File "/home/gordon/projects/galaxy_devel/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py", line 896, in __execute_context self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context) File "/home/gordon/projects/galaxy_devel/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py", line 950, in _cursor_execute self._handle_dbapi_exception(e, statement, parameters, cursor, context) File "/home/gordon/projects/galaxy_devel/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py", line 931, in _handle_dbapi_exception raise exc.DBAPIError.instance(statement, parameters, e, connection_invalidated=is_disconnect) sqlalchemy.exc.ProgrammingError: (ProgrammingError) syntax error at or near "RENAME" LINE 1: ALTER SEQUENCE sequencer_id_seq RENAME TO external_service_i... ^ 'ALTER SEQUENCE sequencer_id_seq RENAME TO external_service_id_seq' {}

2 1

histogram tool not working
by David Hoover 01 Feb '11

01 Feb '11

I think I have everything installed correctly for the histogram tool. However, when I try to generate a histogram with a simple tabular file of numbers, I get the following error message: An error occurred running this job: array() argument 1 must be char, not list Any ideas? David Hoover Helix Systems Staff http://helix.nih.gov

4 5

quality_scores.loc files
by David Hoover 01 Feb '11

01 Feb '11

Where do the files in quality_scores.loc come from? Can they be downloaded from UCSC or somewhere else? David Hoover Helix Systems Staff http://helix.nih.gov

2 5

Data Library - upload files from filesystem paths
by Nicki Gray 01 Feb '11

01 Feb '11

Hi As an admin I have created a data library on our local instance of galaxy, created folders in that library and added datasets using 'upload files from filesystem paths' option, choosing not to copy the data into Galaxy, which then loads all the files in the given directory into the data library. Is there a way to refresh this list automatically? so that if a new file is added into that directory it will appear in the data library? or do I have to add new files in that directory manually, giving the path to the new file? I am also having a problem with creating sub folders - I choose to create sub folder from the drop down list and it reports that the sub folder has been created but I cant see it, and I cant click on the blue triangles to the left of the folders either. Does anyone else have this problem? Any help would be appreciated, Thanks, Nicki --------------------- Nicki Gray MRC Molecular Haematology Unit 01865 222434

2 4

Links to anchors in Pages
by SHAUN WEBB 01 Feb '11

01 Feb '11

Hi, I am creating a tutorial using galaxy pages and I want to add a contents section at the top to link to headings further down the page. I can't quite figure out how to use the anchoring. Thanks for your help. Shaun Webb -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

2 1

Batch download of a history
by Davide Cittaro 01 Feb '11

01 Feb '11

Hi all,

1 1

Re: [galaxy-dev] Question about GC
by Enis Afgan 01 Feb '11

01 Feb '11

Hi Xiandong, Yes, it is possible to run a customized Galaxy instance and also use your own tools on the cloud but there is a bit of a process. Namely, you will need to install the tools by hand, create required EBS volume snapshots, and changing some configuration files in your cluster's S3 bucket. I can work on a wiki page on how to achieve this over the coming days and guide you through the process, if you'd like to proceed with it? Enis (I've CC'd galaxy-dev mailing list because this is probably something other users are interested in as well) On Mon, Jan 31, 2011 at 7:05 PM, Xiandong Meng <xiandongmeng(a)lbl.gov> wrote: > Hello Enis, > > I am Xiandong from Berkeley Lab. I just read the tutorial about how to run > Galaxy Instance on Amazon. I was wondering if it is possible to run > customized Galaxy Instance with our own tools on Amazon. It seems that I can > run a predefined AMI only. > > Thanks a lot, > Xiandong > >

2 2

March 2011 GMOD Meeting: Registration is open
by Dave Clements 31 Jan '11

31 Jan '11

Hello all, Registration is now open for the March 2011 GMOD Meeting ( http://gmod.oicr.on.ca/wiki/March_2011_GMOD_Meeting) This meeting is part of GMOD Americas 2011, an event that includes a day of Satellite Meetings, a GMOD Course (ialready full), and for the first time, an "Introduction to GMOD" session the night before the meeting for GMOD newcomers. GMOD Americas 2011 events are being held at the US National Evolutionary Synthesis Center (NESCent) in Durham, North Carolina, United States. As with previous GMOD meetings, this meeting will have a mixture of project talks, component talks, and user talks. Our guest speaker is Dr. Eric Stone of North Carolina State University. Dr. Stone will talk about his experience on the "Drosophila Genome Reference Panel," a project that is sequencing 192 lines. See http://gmod.oicr.on.ca/wiki/March_2011_GMOD_Meeting#Agenda for more. The agenda is driven by attendee suggestions, and you are encouraged to add your suggestions now ( http://gmod.oicr.on.ca/wiki/March_2011_GMOD_Meeting#Agenda_Proposals) For examples of what happens at a GMOD meeting, see the writeup of the September 2010 GMOD Meeting (http://gmod.oicr.on.ca/wiki/September_2010_GMOD_Meeting) or any previous meeting. GMOD meetings are an excellent way to meet GMOD developers and users and to learn (and affect) what's coming in the project. Registration for the March 2011 GMOD Meeting is $80 on or before February 18 <<<<======= $95 after February 18 Please register early, both to save money, and ensure a spot. You are also strongly encouraged to sign up for (or propose) a Satellite Meeting (more details to come). Details on transportation, suggested lodging, and other logistics are on the GMOD Americas 2011 page. This meeting, and all GMOD Americas 2011 events, are jointly sponsored by NESCent and the Galaxy Project. Dave Clements Galaxy Project -- http://gmod.org/wiki/GMOD_Americas_2011 http://nescent.org http://usegalaxy.org/ http://getgalaxy.org

1 0

January 31, 2011 Galaxy Development News Brief
by Jennifer Jackson 31 Jan '11

31 Jan '11

January 31, 2011 Galaxy Development News Brief http://bitbucket.org/galaxy/galaxy-central/wiki/Features/DevNewsBrief/2011_… ---- Get Galaxy! http://bitbucket.org/galaxy/galaxy-central/wiki/GetGalaxy * new: % hg clone http://www.bx.psu.edu/hg/galaxy galaxy-dist * upgrade: % hg pull -u -r 95d65755ac69 ---- What's New = Workflow Additions= 1) Usability improvements for workflow annotations * Workflow annotation is now shown at the top of the page. * Step annotations are shown in the step header rather than at the bottom. example: Annotation http://bitbucket.org/galaxy/galaxy-central/wiki/Features/DevNewsBrief/2011_… 2) Easier to move workflows directly from one Galaxy instance to another * Workflow download/export page now provides URL that can be used to directly import a workflow from one instance to another. example: URL import http://bitbucket.org/galaxy/galaxy-central/wiki/Features/DevNewsBrief/2011_… 3) New Parameter settings for global application or specific actions * Workflow parameters are a new feature we've added to simplify reuse of workflows, and to allow for easier variation of parameters when re-running a workflow. * Instead of filling in explicit values when building a workflow, you can now use flexible parameters. To specify a workflow parameter, simply use a tag like ${my_variable_name} in any tool input field or in a rename dataset action field. * The workflow shown below has two parameters, as shown in the Workflow Parameters display in the top right of the editor window. You can see the ${filter_condition} parameter in the right panel in both the tool input and the rename action. ** Note that while this ${filter_condition} is only used in a single step in this simple demo workflow, variables can be used across steps. example: Parameters http://bitbucket.org/galaxy/galaxy-central/wiki/Features/DevNewsBrief/2011_… *As the inputs are filled in the Workflow Parameters box, seen in the runtime example below, the new values will be reflected in all workflow steps and will be used when the workflow is executed. example: Runtime display http://bitbucket.org/galaxy/galaxy-central/wiki/Features/DevNewsBrief/2011_… 4) General workflow tuning * HideDataset Action will no longer show in the workflow editor. The ability to manually create one of these actions has been deprecated in favor of the workflow outputs approach. * Workflow run results can now be sent to a new history instead of the current one. * Ordering of workflow steps is now sorted based on the layout in the workflow editor, arranged based on distance from top left corner of the editor. This won't affect existing workflows until re-saved. * Workflows that contain steps expecting tools that are unavailable (as might be the case for a workflow imported from another Galaxy instance) will now have problem nodes marked with an error state. The workflow cannot be saved until the steps are removed or the tools are added to the current Galaxy instance. = Deferred Jobs & Managed Transfers = These components are under rapid development and interfaces should be considered experimental. They can be enabled by setting 'enable_beta_job_managers = True' in universe_wsgi.ini. 1) Deferred Jobs * A generic method for creating a dependency on an event before executing arbitrary code has now been defined in: galaxy-dist/lib/galaxy/job/deferred/__init__.py * The deferred job runner loads plugins found in the same directory which implement the necessary methods check_job() and run_job(). check_job() returns a state which informs the deferred job runner whether it is okay to execute the run_job() method. * The deferred job runner is independent from the regular tool-related job runner and is not coupled with tools, nor does it have any integrated cluster support. * No documentation is provided for the format of a plugin at this time, but a sample plugin will be included at a later date. 2) Transfer Manager * Galaxy can now spawn persistent transfers of (unauthenticated) http and https URLs via the code in: galaxy-dist/lib/galaxy/job/transfer_manager.py * The transfer manager is accessible in-application at app.transfer_manager. * Transfers are daemonized and thus not influenced by Galaxy restarts, although a loss of database connection (restarting the database server) will cause transfers to fail. * A transfer can be polled for progress via JSON-RPC requests to its socket. An interface for this request is available in the transfer_manager class. * Future enhancements will allow for authenticated http/https and scp. ---- Updated & Improved = Data Content = * New "Get Data" source: modENCODE modMine server. * New Genomes: format: Genome [dbkey] ** //Pseudomonas aeruginos (str. PA7) [16720] ** Pseudomonas aeruginos (str. PAO1) [pseuAeru] ** Pseudomonas aeruginos (str. UCBPP-PA14) [386] ** Silkworm (Bombyx mori str. p50T) [Bombyx_mori_p50T_2.0] ** Rice pathogen: Xanthomonas oryzae (str. KACC10331) [12931] ** Rice pathogen: Xanthomonas oryzae (str. MAFF 311018) [16297] ** Rice pathogen: Xanthomonas oryzae (str. PXO99A) [Xanthomonas_oryzae_PXO99A] ** Little Brown Bat (Myotis lucifugus) [myoluc2] = Current Tools = * Optional 'min' and 'max' attributes for tool integer and float parameters. * Add --max and --un options to Bowtie. * Validators enforce min/max values and generated error messages for invalid values. * Add Standard deviation operation to grouping tool. * Enhance histogram tool to allow plotting as frequency/counts. * Allow fastx_toolkit clipper wrapper to work on fastqsanger formatted files. * SAM metadata detection and setting sped up dramatically upon import. * Wig-to-bigWig converter tool default parameters now same as UCSC's. * Cufflinks tool suite updates: ** Update Cufflinks tool suite wrappers to support v0.9.3. ** Add support for bias correction to Cufflinks and Cuffdiff; bias correction improves transcript quantitation results (FPKM values). ** Enable Cuffcompare to use sequence data so that it can generate data for use by Cuffdiff. Cuffcompare uses sam_fa_indices.loc to find locally cached genome sequence data and indices that are needed for this option. ** Add normalization support to Cufflinks and Cuffdiff and replicate support for cuffdiff. = New Tools = * BED-to-bigBed converter tool now under "Convert Formats". ** Converts sorted BED files into UCSC's bigBed format. ** Requires bedToBigBed in PATH. * BLAST+ tools are now commented in tool_conf.xml.sample. ** Not in Galaxy main, for local instance use. ** If you run the BLAST+ tools at your site, please be sure to uncomment them if updating to the latest tool_conf.xml.sample (will be default at next update). * Add SAMtools flagstat tool. * Add CCAT ChIP-seq peak/region caller. * Add BWA wrapper for SOLiD. = Histories = * Estimated size is now displayed for very large text-based (non-binary) datasets. * Always show Galaxy masthead and enable Saved Histories to work with and without panels. * New "Copy Datasets" link added under History Options dropdown. ** Dataset and history IDs are now encoded. ** JS dropdown used to change source history, so you no longer have to switch desired source history to be "active". ** Remove link in "edit attributes" of datasets. ** Simplified destination interface by using a single select box by default, but also providing a link to show checkboxes for multiple destination histories. ** Use newly imported Inflector library to get correct plural/singular nouns on actions. ** Added arrow between source and destination. ** Removed checkbox for copying to a new history. Instead, create new history if new history field is not blank. = Data Libraries= * Creating job information (stdout/stderr) is now available on the library item info page, which is helpful when library uploads fail. = Trackster = * Add bigWig display to trackster. Automatically converts wig to bigwig if needed (NOTE: datatypes_conf.xml.sample has been edited to add the new converter, you must update datatypes_conf.xml to use it). The converter requires that wigToBigWig be in the PATH, but no other tools are needed to view bigwig files as they are provided by bx_python. * Tuning ** Fix track preferences not being applied ** Fix chroms not being selectable when a new track browser is created ** Fix ReferenceTrack not working with filters ** Fix visual analytics error when tool configuration has changed ** Fix visualization saving on Chrome by using $.each instead of for loop ** Fix shared visualizations ** Fix tracks fetching from data provider when indexer returned None, and when ** Fix BAM reads without cigar string. = Sample Tracking = * Additions and tuning to improve tracking (complete documentation will be available soon). = User Interface (UI) = *Version info is now printed in history item for Bowtie, BWA, Lastz, TopHat, Samtools, Cuffdif, Cufflinks, Cuffcompare, BFAST, and PerM. * Turn off web browser auto-complete for tool search (includes workflows). * Grid changes resulting in better readability: ** Better page number display. ** Use "~" instead of "about" example: "about 2 hours ago" -> "~ 2 hours ago". ** Cell padding decreased. ** Added new "nowrap" parameter to prevent text from being wrapped. Currently only used for tags to prevent "X tags" from wrapping in the middle. = Application Programming Interface (API) = * Still 'alpha', but: Initial pass completed for forms, request_type, users and roles. See README and examples in source. = Source = * Galaxy now runs with system python on 64-bit mac kernel. * Enhance select parameter wrapper objects to provide access to additional fields by name for dynamically generated select lists (i.e. dynamic_options). example: use ${param.fields.path} to access a path field * Updated the XML in filter specification for output files. The closing filter tag can now be on separate line to use as an # actual filter. * Implemented 'from_work_dir' attribute for tool outputs. Using this attribute matches a file in the working directory to a tool output/HDA; when a tool finishes, the file in the working directory is automatically copied to the HDA. Hence, it's no longer necessary for tool wrapper scripts to manually copy tool output files to HDA files. * More programmatic control of page numbers on grid (custom UI). * Many formerly undocumented options have been added to the universe_wsgi.ini.sample file. Please compare your working copy with the new .sample and determine whether any of the options are relevant to your environment. *The "welcome page" found at static/welcome.html, has been renamed to static/welcome.html.sample. * The UCSC Genome Browser now supports loading data via https. If you implemented this former Galaxy URLs from https->http method in Apache as per the wiki documentation you will also need to remove it there, and allow the browser through your authentication scheme over https. * Explicitly convert autocomplete dropdown values on refresh_on_change. * Adjust image links in tools to work with a proxy prefix (thanks Brad Chapman). = Bug Fixes = * Ensembl GTF files are recognized correctly. * Make Tophat wrapper compatible with python 2.4 by removing try-catch-finally. * Fix unicode error for dataset peeks. * Sanitize tool links in tool menu so that they can be searched. Use only lowercase letters, numbers, and underscores in links to ensure cross-browser functionality. * Fix for building form element for boolean tool parameter when default state is configured as checked in tool configuration, but user provided value is non-checked. Fixes an issue seen in workflows that prevents saving the unchecked state under described conditions. * Fix typo in BFAST. * Trackster: fix BAM display bugs, Dense display bug, better dataset selection. * Add validator to Tophat's segment mismatches parameter. * Turn off web browser autocomplete for tool search in workflows. * Fix Trackster's BAM display bugs, Dense display bug, better dataset selection. * Add validator to Tophat's segment mismatches parameter. * Added the ctypes egg so the DRMAA egg will now work under Python 2.4. * Bug fixes to the Mac OS X launcher in the galaxy-dist/contrib/ folder (from Florent Angly). * Python reports the wrong platform when running a 32-bit Python on 64-bit Linux, so Galaxy now forces the correct platform in this environment (thanks David Hoover). * Fix Chrome not auto-saving changes to workflow checkboxes. ---- About Galaxy Galaxy is supported in part by NSF, NHGRI, the Huck Institutes of the Life Sciences, and The Institute for CyberScience at Penn State. Core Team http://bitbucket.org/galaxy/galaxy-central/wiki/GalaxyTeam Use Galaxy! http://usegalaxy.org GalaxyProject.org http://galaxyproject.org|GalaxyProject.org Development Home http://bitbucket.org/galaxy/galaxy-central ---- Galaxy Team January 31, 2011

2 1