- galaxy-user - lists.galaxyproject.org

Re: [galaxy-user] [galaxy-dev] input security validation
by Roman Valls 28 Jun '11

28 Jun '11

Sorry for the cross-posting, but I guess that's also interesting to Galaxy users... I've been thinking about input validation too... only a bit more generally, on a server/production basis. Nate, on your production setup[1] (galaxy main), do you use any kind of filter or framework a la modsecurity[2] to prevent security issues on third party tools ? How do you mitigate those security risks ? [1] http://usegalaxy.org/production [2] http://modsecurity.org/ On 2011-06-23 00:27, Jennifer Jackson wrote: > Hi Russell, > > Dan Blankenberg is our ChIP-seq expert and will be able to work with you > when he returns from vacation. > > Thank you for your patience! > > Best, > > Jen > Galaxy team > > On 6/20/11 2:22 PM, Russell Bonneville wrote: >> Hello all, >> >> I am Russell, a student working with Dr. Victor Jin at the Dept. of >> Biomedical Informatics, OSU Medical Center. I am developing a Galaxy >> wrapper for our ChIP-seq peak-calling program BELT (PMID: 21138948), and >> I have a question about input validation. Does Galaxy filter for >> malicious field entries (such as XSS attacks) or is this our wrapper’s >> responsibility (the MACS wrapper appears to not perform any explicit >> checks)? Thank you for your time. >> >> Sincerely, Russell Bonneville >> >> >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> >> http://lists.bx.psu.edu/ >

2 1

FTP and command line access in Galaxy
by Robert Curtis Hendrickson 28 Jun '11

28 Jun '11

Nate, The Galaxy's ability to pull files with user/password from FTP sites as a client is great. However, I need to pull data from an HTTP site at a sequencing center with user/password (already tried to get them to set up an FTP server, no luck). Any way to do this? If not, would it be easy to add? Regards, Curtis > Hi Nate, > >>> We'd like to set up our local Galaxy to be able to import data (fastq >>> sequences) from an ftp site (with a username/password) into a user's >>> account. Does Galaxy support FTP or would we have to write a wrapper >>> script to do it via HTTP and use the standard data connection methods as >>> described in http://bitbucket.org/galaxy/galaxy-central/wiki/DataSources? >> >> Hi Steve, >> >> You can actually put FTP URLs directly in the URL/paste box on the >> upload form. With a username and password, the format would be: >> >> ftp://user:pass@example.org/path/to/file.ext >> >> I haven't tested that the user/pass bit works, but it should. > >Yep. It works. Thanks for the tip!

3 2

SIFT analysis input and output file
by Charlotte Gueydan 28 Jun '11

28 Jun '11

Sometimes when I do a sift analysis, I noticed that the input and output alleles are different. I don't understand why and what does it means ? could you explain it to me ? For example, here you can see that the input allele is G/A for chr2 (first line) and the output is G/T * input file: chr2 230341630 + G/A chr2 43881517 + A/T chr10 115912482 - G/T chr10 115900918 - C/T * output file: #Chrom Position Strand Allele Codons Transcript ID Protein ID Substitution Region dbSNP ID SNP Type Prediction Score Median Info Num seqs at position User Comment chr2 230341630 + G/T - ENST00000389045 ENSP00000373697 NA EXON CDS rs1803846:A Unknown Not scored NA NA NA chr2 43881517 + A/T ATA-tTA ENST00000260605 ENSP00000260605 I230L EXON CDS rs11556157:T Nonsynonymous TOLERATED 0.47 3.19 7 chr10 115912482 + C/A CGA-CtA ENST00000369285 ENSP00000358291 R179L EXON CDS rs12782946:T Nonsynonymous TOLERATED 0.06 4.32 2 chr10 115900918 + G/A CAA-tAA ENST00000369287 ENSP00000358293 Q271* EXON CDS rs7095762:T Nonsynonymous N/A N/A N/A N/A Regards. Charlotte Gueydan -- *Charlotte Gueydan *, PhD. INSERM/Université Pierre et Marie Curie Hôpital Tenon - bâtiment de recherche 4, rue de la Chine 75020 Paris cedex 20 tel : 01 56 01 83 75

2 1

question on cufflinks output
by Wen Huang 27 Jun '11

27 Jun '11

Dear Galaxy team and users, I have a question on the output by cufflinks on Galaxy. I started with about 28M paired-end reads and mapped them to the reference genome using Tophat on Galaxy. The aligned fragments were assembled by cufflinks, again on Galaxy and I got an output with the first few lines on the bottom of this email. I was wondering how could cufflinks possibly estimate FPKM on the order of e+07 when the coverage is between 8-50 fragments per base and the total mapped fragments smaller than 28M. Assuming that 20M fragments were mapped, the FPKM should be something around coverage/28. Was the e in the output the Euler's number or 10? I appreciate your help. Thanks, Wen Huang tracking_id class_code nearest_ref_id gene_id gene_short_name tss_id locus length coverage status FPKM FPKM_conf_lo FPKM_conf_hi CUFF.2.1 - - CUFF.2 - - chr1:90301-90706 405 21.1837 OK 1.84527e+07 1.10716e+07 2.58338e+07 CUFF.1.1 - - CUFF.1 - - chr1:65419-65692 273 30.9833 OK 2.31848e+07 8.52143e+06 3.78481e+07 CUFF.3.1 - - CUFF.3 - - chr1:135255-135896 641 8.61389 OK 6.31907e+06 3.41968e+06 9.21846e+06 CUFF.4.1 - - CUFF.4 - - chr1:155808-156529 721 7.26147 OK 5.32695e+06 2.88278e+06 7.77112e+06 CUFF.5.1 - - CUFF.5 - - chr1:160421-160729 308 17.6004 OK 1.77483e+07 7.50132e+06 2.79953e+07 CUFF.6.1 - - CUFF.6 - - chr1:170695-171212 517 9.16414 OK 8.41605e+06 4.44869e+06 1.23834e+07 CUFF.7.1 - - CUFF.7 - - chr1:180885-181188 303 30.5702 OK 2.6515e+07 1.36533e+07 3.93767e+07 CUFF.8.1 - - CUFF.8 - - chr1:184397-184702 305 26.712 OK 2.13696e+07 9.94707e+06 3.27921e+07 CUFF.10.1 - - CUFF.10 - - chr1:233237-234095 858 3.71208 OK 3.31435e+06 1.60283e+06 5.02588e+06 CUFF.9.1 - - CUFF.9 - - chr1:203688-204070 382 41.6301 OK 5.36082e+07 4.02061e+07 6.70102e+07 CUFF.11.1 - - CUFF.11 - - chr1:239126-239664 538 19.5995 OK 2.0562e+07 1.45634e+07 2.65605e+07 CUFF.12.1 - - CUFF.12 - - chr1:243903-244327 424 10.3509 OK 1.07542e+07 5.37709e+06 1.61313e+07 CUFF.15.1 - - CUFF.15 - - chr1:240487-240995 508 15.8596 OK 1.83065e+07 1.23671e+07 2.42459e+07

4 3

Unable to load FASTA files?
by Laura Elizabeth Spoor 27 Jun '11

27 Jun '11

Is anyone else having trouble uploading FASTA files to the Galaxy server? I thought it was my files at first, but I have since tried to upload other FASTA files that I have previously had no trouble uploading and I get the same error reported in the info section when I click on the pen icon: Unable to queue job for execution. Resubmitting the job may succeed. (Unfortunately re-submitting the job does not succeed!). The file is 50 MB in size. Any help is greatly appreciated as I am unable to progress my work until this problem is corrected. Best Wishes, Laura -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

3 2

Error running cufflinks on Galaxy
by Jennifer Jackson 24 Jun '11

24 Jun '11

Dear all, I tried to follow the Cufflinks tutorial created by Jeremy. The problems start with first "visualization" step. The is no overlapping at all between Cufflinks transcripts and RefSeq Chr19 reference, provided on the tutorial page. Was that the goal, or transcripts should overlap and something did not work? By the way, mapping for cufflinks was done to hg19, but reference annotation is provided at hg18. Maybe, this is the problem? Aleks

1 0

question on cufflinks output
by Wen Huang 24 Jun '11

24 Jun '11

Dear Galaxy team and users, I have a question on the output by cufflinks on Galaxy. I started with about 28M paired-end reads and mapped them to the reference genome using Tophat on Galaxy. The aligned fragments were assembled by cufflinks, again on Galaxy and I got an output with the first few lines on the bottom of this email. I was wondering how could cufflinks possibly estimate FPKM on the order of e+07 when the coverage is between 8-50 fragments per base and the total mapped fragments smaller than 28M. Assuming that 20M fragments were mapped, the FPKM should be something around coverage/28. Was the e in the output the Euler's number or 10? I appreciate your help. Thanks, Wen Huang tracking_id class_code nearest_ref_id gene_id gene_short_name tss_id locus length coverage status FPKM FPKM_conf_lo FPKM_conf_hi CUFF.2.1 - - CUFF.2 - - chr1:90301-90706 405 21.1837 OK 1.84527e+07 1.10716e+07 2.58338e+07 CUFF.1.1 - - CUFF.1 - - chr1:65419-65692 273 30.9833 OK 2.31848e+07 8.52143e+06 3.78481e+07 CUFF.3.1 - - CUFF.3 - - chr1:135255-135896 641 8.61389 OK 6.31907e+06 3.41968e+06 9.21846e+06 CUFF.4.1 - - CUFF.4 - - chr1:155808-156529 721 7.26147 OK 5.32695e+06 2.88278e+06 7.77112e+06 CUFF.5.1 - - CUFF.5 - - chr1:160421-160729 308 17.6004 OK 1.77483e+07 7.50132e+06 2.79953e+07 CUFF.6.1 - - CUFF.6 - - chr1:170695-171212 517 9.16414 OK 8.41605e+06 4.44869e+06 1.23834e+07 CUFF.7.1 - - CUFF.7 - - chr1:180885-181188 303 30.5702 OK 2.6515e+07 1.36533e+07 3.93767e+07 CUFF.8.1 - - CUFF.8 - - chr1:184397-184702 305 26.712 OK 2.13696e+07 9.94707e+06 3.27921e+07 CUFF.10.1 - - CUFF.10 - - chr1:233237-234095 858 3.71208 OK 3.31435e+06 1.60283e+06 5.02588e+06 CUFF.9.1 - - CUFF.9 - - chr1:203688-204070 382 41.6301 OK 5.36082e+07 4.02061e+07 6.70102e+07 CUFF.11.1 - - CUFF.11 - - chr1:239126-239664 538 19.5995 OK 2.0562e+07 1.45634e+07 2.65605e+07 CUFF.12.1 - - CUFF.12 - - chr1:243903-244327 424 10.3509 OK 1.07542e+07 5.37709e+06 1.61313e+07 CUFF.15.1 - - CUFF.15 - - chr1:240487-240995 508 15.8596 OK 1.83065e+07 1.23671e+07 2.42459e+07

1 0

June 23, 2011 Galaxy Development News Brief
by Jennifer Jackson 24 Jun '11

24 Jun '11

June 23, 2011 Galaxy Development News Brief http://galaxyproject.org/wiki/Features/DevNewsBrief/2011_06_23 ---- == Source Distribution == http://getgalaxy.org new: % hg clone http://www.bx.psu.edu/hg/galaxy galaxy-dist upgrade: % hg pull -u -r 720455407d1c ---- == GCC 2011 == We had ~ 140 attendees at the 2011 Galaxy Community Conference, held May 25-26, in the Netherlands! * Please see [[http://wiki.g2.bx.psu.edu/GCC2011]] for links to slides and to videos of the talks. * Feedback has been very positive with 100% of respondents saying they would recommend future Galaxy meetings to others. * Many thanks to the Netherlands Bioinformatics Centre (NBIC) for co-sponsoring and hosting this event. http://www.nbic.nl Note: "#NNN" notation refers to fixed/resolved bitbucket issues at http://galaxyproject.org/issues == What's New == === Python Support === Python 2.7 is now supported in beta. Functional tests with 2.7 are a pass. This also includes an upgrade to Numpy 1.6.0. === Cloudman Admin Functions === Galaxy's Cloudman now has an admin interface that allows you to gain insight and control many of the services underlying your cloud instance. Namely, you can: * Add Galaxy admin users through the web interface. * Update Galaxy source (even from a custom repository). * View logs and configurations for individual services. ** e.g., Galaxy, Postgres, SGE, as well as control some of the system-level services To get to the admin interface, just click the 'Admin' link at the top right corner on the main Cloudman UI. http://bytebucket.org/galaxy/galaxy-central/wiki/Features/DevNewsBrief/2011… === New Galaxy Tool Shed === Rather than rigidly structured tool archives, the foundational component of Galaxy's Tool Shed is now an hg (Mercurial) repository. * The Test/Sandbox Tool Shed available at http://testtoolshed.g2.bx.psu.edu * The Production Tool Shed is available at http://toolshed.g2.bx.psu.edu A new select list was added to the upload form in Galaxy's Tool Shed with the following label: "Remove files in the repository (relative to the upload point) that are not in the uploaded archive?" The selection pertains only to uploaded tar archives, not to single file uploads. If 'Yes' is selected, files that exist in the repository (relative to the selected upload point) but that are not in the uploaded archive will be removed from the repository. Otherwise, all existing repository files will remain and the uploaded archive will be added to the repository. Resolves #585 == Updated & Improved == === Tools === * RNA-seq tools: ** Rename and reorder Cuffdiff outputs for clarity. ** Make Cuffcompare wrapper compatible with v1.0.3. * 2bit/twobit datatypes now supported. (thanks Peter van Heusden!) * New tools for FASTA de(interlacing) added. Resolves #447 === Histories === * Add the download/save icon to datasets in an error state, but that contain data. Fixes #207 and #583 * Disk space usage accounting: ** Users can see how much disk a history is consuming. ** Disk used is also a column on the history grid. ** Users can also forcibly purge a dataset from their history. * Admin notes: ** New script (scripts/set_dataset_sizes.py) which will set the value of the new total_size column on the dataset table (includes the contents of the extra_files_path and metadata files). ** The ability for users to purge their own data is conditional on a config variable. Due to limitations in the grid framework, the button can't be removed from the history grid, but clicking it just does the same thing as delete but adds a message explaining that the purge operation wasn't performed. ** For users who were pulling from galaxy-central and that already upgraded to database version 78 prior to this distribution, the following SQL query should be run to set the default value of the 'HistoryDatasetAssociation' purged column: UPDATE history_dataset_association SET purged=false WHERE purged IS NULL === Trackster === * Can run filters on entire dataset. * When adding tracks to Trackster, add them in the order that they appear in the dataset listing. * Use error padding only when needed by placing error messages in div above tiles rather than in tiles. * Add bookmarking to Trackster. Also group Trackster options into a single menu button. * Add trackster_conf tag to tool definition; this tag indicates that tool is compatible with trackster. * Enable importing of visualizations via a shallow copy. === Workflows === * Include tool annotation and tags when cloning workflows. * The workflow editor is now aware of the format_source tag, and updates the output datatype of tools using it accordingly. Patch from Jim Johnson enhancing format="input" handling in workflows. Resolves #582 === Data Libraries === * Library datasets can now be job inputs. This is needed to be able to use the deferred job runner and transfer manager with set_metadata_externally = True. * Fix for importing library datasets into a new named history. Library datasets will no longer be imported into both the current history and the new named history ( only the latter ) in this scenario. * Determining if an accessible dataset exists somewhere in the data library hierarchy now works all the time. It used to work most of the time, but under corner conditions, it would break. * When importing a dataset from a history into a data library, the dataset permissions are now correctly applied to the library dataset. The DATASET_MANAGE_PERMISSIONS permission is a spoecial case in that if it exists on the dataset, the LIBRARY_MANAGE permission must be applied to the library dataset. * Fix for message display and db flushing when importing a library dataset into a history. * No longer render the folder pop-up menu option for selecting a library dataset to import into histories if the folder does not contain any accessible library datasets. === Source === * Move duplicate data type checker methods from sniff and upload into a new ~/datatypes/checkers.py. * No longer allow Datasets to have DatasetPermissions set such that no roles are associated with the DATASET_MANAGE_PERMISSION. Include the automatic creation of a new DatasetPermission for an hda where the DATASET_MANAGE_PERMISSION permission is associated with the hda.history.user's private role if the hda has no roles associated with the DATASET_MANAGE_PERMISSION permission. The creation of the DatasetPermission occurs when the user clicks the pencil icon for the hda. * SMTP Authentication support for all places where Galaxy sends mail. Use STARTTLS where available, too. Fixes #277 === Bug Fixes === * Fix solid2fastq tool to correctly handle cases when csfasta and QUAL filed are out of sync. Fixes #406 * Applied patch from Peter Cock for 'Cut' tool to ignore commented lines. Resolves #534 * Fix to sam2interval. The bug was causing sam2interval to successfully parse unmapped sam entries, which have no valid coordinate. * Enable Cufflinks to work when setting metadata externally. * Remove obsolete test values for Cufflinks and Cuffdiff tests. * In Trackster, have child track inherit mode from parent track. * Trackster bug fixes when (a) running tools in a shared visualization and (b) when drawing a track created by a tool. * Enable tags and annotations to work for new datasets in history. Fixes #550 * Fix bugs in reading and visualizing GFF3 datasets. * Remove unspecified build validators from Cufflinks/compare/diff tools as these cause trouble when running tools on custom genomes. * Disable update check in cuffdiff wrapper. * Unicode fixes and refactoring. * Disable grid controls and content when new content is being fetched. Fixes #272 * Include post-job actions when importing and exporting workflows. Fixes #518 * Improve filtering tool's stdout reporting of lines filtered. Fixes #536 * Show tool search by default. Fixes #575 * Fixes for editing role name, description, and renaming groups. Resolves #539 * Add baseline support for translating special chars to html escape codes for correct html display in the Galaxy Tool Shed. Fixes #564 * Commit versions of tools from the last_gen Tool Shed in order of oldest -> newest in the next-gen tool shed. Fixes #578 * Fix sample tracking to correctly import samples from a csv file. Add the ability to include field values (in addition to field names) when defining sample form rows by importing from a csv file. Fixes #398 ** The csv file must be in the following format: the [:FieldValue] is optional, the named form field will contain the value after the ':' if included. "SampleName,DataLibraryName,FolderName,HistoryName,WorkflowName,Field1Name:Field1Value,Field2Name:Field2Value..." * Change the Galaxy tool shed search feature on the Categories grid to search Repository names and descriptions rather than Category names and descriptions. The ability to search Category names and descriptions has been eliminated. Fixes #480 * Add support for uncompressing a gz or bz2 compressed file upon upload to a galaxy tool shed repository. Fixes #586 * Add the ability to select files for deletion from a tool shed repository using the built-in file browser. Any number of selected files will be deleted. Selecting a folder will delete all of it's contents. Fixes #584 * Required fields for extra sample form data now enforced. Some minor cleanup included. Fixes #301 * Workflow Parameter bug fix for the case when a parameter isn't used in any workflow step, but should still be available for PJAs. * Proper inheritance in model for APIItem. Fixes #527 * Fix broken user create/login links in the FTP upload form. * Set a catchall content-type (application/octet-stream) on metadata file downloads so Safari doesn't consider them to be HTML. * Fix for library upload when there is no current history. * 'fields' is apparently reserved in MySQL 4, fix migration scripts that use it unaliased. (thanks John Eppley!) * Fix the race condition of the upload tool's outputs not being created before the output size check occurs. (thanks Tao Liu!) * Remove hardcoding of universe_wsgi.ini (except for the sample cleanup scripts which, as samples, are expected to be modified by the site anyway). Fixes #358 * Don't provde the output filename to the upload tool if it's outside Galaxy's files_path, since this means we're only linking data and the output paths are not used (and may contain non-shell-safe characters). Fixes #533 * Uploading a compressed library dataset using either the server directory or path paste methods was previously overwriting the input file, which should never happen. Grouped with this change, it is now possible to upload compressed files that remain compressed (by choosing to "link files only"). However, Galaxy does not yet open the compressed file to properly set metadata on its contents. * Cope with white space at start of command after processing template. Fixes #159 * Support optional integer/float arguments. Resolves #403 * Copying datasets now keeps item order. * Hidden parameters no longer take up space in tool form. Fixes #397 * Sniffers are now properly loaded again from datatypes.conf.xml. === About Galaxy === The Galaxy team is a part of BX at Penn State and the Biology and Mathematics and Computer Science departments at Emory University. http://www.psu.edu http://www.bx.psu.edu http://www.emory.edu http://www.biology.emory.edu http://www.mathcs.emory.edu Galaxy is supported in part by NSF, NHGRI, the Huck Institutes of the Life Sciences, The Institute for CyberScience at Penn State, and Emory University. http://www.emory.edu http://www.huck.psu.edu http://www.nsf.gov http://www.genome.gov http://www.ics.psu.edu Galaxy Team http://galaxyproject.org/wiki/GalaxyTeam Galaxy Home http://galaxyproject.org Use Galaxy! http://usegalaxy.org Join us at Twitter! #usegalaxy http://twitter.com/#!/search/galaxyproject -- Want to learn more or have questions? * Visit our wiki: http://galaxyproject.org/wiki * View our Screencasts: http://usegalaxy.org/screencast * Learn from our Tutorials and Examples: http://usegalaxy.org/page/list_published * Subscribe to or email one of our mailing lists: http://lists.bx.psu.edu Subscribe, view archives, join the community! mailto:galaxy-user@bx.psu.edu Using Galaxy: science, data, and tools mailto:galaxy-dev@bx.psu.edu Galaxy Developers: installation, programming, and technical mailto:galaxy-bugs@bx.psu.edu Bug Reports: email us or even better, click the UI's "green bug" -- Thanks for using Galaxy!!

1 0

Re: [galaxy-user] newbies
by Jennifer Jackson 22 Jun '11

22 Jun '11

Hello Curtis, Please see under "Shared Data" in the public Galaxy for examples that cover a range of analysis. For RNA-seq, this tutorial will likely be helpful (under "Shared Data -> Published Pages -> Galaxy RNA-seq Analysis Exercise"): http://main.g2.bx.psu.edu/u/jeremy/p/galaxy-rna-seq-analysis-exercise Thanks! Jen Galaxy team On 6/22/11 12:48 PM, Robert Curtis Hendrickson wrote: > Jen, > > Are there any pages that give standard workflows for RNAseq, specifically analysis of alternate splicing? > I'm not finding any examples, and it would be quite helpful. > > Regards, > Curtis > > Research Associate > Center for Clinical and Translational Science > University of Alabama at Birmingham -- Jennifer Jackson http://usegalaxy.org/ http://galaxyproject.org/

1 0

GenBank Submission - How to Generate Fasta (not fastq) files
by John David Osborne 22 Jun '11

22 Jun '11

I still haven't found an easy solution to this problem and I am afraid I'm going to have to write one my own - which makes little sense as I bet this has been solved thousands of times! Can anybody point me to a script/software to convert a samtools pileup file into a fasta consensus file? It would be nice to set coverage thresholds, etc... but I'll take anything I can work with. The best google could do for me was this: http://biostar.stackexchange.com/questions/1389/how-to-generate-a-consensus… Not that helpful, -John P.S. If there is a better way of doing this (something other than samtools) I'm all ears.

2 2