galaxy-dev February 2011

galaxy-dev@lists.galaxyproject.org

63 participants
110 discussions

custom datatypes
by Glen Beane 04 Mar '11

04 Mar '11

I've created a custom file type based on the Galaxy Tabular type (this is so some tools I'm developing can declare this type as input or output and prevent any arbitrary tabular file from being used as input for a tool) I have a data source tool that declares its output format to be my type (this tool brings the user to a website where they query a database and then sends the data file back to galaxy) when I load data with this tool the format is set to "tabular" by galaxy and not to my custom type. I have a feeling Galaxy is ignoring what the tool .xml says the format of the output is and tries to autodetect, and comes up with tabular. If a data source says the output is a specific type shouldn't galaxy use that as the format rather than autodetect? Also, I have other tools that declare my type as the input type, yet when I go to select an input file it shows me all tabular files in my history, not just those with my custom type. how do I get the format to be correctly set for a file I load from my data_source tool? And how do I limit selections in an input drop down to just my specific file type? I'm guessing I need to extend the Tabular class, but I don't need to add any additional functionality at this point, I just want to limit how the tools can be chained together. This datatype is probably just a place holder for what will probably end up being a binary type, so I'm not going to put much effort in. -- Glen L. Beane Software Engineer The Jackson Laboratory Phone (207) 288-6153

2 1

requested number of bytes is more than a Python string can hold
by Glen Beane 01 Mar '11

01 Mar '11

one of our users is experiencing this error from the SAM-to-BAM tool: Error extracting alignments from (/hpcdata/galaxy-setup/galaxy-dist/database/files/000/dataset_85.dat), requested number of bytes is more than a Python string can hold The SAM file has 58 million lines of mapped paired end reads. Any suggestions? The galaxy jobs running on our cluster are currently using the version of Python installed with CentOS 5 (python 2.4.3, 64-bit). We have been considering setting up a new python virtualenv for the galaxy user on the cluster that uses a newer version of Python (probably 2.6 so it matches the version on the server running the galaxy web app) -- Glen L. Beane Software Engineer The Jackson Laboratory Phone (207) 288-6153

2 4

uploading gzip compressed datasets
by Ryan Golhar 28 Feb '11

28 Feb '11

Hi all - I'm uploading datasets into my local instance by importing from the filesystem w/o copying into Galaxy. Importing of uncompressed files works properly. The files I'm importing are owned by user1 and a read-able by everyone, including the galaxy user. The galaxy user does not have write permissions in the directory where the files are stored. I have fastq files that are compressed with gzip and owned by user1. I don't want Galaxy to uncompress them, but it appears it is trying to during import, and subsequently fails because of permissions. Is there a way I can force galaxy to import the compressed files w/o uncompressing them? I think most of the mapping programs work on compressed fastq files so I'd rather not uncompressed them to save space. -- CONFIDENTIALITY NOTICE: This email communication may contain private, confidential, or legally privileged information intended for the sole use of the designated and/or duly authorized recipient(s). If you are not the intended recipient or have received this email in error, please notify the sender immediately by email and permanently delete all copies of this email including all attachments without reading them. If you are the intended recipient, secure the contents in a manner that conforms to all applicable state and/or federal requirements related to privacy and confidentiality of such information.

2 1

Re: [galaxy-dev] uploading gzip compressed datasets
by Ryan Golhar 28 Feb '11

28 Feb '11

That makes sense as BAM files are compressed. My files though are fastq. I don't think (or know) that BAM files can be used as input to mapping programs. On 2/28/11 4:01 AM, Christian Hundsrucker wrote: > Hi Ryan, > > thnx for the response. I figured out that this is/sould be solved for > .bam files > I just don't understand how... > > Best, > Christian > > see .../lib/galaxy/datatypes/binary.py > > [...] > def sniff( self, filename ): > # BAM is compressed in the BGZF format, and must not be > uncompressed in Galaxy. > # The first 4 bytes of any bam file is 'BAM\1', and the file is > binary. > try: > header = gzip.open( filename ).read(4) > if binascii.b2a_hex( header ) == binascii.hexlify( 'BAM\1' ): > return True > return False > except: > return False > [...] > > Ryan Golhar wrote: >> No, I never got a response to this...post it to the list and see what >> comes out of it... >> >> On 2/25/11 3:59 AM, Christian Hundsrucker wrote: >>> Hi Ryan, >>> >>> did you get an answer to your questions which didn't appear in the >>> mailing-list? >>> I have a similar Problem: >>> want to load a R-workspace within a galaxy module (.rdat-file, >>> R-Project) and therefore built the galaxy-.rdat datatype (binary). >>> .rdat-files are gzipped and are only recognized within R if they are >>> still zipped... >>> >>> Would be happy for any aswer! >>> If the answer is "no" I'll add this problem to the mailing list. >>> >>> Thnx, >>> Christian >>> >>> Ryan Golhar wrote: >>>> Hi all - I'm uploading datasets into my local instance by importing >>>> from the filesystem w/o copying into Galaxy. >>>> >>>> Importing of uncompressed files works properly. The files I'm >>>> importing are owned by user1 and a read-able by everyone, including >>>> the galaxy user. The galaxy user does not have write permissions in >>>> the directory where the files are stored. >>>> >>>> I have fastq files that are compressed with gzip and owned by user1. >>>> I don't want Galaxy to uncompress them, but it appears it is trying to >>>> during import, and subsequently fails because of permissions. >>>> >>>> Is there a way I can force galaxy to import the compressed files w/o >>>> uncompressing them? I think most of the mapping programs work on >>>> compressed fastq files so I'd rather not uncompressed them to save >>>> space. >>>> >>>> >>>> _______________________________________________ >>>> To manage your subscriptions to this and other Galaxy lists, please >>>> use the interface at: >>>> >>>> http://lists.bx.psu.edu/ >>> >>> > > -- CONFIDENTIALITY NOTICE: This email communication may contain private, confidential, or legally privileged information intended for the sole use of the designated and/or duly authorized recipient(s). If you are not the intended recipient or have received this email in error, please notify the sender immediately by email and permanently delete all copies of this email including all attachments without reading them. If you are the intended recipient, secure the contents in a manner that conforms to all applicable state and/or federal requirements related to privacy and confidentiality of such information.

1 0

How to retain files compressed
by Christian Hundsrucker 28 Feb '11

28 Feb '11

Hi all! I want to load a R-workspace within a galaxy module (.rdat-file, R-Project) and therefore built the galaxy-.rdat datatype (binary). .rdat-files are gzipped and are only recognized within R if they are still zipped. However, the corresponding .dat-file is an uncompressed version of the original .rdat file as I figured out using a hex-editor. I couldn't find any documentation how to change this behaviour, nor answers to similar Questions in this list. Would be happy for any answere that points me in the right direction. Details ##### datatypes_conf.xml: ----------------------------------- <?xml version="1.0"?> <datatypes> <registration converters_path="lib/galaxy/datatypes/converters" display_path="display_applications"> [...] <datatype extension="rdat" type="galaxy.datatypes.binary:Rdat" mimetype="application/octet-stream" display_in_upload="true"/> [...] </registration> <sniffers> [...] <sniffer type="galaxy.datatypes.binary:Rdat"/> [...] </sniffers> </datatypes> binary.py: ------------------ [...] class Rdat( Binary ): """Class describing an rdat binary file (R-workspace)""" file_ext = "rdat" #MetadataElement( name="Rdat", desc="R-workspace", param=metadata.FileParameter, readonly=True, no_value=None, visible=False, optional=True ) """ def __init__( self, **kwd ): Binary.__init__( self, **kwd ) self._name = "Rdat" """ def set_peek( self, dataset, is_multi_byte=False ): if not dataset.dataset.purged: dataset.peek = "Binary rdat file (R-workspace)" dataset.blurb = data.nice_size( dataset.get_size() ) else: dataset.peek = 'file does not exist' dataset.blurb = 'file purged from disk' def display_peek( self, dataset ): try: return dataset.peek except: return "Binary rdat file (%s)" % ( data.nice_size( dataset.get_size() ) ) def get_mime( self ): """Returns the mime type of the datatype""" return 'application/octet-stream' def sniff( self, filename ): # rdat is compressed in the gzip format, and must not be uncompressed in Galaxy. # The first 4 bytes of any rdat file are RDX2 try: header = gzip.open( filename ).read(4) #(4)=>4Bytes if binascii.b2a_hex( header ) == binascii.hexlify( 'RDX2' ): #check if there is the RDX2 signature return True return False except: return False try: header = open( filename ).read(4) #(4)=>4Bytes if binascii.b2a_hex( header ) == binascii.hexlify( 'RDX2' ): #check if there is the RDX2 signature return True return False except: return False -- Dr. Christian Hundsrucker Institute for Functional Genomics Computational Diagnostics Group University of Regensburg Josef Engertstr. 9 93053 Regensburg, Germany

2 1

unable to login user to local instance
by Ryan Golhar 25 Feb '11

25 Feb '11

Ok, so I think I have everything set up correctly. I'll test my cluster integration tomorrow. For now, I can see galaxy from my web browser. First issue - When I go to User -> Login and try to log myself in (I've already registered myself), it looks like it works, but the browser get redirected really fast to another page, then again to the main page. The User menu never shows me as logged in as it does on the public usegalaxy.org site. Is there a way to check that I'm actually logged in? If its not showing, then I assume something is wrong. Where do I look to debug this?

4 5

stagingin and stagingout at DRMAA
by hari krishna 25 Feb '11

25 Feb '11

Hi , Can any one help me how to transfer files from head node to remote location by using DRMAA job template attributes with example.... will DRMAA supports staging in and staging out of files,... -- Thanks & Regards, Hari Krishna .M

2 1

Amazon cloud formation for galaxy
by Ry4an Brase 25 Feb '11

25 Feb '11

Today's release of amazon cloud formation has to make the Galaxy Cloud stuff a bit easier: http://aws.typepad.com/aws/2011/02/cloudformation-create-your-aws-stack-fro… In theory with a description file like this: https://s3.amazonaws.com/cloudformation-templates/CloudFormationSample_Word… Once can define an entirely cluster to bring up from custom .amis with a single click. Exciting. -- Ry4an Brase 612-626-6575 Software Developer Application Development University of Minnesota Supercomputing Institute http://www.msi.umn.edu

2 1

Re: [galaxy-dev] Local Galaxy implimentation
by Anton Nekrutenko 25 Feb '11

25 Feb '11

Lee: I am CC'ing you e-mail to galaxy-dev list. You can get a lot of useful answers by posting to this list. Maintaining Galaxy is not different from maintaining other software resources. To run it locally you will need to have a system administration supervision as with any other high performance system. It is wonderful that you are sending your staff members to Galaxy Community Conference as they will be able to interact with key Galaxy Team members such as Nate Coraor, who runs Penn State instance here. Thanks! anton On Feb 25, 2011, at 6:14 AM, Lee Hazelwood wrote: > Dear Prof Nekrutenko, > > The university where I work is looking to locally implement your elegant software to enable us to view and analyse the rapidly growing amount of sequence data. We will be sending two staff members to the conference in Lunteren in May to get right into the detail on how we would achieve this. > > However, with all things I need to request some time/ support for running such a system in advance of May. > If you could help by briefly answering the questions below I would much appreciate it. > > * What is a realistic amount of someones time to implement Galaxy on a Linux server so that it can be used by researchers using the Web? > * What is the usual time overhead to keep it running and how skilled would they have to be? > > I should say, this will be for local biological researchers only, approx 20-30 dealing in Exome and Whole genome sequences. > > Best regards > Lee > > -- > Dr Lee Hazelwood > BHRC Bioinformatics Technology Group Leader > Biomedical and Health Research Centre > Faculty of Biological Sciences > Garstang Building, > University of Leeds, > Leeds, > LS2 9JT, UK. > E-mail: L.D.Hazelwood(a)leeds.ac.uk<mailto:L.D.Hazelwood@leeds.ac.uk> > > Anton Nekrutenko http://nekrut.bx.psu.edu http://usegalaxy.org

3 2

Only use filepaths and "Copy data into Galaxy?"
by Jean-Baptiste Denis 25 Feb '11

25 Feb '11

Hello everybody, i've got one galaxy server running. The ressource on the wiki are very useful for that purpose, thank you. I've got one question regarding data library when using "upload files from filesystem paths". There is a checkbox, unchecked by default, which prevent file upload into Galaxy. Is there a way to activate this checkbox by default, or to totally remove it on a per library basis ? I'm afraid of data libraries admin forgetting to check this box :D Regards, Jean-Baptiste

2 2

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

galaxy-dev February 2011