- galaxy-dev - lists.galaxyproject.org

requested number of bytes is more than a Python string can hold
by Glen Beane 01 Mar '11

01 Mar '11

one of our users is experiencing this error from the SAM-to-BAM tool: Error extracting alignments from (/hpcdata/galaxy-setup/galaxy-dist/database/files/000/dataset_85.dat), requested number of bytes is more than a Python string can hold The SAM file has 58 million lines of mapped paired end reads. Any suggestions? The galaxy jobs running on our cluster are currently using the version of Python installed with CentOS 5 (python 2.4.3, 64-bit). We have been considering setting up a new python virtualenv for the galaxy user on the cluster that uses a newer version of Python (probably 2.6 so it matches the version on the server running the galaxy web app) -- Glen L. Beane Software Engineer The Jackson Laboratory Phone (207) 288-6153

2 4

Galaxy static assets and redirect woes
by Paul-Michael Agapow 01 Mar '11

01 Mar '11

This is not a Galaxy problem per se, but a problem during Galaxy setup so maybe others will have an insight. So we're installing our own Galaxy for production / general use. I've been following the config and advanced config guides and installed Galaxy in its own account and home directory. It runs off the built-in web-server fine - the problem comes when I try to put it through an Apache proxy. Galaxy has to run off a subdirectory (http://foobar/galaxy) and if I just rewrite everything through there (RewriteRule ^/galaxy(.*) http://localhost:7070$1 [P]), it works fine. The problem comes when I introduce the rewrite rules to have Apache serve the static content directly. None of the static content shows up and the error log shows "(13) permission denied" errors. Things I have tried: * Logged the rewrite calls to see they rewrite to the correct paths for the static content * Checked the proxy-filter and filter-with declarations * Checked the unix permissions on the static dir * Tested for non-existent or incorrect paths (which generate a different error) * Inserted directory declarations to "Allow from all" for the static dir * Checked for and tried .htaccess files * SELinux is disabled I feel I must have a blindspot about the solution - any ideas about where else to look? Technical details: cloned the latest Galaxy distribution, running on Fedora 9, Apache 2.2. ---- Paul Agapow (paul-michael.agapow(a)hpa.org.uk) Bioinformatics, Centre for Infections, Health Protection Agency ----------------------------------------- ************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of the HPA, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses, but please re-sweep any attachments before opening or saving. HTTP://www.HPA.org.uk **************************************************************************

2 1

uploading gzip compressed datasets
by Ryan Golhar 28 Feb '11

28 Feb '11

Hi all - I'm uploading datasets into my local instance by importing from the filesystem w/o copying into Galaxy. Importing of uncompressed files works properly. The files I'm importing are owned by user1 and a read-able by everyone, including the galaxy user. The galaxy user does not have write permissions in the directory where the files are stored. I have fastq files that are compressed with gzip and owned by user1. I don't want Galaxy to uncompress them, but it appears it is trying to during import, and subsequently fails because of permissions. Is there a way I can force galaxy to import the compressed files w/o uncompressing them? I think most of the mapping programs work on compressed fastq files so I'd rather not uncompressed them to save space. -- CONFIDENTIALITY NOTICE: This email communication may contain private, confidential, or legally privileged information intended for the sole use of the designated and/or duly authorized recipient(s). If you are not the intended recipient or have received this email in error, please notify the sender immediately by email and permanently delete all copies of this email including all attachments without reading them. If you are the intended recipient, secure the contents in a manner that conforms to all applicable state and/or federal requirements related to privacy and confidentiality of such information.

2 1

Re: [galaxy-dev] uploading gzip compressed datasets
by Ryan Golhar 28 Feb '11

28 Feb '11

That makes sense as BAM files are compressed. My files though are fastq. I don't think (or know) that BAM files can be used as input to mapping programs. On 2/28/11 4:01 AM, Christian Hundsrucker wrote: > Hi Ryan, > > thnx for the response. I figured out that this is/sould be solved for > .bam files > I just don't understand how... > > Best, > Christian > > see .../lib/galaxy/datatypes/binary.py > > [...] > def sniff( self, filename ): > # BAM is compressed in the BGZF format, and must not be > uncompressed in Galaxy. > # The first 4 bytes of any bam file is 'BAM\1', and the file is > binary. > try: > header = gzip.open( filename ).read(4) > if binascii.b2a_hex( header ) == binascii.hexlify( 'BAM\1' ): > return True > return False > except: > return False > [...] > > Ryan Golhar wrote: >> No, I never got a response to this...post it to the list and see what >> comes out of it... >> >> On 2/25/11 3:59 AM, Christian Hundsrucker wrote: >>> Hi Ryan, >>> >>> did you get an answer to your questions which didn't appear in the >>> mailing-list? >>> I have a similar Problem: >>> want to load a R-workspace within a galaxy module (.rdat-file, >>> R-Project) and therefore built the galaxy-.rdat datatype (binary). >>> .rdat-files are gzipped and are only recognized within R if they are >>> still zipped... >>> >>> Would be happy for any aswer! >>> If the answer is "no" I'll add this problem to the mailing list. >>> >>> Thnx, >>> Christian >>> >>> Ryan Golhar wrote: >>>> Hi all - I'm uploading datasets into my local instance by importing >>>> from the filesystem w/o copying into Galaxy. >>>> >>>> Importing of uncompressed files works properly. The files I'm >>>> importing are owned by user1 and a read-able by everyone, including >>>> the galaxy user. The galaxy user does not have write permissions in >>>> the directory where the files are stored. >>>> >>>> I have fastq files that are compressed with gzip and owned by user1. >>>> I don't want Galaxy to uncompress them, but it appears it is trying to >>>> during import, and subsequently fails because of permissions. >>>> >>>> Is there a way I can force galaxy to import the compressed files w/o >>>> uncompressing them? I think most of the mapping programs work on >>>> compressed fastq files so I'd rather not uncompressed them to save >>>> space. >>>> >>>> >>>> _______________________________________________ >>>> To manage your subscriptions to this and other Galaxy lists, please >>>> use the interface at: >>>> >>>> http://lists.bx.psu.edu/ >>> >>> > > -- CONFIDENTIALITY NOTICE: This email communication may contain private, confidential, or legally privileged information intended for the sole use of the designated and/or duly authorized recipient(s). If you are not the intended recipient or have received this email in error, please notify the sender immediately by email and permanently delete all copies of this email including all attachments without reading them. If you are the intended recipient, secure the contents in a manner that conforms to all applicable state and/or federal requirements related to privacy and confidentiality of such information.

1 0

How to retain files compressed
by Christian Hundsrucker 28 Feb '11

28 Feb '11

Hi all! I want to load a R-workspace within a galaxy module (.rdat-file, R-Project) and therefore built the galaxy-.rdat datatype (binary). .rdat-files are gzipped and are only recognized within R if they are still zipped. However, the corresponding .dat-file is an uncompressed version of the original .rdat file as I figured out using a hex-editor. I couldn't find any documentation how to change this behaviour, nor answers to similar Questions in this list. Would be happy for any answere that points me in the right direction. Details ##### datatypes_conf.xml: ----------------------------------- <?xml version="1.0"?> <datatypes> <registration converters_path="lib/galaxy/datatypes/converters" display_path="display_applications"> [...] <datatype extension="rdat" type="galaxy.datatypes.binary:Rdat" mimetype="application/octet-stream" display_in_upload="true"/> [...] </registration> <sniffers> [...] <sniffer type="galaxy.datatypes.binary:Rdat"/> [...] </sniffers> </datatypes> binary.py: ------------------ [...] class Rdat( Binary ): """Class describing an rdat binary file (R-workspace)""" file_ext = "rdat" #MetadataElement( name="Rdat", desc="R-workspace", param=metadata.FileParameter, readonly=True, no_value=None, visible=False, optional=True ) """ def __init__( self, **kwd ): Binary.__init__( self, **kwd ) self._name = "Rdat" """ def set_peek( self, dataset, is_multi_byte=False ): if not dataset.dataset.purged: dataset.peek = "Binary rdat file (R-workspace)" dataset.blurb = data.nice_size( dataset.get_size() ) else: dataset.peek = 'file does not exist' dataset.blurb = 'file purged from disk' def display_peek( self, dataset ): try: return dataset.peek except: return "Binary rdat file (%s)" % ( data.nice_size( dataset.get_size() ) ) def get_mime( self ): """Returns the mime type of the datatype""" return 'application/octet-stream' def sniff( self, filename ): # rdat is compressed in the gzip format, and must not be uncompressed in Galaxy. # The first 4 bytes of any rdat file are RDX2 try: header = gzip.open( filename ).read(4) #(4)=>4Bytes if binascii.b2a_hex( header ) == binascii.hexlify( 'RDX2' ): #check if there is the RDX2 signature return True return False except: return False try: header = open( filename ).read(4) #(4)=>4Bytes if binascii.b2a_hex( header ) == binascii.hexlify( 'RDX2' ): #check if there is the RDX2 signature return True return False except: return False -- Dr. Christian Hundsrucker Institute for Functional Genomics Computational Diagnostics Group University of Regensburg Josef Engertstr. 9 93053 Regensburg, Germany

2 1

unable to login user to local instance
by Ryan Golhar 25 Feb '11

25 Feb '11

Ok, so I think I have everything set up correctly. I'll test my cluster integration tomorrow. For now, I can see galaxy from my web browser. First issue - When I go to User -> Login and try to log myself in (I've already registered myself), it looks like it works, but the browser get redirected really fast to another page, then again to the main page. The User menu never shows me as logged in as it does on the public usegalaxy.org site. Is there a way to check that I'm actually logged in? If its not showing, then I assume something is wrong. Where do I look to debug this?

4 5

stagingin and stagingout at DRMAA
by hari krishna 25 Feb '11

25 Feb '11

Hi , Can any one help me how to transfer files from head node to remote location by using DRMAA job template attributes with example.... will DRMAA supports staging in and staging out of files,... -- Thanks & Regards, Hari Krishna .M

2 1

Amazon cloud formation for galaxy
by Ry4an Brase 25 Feb '11

25 Feb '11

Today's release of amazon cloud formation has to make the Galaxy Cloud stuff a bit easier: http://aws.typepad.com/aws/2011/02/cloudformation-create-your-aws-stack-fro… In theory with a description file like this: https://s3.amazonaws.com/cloudformation-templates/CloudFormationSample_Word… Once can define an entirely cluster to bring up from custom .amis with a single click. Exciting. -- Ry4an Brase 612-626-6575 Software Developer Application Development University of Minnesota Supercomputing Institute http://www.msi.umn.edu

2 1

Re: [galaxy-dev] Local Galaxy implimentation
by Anton Nekrutenko 25 Feb '11

25 Feb '11

Lee: I am CC'ing you e-mail to galaxy-dev list. You can get a lot of useful answers by posting to this list. Maintaining Galaxy is not different from maintaining other software resources. To run it locally you will need to have a system administration supervision as with any other high performance system. It is wonderful that you are sending your staff members to Galaxy Community Conference as they will be able to interact with key Galaxy Team members such as Nate Coraor, who runs Penn State instance here. Thanks! anton On Feb 25, 2011, at 6:14 AM, Lee Hazelwood wrote: > Dear Prof Nekrutenko, > > The university where I work is looking to locally implement your elegant software to enable us to view and analyse the rapidly growing amount of sequence data. We will be sending two staff members to the conference in Lunteren in May to get right into the detail on how we would achieve this. > > However, with all things I need to request some time/ support for running such a system in advance of May. > If you could help by briefly answering the questions below I would much appreciate it. > > * What is a realistic amount of someones time to implement Galaxy on a Linux server so that it can be used by researchers using the Web? > * What is the usual time overhead to keep it running and how skilled would they have to be? > > I should say, this will be for local biological researchers only, approx 20-30 dealing in Exome and Whole genome sequences. > > Best regards > Lee > > -- > Dr Lee Hazelwood > BHRC Bioinformatics Technology Group Leader > Biomedical and Health Research Centre > Faculty of Biological Sciences > Garstang Building, > University of Leeds, > Leeds, > LS2 9JT, UK. > E-mail: L.D.Hazelwood(a)leeds.ac.uk<mailto:L.D.Hazelwood@leeds.ac.uk> > > Anton Nekrutenko http://nekrut.bx.psu.edu http://usegalaxy.org

3 2

Only use filepaths and "Copy data into Galaxy?"
by Jean-Baptiste Denis 25 Feb '11

25 Feb '11

Hello everybody, i've got one galaxy server running. The ressource on the wiki are very useful for that purpose, thank you. I've got one question regarding data library when using "upload files from filesystem paths". There is a checkbox, unchecked by default, which prevent file upload into Galaxy. Is there a way to activate this checkbox by default, or to totally remove it on a per library basis ? I'm afraid of data libraries admin forgetting to check this box :D Regards, Jean-Baptiste

2 2