- galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] sort error: expansion of \t character in system call?
by Jacob Biesinger 03 Aug '10

03 Aug '10

Well would you look a that. $ ls -l /bin/sh lrwxrwxrwx 1 root root 4 2010-05-28 12:07 /bin/sh -> dash I assumed having bash set in /etc/passwd was enough-- I guess that's only for login shells? This also explains why I've always had trouble starting galaxy --> `sh run.sh` would not have environment variables, etc so I always did `bash run.sh` After: $ sudo rm /bin/sh $ sudo ln -s /bin/bash /bin/sh Things work wonderfully. Thanks for that. Perhaps there should be a disclaimer on the wiki somewhere? I imagine Ubuntu is fairly large portion of galaxy's target audience, and it sure can cause funny errors... -- Jake Biesinger Graduate Student Xie Lab, UC Irvine (949) 231-7587 On Tue, Aug 3, 2010 at 3:30 PM, Assaf Gordon <gordon(a)cshl.edu> wrote: > I don't think so. > It actually shows that your "sh" is not "bash". > > run: > $ ls -l /bin/sh > > and see if you get: > /bin/sh -> bash > or > /bin/sh -> dash > > > Jacob Biesinger wrote, On 08/03/2010 06:27 PM: >> No, my default shell (listed in /etc/passwd) is definitely /bin/bash >> and I can run the sort command just fine from the command line: >> $ sort -f -t $'\t' -k1,1 -k2,2n -o >> /home/wbiesing/galaxy_dist/database/files/000/dataset_190.dat >> /home/wbiesing/galaxy_dist/database/files/000/dataset_171.dat >> >> Though running just sh, the command fails: >> $ sh >> $ sort -f -t $'\t' -k1,1 -k2,2n -o >> /home/wbiesing/galaxy_dist/database/files/000/dataset_190.dat >> /home/wbiesing/galaxy_dist/database/files/000/dataset_171.dat >> sort: multi-character tab `$\\t' >> >> Seems like python's os.system(...) is the weak link. >> -- >> Jake Biesinger >> Graduate Student >> Xie Lab, UC Irvine >> (949) 231-7587 >> >> >> >> On Tue, Aug 3, 2010 at 3:16 PM, Assaf Gordon <gordon(a)cshl.edu> wrote: >>> One thing to try: >>> >>> see if your shell is 'dash' (which is the default is recent Ubuntus) and switch it to 'bash' . >>> Make sure to change not only your currently running shell, but also the "/bin/sh" link . >>> >>> "dash" does not respect the $'\t' trick . >>> >>> -gordon >>> >>> >>> Jacob Biesinger wrote, On 08/03/2010 06:01 PM: >>>> Hi! On my local galaxy_dist, doing a simple sort on a bed file is >>>> failing. I think the culprit is a tab expansion of the system call on >>>> line 42 of tools/filters/sorter.py. >>>> >>>> The Galaxy command that is being run is: >>>> python /home/wbiesing/galaxy_dist/tools/filters/sorter.py >>>> --input=/home/wbiesing/galaxy_dist/database/files/000/dataset_171.dat >>>> --out_file1=/home/wbiesing/galaxy_dist/database/files/000/dataset_190.dat >>>> --column=1 --style=alpha --order=DESC 2 >>>> num DESC >>>> >>>> and the stderr output from galaxy is: >>>> sort: multi-character tab `$\t' >>>> >>>> I can repro the system call within python: >>>> In [1]: import os >>>> In [3]: cmd = "sort -f -t $'\t' -k1,1 -k2,2n -o >>>> /home/wbiesing/galaxy_dist/database/files/000/dataset_190.dat >>>> /home/wbiesing/galaxy_dist/database/files/000/dataset_171.dat" >>>> In [5]: os.system(cmd) >>>> sort: multi-character tab `$\t' >>>> Out[5]: 512 >>>> >>>> Running python2.6.5 with recent co of galaxy_dist. Ubuntu 10.04 x86_64. >>>> >>>> -- >>>> Jake Biesinger >>>> Graduate Student >>>> Xie Lab, UC Irvine >>>> (949) 231-7587 >>>> >>>> _______________________________________________ >>>> galaxy-dev mailing list >>>> galaxy-dev(a)lists.bx.psu.edu >>>> http://lists.bx.psu.edu/listinfo/galaxy-dev >>> >>> > >

1 1

sort error: expansion of \t character in system call?
by Jacob Biesinger 03 Aug '10

03 Aug '10

Hi! On my local galaxy_dist, doing a simple sort on a bed file is failing. I think the culprit is a tab expansion of the system call on line 42 of tools/filters/sorter.py. The Galaxy command that is being run is: python /home/wbiesing/galaxy_dist/tools/filters/sorter.py --input=/home/wbiesing/galaxy_dist/database/files/000/dataset_171.dat --out_file1=/home/wbiesing/galaxy_dist/database/files/000/dataset_190.dat --column=1 --style=alpha --order=DESC 2 num DESC and the stderr output from galaxy is: sort: multi-character tab `$\t' I can repro the system call within python: In [1]: import os In [3]: cmd = "sort -f -t $'\t' -k1,1 -k2,2n -o /home/wbiesing/galaxy_dist/database/files/000/dataset_190.dat /home/wbiesing/galaxy_dist/database/files/000/dataset_171.dat" In [5]: os.system(cmd) sort: multi-character tab `$\t' Out[5]: 512 Running python2.6.5 with recent co of galaxy_dist. Ubuntu 10.04 x86_64. -- Jake Biesinger Graduate Student Xie Lab, UC Irvine (949) 231-7587

2 3

running DRMAA (sge) jobs as different users
by Assaf Gordon 03 Aug '10

03 Aug '10

Hi all, Attached is a patch that allows galaxy to qsub jobs as different users. It's just a proof-of-concept, but it works. The changes to Galaxy are minimal: instead of calling "drmaa.session.runJob(JobTemplate)", it saves the jobTemplate parameters to a file, the run (with sudo) a small scripts that changes the real user-id and then calls 'qsub'. from there on, galaxy picks-up the JobID, and continues as usual. Note that the logic to actually select which user to use is not implemented, the method get_qsub_user() in drmaa.py always returns the current user, but one can change it to run as a different user (or implement a more sophisticated mechanism, as I plan to). To try this out: 1. import the patch 2. edit your 'universe_wsgi.ini' and add "drmaa_external_runjob_script" key (see "universe_wsgi.ini.sample") 3. edit /etc/sudoers to allow the script (drmaa_external_runner.py) to run as root without asking for password, by adding a line similar to this: gordon ALL = (root) NOPASSWD: SETENV: /data/hannon/gordon/galaxy/galaxy_dev_external_qsub/scripts/drmaa_external_runner.py Also available here: http://cancan.cshl.edu/labmembers/gordon/files/galaxy_drmaa_setuid_2010_08_… comments are welcomed, -gordon P.S. This was tested only on SGE 6.2 and CentOS linux. Other machines may have a slighly different 'sudo' (I use "sudo -E -S"), and other DRMAAs might not work.

1 0

Variables always match the empty string?
by Jacob Biesinger 03 Aug '10

03 Aug '10

I have a script that allows arbitrary filtering on a resource within a more complicated step. in the command section, I have: #if $filter_fxn1 != "" --filter_fxn1="$filter_fxn1" #end if and <param name='filter_fxn1' type='text' label='Filter expression for resource1' /> in the inputs section. The trouble is that when there is no text in the input text box, I'm still getting --filter_fxn1="" in my parameter string. galaxy.jobs.runners.local DEBUG 2010-08-02 16:06:33,027 execution finished: python /home/wbiesing/galaxy_dist/tools/chipSeq/python/enrichment_pairwise.py --quiet --filter_fxn1="" PS-- for simple questions like this on developing tools, should I be deferring to the galaxy-users list? Thanks! -- Jake Biesinger Graduate Student Xie Lab, UC Irvine (949) 231-7587

3 2

Conditional variables not available on the command line?
by Jacob Biesinger 02 Aug '10

02 Aug '10

Hi! I have a tool with three possible types of input. A flag is set to either 'bed', 'resource' or 'file' for the two inputs. I've noticed that the conditional variables are not available on the command line, so is there some other way of setting these flags? So far, the UI handles the 3 possible input types nicely, but I'd like to set the format1 and format2 flags directly from the conditional. It seems that the conditional parameters aren't seen by cheetah. I also tried using 'hidden' parameters, but those seem to be for totally different uses. # USAGE: # python enrichment_pairwise.py myfile.bed geneExons --format1=bed --format2=resource --genome_resource=hg18 --output_file=overlap.txt Trying to run this in galaxy gives: Traceback (most recent call last): File "/home/wbiesing/galaxy_dist/lib/galaxy/jobs/runners/local.py", line 55, in run_job job_wrapper.prepare() File "/home/wbiesing/galaxy_dist/lib/galaxy/jobs/__init__.py", line 355, in prepare self.command_line = self.tool.build_command_line( param_dict ) File "/home/wbiesing/galaxy_dist/lib/galaxy/tools/__init__.py", line 1397, in build_command_line command_line = fill_template( self.command, context=param_dict ) File "/home/wbiesing/galaxy_dist/lib/galaxy/util/template.py", line 9, in fill_template return str( Template( source=template_text, searchList=[context] ) ) File "/home/wbiesing/galaxy_dist/eggs/Cheetah-2.2.2-py2.6-linux-x86_64-ucs4.egg/Cheetah/Template.py", line 1004, in __str__ return getattr(self, mainMethName)() File "cheetah_DynamicallyCompiledCheetahTemplate_1280646041_98_13115.py", line 89, in respond NotFound: cannot find 'res1_select My tool's XML file looks like: <command interpreter="python">enrichment_pairwise.py $resource1 $resource2 --format1=$res1_select --format2=$res2_select --genome_resource=$dbkey --output_file=$outputFile</command> <inputs> <conditional name="resource1_test"> <param name="res1_select" type="select" label="Input type for resource1"> <option value="bed">BED File</option> <option value="resource">Worldbase Resource</option> <option value="file">File with list of Worldbase Resources</option> </param> <when value="bed"> <param name="resource1" type="data" format="bed" label="Bed file for resource1" /> </when> <when value="resource"> <param name="resource1" type="text" label="Worldbase ID for resource1" /> </when> <when value="file"> <param name="resource1" type="data" format="text" label="Text file with Worldbase IDs to iterate over for resource1" /> </when> </conditional> <conditional name="resource2_test"> <param name="res2_select" type="select" label="Input type for resource2"> <option value="bed">BED File</option> <option value="resource">Worldbase Resource</option> <option value="file">File with list of Worldbase Resources</option> </param> <when value="bed"> <param name="resource2" type="data" format="bed" label="Bed file for resource2" /> </when> <when value="resource"> <param name="resource2" type="text" label="Worldbase ID for resource2" /> </when> <when value="file"> <param name="resource2" type="data" format="text" label="Text file with Worldbase IDs to iterate over for resource2" /> </when> </conditional> </inputs> Thanks for your help! -- Jake Biesinger Graduate Student Xie Lab, UC Irvine (949) 231-7587

3 3

Re: [galaxy-dev] is it possible to upload files to different local directories based on the galaxy upload.py script?
by John Brunelle 02 Aug '10

02 Aug '10

Hello, I would love to be able to have all datasets, job output, etc. go to a different directory for each user. Some of our cluster users have access to much more storage than others, and it would be nice if they could take advantage of that storage through galaxy, too. I was thinking of inserting an extra level of directory structure below galaxy's database directory so that, for example, instead of just database/files there would be database/user1/files, database/user2/files, and so on (and likewise for job_working_directory, pbs, etc.). That way we could make database/userX a symbolic link to a different filesystem with higher quotas when appropriate. I was looking in the code for where the external_filename attribute you mention is handled, and I see a couple key places where this can be hacked in -- JobExternalOutputMetadataWrapper.setup_external_metadata (in each place where I see it called the username could be passed along) and in JobWrapper.__init__, where it sets the job_working_directory (it has the job, so it has the username). However, it seems that's too late -- I see galaxy tring to create things in the normal database locations before any of these are called and the output paths adjusted. It would be nice if this could be coded into the Dataset class itself, but I don't know how to get the appropriate username there... Any suggestions for how to hack something together to make this work? Thanks a lot, John On Thu, 24 Jun 2010, Nate Coraor wrote: > amenda lee wrote: > > Hi; > > > > Is it poosible to upload files to the different local directories by > > Galaxy upload.py script? It seems to upload all files to the same > > directory based on the file path in universe_wsgi.ini file. > > > > if yes, what part of upload.py needs to be edited? also is it > > possible to keep name of uploaded file unchanged? > > Hi Amenda, > > Galaxy manages all of its data internally. If it was not done like > this, I am not sure how you would define which directory each upload > should go into. > > The exception to this is that when using Galaxy Data Libraries, you can > choose to leave files in their original locations when uploading them on > the server side (via the "Upload a directory of files" and "Upload from > filesystem paths" options). > > That said, if you wanted to write some sort of custom method for placing > files based on some rules at upload, it could be done. Dataset objects > have an 'external_filename' attribute which can be set to any filesystem > path. This value is stored in the database and will be automatically > used by anything which needs access to the disk file. > > --nate > > > > > Thanks in advance. > > > > Amenda Lee > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > galaxy-dev mailing list > > galaxy-dev at lists.bx.psu.edu > > http://lists.bx.psu.edu/listinfo/galaxy-dev

2 1

Re: [galaxy-dev] upload files from filesystems path
by SHAUN WEBB 02 Aug '10

02 Aug '10

I've only just got around to looking at this problem and thanks to Gordons suggestion I have found a solution: The error is thrown in library_common.py, the upload proceeds through this line at 1134: if response_code == 200: Then it is redirected to the get_path_paste_uploaded_datasets method at 1142: uploaded_datasets, response_code, message = self.get_path_paste_uploaded_datasets( trans, cntrller, params, library_bunch, response_code, message ) If everything goes ok here then a response code 'None' is returned at 1264: return uploaded_datasets, None, None Which means the upload now fails at 1147 and goes in to error handling: if response_code != 200: I have replaced line 1264 with: return uploaded_datasets, 200, None This seems to have solved the problem. Hopefully it is this simple, I'd appreciate if some one from the Galaxy team could check if this is a bug and let me know. Thanks Shaun Webb Quoting Assaf Gordon <gordon(a)cshl.edu>: > Hello Shaun, > > I'm not from the galaxy team, but I've looked at a similar exception > for Juan Perin: > http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-July/002948.html > > It looks like you get the exact same exception as he does, > indicating that the upload has failed but galaxy throws and > exception while trying to show you the error. > > If you feel comfortable with tweaking the python code you can try to > further pin-point the problem. > > -gordon > > SHAUN WEBB wrote, On 07/15/2010 04:39 PM: >> >> Hi, >> >> after updating to the latest galaxy changeset I am getting an error when >> trying to upload files from system path, the traceback is below. >> >> The new files then appear in the library as queued jobs. >> >> Thanks in advance for your help >> >> Shaun Webb >> >> >> URL: http://bifx3.bio.ed.ac.uk:8080/library_common/upload_library_dataset >> File >> '/opt/galaxy_dist/eggs/Paste-1.6-py2.5.egg/paste/exceptions/errormiddleware.py', >> line 143 in __call__ >> app_iter = self.application(environ, start_response) >> File '/opt/galaxy_dist/eggs/Paste-1.6-py2.5.egg/paste/debug/prints.py', >> line 98 in __call__ >> environ, self.app) >> File '/opt/galaxy_dist/eggs/Paste-1.6-py2.5.egg/paste/wsgilib.py', line >> 539 in intercept_output >> app_iter = application(environ, replacement_start_response) >> File '/opt/galaxy_dist/eggs/Paste-1.6-py2.5.egg/paste/recursive.py', >> line 80 in __call__ >> return self.application(environ, start_response) >> File >> '/opt/galaxy_dist/eggs/Paste-1.6-py2.5.egg/paste/httpexceptions.py', >> line 632 in __call__ >> return self.application(environ, start_response) >> File '/opt/galaxy_dist/lib/galaxy/web/framework/base.py', line 145 in >> __call__ >> body = method( trans, **kwargs ) >> File '/opt/galaxy_dist/lib/galaxy/web/controllers/library_common.py', >> line 978 in upload_library_dataset >> **kwd ) >> File '/opt/galaxy_dist/lib/galaxy/web/controllers/library_common.py', >> line 1158 in upload_dataset >> message=util.sanitize_text( message ), >> File '/opt/galaxy_dist/lib/galaxy/util/__init__.py', line 138 in >> sanitize_text >> for c in text: >> TypeError: 'NoneType' object is not iterable >> >> >> > > > -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

2 2

Upload issue: 'Upload a directory of files' and compressed files
by padioleau 02 Aug '10

02 Aug '10

Dear Galaxy developer, Using the data-library upload, 'Upload a directory of files', I encounter an issue. When uploading compressed files, without copying them in galaxy folder, they are uncompressed in my folder but the file name extension is not changed. This mean that I have files with 'gz' extension that are in fact uncompressed. Is there a possibility to avoid this issue? I'm using tools which can use compressed files, so for me the best would be to just add the compressed file in my data-library. I believe galaxy can accept compressed files, but I didn't found any explanation in your wiki (index: compress, gzip or zip) to configure galaxy this way. Thanks, Ismael -- Padioleau Ismael EMBL Heidelberg ( V205)

2 1

opening an item (or an history) in IGV
by Davide Cittaro 01 Aug '10

01 Aug '10

Hi all, I'm thinking about using a stand-alone genome browser (such as IGV) coupled with galaxy... do you think it would be possible (in a way similar to the "display in ucsc genome browser")? Is there anybody working on this? d /* Davide Cittaro Cogentech - Consortium for Genomic Technologies via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro(a)ifom-ieo-campus.it */

1 0

Re: [galaxy-dev] Fwd: Re: Help about zip upload (esd files)
by Adhemar 30 Jul '10

30 Jul '10

Dear Nate, We're trying to implement different tools to handle sequences generated by Megabace/AB1 sequencers. The very first step in this process is to load the chromatograms provided by the machines. The thing is these machines produce an individual file per sequence and the phred software (base caller) receive as parameters a compressed file or a directory path. Since galaxy 'get data' feature loads only individual files and we're not able to implement phred to do such things, what options are left? Thank you so much, Adhemar On 07/29/2010 01:57 PM, Eric Aguiar wrote: > > > -------- Original Message -------- > Subject: Re: [galaxy-dev] Help about zip upload (esd files) > Date: Wed, 28 Jul 2010 12:09:06 -0400 > From: > To: Eric Aguiar <eric(a)cebio.org> > CC: List > > > > Eric Aguiar wrote: > > Dear, > > > > I followed all steps in the galaxy tutorial, but I didn't have success. > > I'm trying to create a datatype for megabase chromatograms (.esd) very > > similar to the Ab1 ones. > > > > Here is my configurations. > > > > > > *1 - Creating datatypes in datatypes_conf.xml* > > > > <datatype extension="zip" type="galaxy.datatypes.binary:Esd" > > mimetype="application/zip" display_in_upload="true"/> > > <datatype extension="esd" type="galaxy.datatypes.binary:Esd" > > mimetype="application/octet-stream" display_in_upload="true"/> > > > > *2 - Defining types in lib/galaxy/datatypes/binary.py* > > > > class Esd( Binary ): > > """Class describing an ab1 binary sequence file""" > > file_ext = "esd" > > > > def set_peek( self, dataset, is_multi_byte=False ): > > if not dataset.dataset.purged: > > dataset.peek = "Binary chromatograms sequence file" > > dataset.blurb = data.nice_size( dataset.get_size() ) > > else: > > dataset.peek = 'file does not exist' > > dataset.blurb = 'file purged from disk' > > def display_peek( self, dataset ): > > try: > > return dataset.peek > > except: > > return "Binary esd sequence file (%s)" % ( data.nice_size( > > dataset.get_size() ) ) > > > > > > class Zip( Binary ): > > """Class describing a zip archive of binary sequence files""" > > file_ext = "zip" > > > > def set_peek( self, dataset, is_multi_byte=False ): > > if not dataset.dataset.purged: > > zip_file = zipfile.ZipFile( dataset.file_name, "r" ) > > num_files = len( zip_file.namelist() ) > > dataset.peek = "Archive of %s binary sequence files" % ( > > str( num_files ) ) > > dataset.blurb = data.nice_size( dataset.get_size() ) > > else: > > dataset.peek = 'file does not exist' > > dataset.blurb = 'file purged from disk' > > def display_peek( self, dataset ): > > try: > > return dataset.peek > > except: > > return "Binary sequence file archive (%s)" % ( > > data.nice_size( dataset.get_size() ) ) > > def get_mime( self ): > > """Returns the mime type of the datatype""" > > return 'application/zip' > > > > > > When I'm going to send the file in zip format (.esd files compressed),it > > shows me the following error: > > *"An error occurred running this job: Invalid 'File Format' for archive > > consisting of binary files - use 'Binseq.zip'"* > > Hi Eric, > > Have you updated to a recent revision of Galaxy? Binseq.zip and the > code that prevents uploading zip files has been removed. Currently, > though, a zip file can only contain one member. > > --nate > > > > > I tried somethings, but I don't have success. > > > > Thank you, > > -- > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > galaxy-dev mailing list > > galaxy-dev(a)lists.bx.psu.edu > > http://lists.bx.psu.edu/listinfo/galaxy-dev > -- Adhemar Zerlotini Neto, PhD FIOCRUZ - Fundação Oswaldo Cruz CPqRR - Centro de Pesquisas René Rachou http://www.cpqrr.fiocruz.br http://www.cebio.org +55 31 3337 3863 +55 31 3337 3649

2 1