August 2010 - galaxy-dev - lists.galaxyproject.org

Dynamically create Workflows
by Michael Siebauer 05 Aug '10

05 Aug '10

Hey, I wonder if it is possible to let a self-written script, dynamically create workflows. The idea is to create a pipeline, by providing the user a kind of analysis webpage where he can select a certain set of tools and parameters and the output of this interface should be a workflow. And another question. If if have a workflow consisting of a set of tools, will galaxy create a single cluster job consisting of all tools, or one job for each tool and the run them sequential? This would be important if one script pipes directly to another. Thx, Michael :-)

2 1

GMOD Europe 2010 Registration Now Open
by Dave Clements, GMOD Help Desk 04 Aug '10

04 Aug '10

GMOD Europe 2010 ================ 13-16 September 2010 Cambridge, UK http://gmod.org/wiki/GMOD_Europe_2010 We are pleased to announce GMOD Europe 2010, four days of GMOD events being held 13-16 September 2010, at the University of Cambridge. GMOD Europe 2010 includes: 1) GMOD Community Meeting, Monday & Tuesday: Project updates, developer and user presentations and best practices, project direction. 2) GMOD Satellite Meetings, Wednesday: Special interest groups where GMOD community members meet to discuss specific topics of interest. 3) InterMine Workshop, Wednesday: A one day workshop on installing, configuring and using the InterMine biological data warehouse system. 4) BioMart Workshop, Thursday: A one day workshop on using the BioMart biological data warehouse system, including accessing data through APIs. Registration is now open for these events. There is a £50 registration fee for the GMOD Meeting to cover catered lunches, coffee/tee breaks, and other expenses. Registration for all other events is free, but still required, as space is limited. These events are open to all: GMOD users, developers, prospective users, biologists, and computer scientists. For an idea of what goes on at GMOD meetings, see http://gmod.org/wiki/January_2010_GMOD_Meeting. See you in Cambridge! Scott Cain and Dave Clements PS: If you have topics you want to have covered (or you want to cover) at the GMOD Meeting, or at the Satellite Meetings, please add them to the wiki pages, or contact us. We are also asking for guest speaker nominations for the meeting. -- http://gmod.org/wiki/GMOD_News <http://gmod.org/wiki/GMOD_News>http://gmod.org/wiki/GMOD_Evo_Hackathon http://gmod.org/wiki/GMOD_Europe_2010 http://gmod.org/wiki/Help_Desk_Feedback

1 0

adding another UCSC mirror to display
by Davide Cittaro 04 Aug '10

04 Aug '10

Hi again, this may be a FAQ, sorry for that... How can I add an alternate UCSC mirror site (our local mirror) to galaxy so that everything that could be displayed at UCSC-main will have the UCSC-localmirror option? Besides... how can I remove the bx main mirror? I've tried to add a ${GALAXYROOT}/tool-data/shared/campus/campus_build_sites.txt but that's not enough... I've modified the ucsc/bam.xml file adding another dynamic_link section but that's not enough... any hint? Thanks again d /* Davide Cittaro Cogentech - Consortium for Genomic Technologies via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro(a)ifom-ieo-campus.it */

2 1

Re: [galaxy-dev] sort error: expansion of \t character in system call?
by Jacob Biesinger 03 Aug '10

03 Aug '10

Well would you look a that. $ ls -l /bin/sh lrwxrwxrwx 1 root root 4 2010-05-28 12:07 /bin/sh -> dash I assumed having bash set in /etc/passwd was enough-- I guess that's only for login shells? This also explains why I've always had trouble starting galaxy --> `sh run.sh` would not have environment variables, etc so I always did `bash run.sh` After: $ sudo rm /bin/sh $ sudo ln -s /bin/bash /bin/sh Things work wonderfully. Thanks for that. Perhaps there should be a disclaimer on the wiki somewhere? I imagine Ubuntu is fairly large portion of galaxy's target audience, and it sure can cause funny errors... -- Jake Biesinger Graduate Student Xie Lab, UC Irvine (949) 231-7587 On Tue, Aug 3, 2010 at 3:30 PM, Assaf Gordon <gordon(a)cshl.edu> wrote: > I don't think so. > It actually shows that your "sh" is not "bash". > > run: > $ ls -l /bin/sh > > and see if you get: > /bin/sh -> bash > or > /bin/sh -> dash > > > Jacob Biesinger wrote, On 08/03/2010 06:27 PM: >> No, my default shell (listed in /etc/passwd) is definitely /bin/bash >> and I can run the sort command just fine from the command line: >> $ sort -f -t $'\t' -k1,1 -k2,2n -o >> /home/wbiesing/galaxy_dist/database/files/000/dataset_190.dat >> /home/wbiesing/galaxy_dist/database/files/000/dataset_171.dat >> >> Though running just sh, the command fails: >> $ sh >> $ sort -f -t $'\t' -k1,1 -k2,2n -o >> /home/wbiesing/galaxy_dist/database/files/000/dataset_190.dat >> /home/wbiesing/galaxy_dist/database/files/000/dataset_171.dat >> sort: multi-character tab `$\\t' >> >> Seems like python's os.system(...) is the weak link. >> -- >> Jake Biesinger >> Graduate Student >> Xie Lab, UC Irvine >> (949) 231-7587 >> >> >> >> On Tue, Aug 3, 2010 at 3:16 PM, Assaf Gordon <gordon(a)cshl.edu> wrote: >>> One thing to try: >>> >>> see if your shell is 'dash' (which is the default is recent Ubuntus) and switch it to 'bash' . >>> Make sure to change not only your currently running shell, but also the "/bin/sh" link . >>> >>> "dash" does not respect the $'\t' trick . >>> >>> -gordon >>> >>> >>> Jacob Biesinger wrote, On 08/03/2010 06:01 PM: >>>> Hi! On my local galaxy_dist, doing a simple sort on a bed file is >>>> failing. I think the culprit is a tab expansion of the system call on >>>> line 42 of tools/filters/sorter.py. >>>> >>>> The Galaxy command that is being run is: >>>> python /home/wbiesing/galaxy_dist/tools/filters/sorter.py >>>> --input=/home/wbiesing/galaxy_dist/database/files/000/dataset_171.dat >>>> --out_file1=/home/wbiesing/galaxy_dist/database/files/000/dataset_190.dat >>>> --column=1 --style=alpha --order=DESC 2 >>>> num DESC >>>> >>>> and the stderr output from galaxy is: >>>> sort: multi-character tab `$\t' >>>> >>>> I can repro the system call within python: >>>> In [1]: import os >>>> In [3]: cmd = "sort -f -t $'\t' -k1,1 -k2,2n -o >>>> /home/wbiesing/galaxy_dist/database/files/000/dataset_190.dat >>>> /home/wbiesing/galaxy_dist/database/files/000/dataset_171.dat" >>>> In [5]: os.system(cmd) >>>> sort: multi-character tab `$\t' >>>> Out[5]: 512 >>>> >>>> Running python2.6.5 with recent co of galaxy_dist. Ubuntu 10.04 x86_64. >>>> >>>> -- >>>> Jake Biesinger >>>> Graduate Student >>>> Xie Lab, UC Irvine >>>> (949) 231-7587 >>>> >>>> _______________________________________________ >>>> galaxy-dev mailing list >>>> galaxy-dev(a)lists.bx.psu.edu >>>> http://lists.bx.psu.edu/listinfo/galaxy-dev >>> >>> > >

1 1

sort error: expansion of \t character in system call?
by Jacob Biesinger 03 Aug '10

03 Aug '10

Hi! On my local galaxy_dist, doing a simple sort on a bed file is failing. I think the culprit is a tab expansion of the system call on line 42 of tools/filters/sorter.py. The Galaxy command that is being run is: python /home/wbiesing/galaxy_dist/tools/filters/sorter.py --input=/home/wbiesing/galaxy_dist/database/files/000/dataset_171.dat --out_file1=/home/wbiesing/galaxy_dist/database/files/000/dataset_190.dat --column=1 --style=alpha --order=DESC 2 num DESC and the stderr output from galaxy is: sort: multi-character tab `$\t' I can repro the system call within python: In [1]: import os In [3]: cmd = "sort -f -t $'\t' -k1,1 -k2,2n -o /home/wbiesing/galaxy_dist/database/files/000/dataset_190.dat /home/wbiesing/galaxy_dist/database/files/000/dataset_171.dat" In [5]: os.system(cmd) sort: multi-character tab `$\t' Out[5]: 512 Running python2.6.5 with recent co of galaxy_dist. Ubuntu 10.04 x86_64. -- Jake Biesinger Graduate Student Xie Lab, UC Irvine (949) 231-7587

2 3

running DRMAA (sge) jobs as different users
by Assaf Gordon 03 Aug '10

03 Aug '10

Hi all, Attached is a patch that allows galaxy to qsub jobs as different users. It's just a proof-of-concept, but it works. The changes to Galaxy are minimal: instead of calling "drmaa.session.runJob(JobTemplate)", it saves the jobTemplate parameters to a file, the run (with sudo) a small scripts that changes the real user-id and then calls 'qsub'. from there on, galaxy picks-up the JobID, and continues as usual. Note that the logic to actually select which user to use is not implemented, the method get_qsub_user() in drmaa.py always returns the current user, but one can change it to run as a different user (or implement a more sophisticated mechanism, as I plan to). To try this out: 1. import the patch 2. edit your 'universe_wsgi.ini' and add "drmaa_external_runjob_script" key (see "universe_wsgi.ini.sample") 3. edit /etc/sudoers to allow the script (drmaa_external_runner.py) to run as root without asking for password, by adding a line similar to this: gordon ALL = (root) NOPASSWD: SETENV: /data/hannon/gordon/galaxy/galaxy_dev_external_qsub/scripts/drmaa_external_runner.py Also available here: http://cancan.cshl.edu/labmembers/gordon/files/galaxy_drmaa_setuid_2010_08_… comments are welcomed, -gordon P.S. This was tested only on SGE 6.2 and CentOS linux. Other machines may have a slighly different 'sudo' (I use "sudo -E -S"), and other DRMAAs might not work.

1 0

Variables always match the empty string?
by Jacob Biesinger 03 Aug '10

03 Aug '10

I have a script that allows arbitrary filtering on a resource within a more complicated step. in the command section, I have: #if $filter_fxn1 != "" --filter_fxn1="$filter_fxn1" #end if and <param name='filter_fxn1' type='text' label='Filter expression for resource1' /> in the inputs section. The trouble is that when there is no text in the input text box, I'm still getting --filter_fxn1="" in my parameter string. galaxy.jobs.runners.local DEBUG 2010-08-02 16:06:33,027 execution finished: python /home/wbiesing/galaxy_dist/tools/chipSeq/python/enrichment_pairwise.py --quiet --filter_fxn1="" PS-- for simple questions like this on developing tools, should I be deferring to the galaxy-users list? Thanks! -- Jake Biesinger Graduate Student Xie Lab, UC Irvine (949) 231-7587

3 2

Conditional variables not available on the command line?
by Jacob Biesinger 02 Aug '10

02 Aug '10

Hi! I have a tool with three possible types of input. A flag is set to either 'bed', 'resource' or 'file' for the two inputs. I've noticed that the conditional variables are not available on the command line, so is there some other way of setting these flags? So far, the UI handles the 3 possible input types nicely, but I'd like to set the format1 and format2 flags directly from the conditional. It seems that the conditional parameters aren't seen by cheetah. I also tried using 'hidden' parameters, but those seem to be for totally different uses. # USAGE: # python enrichment_pairwise.py myfile.bed geneExons --format1=bed --format2=resource --genome_resource=hg18 --output_file=overlap.txt Trying to run this in galaxy gives: Traceback (most recent call last): File "/home/wbiesing/galaxy_dist/lib/galaxy/jobs/runners/local.py", line 55, in run_job job_wrapper.prepare() File "/home/wbiesing/galaxy_dist/lib/galaxy/jobs/__init__.py", line 355, in prepare self.command_line = self.tool.build_command_line( param_dict ) File "/home/wbiesing/galaxy_dist/lib/galaxy/tools/__init__.py", line 1397, in build_command_line command_line = fill_template( self.command, context=param_dict ) File "/home/wbiesing/galaxy_dist/lib/galaxy/util/template.py", line 9, in fill_template return str( Template( source=template_text, searchList=[context] ) ) File "/home/wbiesing/galaxy_dist/eggs/Cheetah-2.2.2-py2.6-linux-x86_64-ucs4.egg/Cheetah/Template.py", line 1004, in __str__ return getattr(self, mainMethName)() File "cheetah_DynamicallyCompiledCheetahTemplate_1280646041_98_13115.py", line 89, in respond NotFound: cannot find 'res1_select My tool's XML file looks like: <command interpreter="python">enrichment_pairwise.py $resource1 $resource2 --format1=$res1_select --format2=$res2_select --genome_resource=$dbkey --output_file=$outputFile</command> <inputs> <conditional name="resource1_test"> <param name="res1_select" type="select" label="Input type for resource1"> <option value="bed">BED File</option> <option value="resource">Worldbase Resource</option> <option value="file">File with list of Worldbase Resources</option> </param> <when value="bed"> <param name="resource1" type="data" format="bed" label="Bed file for resource1" /> </when> <when value="resource"> <param name="resource1" type="text" label="Worldbase ID for resource1" /> </when> <when value="file"> <param name="resource1" type="data" format="text" label="Text file with Worldbase IDs to iterate over for resource1" /> </when> </conditional> <conditional name="resource2_test"> <param name="res2_select" type="select" label="Input type for resource2"> <option value="bed">BED File</option> <option value="resource">Worldbase Resource</option> <option value="file">File with list of Worldbase Resources</option> </param> <when value="bed"> <param name="resource2" type="data" format="bed" label="Bed file for resource2" /> </when> <when value="resource"> <param name="resource2" type="text" label="Worldbase ID for resource2" /> </when> <when value="file"> <param name="resource2" type="data" format="text" label="Text file with Worldbase IDs to iterate over for resource2" /> </when> </conditional> </inputs> Thanks for your help! -- Jake Biesinger Graduate Student Xie Lab, UC Irvine (949) 231-7587

3 3

Re: [galaxy-dev] is it possible to upload files to different local directories based on the galaxy upload.py script?
by John Brunelle 02 Aug '10

02 Aug '10

Hello, I would love to be able to have all datasets, job output, etc. go to a different directory for each user. Some of our cluster users have access to much more storage than others, and it would be nice if they could take advantage of that storage through galaxy, too. I was thinking of inserting an extra level of directory structure below galaxy's database directory so that, for example, instead of just database/files there would be database/user1/files, database/user2/files, and so on (and likewise for job_working_directory, pbs, etc.). That way we could make database/userX a symbolic link to a different filesystem with higher quotas when appropriate. I was looking in the code for where the external_filename attribute you mention is handled, and I see a couple key places where this can be hacked in -- JobExternalOutputMetadataWrapper.setup_external_metadata (in each place where I see it called the username could be passed along) and in JobWrapper.__init__, where it sets the job_working_directory (it has the job, so it has the username). However, it seems that's too late -- I see galaxy tring to create things in the normal database locations before any of these are called and the output paths adjusted. It would be nice if this could be coded into the Dataset class itself, but I don't know how to get the appropriate username there... Any suggestions for how to hack something together to make this work? Thanks a lot, John On Thu, 24 Jun 2010, Nate Coraor wrote: > amenda lee wrote: > > Hi; > > > > Is it poosible to upload files to the different local directories by > > Galaxy upload.py script? It seems to upload all files to the same > > directory based on the file path in universe_wsgi.ini file. > > > > if yes, what part of upload.py needs to be edited? also is it > > possible to keep name of uploaded file unchanged? > > Hi Amenda, > > Galaxy manages all of its data internally. If it was not done like > this, I am not sure how you would define which directory each upload > should go into. > > The exception to this is that when using Galaxy Data Libraries, you can > choose to leave files in their original locations when uploading them on > the server side (via the "Upload a directory of files" and "Upload from > filesystem paths" options). > > That said, if you wanted to write some sort of custom method for placing > files based on some rules at upload, it could be done. Dataset objects > have an 'external_filename' attribute which can be set to any filesystem > path. This value is stored in the database and will be automatically > used by anything which needs access to the disk file. > > --nate > > > > > Thanks in advance. > > > > Amenda Lee > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > galaxy-dev mailing list > > galaxy-dev at lists.bx.psu.edu > > http://lists.bx.psu.edu/listinfo/galaxy-dev

2 1

Re: [galaxy-dev] upload files from filesystems path
by SHAUN WEBB 02 Aug '10

02 Aug '10

I've only just got around to looking at this problem and thanks to Gordons suggestion I have found a solution: The error is thrown in library_common.py, the upload proceeds through this line at 1134: if response_code == 200: Then it is redirected to the get_path_paste_uploaded_datasets method at 1142: uploaded_datasets, response_code, message = self.get_path_paste_uploaded_datasets( trans, cntrller, params, library_bunch, response_code, message ) If everything goes ok here then a response code 'None' is returned at 1264: return uploaded_datasets, None, None Which means the upload now fails at 1147 and goes in to error handling: if response_code != 200: I have replaced line 1264 with: return uploaded_datasets, 200, None This seems to have solved the problem. Hopefully it is this simple, I'd appreciate if some one from the Galaxy team could check if this is a bug and let me know. Thanks Shaun Webb Quoting Assaf Gordon <gordon(a)cshl.edu>: > Hello Shaun, > > I'm not from the galaxy team, but I've looked at a similar exception > for Juan Perin: > http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-July/002948.html > > It looks like you get the exact same exception as he does, > indicating that the upload has failed but galaxy throws and > exception while trying to show you the error. > > If you feel comfortable with tweaking the python code you can try to > further pin-point the problem. > > -gordon > > SHAUN WEBB wrote, On 07/15/2010 04:39 PM: >> >> Hi, >> >> after updating to the latest galaxy changeset I am getting an error when >> trying to upload files from system path, the traceback is below. >> >> The new files then appear in the library as queued jobs. >> >> Thanks in advance for your help >> >> Shaun Webb >> >> >> URL: http://bifx3.bio.ed.ac.uk:8080/library_common/upload_library_dataset >> File >> '/opt/galaxy_dist/eggs/Paste-1.6-py2.5.egg/paste/exceptions/errormiddleware.py', >> line 143 in __call__ >> app_iter = self.application(environ, start_response) >> File '/opt/galaxy_dist/eggs/Paste-1.6-py2.5.egg/paste/debug/prints.py', >> line 98 in __call__ >> environ, self.app) >> File '/opt/galaxy_dist/eggs/Paste-1.6-py2.5.egg/paste/wsgilib.py', line >> 539 in intercept_output >> app_iter = application(environ, replacement_start_response) >> File '/opt/galaxy_dist/eggs/Paste-1.6-py2.5.egg/paste/recursive.py', >> line 80 in __call__ >> return self.application(environ, start_response) >> File >> '/opt/galaxy_dist/eggs/Paste-1.6-py2.5.egg/paste/httpexceptions.py', >> line 632 in __call__ >> return self.application(environ, start_response) >> File '/opt/galaxy_dist/lib/galaxy/web/framework/base.py', line 145 in >> __call__ >> body = method( trans, **kwargs ) >> File '/opt/galaxy_dist/lib/galaxy/web/controllers/library_common.py', >> line 978 in upload_library_dataset >> **kwd ) >> File '/opt/galaxy_dist/lib/galaxy/web/controllers/library_common.py', >> line 1158 in upload_dataset >> message=util.sanitize_text( message ), >> File '/opt/galaxy_dist/lib/galaxy/util/__init__.py', line 138 in >> sanitize_text >> for c in text: >> TypeError: 'NoneType' object is not iterable >> >> >> > > > -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

2 2