June 2012 - galaxy-dev - lists.galaxyproject.org

Giardine et al Galaxy paper (2005)
by Peter van Heusden 12 Jun '12

12 Jun '12

Hi everyone The Galaxy system as described in Belinda Giardine et al's 2005 Genome Research paper ("Galaxy: A platform for interactive large-scale genome analysis") appears to be radically different from the current Galaxy, at least in its technical specification. The paper mentions a C core spoken to by a Perl-based web user interface (referred to as the History User Interface). The co-authors on the paper are, however, familiar names from the current Galaxy team. Was this paper describing something in the pre-history of current Galaxy (it sounds rather like the current History but without workflows and of course implemented on a different platform)? Peter

3 2

Composite output with self-declarated datatype
by Marine Rohmer 12 Jun '12

12 Jun '12

Hi everyone, I'm trying to add a tool which generates 2 files, that I will call ".xxx" (a text file) and ".yyy" (a binary file) . Both files are needed to use the result of my tool with an other tool I've added. So I wanted to create a composite datatype , that I will call ".composite", whose components are ".xxx" and ".yyy". I've declared the datatype ".xxx", ".yyy" and ".composite" in the datatypes_conf.xml file, and written the required python files . Now, ".xxx", ".yyy" and ".composite" appear in Get Data's "file format" . These are my files : In datatype_conf.xml : <datatype extension="xxx" type="galaxy.datatypes.xxx:xxx" mimetype="text/html" display_in_upload = "True" subclass="True"/> <datatype extension="yyy" type="galaxy.datatypes.yyy:yyy" mimetype="application/octet-stream" display_in_upload = "True" subclass="True" /> <datatype extension="composite" type="galaxy.datatypes.composite:Composite" mimetype="text/html" display_in_upload="True"/> xxx.py (summarized) : import logging from metadata import MetadataElement from data import Text log = logging.getLogger(__name__) class xxx(Text): file_ext = "xxx" def __init__( self, **kwd ): Text.__init__( self, **kwd ) yyy.py (summarized) : import logging from metadata import MetadataElement from data import Text log = logging.getLogger(__name__) # yyy is a binary file, don't know what to put instead of "Text". "Binary" and "Bin" don't work. class yyy(Text): file_ext = "yyy" def __init__( self, **kwd ): Text.__init__( self, **kwd ) composite.py (summarized) : import logging from metadata import MetadataElement from data import Text log = logging.getLogger(__name__) class Composite(Text): composite_type = 'auto_primary_file' MetadataElement( name="base_name", desc="base name for all transformed versions of this index dataset", default="your_index", readonly=True, set_in_upload=True) file_ext = 'composite' def __init__( self, **kwd ): Text.__init__( self, **kwd ) self.add_composite_file( '%s.xxx', description = "XXX file", substitute_name_with_metadata = 'base_name') self.add_composite_file( '%s.yyy', description = "YYY file", substitute_name_with_metadata = 'base_name', is_binary = True ) Atfer having read Composite Datatypes in the wiki, my myTool.xml looks like : <tool id="my tool"> <command> path/to/crac-index-wrapper.sh ${os.path.join( $output_name_yyy.extra_files_path, '%s.yyy')} ${os.path.join( $output_name_xxx.extra_files_path, '%s.xxx' )} $input_file </command> <inputs> <param name="output_name" type="text" value ="IndexOutput" label="Output name"/> <param name="input_file" type="data" label="Source file" format="fasta"/> </inputs> <outputs> <data format="ssa" name="output_name_ssa" from_work_dir="crac-index_output.ssa" label="CRAC-index: ${output_name}.ssa"> </data> <data format="conf" name="output_name_conf" from_work_dir="crac-index_output.conf" label="CRAC-index: ${output_name}.conf"> </data> </outputs> </tool> I have 2 main problems : When I upload a xxx file via "Get Data", there's no problem. However, when I upload a yyy file (the binary one),history bloc rests eternally blue ("uploading dataset") , even for a small file. The second problem is that I want my tool to only generate the .composite file on the history, and not each of .xxx and .yyy. . But when I run my tool I still have 2 outputs displayed in the history : one for xxx and one for yyy. Furthermore, neither of them work, and I have the following message : path/to/myTool-wrapper.sh: 6: path/to/myTool-wrapper.sh.sh: cannot create /home/myName/work/galaxy-dist/database/files/000/dataset_302_files/%s.yyy.xxx: Directory nonexistent path/to/myTool-wrapper.sh: 6: path/to/myTool-wrapper.sh: cannot create /home/myName/work/galaxy-dist/database/files/000/dataset_302_files/%s.yyy.yyy: Directory nonexistent path/to/myTool-wrapper.sh: 11: path/to/myTool-wrapper.sh: Syntax error: redirection unexpected So I've checked manually in /home/myName/work/galaxy-dist/database/files/000/ and there's only "dataset_302.dat", an empty file. (And whatsmore, I don't understand why I get in the message "%s.yyy.xxx" and "%s.yyy.yyy" instead of "%s.yyy" and "%s.xxx" ...) Then I've looked the example of rgenetics.xml, and tried to change the command line and the output : <tool id="my tool"> <command> path/to/myTool-wrapper.sh '$output_name.extra_files_path/$output_name.metadata.base_name' $input_file </command> <inputs> <param name="output_name" type="text" value ="IndexOutput" label="Output name" /> <param name="input_file" type="data" label="Source file" /> </inputs> <outputs> <data format="html" name="output" label="myTool: ${output_name}.html" metadata_source="input_file"/> </outputs> </tool> This gave me : Traceback (most recent call last): File "/home/myName/work/galaxy-dist/lib/galaxy/jobs/runners/local.py", line 59, in run_job job_wrapper.prepare() File "/home/myName/work/galaxy-dist/lib/galaxy/jobs/__init__.py", line 429, in prepare self.command_line = self.tool.build_command_line( param_dict ) File "/home/myName/work/galaxy-dist/lib/galaxy/tools/__init__.py", line 1971, in build_command_line command_line = fill_template( self.command, context=param_dict ) File "/home/myName/work/galaxy-dist/lib/galaxy/util/template.py", line 9, in fill_template return str( Template( source=template_text, searchList=[context] ) ) File "/home/myName/work/galaxy-dist/eggs/Cheetah-2.2.2-py2.7-linux-x86_64-ucs4.egg/Cheetah/Template.py", line 1004, in __str__ return getattr(self, mainMethName)() File "cheetah_DynamicallyCompiledCheetahTemplate_1339157051_58_87978.py", line 83, in respond NotFound: cannot find 'extra_files_path' while searching for 'output_name.extra_files_path' So now I don't know which way is the one to follow : the first one inspired by the example in the wiki, or the second one inspired by rgenetics.xml. And what's wrong with it... I will really appreciate any suggestion ! Best regards, Marine

1 1

Re: [galaxy-dev] disk space for local galaxy
by Hans-Rudolf Hotz 12 Jun '12

12 Jun '12

- Please keep all replies on the list by using "reply all" - check your "universe_wsgi.ini" file: # Allow users to remove their datasets from disk immediately (otherwise, # datasets will be removed after a time period specified by an administrator in # the cleanup scripts run via cron) #allow_user_dataset_purge = False Regards, Hans On 06/11/2012 05:08 PM, Xu, Jianpeng wrote: > Hi, > > I read them and tried to work according to the instruction below on the > http://wiki.g2.bx.psu.edu/Learn/Managing%20Datasets#Delete_vs_Delete_Perman… > > *Active* and *Deleted* histories can be *permanently deleted *using from > the History pane "Options -> Saved Histories", then click on "Advanced > Search", then click on "status: all". Check the box for the histories to > be discarded and then click on the button "Permanently delete". I do not > find the "Permanently delete" button. I found a button "Delete and > remove datasets from disk", but I click on it and it did not work. > > Do you know how to fix it ? > > Actually I want to delete some histories permanently and get some disk > space. > > Thanks a lot. > > Jianpeng > > > > Hi > > Have you looked at the wiki pages? > > http://wiki.g2.bx.psu.edu/Learn/Managing%20Datasets#Delete_vs_Delete_Perman… > > and > > http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Purge%20Histories%20and%… > > > > Regards, Hans > > On 06/04/2012 05:29 PM, Xu, Jianpeng wrote: >> Hi, >> >> I have installed the local galaxy. I deleted a few histories in order to >> get some disk space. However, the disk space did not change after the >> deletion of the histories. >> >> Do you why ? How can I get some disk space by deleting histories ? >> >> Thanks, >> >> >> >> ------------------------------------------------------------------------ >> >> This e-mail message (including any attachments) is for the sole use of >> the intended recipient(s) and may contain confidential and privileged >> information. If the reader of this message is not the intended >> recipient, you are hereby notified that any dissemination, distribution >> or copying of this message (including any attachments) is strictly >> prohibited. >> >> If you have received this message in error, please contact >> the sender by reply e-mail message and destroy all copies of the >> original message (including attachments). >> >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> >> http://lists.bx.psu.edu/

1 0

Re: [galaxy-dev] Tool Shed Workflow
by Greg Von Kuster 11 Jun '12

11 Jun '12

Hi John, I feel this is an important topic and that others in the community are undoubtedly benefitting from it, so I'm glad you started this discussion. On Jun 9, 2012, at 12:36 AM, John Chilton wrote: > We don't pull down from bitbucket directly to our production > environment, we pull galaxy-dist changes into our testing repository, > merge (that can be quite complicated, sometimes a multihour process), > auto-deploy to a testing server, and then finally we push the tested > changes into a bare production repo. Our sys admins then pull in > changes from that bare production repo in our production environment. > We also prebuild eggs in our testing environment not live on our > production system. Given the complicated merges we need to do and the > configuration files that need to be updated each dist update it would > seem making those changes on a live production system would be > problematic. > > Even if one was pulling changes directly from bitbucket into a > production codebase, I think the dependency on bitbucket would be very > different than on N toolsheds. These tool migrations will only interact with a single tool shed, the main Galaxy tool shed. > If our sys admin is going to update > Galaxy and bitbucket is down, that is no problem he or she can just > bring Galaxy back up and update later. Now lets imagine they shutdown > our galaxy instance, updated the code base, did a database migration, > and went to do a toolshed migration and that failed. In this case > instead of just bringing Galaxy back up they will now need to restore > the database from backup and pullout of the mercurial changes. In your scenario, if everything went well except the tool shed migration, an option that would be less intrusive than reverting back to the previous Galaxy release would be to just bring up your server without the migrated tools for a temporary time. When the tool shed migration process is corrected (generally, the only reason it would break is if the tool shed was down), you could run it at that time. So the worst case scenario is that the specific migrated tools will be temporarily unavailable from your production Galaxy instance. A nice feature of these migrated tool scripts is that they are very flexible in when they can be run, which is any time. They also do not have to be run in any specific order. So, for example, you could run tool migration script "0002" six months after you've run migration script "0003", "0004", etc. These scripts do affect the Galaxy database by adding new records to certain tables, but if the script fails, no database corrections are necessary in order to prepare for running the script again. You can just run the same script later, and the script will handle whatever database state exists at that time. > > Anyway all of that is a digression right, I understand that we will > need to have the deploy-time dependencies on tool sheds and make these > tool migration script calls part of our workflow. My lingering hope is > for a way of programmatically importing and updating new tools that > were never part of Galaxy (Qiime, upload_local_file, etc...) using > tool sheds. This is currently possible (and very simple to do) using the Galaxy Admin UI. See the following sections of the tool shed wiki for details. http://wiki.g2.bx.psu.edu/Tool%20Shed#Automatic_installation_of_Galaxy_tool… http://wiki.g2.bx.psu.edu/Tool%20Shed#Automatic_installation_of_Galaxy_tool… http://wiki.g2.bx.psu.edu/Tool%20Shed#Getting_updates_for_tool_shed_reposit… I'm currently writing the following new section - it should be available within the next week or so. http://wiki.g2.bx.psu.edu/Tool%20Shed#Automatic_3rd_party_tool_dependency_i… > My previous e-mail was proposing or positing a mechanism > for doing that, but I think you read it like I was trying to describe > a way to script the migrations of the existing official Galaxy tools > (I definitely get that you have done that). > > Thanks again for your time and detailed responses, > -John >

2 1

Task Manager: This Galaxy instance is not the job manager.
by Dave Lin 11 Jun '12

11 Jun '12

Dear Galaxy Team, I've been getting the following error for some time when I try to access the Manage Jobs Task. "This Galaxy instance is not the job manager. If using multiple servers, please directly access the job manager instance to manage jobs." For debugging purposes, I'm only running the single master instance. I'm using CloudMan/Amazon EC2. I traced the code and suspect it might have something to do with an Amazon IP Address/hostname discrepancy, but am not sure how to go about fixing this. On a related note, if I can't access this page, what is the best way to clear/cancel jobs via the command line? Thanks in advance, Dave Thanks, Dave

5 5

interactive debug of a tool
by Cittaro Davide 11 Jun '12

11 Jun '12

Hi all, I remember there was be a way to run a debug shell on tools that fail to run in galaxy. Unfortunately I have a bad memory and google is not helping: how can I run it? d /* Davide Cittaro, PhD Coordinator of Bioinformatics Core Center for Translational Genomics and Bioinformatics San Raffaele Scientific Institute Via Olgettina 58 20132 Milano Italy Office: +39 02 26439140 Mail: cittaro.davide(a)hsr.it Skype: daweonline */ -------------------------------------------------------------------------- LA TUA CURA E' SCRITTA NEL TUO DNA. AL SAN RAFFAELE LA STIAMO REALIZZANDO. AIUTA LA RICERCA, DAI IL TUO 5XMILLE - CF: 03 06 80 153 info:www.5xmille@hsr.it - www.5xmille.org Disclaimer added by CodeTwo Exchange Rules 2007 http://www.codetwo.com

1 0

About FTP Upload
by Ciara Ledero 11 Jun '12

11 Jun '12

Hi all, I checked basics.py after reading a similar post and saw this: self.ftp_upload_dir = kwargs.get( 'ftp_upload_dir', None ) self.ftp_upload_site = kwargs.get( 'ftp_upload_site', None ) Can you guys tell me what the None part means? Cheers, CL

2 1

Galaxy cluster jobs - automatic priority setting
by Peter Cock 11 Jun '12

11 Jun '12

Dear all, Does Galaxy have any mechanisms to set the priority of jobs submitted to the cluster? Specifically I am interested in SGE via DRMAA, but this is a general issue. If there is some existing code, I might be able to use it for the following situation: The specific motivation is for balancing competing BLAST searches from multiple users: I want to be able to prioritize short jobs (e.g. under 100 queries) over large jobs (tens of thousands of queries). In the (experimental) task splitting code, I would like to be able to give short jobs higher priority (e.g. normal) than large jobs (e.g. low priority) based on the size of the split file. One idea would be a scaling bases on the (split) input file's size (in bytes), or perhaps a per-file format size threshold, e.g. when splitting FASTA files, 1000 sequences might trigger lower priority. The advantage of this kind of assessment is it would also work on both the current split mechanisms, "number_of_parts" and "to_size", as well as any hybrid of the two: <parallelism method="multi" split_mode="number_of_parts" split_size="8" ..." /> <parallelism method="multi" split_mode="to_size" split_size="1000" ... /> http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-May/009647.html So, is there any existing code in Galaxy that would be helpful here - or existing plans in this area? Thanks, Peter

2 2

Adding mm10 to available genomes
by Larry Helseth 11 Jun '12

11 Jun '12

I'm trying to add mm10 to a Galaxy instance built using Mercurial on May 31st (using Postgres). I've added the .fa file, prepared bwa, bowtie, samtool indexes, etc., edited all of the .loc files for individual tools and restarted the server daemon. mm10 appears in the list of genomes available for these tools but doesn't appear in the list of all genomes when I try uploading a file (Get Data/Upload File from your computer) OR when I try adding the file through the admin library interface (Administration/Data/Manage libraries/Add datasets). I tried editing ~/galaxy-dist/scripts/loc_files/create_all_fasta_loc.py to add a line for "mm10". restarting Galaxy but that genome still doesn't appear in the list. Do I need to edit an entry in Postgres? When I restart Galaxy without the daemon mode (or tail the paster.log) I see that clicking on Get Data/Upload File is calling GET /tool_runner?tool_id=upload1 for the <my URL>/root/tool_menu. Am I supposed to upload new genomes through the Library? screencast.g2.bx.psu.edu is still down so I can't watch the video tutorials about configuring custom genomes. I'd appreciate any suggestions. Thanks in advance! Adios, Larry Larry Helseth, Ph.D. Center for Molecular Medicine NorthShore University HealthSystem Evanston, IL

2 2

Dynamic job runner configuration followup
by John Chilton 11 Jun '12

11 Jun '12

Following up on some recent threads that have referenced my dynamic job runner configuration work, Nate and I have discussed these issues and I have created a new pull request based on those discussions and I am confident these changes will be accepted soon. Things are basically as I outlined them in my previous description: http://www.mail-archive.com/galaxy-dev@lists.bx.psu.edu/msg03010.html except for where to place the rules has changed. Now instead of placing them in lib/galaxy/jobs/rules.py you will need to create a file (or multiple files for them) in your lib/galaxy/jobs/rules the directory. My previous e-mail was a technical description of how it worked, I think maybe that is why it didn't generate the excitement I had hoped :). I think instead describing some concrete use cases might be better. So here are six cool things you can do with dynamic job runners. 1) Change maximum walltime based on job parameters or file sizes. 2) Implement wild card like configuration of job runners instead of configuring one tool at a time. 3) Create queues with different priorities, and then give higher priorities to people giving demos (or directors or testers etc...). 4) Utilize environment variables to determine job runner configurations. 5) Limit a particular tool's use to only white-listed users. 6) Tie into Galaxy's job history tables to throttle those problem users clogging up your Galaxy instance. To do any of these you will need pull in the changes from bitbucket, add dynamic to the start_job_runners configuration option in universe_wsgi.ini, and create a file such as lib/galaxy/jobs/rules/200_runners.py for your rules. Below I describe how to do these, though I haven't actually tested the code snippets so they should be considered just an outline of the idea, your mileage may vary. 1) Change maximum walltime based on job parameters or file sizes. Lets say you want to change the max walltime of the BlastN based on the size of the input query. First you would add the line ncbi_blastn_wrapper=dynamic:///python to universe_wsgi.ini. Next in 200_runners.py you would add a function such as following: import os def ncbi_blastn_wrapper(job): inp_data = dict( [ ( da.name, da.dataset ) for da in job.input_datasets ] ) inp_data.update( [ ( da.name, da.dataset ) for da in job.input_library_datasets ] ) query_file = inp_data[ "query" ].file_name query_size = os.path.getsize( query_file ) if query_size > 1024 * 1024: return 'pbs:////-l walltime=24:00:00/' else: return 'pbs:////-l walltime=12:00:00/' 2) Implement wild card like configuration of job runners instead of configuring one tool at a time. Lets say you have a coworker called J. Johnson ummm wait no Jim J. and he maintains a tool suite for a fictitious metagenomics application called fathur. Assume also that this fathur suite has dozens of tools clogging up your configuration file because they need to all use pbs:////-l procs=8/ instead of the default pbs://///. To configure all the fathur tools at once, in the [app:main] of universe_wsgi.ini would would change default_cluster_job_runner from pbs:///// to dynamic:///python/default_runner and then add the following function to 200_runners.py. def default_runner(tool_id): if tool_id.startswith('fathur_'): return 'pbs:////-l procs=8/' else: return 'pbs://///' 3) Create queues with different priorities, and then give higher priorities to people giving demos. Lets say the users defined by the admin_users configuration property in universe_wsgi.ini are the ones that give demos and do testing and so you want to increase their priority for all jobs, and lets say to do this you have created queues gx_normal and gx_important in your queue manager with differing priorities. You could then take the default_runner concept from the previous example and do something like this: def default_runner(app, user_email): admin_users = app.config.get( "admin_users", "" ).split( "," ) if user_email in admin_users: return 'pbs:///gx_important//' else: return 'pbs:///gx_normal//' You could define the list of users right in this file instead of pulling it in from admin_users and then apply this concept to give higher priority to director or testers or paying users. Alternatively, you could give lower priority to external users, people you just don't like, etc.... 4) Utilize environment variables to determine job runner configurations. Lets say you want cufflinks to always use as many cores as are available, but in your testing environment you only have 4 cores available whereas in production you have 16. Lets also say you have the environment variable MAX_CORES set and this will be different on each machine. You would then update universe_wsgi.ini to have cufflinks use the dynamic job config (cufflinks=dynamic:///python) and then add the following to 200_runners.py import os def cufflinks(): return 'pbs:////-l procs=%s/' % os.environ['MAX_CORES'] (Warning you would need to update the cufflinks wrapper to respect this environment variable also). 5) Limit a particular tool's use to only white-listed users. Say you are developing top_secret_tool and you only want chilton(a)msi.umn.edu to be able to use it, you could add the line top_secret_tool=dynamic:///python/only_john to universe_wsgi.ini and then the following function to 200_runners.py. def only_john(user_email): if user_email == "chilton(a)msi.umn.edu": return 'pbs://///' else: return 'noway://///' This is admittedly quite the hack but it works. 6) Tie into Galaxy's job history tables to throttle those problem users clogging up your Galaxy instance. Okay, I haven't actually worked through how to do this one, but I am sure it is possible. -John ------------------------------------------------ John Chilton Senior Software Developer University of Minnesota Supercomputing Institute Office: 612-625-0917 Cell: 612-226-9223

2 1