- galaxy-dev - lists.galaxyproject.org

Support for subdirs in dataset extra_files_path
by Jim Johnson 03 Jun '11

03 Jun '11

Request is issue#494 https://bitbucket.org/galaxy/galaxy-central/issue/494/support-sub-dirs-in-e… I'm finding that some qiime metagenomics applications build HTML results with an inherent directory structure. For some other applications, e.g. FastQC, I've been able to flatten the hierarchy and edit the html, but that appears problematic for qiime. Galaxy hasn't supported a dataset extra_files_path hierarchy, though the developers don't seem opposed to the idea: http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-October/003605.html I added a route in lib/galaxy/web/buildapp.py and modified the dataset download code to traverse a hierarchy in lib/galaxy/web/controllers/dataset.py I don't think these add any security vulnerabilities, (I tried the obvious ../../ ). $ hg diff lib/galaxy/web/buildapp.py diff -r 6ae06d89fec7 lib/galaxy/web/buildapp.py --- a/lib/galaxy/web/buildapp.py Wed Mar 16 09:01:57 2011 -0400 +++ b/lib/galaxy/web/buildapp.py Wed Mar 16 10:24:13 2011 -0500 @@ -94,6 +94,8 @@ webapp.add_route( '/async/:tool_id/:data_id/:data_secret', controller='async', action='index', tool_id=None, data_id=None, data_secret=None ) webapp.add_route( '/:controller/:action', action='index' ) webapp.add_route( '/:action', controller='root', action='index' ) + # allow for subdirectories in extra_files_path + webapp.add_route( '/datasets/:dataset_id/display/{filename:.+?}', controller='dataset', action='display', dataset_id=None, filename=None) webapp.add_route( '/datasets/:dataset_id/:action/:filename', controller='dataset', action='index', dataset_id=None, filename=None) webapp.add_route( '/display_application/:dataset_id/:app_name/:link_name/:user_id/:app_action/:action_param', controller='dataset', action='display_application', dataset_id=None, user_id=None, app_name = None, link_name = None, app_action = None, action_param = None ) webapp.add_route( '/u/:username/d/:slug', controller='dataset', action='display_by_username_and_slug' ) $ $ hg diff lib/galaxy/web/controllers/dataset.py diff -r 6ae06d89fec7 lib/galaxy/web/controllers/dataset.py --- a/lib/galaxy/web/controllers/dataset.py Wed Mar 16 09:01:57 2011 -0400 +++ b/lib/galaxy/web/controllers/dataset.py Wed Mar 16 10:24:29 2011 -0500 @@ -266,17 +266,18 @@ log.exception( "Unable to add composite parent %s to temporary library download archive" % data.file_name) msg = "Unable to create archive for download, please report this error" messagetype = 'error' - flist = glob.glob(os.path.join(efp,'*.*')) # glob returns full paths - for fpath in flist: - efp,fname = os.path.split(fpath) - try: - archive.add( fpath,fname ) - except IOError: - error = True - log.exception( "Unable to add %s to temporary library download archive" % fname) - msg = "Unable to create archive for download, please report this error" - messagetype = 'error' - continue + for root, dirs, files in os.walk(efp): + for fname in files: + fpath = os.path.join(root,fname) + rpath = os.path.relpath(fpath,efp) + try: + archive.add( fpath,rpath ) + except IOError: + error = True + log.exception( "Unable to add %s to temporary library download archive" % rpath) + msg = "Unable to create archive for download, please report this error" + messagetype = 'error' + continue if not error: if params.do_action == 'zip': archive.close()

2 1

Sharing tool definitions between XML files
by Peter 03 Jun '11

03 Jun '11

Hi all, I'm wondering if there is any established way to share parameter definitions between tool wrapper XML files? For example, I am currently working on NCBI BLAST+ wrappers, and these tools have a lot in common. For example, the output format option will be a select parameter, and it will be the same for blastn, blastp, tblastn, etc. I can just cut and paste the definition but this seems inelegant and will be a long term maintenance burden if it ever needs updating (lots of places would need to be updated the same way). I'd like to have a shared XML snippet defining this parameter which I could then reference/include from each tool wrapper than needs it. I'm thinking of something like XML's <!ENTITY ...> might work. Is this possible? Peter

2 2

recovery: galaxy restart
by Harendra chawla 02 Jun '11

02 Jun '11

Hi everyone, I am trying to modify the *recover* function from the drmaa.py (/galaxy_central/lib/galaxy/job/runners/drmaa.py) as per my requirements. But I am not ale to understand the flow of that function. The recover function is called when the galaxy server is restarted. It first looks for the running jobs from the database. Then my problem is how it regains the same old state of the galaxy (specially the GUI) which was before the galaxy got restarted. Can anyone explain me the flow of the recover function and how the old state is regained. Regards Harendra

2 7

sqlite to postgres
by Harendra chawla 02 Jun '11

02 Jun '11

Hi, I want to convert my database from sqlite to postgres. I am able to do it as far as schema is concerned, but I want to preserve the data as well. When I changed the database to postgres, all my history got lost. Is there any way I can preserve the data in sqlite database while converting to postgres. Regards Harendra

2 1

listing universe_wsgi.ini configuration
by Shantanu Pavgi 02 Jun '11

02 Jun '11

Hi, I have started galaxy daemon using 'reload' option. I am wondering if there is a command to print currently used configuration values in the universe_wsgi.ini file. It may help verify that configuration changes were reloaded. -- Thanks, Shantanu.

2 2

separate galaxy tools directory
by Shantanu Pavgi 02 Jun '11

02 Jun '11

I am wondering if galaxy tools directory structure can be changed so that tool configurations resides outside galaxy code base. Right now the galaxy code comes with some default/pre-installed tools in the $GALAXY_DIST/tools directory. Any additional tool configurations are also defined in the same directory. I think if we use a separate directory for additional tools then core galaxy code base can remain clean. This may help in updating the galaxy code as well. Any thoughts? -- Thanks, Shantanu.

2 1

user and dataset management with LDAP: some questions
by Louise-Amélie Schmitt 02 Jun '11

02 Jun '11

Hello everyone I have a couple of questions regarding user and dataset management. 1) We use LDAP for user registration and logging, would it be possible to retrieve automatically the LDAP users' groups and create the groups in Galaxy accordigly? (and of course put the user in their respective groups) 2) Is it possible, still using LDAP, to delete a user and all his datasets? 3) Is it possible to automatically suppress, for instance, any dataset that was added more than a couple of months ago? 4) Is there a not-so-intrusive way to add a column in the user list showing the disk space they use respectively? 5) I tried to see how the API works but i have to admit I didn't get a thing. I read the scripts/api/README file and there I saw one needs to access the user preferences to generate an API key. What is its purpose? Is there a way to do it when you use LDAP (therefore no access to that key generator) ? Sorry, this is a bit random but I'm kinda drowning here, since I'm not used to manipulating apps this huge. Thanks for your help and patience. Cheers, L-A

2 2

galaxy deployment workflow or best-practices
by Shantanu Pavgi 02 Jun '11

02 Jun '11

I looked at the Galaxy deployment presentation from the Galaxy Community Conference ( http://wiki.g2.bx.psu.edu/GCC2011?action=AttachFile&do=view&target=GalaxyDe… ) and it was really helpful. I am wondering if you could share any details or best practices for maintaing the galaxy install base. For example, do you use any build-deploy tools or workflow for updating the galaxy install?? Any comments or suggestions on this will be really helpful. -- Thanks, Shantanu.

1 0

downloading bam and bai files from sam-to-bam wrapper
by Ryan Golhar 02 Jun '11

02 Jun '11

In my local instance of Galaxy, I'm trying to download the BAM and BAI file from the tool SAM-to-BAM. I see the links for downloading the Dataset and bam_index. BTW - I'm running Safari on Mac OS X. When I download the Dataset, I get the BAM file. When I download the bam_index, it gets renamed to Galaxy13-[SAM-to-BAM_on...].bam_index.html The error is the extension, .bam_index.html. This should be .bam.bai. The BAM file gets saved correctly with .bam. I check the sam-to-bam XML wrapper but can't determine where this filename comes from or how to fix it. Has anyone else run into this? -- CONFIDENTIALITY NOTICE: This email communication may contain private, confidential, or legally privileged information intended for the sole use of the designated and/or duly authorized recipient(s). If you are not the intended recipient or have received this email in error, please notify the sender immediately by email and permanently delete all copies of this email including all attachments without reading them. If you are the intended recipient, secure the contents in a manner that conforms to all applicable state and/or federal requirements related to privacy and confidentiality of such information.

2 1

Multiple galaxy instances
by Jean-Baptiste Denis 02 Jun '11

02 Jun '11

Hello everybody, i'm in the process to provide Galaxy for multiple team. I've already setup a testing instance using the production setup page on the wiki (apache + sge) and it works quite well if i'm refearing to the users feedback. This setup is typically use for NGS dealing with data on a NFS share without uploading them to the instance. Why do i need multiple instance ? Maybe i'm not using galaxy correctly. Correct me if i'm wrong. My goal is to delagate the management of library/datasets to a galaxy admin of each team from the beginning : i do NOT want a SINGLE independant super admin to manage the access for multiple team, it doesn't scale. The galaxy instance and the underlying "galaxy" system user must access to the NGS data on the NFS (v3) share. This means that the galaxy user must be in a group that has access to the data. I can delegate the process of managing datasets and library to a dedicated galaxy admin. This setup is working quite well with the single instance setup. My job as a sysadmin is reduced to galaxy setup and maintenance : i'm not involved in the library/dataset management. The problem with this setup does not work if there is another team with data they don't want to share with others (don't blame me on that) : the galaxy system user must access the data of the first team AND the second team, this means that the galaxy admin of each team could access everything. One solution to this problem would be to have an independant galaxy super admin with access to everything which manage data access to each team. I don't like this solution, like i said, it doesn't scale. So, another way to deal with that is to give each team its own galaxy instance (each running with a specific system galaxy user) with a dedicated galaxy admin. Two possibility : - N galaxy tree, each with a different tuned universe_wsgi.ini init file (dedicated path, port, database, etc...). The problem here is on the sysadmin side : the update process effort must be repeated N times. - A unique galaxy tree, and N tuned (dedicated path, port, database, etc...) universe_wsgi.ini files. This seems the best to me but i need to know if galaxy internals can managed that kind of setup ? What do you think ? Any inputs, remarks or advices are welcome ! Regards, Jean-Baptiste

3 2