October 2013 - galaxy-dev - lists.galaxyproject.org

How we found the filenames of all of the Galaxy datasets
by Yaron de Leeuw 30 Oct '13

30 Oct '13

Hi. During IT housekeeping, we have decided that we must know the filenames of all of the galaxy datasets in our local deployment. Searching the docs and the web yielded no reasonable answer, so we figured we'll just write our own tool. At first, we thought about a Galaxy tool, but found no way to extract all the datasets of all histories. So we went deeper. To the SQL itself. Without further ado, I give you the query: SELECT substring(to_char(history_dataset_association.dataset_id, '000000') from 2 for 3) || '/dataset_' || history_dataset_association.dataset_id || '.dat' , history_dataset_association.create_time, history_dataset_association.name, history_dataset_association.extension, history.name AS history_name,galaxy_user.username FROM history_dataset_association JOIN history ON history_dataset_association.history_id = history.id JOIN galaxy_user ON history.user_id = galaxy_user.id WHERE history_dataset_association.deleted = false AND history_dataset_association.purged = false ORDER BY username, history_name, create_time; We use PostgreSQL, so to run this you save the query as `get_filenames.sql`, and run: psql -U galaxy galaxy_prod -f get_filenames.sql -A > galaxy_export_filenames.csv This prints out a line structured as: 003/dataset_3748.dat|2012-10-09 12:07:37.786751| index.fastq|fastq|nan|sean Which means: folder/filename | time of creation | name as appears in history | format | history name | history owner Only files that were not deleted in galaxy are printed (you can remove the "WHERE" line if you want them all). By the way, one of the main struggles was finding the folder/filename. We have searched the database repeatedly for a filename field to no avail, until someone recognised the fact that they can be derived from the dataset_id. If you zero pad the id to six chars, the first three chars would be your folder name. The whole id is the file name. e.g.: dataset_id = 12345 folder = 012 filename = dataset_12345.dat Yaron. Search engine candies: export migrate files datasets filenames on disk

2 1

Tabular data not displaying in main panel
by Peter Cock 30 Oct '13

30 Oct '13

Hi all, Multiple users of our internal Galaxy server have reported with multiple browsers that tabular data is not displayed in the main central panel. The preview in the history works, but clicking on the "eye" icon gives a blank page. I can confirm this in Firefox 24.0 and Safari 6.0.5, the HTML for the central frame looks blank but is actually a script when viewed via the frame source. The tabular file data is there in the script's first_data_chunk entry - but does not get shown. An except is shown below. Has anyone else noticed this problem? I believe it began when I last updated this Galaxy instance, but was not noticed immediately. Thanks, Peter ----------------------- I believe we are running the current stable release, or at least the latest commit on the default branch from galaxy-dist $ hg pull warning: bitbucket.org certificate with fingerprint 24:9c:45:8b:9c:aa:ba:55:4e:01:6d:58:ff:e4:28:7d:2a:14:ae:3b not verified (check hostfingerprints or web.cacerts config setting) real URL is https://bitbucket.org/galaxy/galaxy-dist pulling from http://bitbucket.org/galaxy/galaxy-dist searching for changes no changes found $ hg update 0 files updated, 0 files merged, 0 files removed, 0 files unresolved $ hg branch default [galaxy@ppserver galaxy-dist]$ hg log -b default | head changeset: 10411:c42567f43aa7 user: greg date: Mon Aug 19 13:19:56 2013 -0400 summary: Filter invalid objects when generating the list of repository_dependencies objects that are associated with a tool shed repository installed into Galaxy. changeset: 10410:e9ee9c5d30ae user: Dave Bouvier <dave(a)bx.psu.edu> date: Mon Aug 19 13:08:45 2013 -0400 summary: Remove duplicate method from the install and test's functional_tests.py. ... ----------------------- <!DOCTYPE HTML> <html>  <head> <title>Dataset Display</title> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <link href="/galaxy/static/style/base.css?v=1380731103" media="screen" rel="Stylesheet" type="text/css" /> <script src="/galaxy/static/scripts/libs/jquery/jquery.js?v=1380731103" type="text/javascript"></script> <script src="/galaxy/static/scripts/libs/jquery/jquery.migrate.js?v=1380731103" type="text/javascript"></script> <script src="/galaxy/static/scripts/libs/jquery/select2.js?v=1380731103" type="text/javascript"></script> <script src="/galaxy/static/scripts/libs/bootstrap.js?v=1380731103" type="text/javascript"></script> <script src="/galaxy/static/scripts/libs/underscore.js?v=1380731103" type="text/javascript"></script> <script src="/galaxy/static/scripts/libs/backbone/backbone.js?v=1380731103" type="text/javascript"></script> <script src="/galaxy/static/scripts/libs/backbone/backbone-relational.js?v=1380731103" type="text/javascript"></script> <script src="/galaxy/static/scripts/libs/handlebars.runtime.js?v=1380731103" type="text/javascript"></script> <script src="/galaxy/static/scripts/galaxy.base.js?v=1380731103" type="text/javascript"></script> <script src="/galaxy/static/scripts/templates/compiled/template-popupmenu-menu.js?v=1380731103" type="text/javascript"></script> <script src="/galaxy/static/scripts/mvc/ui.js?v=1380731103" type="text/javascript"></script> <script type="text/javascript"> // console protection window.console = window.console || { log : function(){}, debug : function(){}, info : function(){}, warn : function(){}, error : function(){}, assert : function(){} }; // Set up needed paths. var galaxy_paths = new GalaxyPaths({ root_path: '/galaxy/', image_path: '/galaxy/static/images' }); </script> <script src="/galaxy/static/scripts/libs/require.js?v=1380731103" type="text/javascript"></script> <script src="/galaxy/static/scripts/templates/compiled/helpers-common-templates.js?v=1380731103" type="text/javascript"></script> <script type="text/javascript"> require.config({ baseUrl: "/galaxy/static/scripts", shim: { "libs/backbone/backbone": { exports: "Backbone" }, "libs/backbone/backbone-relational": ["libs/backbone/backbone"] } }); require(['mvc/data'], function(data) { data.createTabularDatasetChunkedView( _.extend( {"misc_blurb": "7,891 lines", "peek": "<table cellspacing=\"0\" cellpadding=\"3\"><tr><th>1</th><th>2</th><th>3</th></tr><tr><td>Gpa_EST_02_04___C05_022</td><td>GO:0005737</td><td>...</td></tr></table>", "data_type": "tabular", "deleted": false, "history_id": "cdbf837ba6516a14", "metadata_column_names": null, "visible": true, "genome_build": "?", "hid": 10, "file_size": 509095, "metadata_data_lines": 7891, "file_ext": "tabular", "id": "0ddcd2cf2375e24a", "misc_info": "\nBLASTX output identified\nInput has 4507 queries\nDone\nStandard out:\nBlast2GO Pipeline Version 2.5.0\n--------------------------------------\n\nHash for resource ec2go_19072011.txt of type ec created. 4003 associations loaded.\nBlast2GO build: 23092011\nChange ", "hda_ldda": "hda", "name": "Blast2GO Local database (May 2011)", "metadata_columns": 3, "uuid": null, "state": "ok", "metadata_comment_lines": null, "model_class": "HistoryDatasetAssociation", "metadata_dbkey": "?", "purged": false, "metadata_column_types": ["str", "str", "str"]}, { url_viz: "/galaxy/visualization", chunk_url: "/galaxy/dataset/display?dataset_id=0ddcd2cf2375e24a", first_data_chunk: {"ck_index": 1, "ck_data": "Gpa_EST_02_04___C05_022\tGO:0005737\tn-dimethylarginine dimethylaminohydrolase\n...\n"} } ), // Append view to body. $('body') ); }); </script> </head> <body> </body> </html> --

5 17

Problem With Galaxy
by Caitlin Grube 29 Oct '13

29 Oct '13

I am having an issue trying to do anything. I click on a tool, the screen pops up in the middle but then when I move my mouse over to that screen, the drop down menus all pull down in a large black screen so that I cannot get to anything. The screen also does not go away if I try to click away from it, but only if I choose one of the options from the list. I would click analyze data which would take back to where I started. This happens for every tool that I try. I did not have this problem this morning or afternoon, but I've been trying to get it to work for almost an hour. As soon as my mouse hits that center area (top, bottom, middle), the screen comes down, but only after I select a tool. Please help. Caitlin Grube

2 1

cloning Galaxy cluster on cloud Amazon EC2
by Manisha Sapre 29 Oct '13

29 Oct '13

Hello Team Galaxy trying to launch a clone cluster of galaxy on cloud amazon EC2 (Scr.1),launching instance on US East region getting error unable to console (Scr.2), insights of team will be very helpful. [cid:DC3AB594-6DEF-4B12-B6C1-6D5ACEC22EEB] [cid:CD19EA5C-961A-4777-A811-DC0FFEBAD796] Thanks Manisha

2 7

Bam File
by Arshad Rafiq 29 Oct '13

29 Oct '13

I am trying to upload a bam file for my data analysis (size is about 9GB) I am trying URL method to up load and getting error message, can you please help me to sort out this problem. I am seeing following message "An error occurred setting the metadata for this dataset. You may be able to set it manually or retry auto-detection" Thanks *********** Arshad -- Muhammad Arshad Rafiq, PhD Research Associate Laboratory of Dr. R. Hamilton Physiology and Experimental Medicine Research Institute The Hospital for Sick Children McMaster Building, Room 7005 88 Elm St. Toronto, ON M5G 1X8 Canada 647-237-4915 arshadrafiq68(a)gmail.com

2 1

Why are Custom Builds disabled when
by Peter van Heusden 28 Oct '13

28 Oct '13

In galaxy.masthead.mako, from line 194, the following code appears: if app.config.use_remote_user: if app.config.remote_user_logout_href: menu_options.append( [ _('Logout'), app.config.remote_user_logout_href, "_top" ] ) else: menu_options.append( [ _('Preferences'), h.url_for( controller='/user', action='index', cntrller='user' ), "galaxy_main" ] ) menu_options.append( [ 'Custom Builds', h.url_for( controller='/user', action='dbkeys' ), "galaxy_main" ] ) logout_url = h.url_for( controller='/user', action='logout' ) menu_options.append( [ 'Logout', logout_url, "_top" ] ) menu_options.append( None ) Why are Custom Builds only enabled if you're *not* using remote_user ? Is this because they're related to user preferences (as per the code in lib/galaxy/webapps/galaxy/controllers/user.py)? And then a followup, if I have to add a build using either a FASTA file or a lenfile and editing the Galaxy install on the server, where do I need to make changes? Thanks, Peter

2 1

Upload via URL returns empty output
by Nikos Sidiropoulos 28 Oct '13

28 Oct '13

Hi all, I have created a Data library and uploaded (linked) some fastq files. After applying the bar code splitter tool, when I click on the output links I can get the output on my browser. I also get them by copy pasting the address in a new tab. But when I try the upload via URL by copy pasting the address I get an empty output. I haven't setup an ftp server neither I'm using and apache proxy. Could this be the problem? Bests, Nikos

1 0

trackster: missing graphs for large number of contigs
by Berner, Thomas 28 Oct '13

28 Oct '13

Hey guys, we want to use trackster in our local galaxy instance (rev. 10392:1ae95b3aa98d release_2013.08.12) and followed the instructions at http://wiki.g2.bx.psu.edu/Learn/Visualization . We tested it with a custom Build of 96 sequences in a fasta file, where all bams were mapped against before, everything worked fine. But now we want to do the same for a much larger file (about 29.000 contigs) and no graph is displayed at all, no matter which contig is used. No error appears, only the graphs are missing. Any ideas what this could be? We would be grateful for any advice. Thanks Thomas --- Thomas Berner Julius Kühn-Institut (JKI) - Federal Research Centre for Cultivated Plants - Erwin Baur-Straße 27 06484 Quedlinburg - Germany - Phone: ++49 ( 0 ) 3946 47 562 EMail: thomas.berner(a)jki.bund.de

2 3

Galaxy and ProFTP
by Jurgens de Bruin 28 Oct '13

28 Oct '13

I am having problems getting ProFTP to work: This is the output of ProFTP: FTP session opened. manager proftpd[33551] manager (155.240.187.111[155.240.187.111]): dispatching PRE_CMD command 'USER username' to mod_core manager proftpd[33551] manager (155.240.187.111[155.240.187.111]): dispatching PRE_CMD command 'USER username' to mod_core manager proftpd[33551] manager (155.240.187.111[155.240.187.111]): dispatching PRE_CMD command 'USER username' to mod_delay manager proftpd[33551] manager (155.240.187.111[155.240.187.111]): dispatching PRE_CMD command 'USER username' to mod_auth manager proftpd[33551] manager (155.240.187.111[155.240.187.111]): dispatching CMD command 'USER username' to mod_auth manager proftpd[33551] manager (155.240.187.111[155.240.187.111]): dispatching POST_CMD command 'USER username' to mod_sql manager proftpd[33551] manager (155.240.187.111[155.240.187.111]): dispatching POST_CMD command 'USER username' to mod_delay manager proftpd[33551] manager (155.240.187.111[155.240.187.111]): dispatching LOG_CMD command 'USER username' to mod_sql manager proftpd[33551] manager (155.240.187.111[155.240.187.111]): dispatching LOG_CMD command 'USER username' to mod_log manager proftpd[33551] manager (155.240.187.111[155.240.187.111]): dispatching PRE_CMD command 'PASS (hidden)' to mod_core manager proftpd[33551] manager (155.240.187.111[155.240.187.111]): dispatching PRE_CMD command 'PASS (hidden)' to mod_core manager proftpd[33551] manager (155.240.187.111[155.240.187.111]): dispatching PRE_CMD command 'PASS (hidden)' to mod_sql_passwd manager proftpd[33551] manager (155.240.187.111[155.240.187.111]): dispatching PRE_CMD command 'PASS (hidden)' to mod_sql manager proftpd[33551] manager (155.240.187.111[155.240.187.111]): dispatching PRE_CMD command 'PASS (hidden)' to mod_delay manager proftpd[33551] manager (155.240.187.111[155.240.187.111]): dispatching PRE_CMD command 'PASS (hidden)' to mod_auth manager proftpd[33551] manager (155.240.187.111[155.240.187.111]): dispatching CMD command 'PASS (hidden)' to mod_auth manager proftpd[33551] manager (155.240.187.111[155.240.187.111]): no supplemental groups found for user 'username' manager proftpd[33551] manager (155.240.187.111[155.240.187.111]): mod_sql_passwd/0.4: expected '982673e732e72d0646575025d72ee4c811a547f6', got 'c4233ae0617d48182fdca25e1488871a008ad783' manager proftpd[33551] manager (155.240.187.111[155.240.187.111]): USER username (Login failed): No such user found. This is proftpd.conf: This is a basic ProFTPD configuration file (rename it to # 'proftpd.conf' for actual use. It establishes a single server # and a single anonymous login. It assumes that you have a user/group # "nobody" and "ftp" for normal operation and anon. ServerName "Galaxy FTP" ServerType standalone DefaultServer on # Port 21 is the standard FTP port. Port 21 # Don't use IPv6 support by default. UseIPv6 off # Umask 022 is a good standard umask to prevent new dirs and files # from being group and world writable. Umask 022 # To prevent DoS attacks, set the maximum number of child processes # to 30. If you need to allow more than 30 concurrent connections # at once, simply increase this value. Note that this ONLY works # in standalone mode, in inetd mode you should use an inetd server # that allows you to limit maximum number of processes per service # (such as xinetd). MaxInstances 30 # Set the user and group under which the server will run. User galaxy Group galaxy # To cause every FTP user to be "jailed" (chrooted) into their home # directory, uncomment this line. DefaultRoot ~ # Automatically create home directory if it doesn't exist CreateHome on dirmode 700 # Normally, we want files to be overwriteable. AllowOverwrite on # Allow users to resume interrupted uploads AllowStoreRestart on # Bar use of SITE CHMOD by default <Limit SITE_CHMOD> DenyAll </Limit> # Bar use of RETR (download) since this is not a public file drop <Limit RETR> DenyAll </Limit> # Do not authenticate against real (system) users AuthPAM off # Set up mod_sql_password - Galaxy passwords are stored as hex-encoded SHA1 SQLPasswordEngine on SQLPasswordEncoding hex # Set up mod_sql to authenticate against the Galaxy database SQLEngine on SQLBackend postgres SQLConnectInfo galaxy_prod@localhost:5432 galaxyftp ******* SQLAuthTypes SHA1 SQLAuthenticate users # An empty directory in case chroot fails SQLDefaultHomedir /scratch/sysusers/galaxy/ftp #Define a custom query for lookup that returns a passwd-like entry. UID and GID should match your Galaxy user. SQLUserInfo custom:/LookupGalaxyUser SQLNamedQuery LookupGalaxyUser SELECT "email,password,'600','601','/scratch/sysusers/galaxy/ftp/%U','/bin/bash' FROM galaxy_user WHERE email='%U'" # Create log ExtendedLog /var/log/proftpd.log read,write Include /etc/proftpd/modules.conf AuthOrder mod_sql.c SQLDefaultGID 601 SQLDefaultUID 600 I have added use_pbkdf2 = false to my universe_wsgi.ini and the password have been reset. Any help would be appreciated!! -- Regards/Groete/Mit freundlichen Grüßen/recuerdos/meilleures salutations/ distinti saluti/siong/duì yú/привет Jurgens de Bruin

1 0

pass regular expression
by Jun Fan 26 Oct '13

26 Oct '13

Hi all, I am trying to develop a tool which allows the user to use regular expression to only keep the lines matching the given pattern in the output. There are many special characters, e.g. \ in \d. In the perl script the wrapper invokes, the regular expression is printed as Xd+ for the regular expression \d+. My question is how to pass these special characters including \, (, ), ^ etc. correctly into the perl script. Best regards! Jun

4 4