July 2014 - galaxy-dev - lists.galaxyproject.org

XML/json code to parse/get/create wrapper files to/from galaxy
by Rémy Dernat 22 Jul '14

22 Jul '14

Hi, I created a little repository on github to parse or create wrapper files. I have not implemented everything, but it is a good base, i.e. if you want to create a galaxy wrapper from a standard help output ! Just clone it and have fun : git clone https://github.com/remyd1/XMLparser-wrapper.git cd XMLparser-wrapper python ParserConverterXML.py --help python ParserConverterXML.py h2gw -c "ls --help" -o ../ls.gw.xml Any help is welcome on this little project. Exemples : json for workflows... Best regards, Remy

2 2

Customizing the HTML datatype and altering the behavior of "download results" link.
by Renato Alves 21 Jul '14

21 Jul '14

Hi all, TLDR - Last 2 lines of this email Some time ago we created a wrapper script for breseq, a tool that generates a friendly HTML report of mutations. This was a perfect match for Galaxy given that we could simply set the "summary.html" file to be displayed using Galaxy's HTML datatype and place all extra media files in $htmlfile.files_path. However the friendliness quickly became a pain with large datasets. The size of the "files_path" folder quickly grew to over 500MB if multiple references are used. Still this was not the biggest problem since the size is spread over lots of tiny HTML files which work fine in Galaxy's preview. The real nasty hit us when we found that with the HTML format Galaxy does on-the-fly compression of the whole folder if a user clicks the "download" button. Needless to say most of the time users end up getting a server time-out and we get lots of broken pipes on the logs. An additional complication of the HTML format is that only the size of the primary html file is taken into account. So most users didn't understand why a 10KB file would cause so much pain to download and why was their quota being taken for no apparent reason. So after this we got stuck with 2 alternatives: 1) Customize the HTML datatype to do /a priori/ zipping instead of on-the-fly. Provide this pre-zipped file if users click on the download button. Fix the HTML datatype to display the correct size of the full dataset. 2) Provide 2 outputs, the usual HTML file output and an additional "complete report" which is a pre-packaged zip file generated by the wrapper. After spending quite some time browsing the existing documentation and trying to make sense of the code we gave up on 1). Looking at datatype class definitions, in particular HTML and the class hierarchy, we couldn't even figure out how the "files_path" mechanism worked. Documentation or lack thereof didn't help either. Knowing what methods are mandatory and what, when and how are they called would be a major help. So we jumped on 2). Getting the pre-packaged zip was quick and easy. We had to create a zip datatype but the existing Binary class worked fine: <datatype extension="zip" type="galaxy.datatypes.binary:Binary" mimetype="application/zip" subclass="True" /> The second part of the problem was to disable the HTML "download" link. And now we are back into customizing the HTML datatype... So long story short, if I want to customize what happens when a user clicks the download link of a given format, which methods should I be looking at? Can you think of any alternative (better) way to solve the original problem? Thanks, Renato

2 2

Galaxy+Toolshed Docker image?
by Kyle Ellrott 21 Jul '14

21 Jul '14

I know there is a base docker image for Galaxy stable ( https://registry.hub.docker.com/u/bgruening/galaxy-stable/) but is there a docker image that will start both a server and a toolshed sever and link them? I was hoping that I could use something like that for testing shed based tool deployment. Kyle

2 1

bugs with relative path in library_import_dir
by Wolfgang Maier 21 Jul '14

21 Jul '14

Hi, when you specify a library_import_dir in universe_wsgi.ini with a path relative to galaxy-dist, i.e. a directory: /home/galaxy/galaxy-dist/uploads is specified as: library_import_dir = uploads and you then try to Upload directory of files, everything's reported as fine, but you end up with a 0 bytes file (which gives an error when trying to import it into any history). This is the case with Copy files into Galaxy selected. If you instead try to Link to files without copying into Galaxy, the error is even more subtle because you can import the dataset into histories and even work with it, but auto-detection of the file format guesses the wrong datatype (just data for e.g. fasta format) and the dataset carries the message: Uploaded temporary file (path/filename) does not exist. Everything's working just fine with an absolute path in universe_wsgi.ini, but I think the behaviour with relative paths is just bizarre. Apparently, some parts of Galaxy do use os.path.abspath, while others don't ? My version of Galaxy is changeset 68a8b0397947 just in case this has recently been fixed. Best, Wolfgang

1 0

questions about cloudman
by Pablo Escobar 20 Jul '14

20 Jul '14

Hi, I have been reading some docs about cloudman and I am thinking to give it a try, but I have a few doubts about it. 1) I have read in the wiki that it's only available in region US East (Virginia). Will it be available in Ireland region? 2) I see I need to attach EBS volumes to my head node to host the software and data. I think EBS volumes have a max size of 1TB. Which would be the approach if I need a filesystem bigger than 1TB? 3) Is possible to customize compute nodes with extra packages or config files? I suppose once the compute nodes are up and running I could ssh to them and do any modification but if the auto-scaling feature launch new compute nodes those new lauched nodes would not have my custom modifications. Is possible to do modifications that keep persistent even for new launched compute nodes? thank you very much for your help. Pablo.

2 1

Error on Wiki
by Inus Scheepers 19 Jul '14

19 Jul '14

This must be a mistake on https://wiki.galaxyproject.org/ -- $dig -x 54.243.198.183 183.198.243.54.in-addr.arpa. 300 IN PTR chouffe.galaxyproject.org. -- Inus Scheepers +27 82 658 1961 Skype:inus.s PO Box 21721, Kloof Street, 8008

2 1

API v/s twill based testing
by Saket Choudhary 18 Jul '14

18 Jul '14

I recently updated tests for one of my wrappers and came across this strange behaviour: The twill based testing reports a failure: https://travis-ci.org/saketkc/galaxy_tools/jobs/29956682#L1463 whereas, the API based testing shows success: https://travis-ci.org/saketkc/galaxy_tools/jobs/29956683 Unfortunately I cannot run these tests locally since I am behind a system proxy [Refer: http://dev.list.galaxyproject.org/Functional-Tests-and-ftype-td4664233.html] and have to rely on travis.. The place where twii tests fails shows that it is trying to compare the diff between 'chasm_output_genes.tabular' and 'chasm_output_variants.tabular' instead of 'chasm_output_genes.tabular'. [See : https://travis-ci.org/saketkc/galaxy_tools/jobs/29956682#L1469] I tried running my tools locally and I did not come across any case where the 'variants' output gets replaced by the 'genes' output, thus possibly ruling out unexpected behavior from the tool's server end. Is this a possible bug or am I missing something? Saket

4 6

Unable to update to latest version of galaxy
by Eleanor Morgan 17 Jul '14

17 Jul '14

Hi all, I am new to galaxy. We have a local install that we intend to turn into a production server at some stage. We initially installed galaxy using the April 2014 version. We set up a Virtual Python Environment following the instructions set out on this page: https://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer#Us… When I updated to the June version of galaxy and tried to start up the server using the recommended shell script placed in /etc/init.d I got the following error message: Starting galaxy... Some eggs are out of date, attempting to fetch... Traceback (most recent call last): File "./scripts/fetch_eggs.py", line 37, in <module> c.resolve() # Only fetch eggs required by the config File "/galaxy/galaxy-dist/lib/galaxy/eggs/__init__.py", line 345, in resolve egg.resolve() File "/galaxy/galaxy-dist/lib/galaxy/eggs/__init__.py", line 195, in resolve return self.version_conflict( e.args[0], e.args[1] ) File "/galaxy/galaxy-dist/lib/galaxy/eggs/__init__.py", line 226, in version_conflict r = pkg_resources.working_set.resolve( ( dist.as_requirement(), ), env, egg.fetch ) File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 550, in resolve raise VersionConflict(dist,req) # XXX put more info here pkg_resources.VersionConflict: (pycrypto 2.0.1 (/usr/lib64/python2.6/site-packages), Requirement.parse('pycrypto>=2.1,!=2.4')) Fetch failed. ...done. This error message implies that I need to update pycrypto on our system, but my understanding is that the virtual python environment takes care of these sorts of dependencies. Can someone please advise? Here are two series of commands that I tried to start the system: First attempt: sudo bash /etc/init.d/galaxy stop su - galaxy cd /galaxy/galaxy-dist hg pull hg update latest_2014.06.02 exit /etc/init.d/galaxy start After galaxy start failed I tried: su - galaxy cd /galaxy . ./galaxy_env/bin/activate cd galaxy_dist sh run.sh I also tried reinstalling the virtual environment to no avail. Regards, Eleanor

3 3

cleanup_datasets.py not deleting files...
by Neil.Burdett＠csiro.au 17 Jul '14

17 Jul '14

Hi, I'm trying to use the cleanup_datasets.py file to remove all files on my system older than 20 days. My crontab looks like this: # m h dom mon dow command 34 10 * * * cd /export/barium-data3/galaxy-suvr && python scripts/cleanup_datasets/cleanup_datasets.py universe_wsgi.ini -d 20 -1 > /home/galaxy/crontab_purge_milxcloud.log 2>&1 && python scripts/cleanup_datasets/cleanup_datasets.py universe_wsgi.ini -d 20 -2 -r >> /home/galaxy/crontab_purge_milxcloud.log 2>&1 && python scripts/cleanup_datasets/cleanup_datasets.py universe_wsgi.ini -d 20 -3 -r >> /home/galaxy/crontab_purge_milxcloud.log 2>&1 && python scripts/cleanup_datasets/cleanup_datasets.py universe_wsgi.ini -d 20 -5 -r >> /home/galaxy/crontab_purge_milxcloud.log 2>&1 && python scripts/cleanup_datasets/cleanup_datasets.py universe_wsgi.ini -d 20 -4 -r >> /home/galaxy/crontab_purge_milxcloud.log 2>&1 && python scripts/cleanup_datasets/cleanup_datasets.py universe_wsgi.ini -d 20 -6 -r >> /home/galaxy/crontab_purge_milxcloud.log 2>&1 The crontab executes and the contents of /home/galaxy/crontab_purge_milxcloud.log is: cat /home/galaxy/crontab_purge_milxcloud.log psycopg2 egg successfully loaded for postgres dialect ########################################## # 2014-07-17 10:34:02 - Handling stuff older than 20 days Datasets will NOT be removed from disk. Deleted 0 histories Elapsed time: 0.170083045959 ########################################## psycopg2 egg successfully loaded for postgres dialect ########################################## # 2014-07-17 10:34:02 - Handling stuff older than 20 days Datasets will be removed from disk. Purged 0 histories. Elapsed time: 0.174137830734 ########################################## psycopg2 egg successfully loaded for postgres dialect ########################################## # 2014-07-17 10:34:03 - Handling stuff older than 20 days Datasets will be removed from disk. Purged 0 datasets Freed disk space: 0 Elapsed time: 0.168104887009 ########################################## psycopg2 egg successfully loaded for postgres dialect ########################################## # 2014-07-17 10:34:04 - Handling stuff older than 20 days Datasets will be removed from disk. # Purged 0 folders. Elapsed time: 0.168635129929 ########################################## psycopg2 egg successfully loaded for postgres dialect ########################################## # 2014-07-17 10:34:05 - Handling stuff older than 20 days Datasets will be removed from disk. # Purged 0 libraries . Elapsed time: 0.166506052017 ########################################## psycopg2 egg successfully loaded for postgres dialect ########################################## # 2014-07-17 10:34:06 - Handling stuff older than 20 days Datasets will be removed from disk. Examined 0 datasets, marked 0 datasets and 0 dataset instances (HDA) as deleted Total elapsed time: 0.00744795799255 ########################################## However, today is the 17th July and when I look into the database directory i.e. ~/database/files/000 I still have files from the 18th June so older than 20 days i.e.: drwxr-xr-x 2 galaxy nogroup 4096 Jun 18 09:19 dataset_213_files -rw-r--r-- 1 galaxy nogroup 1061 Jun 18 09:19 dataset_213.dat drwxr-xr-x 2 galaxy nogroup 4096 Jun 18 09:19 dataset_215_files -rw-r--r-- 1 galaxy nogroup 270 Jun 18 09:19 dataset_215.dat drwxr-xr-x 2 galaxy nogroup 4096 Jun 18 09:19 dataset_221_files drwxr-xr-x 2 galaxy nogroup 4096 Jun 18 09:19 dataset_219_files drwxr-xr-x 2 galaxy nogroup 4096 Jun 18 09:19 dataset_220_files drwxr-xr-x 2 galaxy nogroup 4096 Jun 18 09:19 dataset_222_files drwxr-xr-x 2 galaxy nogroup 4096 Jun 18 09:19 dataset_218_files -rw-r--r-- 1 galaxy nogroup 994 Jun 18 09:19 dataset_216.dat -rw-r--r-- 1 galaxy nogroup 161 Jun 18 09:19 dataset_214.dat Am I doing something wrong? Using wrong arguments/file etc ? Thanks Neil

3 6

Wiki datatypes tutorial
by Eric Rasche 17 Jul '14

17 Jul '14

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Thanks to everyone for their assistance in my adventure of custom local datatypes. In response to this, I've added a new wiki page with a basic MWE/tutorial on adding datatypes. A complete example is at the end, because most people like copy+paste code to get them started. https://wiki.galaxyproject.org/Admin/Datatypes/AddingCompleteDatatypes Please feel free to add to it/fix things I completely misunderstood. I'm not sure what 80% of the functions that get called in datatypes do, nor where they're called from, so I can't offer much more detail in this wiki page than I already have. (E.g., when is split called? If I write a split method, how can I test it? What other methods should I implement?) Cheers, Eric -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAEBAgAGBQJTyBO2AAoJEMqDXdrsMcpV2lcQAJAm0V3uFHzNIovEVoA7HvL6 WAaeVLjsUjEqiGl4F2WECzB4kXyGWN2N9t3QaBH0SCMFRzdrMmvoM7Bmw8/rmWcY xCFfgdP1JS5cxAJlCAzcWw6VEq9eWW8yHv+9QLc3E++X8R458n1SfsCV2IehPEcw bJSmv3FbhF7dLwmW2XjwRmIl6mitbB37eVa8w7z+NWgYtlCuFBzdZztHvPhULNyY s2mUf1T9KsH7Pdoho7op82MSBlLyKyHr6oweRpxgHHGUk+VkozAuLg6BEV+/YDQ1 Ulvyz4NOO/l8aRG+NQS6SzzXHfZhh+FFCfludrNpy9LJ1rFxqwOlex8tgl1XymXU vDzjo+yT4ypdYrRrOz0QE7W52JvVzNK1jk83454j1KTI2ZJnis3RfUJ6xeXwNVW4 LKyfo6RYJ8eRvCiJHXwuCRpXKFUKuUwxY/2due0P5rI9B4432X8FglK15q0AFd9K g7ChG+gNfiWNUjvxOVHM5ZCa/1SSTtw4khSwbeTX9n5sjbTNzu2eOr+ZFxSeCp2U vC2tqKR6NJUU/T16OYPMT4mGnVCrfX10CnwxnDR17ekzJLhTHd0WAh9qSYXnLFgQ wkint9A4N4RmqgD9+o1B0HCRGlKPFhUJ4gI4M+HnHS6QXBMDtq+0V3vX8Yx+DYKN Gpx/gp3ns8bJkRarha7m =2tCq -----END PGP SIGNATURE-----

2 1