- galaxy-dev - lists.galaxyproject.org

uploads stuck in history
by Jorrit Boekel 08 Mar '12

08 Mar '12

Dear list, I have stumbled on some strange behaviour when uploading files to galaxy via the upload.py tool. At times, the upload seems to be stalled in history and is never actually performed, followed by a seemingly infinite history update (see log below). My system is Ubuntu 11.10 and runs Python 2.7.2. I find the behaviour in both my own modified galaxy install (based on galaxy-dist), and in a fresh clone from galaxy-central. I have tried to upload different files, and all seem to sometimes trigger the behaviour, but not all the time. A restart of galaxy sometimes sorts things out. Common for the debug messages is that it seems there is never a job id generated as in "galaxy.tools.actions.upload_common INFO 2012-02-22 10:06:36,186 tool upload1 created job id 2". Has anyone seen similar things or can it be a problem with my system? cheers, jorrit --------------Debug messa: galaxy.web.framework DEBUG 2012-02-22 09:47:43,730 Error: this request returned None from get_history(): http://localhost:8080/ 127.0.0.1 - - [22/Feb/2012:09:47:43 +0200] "GET / HTTP/1.1" 200 - "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:47:44 +0200] "GET /root/tool_menu HTTP/1.1" 200 - "http://localhost:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:47:44 +0200] "GET /history HTTP/1.1" 200 - "http://localhost:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:47:44 +0200] "POST /root/user_get_usage HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:47:50 +0200] "GET /tool_runner?tool_id=upload1 HTTP/1.1" 200 - "http://localhost:8080/root/tool_menu" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:11 +0200] "POST /tool_runner/upload_async_create HTTP/1.1" 200 - "http://localhost:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:11 +0200] "GET /tool_runner/upload_async_message HTTP/1.1" 200 - "http://localhost:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:11 +0200] "GET /history HTTP/1.1" 200 - "http://localhost:8080/tool_runner/upload_async_message" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:11 +0200] "POST /root/user_get_usage HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:13 +0200] "GET /tool_runner?tool_id=upload1 HTTP/1.1" 200 - "http://localhost:8080/root/tool_menu" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:15 +0200] "POST /root/history_item_updates HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:19 +0200] "POST /root/history_item_updates HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:23 +0200] "POST /root/history_item_updates HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:27 +0200] "POST /root/history_item_updates HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:31 +0200] "POST /root/history_item_updates HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:35 +0200] "POST /root/history_item_updates HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:39 +0200] "POST /root/history_item_updates HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:43 +0200] "POST /root/history_item_updates HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:47 +0200] "POST /root/history_item_updates HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2"

6 10

Welcome.html page
by huayan gao 08 Mar '12

08 Mar '12

Hi Nate, I got a silly question for you. My galaxy is running now but I'd like to customize it. I've changed the welcome.html page but the galaxy mirror site is showing the old one. Do I need to change some other files or other configurations? Also, I add <lable> in tool_conf.xml file but it does not show up either. Is there another file I need to change too? Best, Huayan

2 1

Re: [galaxy-dev] [galaxy-user] Problem using Galaxy on the cloud
by Brad Chapman 07 Mar '12

07 Mar '12

Jan; Thanks for getting back with all the detailed information. I dug into this further and understand what is happening: - tools/data_source/upload.py calls lib/galaxy/datatypes/sniff.py:stream_to_file - stream_to_file uses pythons tempfile module - tempfile defaults to using /tmp - As large files stream in the temporary space fills up, causing the issue you are seeing. The best way to work around this is to have the galaxy user on Amazon export TMPDIR to point at a temporary directory on /mnt/galaxyData instead of the root filesystem. I'm hoping that Enis or Dannon might be able to help out with the best place to set this in CloudMan to avoid the issue, I've cc'ed them in. If you want to manually fix it to get some work done, you could create a directory /mnt/galaxyData/tmp and then symlink /tmp there: ln -s /mnt/galaxyData/tmp /tmp Hope this helps and we can come up with a more permanent fix. Thanks again, Brad > Thanks Brad, > Sorry it has taken me so long to respond. I had a meeting sort of day. I started another instance to recreate what is happening to me in more detail. I hope this helps. I tried to color code things so you could follow everything more easily. > > This is what I have when I start up Galaxy through BioCloudCentral before I import any files... > > Get cloud support with Ubuntu Advantage Cloud Guest > http://www.ubuntu.com/business/services/cloud > ubuntu@ip-10-44-78-218:~$ df -h > Filesystem Size Used Avail Use% Mounted on > /dev/xvda1 20G 13G 6.1G 68% / > udev 8.4G 4.0K 8.4G 1% /dev > tmpfs 3.4G 644K 3.4G 1% /run > none 5.0M 0 5.0M 0% /run/lock > none 8.4G 0 8.4G 0% /run/shm > /dev/xvdb 404G 201M 383G 1% /mnt > > After starting to import data (pasted https://s3.amazonaws.com/thunnus/BluefinAQ30_shuffled.fastq) it starts filling up immediately > > Filesystem Size Used Avail Use% Mounted on > /dev/xvda1 20G 19G 0 100% / > udev 8.4G 4.0K 8.4G 1% /dev > tmpfs 3.4G 660K 3.4G 1% /run > none 5.0M 0 5.0M 0% /run/lock > none 8.4G 0 8.4G 0% /run/shm > /dev/xvdb 404G 201M 383G 1% /mnt > /dev/xvdg1 700G 654G 47G 94% /mnt/galaxyIndices > /dev/xvdg2 10G 1.7G 8.4G 17% /mnt/galaxyTools > /dev/xvdg3 500G 81M 500G 1% /mnt/galaxyData >

1 0

Running SICER
by Vidya .H.K 07 Mar '12

07 Mar '12

Hi, I am trying to run SICER-peak calling tool in galaxy. I am not able run the analysis, as an error message " SICER is not available for the genome" is frequently flagged. Can any one suggest me a soultion for the same. Regards VIDYA

2 1

toolsectionlabel error (problem configuring toolshed hotplug?)
by Bossers, Alex 07 Mar '12

07 Mar '12

Somehow I seem to miss something when configuring our latest up-to-date galaxy_central version to use the galaxy toolshed. When as admin I want to install a tool I can browse the given 2 repositories and select view tools. When I select install to local galaxy it drops an error on 2 tools I tested thus far: --------------------------- Error Traceback: View as: Interactive | Text | XML (full) ⇝ AttributeError: 'ToolSectionLabel' object has no attribute 'name' URL: http://galaxy02:8080/galaxy/admin_toolshed/install_repository?tool_shed_url… Module weberror.evalexception.middleware:364 in respond view >> app_iter = self.application(environ, detect_start_response) Module paste.debug.prints:98 in __call__ view >> environ, self.app) Module paste.wsgilib:539 in intercept_output view >> app_iter = application(environ, replacement_start_response) Module paste.recursive:80 in __call__ view >> return self.application(environ, start_response) Module paste.httpexceptions:632 in __call__ view >> return self.application(environ, start_response) Module galaxy.web.framework.base:160 in __call__ view >> body = method( trans, **kwargs ) Module galaxy.web.framework:173 in decorator view >> return func( self, trans, *args, **kwargs ) Module galaxy.web.controllers.admin_toolshed:322 in install_repository view >> tool_panel_section_select_field = build_tool_panel_section_select_field( trans ) Module galaxy.web.controllers.admin_toolshed:804 in build_tool_panel_section_select_field view >> options.append( ( tool_section.name, tool_section.id ) ) AttributeError: 'ToolSectionLabel' object has no attribute 'name' --------------------------- The steps I took: Updated the universe to load the extra shed_tool_conf.xml file. Set the toolpath to "../toolshedtools" in shed_tool_conf.xml Gave the directory read write permissions for galaxy user:dev_group Restarted galaxy. Any additional configs to make somewhere? In the original tool_conf.xml file? Alex

4 7

Gal-Cloudman AMI Fails
by John Major 07 Mar '12

07 Mar '12

Hello GalFolks- I saw a new galaxy-cloudman AMI appear ( 072133624695/galaxy-cloudman-2012-02-26 ) and was interested in giving it a whirl. But when I tried to launch it got the following error: """ The requested instance type's architecture (x86_64) does not match the architecture in the manifest for aki-407d9529 (i386) """ Is this a dev-AMI? Or should I be expecting this to run? Thanks- John

2 2

babel egg: epiphany locale setting fix
by Jorrit Boekel 07 Mar '12

07 Mar '12

Dear list, I encountered problems when using galaxy with ubuntu's epiphany browser. It seems that when using this browser, HTTP_ENVIRON's locale setting is sometimes 'en-us', but sometimes ' en-us'. That's a whitespace before the locale. I solved the problem by adding a simple .strip() in eggs/babel/core.py on line 763: lang = parts.pop(0).lower().strip() I assume the egg is not maintained by the galaxy team, so maybe I should forward this to Babel developers, but if anyone runs into the same problem, here's a solution. cheers, jorrit

2 1

How to hide an output in the history depending on its size ?
by Aurélien Bernard 07 Mar '12

07 Mar '12

Hello everybody, My question is included in the subject of this email: Is there a way to force Galaxy to hide a specific output (of a tool) if this output is empty (or if the file does not exist in the job_working_directory repository) ? Thank you in advance, Have a nice day A. Bernard -- Aurélien Bernard IE Bioprogrammeur - CNRS Université des sciences Montpellier II Institut des Sciences de l'Evolution France

1 0

Problem with filter tool accepting all rows
by Peter Cock 07 Mar '12

07 Mar '12

Hi all, I just debugged a problem one of our users was having filtering BLAST tabular output using the "Filter" tool (version 1.1.0), where it was accepting 100% of the data. He had entered the filter condition as: 'c8-c7>=0.8*c23' If you know how the tool works internally (essentially evaluating the string as a Python expression where c1 etc are variables), you can guess what happened. This evaluates as a non-empty string, which Python regards as True. Therefore every row in the file was accepted. What he should have used was 'c8-c7>=0.8*c23' without the quotes. I think this is an easy mistake to make, especially if copy & pasting an example from documentation. Can Galaxy do anything about this? Note that a simple hack to remove leading and closing quote characters would break some valid use cases, e.g. 'gi:' + c1 == c2 + '|D1', where there happens to be a quote character at the start and end. Peter

1 0

Re: [galaxy-dev] [galaxy-user] Problem using Galaxy on the cloud
by Brad Chapman 06 Mar '12

06 Mar '12

Jan; > I appreciate the help, I am in no way a competent UNIX/Linux user. I > just have 2 shuffled fastq files in an S3 bucket-I pasted the URL for > these into Galaxy's upload file URL/Text box and that seems to be > where the trouble started. On my first try loading the data files, I > got a message saying there was no disk space. I'm trying to reproduce here but can't seem to figure out how the disk is filling up. A new filesystem should look like: $ df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda1 20G 13G 6.1G 68% / udev 3.7G 4.0K 3.7G 1% /dev tmpfs 1.5G 660K 1.5G 1% /run none 5.0M 0 5.0M 0% /run/lock none 3.7G 0 3.7G 0% /run/shm /dev/xvdb 414G 201M 393G 1% /mnt /dev/xvdg1 700G 654G 47G 94% /mnt/galaxyIndices /dev/xvdg2 10G 1.7G 8.4G 17% /mnt/galaxyTools /dev/xvdg3 5.0G 81M 5.0G 2% /mnt/galaxyData with 6Gb of free space on '/'. If you pasted a file, like: https://s3.amazonaws.com/chapmanb/example.fastq into the text box, it should show up as: /mnt/galaxyData/files/000/dataset_1.dat in the /mnt/galaxyData mount, which has all of the space for storing files. So I'm not sure exactly what is filling up your root partition. Could you try checking a couple of places and see if any of the filesystem usage is different from what we expect: $ du -sh /home/ubuntu/ 361M /home/ubuntu/ $ sudo du -sh /var/log/ 3.1M /var/log/ I'm cc'ing Enis in case he has any other ideas where we might be accidentally filling up the root filesystem. Thanks again for the feedback and sorry about the problems, Brad > I tried again, and it > worked. Then, when I tried to run velveth, it got hung up again with > the no disk space issue. I browsed around in the /dev/xvda1 file > system and there were quite a few files (I pasted them below). I > assumed they were files relating to BloudBioLinux and I was not sure > if I could get rid of any of the files. It was nothing I put there. I > started the instance using BioCloudCentral. Maybe it would be better > to just start Galaxy the long way (choosing a public AMI under launch > instance...)? > > > > ubuntu@ip-10-44-117-85:/$ ls > bin etc initrd.img lib64 mnt proc sbin sys var > boot export lib lost+found opt root selinux tmp vmlinuz > dev home lib32 media pkg run srv usr > > > Thanks again, > Jan > > On Mar 6, 2012, at 8:47 PM, Brad Chapman wrote: > > > > > Jan; > > Glad to hear you got Galaxy running successfully. It sounds like > > everything is good to go once we sort out the disk space issue. > > > >>> However, when I try to use NX to get the virtual desktop going I get the > >>> message usr/bin/nxserver: line 381: echo: write error: No space left > >>> on device. > >>> > >>> Using the df -h command, I get: > >>> > >>> Filesystem Size Used Avail Use% Mounted on > >>> /dev/xvda1 20G 19G 0 100% / > >>> udev 8.4G 4.0K 8.4G 1% /dev > >>> tmpfs 3.4G 660K 3.4G 1% /run > >>> none 5.0M 0 5.0M 0% /run/lock > >>> none 8.4G 0 8.4G 0% /run/shm > >>> /dev/xvdb 404G 202M 383G 1% /mnt > >>> /dev/xvdg1 700G 654G 47G 94% /mnt/galaxyIndices > >>> /dev/xvdg2 10G 1.7G 8.4G 17% /mnt/galaxyTools > >>> /dev/xvdg3 200G 11G 190G 6% /mnt/galaxyData > >>> > >>> > >>> So, I guess my question as a new user is: How do I point Galaxy and > >>> CloudBioLinux to all of this unused space? > > > > By default CloudMan will put files into /mnt/galaxyData. However, as > > you noticed the main filesystem got filled up at some point. Could this > > have happened while transferring files over from S3? Are there files in > > your home directory that you could delete or move to /mnt/galaxyData to > > free up space? > > > > CloudBioLinux and CloudMan shouldn't put a large number of files in the > > root directory, but when the root filesystem is full it's going to be > > very unhappy. Once you manually clear up some room there hopefully > > things will run smoother. > > > > If that doesn't help let us know and we can dig into it further. Thanks, > > Brad > >

1 0