May 2013 - galaxy-dev - lists.galaxyproject.org

Conditional tool options based on input file format?
by Peter Cock 06 May '13

06 May '13

Dear all, I've looked over the wiki and as far as I can see, 'when' tags used inside 'conditional' tags only work on another input variable's values. I would like to be able to do is a conditional based on the file format of an input file. For instance, a tool might take XML or tabular files, and need to show additional parameters for one input type but not the other. Does this enhancement idea seem useful? Peter

4 6

LWR runner configuration for shared folder in cluster
by Krampis, Konstantinos 06 May '13

06 May '13

Hi all, I am trying to set up a Galaxy cluster using the LWR runner. The nodes have a shared filesystem and in universe.wsgi this parameter is set : job_working_directory = /mnt/shared ... clustalw = lwr://http://192.168.33.12:8913 .... this folder has been "chown-ed" to the galaxy user, and also is "a+w", while it has been verified that can been read / written by ssh-ing to each node of the cluster. The sticky bit is set. When I try to run jobs (I used clustalw as example) there seems to be confusion between where Galaxy puts files and where LWR tries to read them from. Here are two setups that error out: 1). When in server.ini for LWR the following is set as: staging_directory = /mnt/shared/000 galaxy error: galaxy.jobs DEBUG 2013-05-06 10:21:22,320 (128) Working directory for job is: /mnt/shared/000/128 galaxy.jobs.handler DEBUG 2013-05-06 10:21:22,320 dispatching job 128 to lwr runner galaxy.jobs.handler INFO 2013-05-06 10:21:22,427 (128) Job dispatched galaxy.datatypes.metadata DEBUG 2013-05-06 10:21:22,875 Cleaning up external metadata files galaxy.jobs.runners.lwr ERROR 2013-05-06 10:21:22,902 failure running job 128 lwr error (on the cluster node): File "/home/vagrant/jmchilton-lwr-5213f6dce32d/lwr/app.py", line 81, in setup manager.setup_job_directory(job_id) File "/home/vagrant/jmchilton-lwr-5213f6dce32d/lwr/manager.py", line 101, in setup_job_directory os.mkdir(job_directory) OSError: [Errno 17] File exists: '/mnt/shared/000/128' 2). When in server.ini for LWR the following is set as: staging_directory = /mnt/shared galaxy error: galaxy.jobs DEBUG 2013-05-06 10:28:46,872 (129) Working directory for job is: /mnt/shared/000/129 galaxy.jobs.handler DEBUG 2013-05-06 10:28:46,872 dispatching job 129 to lwr runner galaxy.jobs.handler INFO 2013-05-06 10:28:46,967 (129) Job dispatched 192.168.33.1 - - [06/May/2013:10:28:48 -0200] "GET /api/histories/2a56795cad3c7db3 HTTP/1.1" 200 - "http://192.168.33.11:8080/history" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.65 Safari/537.31" galaxy.jobs.runners.lwr DEBUG 2013-05-06 10:28:50,653 run_results {'status': 'status', 'returncode': 0, 'complete': 'true', 'stderr': '', 'stdout': ''} galaxy.datatypes.metadata DEBUG 2013-05-06 10:28:50,970 Cleaning up external metadata files galaxy.jobs.runners.lwr ERROR 2013-05-06 10:28:51,050 failure running job 129 lwr error (on the cluster node): resp.app_iter = FileIterator(result) File "/home/vagrant/jmchilton-lwr-5213f6dce32d/lwr/framework.py", line 111, in __init__ self.input = open(path, 'rb') IOError: [Errno 2] No such file or directory: u'/mnt/shared/129/outputs/dataset_170.dat' The full error stacks are at the end of this email. It might be something very simple that I am missing, but any feedback would be greatly appreciated. Thanks ! Ntino -- Konstantinos (Ntino) Krampis, Ph.D. Asst. Professor, Informatics J.Craig Venter Institute kkrampis(a)jcvi.org agbiotec(a)gmail.com +1-540-200-8277 Web: http://bit.ly/cloud-research http://cloudbiolinux.org/ http://twitter.com/agbiotec ---- GALAXY ERROR galaxy.jobs DEBUG 2013-05-06 10:21:22,320 (128) Working directory for job is: /mnt/shared/000/128 galaxy.jobs.handler DEBUG 2013-05-06 10:21:22,320 dispatching job 128 to lwr runner galaxy.jobs.handler INFO 2013-05-06 10:21:22,427 (128) Job dispatched galaxy.datatypes.metadata DEBUG 2013-05-06 10:21:22,875 Cleaning up external metadata files galaxy.jobs.runners.lwr ERROR 2013-05-06 10:21:22,902 failure running job 128 Traceback (most recent call last): File "/home/vagrant/galaxy-dist/lib/galaxy/jobs/runners/lwr.py", line 286, in run_job file_stager = FileStager(client, command_line, job_wrapper.extra_filenames, input_files, output_files, job_wrapper.tool.tool_dir) File "/home/vagrant/galaxy-dist/lib/galaxy/jobs/runners/lwr.py", line 40, in __init__ job_config = client.setup() File "/home/vagrant/galaxy-dist/lib/galaxy/jobs/runners/lwr.py", line 212, in setup return self.__raw_execute_and_parse("setup", { "job_id" : self.job_id }) File "/home/vagrant/galaxy-dist/lib/galaxy/jobs/runners/lwr.py", line 150, in __raw_execute_and_parse response = self.__raw_execute(command, args, data) File "/home/vagrant/galaxy-dist/lib/galaxy/jobs/runners/lwr.py", line 146, in __raw_execute response = self.url_open(request, data) File "/home/vagrant/galaxy-dist/lib/galaxy/jobs/runners/lwr.py", line 134, in url_open return urllib2.urlopen(request, data) File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen return _opener.open(url, data, timeout) File "/usr/lib/python2.7/urllib2.py", line 406, in open response = meth(req, response) File "/usr/lib/python2.7/urllib2.py", line 519, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python2.7/urllib2.py", line 444, in error return self._call_chain(*args) File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain result = func(*args) File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) HTTPError: HTTP Error 500: Internal Server Error ---- LWR ERROR Exception happened during processing of request from ('192.168.33.11', 44802) Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/Paste-1.7.5.1-py2.7.egg/paste/httpserver.py", line 1068, in process_request_in_thread self.finish_request(request, client_address) File "/usr/lib/python2.7/SocketServer.py", line 323, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib/python2.7/SocketServer.py", line 638, in __init__ self.handle() File "/usr/local/lib/python2.7/dist-packages/Paste-1.7.5.1-py2.7.egg/paste/httpserver.py", line 442, in handle BaseHTTPRequestHandler.handle(self) File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle self.handle_one_request() File "/usr/local/lib/python2.7/dist-packages/Paste-1.7.5.1-py2.7.egg/paste/httpserver.py", line 437, in handle_one_request self.wsgi_execute() File "/usr/local/lib/python2.7/dist-packages/Paste-1.7.5.1-py2.7.egg/paste/httpserver.py", line 287, in wsgi_execute self.wsgi_start_response) File "/home/vagrant/jmchilton-lwr-5213f6dce32d/lwr/framework.py", line 35, in __call__ return controller(environ, start_response, **request_args) File "/home/vagrant/jmchilton-lwr-5213f6dce32d/lwr/framework.py", line 90, in controller_replacement result = func(**args) File "/home/vagrant/jmchilton-lwr-5213f6dce32d/lwr/app.py", line 81, in setup manager.setup_job_directory(job_id) File "/home/vagrant/jmchilton-lwr-5213f6dce32d/lwr/manager.py", line 101, in setup_job_directory os.mkdir(job_directory) OSError: [Errno 17] File exists: '/mnt/shared/000/128'

2 4

galaxysession cookie secure flag
by Vipin TS 06 May '13

06 May '13

Hi dev-team, We have placed our galaxy instance ssl and I need to make sure that the secure flag is set on the cookie (commonly represented by the word “secure” under the Security column) but I am not able to do the same. something like below: [image: Inline image 2] when I checked on my instance I saw as below: [image: Inline image 3] I have made necessary changes to my ssl.conf to put the flag as secure, but it seems not appearing here. Header edit Set-Cookie ^(.*)$ $1;Secure;HttpOnly does anybody have an experience in setting up the same. thanks in advance, --/Vipin

1 1

Any hint on how to use Galaxy within virtualenv?
by Jean-Francois Payotte 06 May '13

06 May '13

Dear Galaxy mailing list, We ran into the same issue described in this post: http://dev.list.galaxyproject.org/markupsafe-module-conflicts-tc4656214.html The exact error message we are getting, with a few different tools is: /Galaxy/galaxy_dist/lib/galaxy/__init__.py:79: UserWarning: Module markupsafe was already imported from /usr/lib64/python2.6/site-packages/markupsafe/__init__.pyc, but /Galaxy/galaxy_dist/eggs/MarkupSafe-0.12-py2.6-linux-x86_64-ucs4.egg is being added to sys.path self.check_version_conflict() >From the solution given in the above-mentioned post, I guess we would have to run Galaxy inside virtualenv. (Details of the procedure to install virtualenv, are given here: http://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer) However, I am not sure to understand completely how it works. Can we use our already existing Galaxy instance with virtualenv? And how can we achieve this? For the installation of "virtualenv" part, I think we are ok. It is the next step: how to tell Galaxy to run in this virtualenv, that I am not sure about. Thank you in advance for your help! Jean-Francois

2 4

May 2013 Galaxy Update
by Dave Clements 06 May '13

06 May '13

1 0

user creation using API
by Hagai Cohen 06 May '13

06 May '13

Hi, I would like to auto create users on my local Galaxy instance. I saw that in the "create user" on the API, I should enable use_remote_user (which as I understand means that I have to use Apache or nginx). Is there another way besides using Apache to create users automatically? Thanks, Hagai

3 3

(no subject)
by sridhar srinivasan 06 May '13

06 May '13

Dear All, i am installing galaxy first time. i face problem while installing the python eggs. while running the Run.sh command it gives as /home/sbsuser/.python-eggs/pysam-0.4.2_kanwei_b10f6e722e9a-py2.6-linux-x86_64-ucs4.egg-tmp/csamtools.so:6: RuntimeWarning: __builtin__.file size changed, may indicate binary incompatibility �b�n 640�c��?��U�#k�!\jZo�[&��[<��9-��ra#��8n��C�U�;B�SlK(p��U�&�P w 5 ≠ ' H è π ø * ─ ␋ Ð Ä @ ê i couldn't read the message and also i couldn't open the galaxy page http://localhost:8080 Could you please suggest tth epossibilities to install. Thanks in advance Regards Sridhar

2 2

Running galaxy using Apache as proxy
by Michael Place 04 May '13

04 May '13

Hello, I am trying to run galaxy using Apache as the proxy server. This is intended to be internal to our lab. I have galaxy running and working. The problem is that the pages load slowly and then seem to time out after 10 min or so. If I login and click get data then go away for a few minutes , then I come back and click a tool I get: The connection to the server was reset while the page was loading. If I retry the page loads. This makes the page very clunky. httpd.conf changes: <IfModule mod_proxy.c> ProxyRequests On <Proxy * > Order deny,allow Deny from all Allow from * </Proxy> <VirtualHost *:8080> ServerName 192.168.0.240 RewriteEngine on RewriteRule ^/galaxy$ /galaxy/ [R] RewriteRule ^/galaxy/static/style/(.*) /opt/galaxy-dist/static/june_2007_style/blue/$1 [L] RewriteRule ^/galaxy/static/scripts/(.*) /opt/galaxy-dist/static/scripts/packed/$1 [L] RewriteRule ^/galaxy/static/(.*) /opt/galaxy-dist/galaxy-dist/static/$1 [L] RewriteRule ^/galaxy/favicon.ico /opt/galaxy-dist/static/favicon.ico [L] RewriteRule ^/galaxy/robots.txt /opt/galaxy-dist/static/robots.txt [L] RewriteRule ^/galaxy(.*) http://localhost:8080$1 [P] </VirtualHost> I have attached the universe_wsgi.ini and my httpd.conf I am running CentOS. Thank you, Mike

2 1

Re: [galaxy-dev] [galaxy-user] Amazon EC2: An error occurred running this job: Job output not returned from cluster
by Brian Lin 04 May '13

04 May '13

I am unfortunately not suuuper sure how I fixed this issue, as I was doing some pretty bad troubleshooting techniques and changing a ton of things at once. There are two things I may be able to suggest. When you create the amazon instance, there is an option to use optimized EBS storage. There is a possibility that that option combined with the retry option>30 in my first email are able to solve the issue. If you haven't already, it would be worth it to try just adding that extra line, commit changes and restart the service afterwards. I noticed the first two times I tried modifying the ini file, the changes were not committed, so that may contribute to it not working. Good luck, Brian -- Brian Lin contact(a)brian-lin.com brian.lin(a)tufts.edu On Fri, May 3, 2013 at 4:33 PM, Dave Lin <dave(a)verdematics.com> wrote: > Dear Galaxy-Dev > > I was hoping to see if anybody had any suggestions to resolve this error. > > To summarize, I'm using cloudman/Amazon EC2. > I typically batch analyze 20-100 data sets against a workflow. (launched > serially using bioblend script) > > I'm consistently seeing the following error "An error occurred running > this job: Job output not returned from cluster" when I launch a large > number of samples. > > If I analyze the same data sets/workflow, but launch 5 at a time, the > analysis proceeds smoothly. > > Any pointers would be appreciated. > > Thanks > Dave > > > > > On Thu, May 2, 2013 at 1:51 PM, Dave Lin <dave(a)verdematics.com> wrote: > >> I am getting similar errors as Brian reported back in March. (Note, we >> appear to have the same last name, but no relation) >> >> An error occurred with this dataset: *Job output not returned from >> cluster* >> * >> * >> - Running on Cloudman with 5-6 nodes. (xlarge) >> - The error seems to occur consistently when I launch multiple workflows >> in batch (using bioblend) >> - Probably not relevant, but is failing on a BWA step. >> - I am able to run successfully the same workflow against one of the >> datasets that failed in batch. >> - Change-set is from Feb 8, 2013. 8794:1c7174911392. Stable branch. Prior >> to that, I was running different galaxy instances using changesets from >> last year and never ran into this problem. >> - I'm seeing errors like: >> galaxy.jobs.runners.drmaa WARNING 2013-05-02 17:07:51,991 Job output not >> returned from cluster: [Errno 2] No such file or directory\ >> : '/mnt/galaxyData/tmp/job_working_directory/002/2066/2066.drmec' >> - In this example, the /mnt/galaxyData/tmp/job_working_directory/002/2066 >> folder and /mnt/galaxyData/tmp/job_working_directory/002/2066/2066.drmec >> files do not exist. >> >> >> Any suggestions? Seems like this might be some type of resource >> contention issue, but I'm not sure where to investigate next. >> >> Thanks in advance, >> Dave >> >> >> >> On Mon, Mar 11, 2013 at 9:04 AM, Brian Lin <Brian.Lin(a)tufts.edu> wrote: >> >>> Hi guys, I'm running a galaxy cloudman instance and running the usual >>> tophat->cufflinks->cuffdiff workflow from RNAseq data. >>> I am using a m2.4xlarge as a master node, and autoscaling from 0-4 >>> workers of the m2.xlarge type. >>> I have gotten the error: An error occurred running this job: *Job >>> output not returned from cluster* >>> when running fasta groomer, tophat, and now cufflinks. >>> Following up troubleshooting from other people in the mailing list, I >>> have set a new line in universe_wsgi.ini of retry_job_output_collection=30 >>> Unfortunately, this does not seem to have fixed the problem. >>> The stdout is blank, and stderr gives Job output not returned from >>> cluster >>> >>> Under manage jobs in the admin panel, it lists 4 out of the 6 jobs as >>> currently running. What is confusing is that of the 4 "running," one has >>> already returned the error in the user dataset panel and yet is still >>> listed as running. >>> >>> From the SGE log, I see these errors: >>> >>> >>> 03/11/2013 14:32:52|worker|ip-10-159-47-223|W|job 42.1 failed on host ip-10-30-130-84.ec2.internal before writing exit_status because: shepherd exited with exit status 19: before writing exit_status >>> 03/11/2013 14:32:52|worker|ip-10-159-47-223|W|job 43.1 failed on host ip-10-30-130-84.ec2.internal before writing exit_status because: shepherd exited with exit status 19: before writing exit_status >>> 03/11/2013 14:39:41|worker|ip-10-159-47-223|E|adminhost "ip-10-30-130-84.ec2.internal" already exists >>> 03/11/2013 14:40:07|worker|ip-10-159-47-223|E|exechost "ip-10-30-130-84.ec2.internal" already exists >>> 03/11/2013 14:50:54|worker|ip-10-159-47-223|E|adminhost "ip-10-30-130-84.ec2.internal" already exists >>> 03/11/2013 14:50:55|worker|ip-10-159-47-223|E|exechost "ip-10-30-130-84.ec2.internal" already exists >>> >>> >>> Does anyone have any idea how to solve this error? It has removed my >>> ability to use workflows completely and I still have not been able to run a >>> single analysis to completion due to it. >>> >>> Thanks for any insight anyone provide! >>> >>> Brian >>> -- >>> Brian Lin >>> contact(a)brian-lin.com >>> brian.lin(a)tufts.edu >>> >>> ___________________________________________________________ >>> The Galaxy User list should be used for the discussion of >>> Galaxy analysis and other features on the public server >>> at usegalaxy.org. Please keep all replies on the list by >>> using "reply all" in your mail client. For discussion of >>> local Galaxy instances and the Galaxy source code, please >>> use the Galaxy Development list: >>> >>> http://lists.bx.psu.edu/listinfo/galaxy-dev >>> >>> To manage your subscriptions to this and other Galaxy lists, >>> please use the interface at: >>> >>> http://lists.bx.psu.edu/ >>> >> >> >

1 0

Re: [galaxy-dev] [galaxy-user] Amazon EC2: An error occurred running this job: Job output not returned from cluster
by Dave Lin 03 May '13

03 May '13

I am getting similar errors as Brian reported back in March. (Note, we appear to have the same last name, but no relation) An error occurred with this dataset: *Job output not returned from cluster* * * - Running on Cloudman with 5-6 nodes. (xlarge) - The error seems to occur consistently when I launch multiple workflows in batch (using bioblend) - Probably not relevant, but is failing on a BWA step. - I am able to run successfully the same workflow against one of the datasets that failed in batch. - Change-set is from Feb 8, 2013. 8794:1c7174911392. Stable branch. Prior to that, I was running different galaxy instances using changesets from last year and never ran into this problem. - I'm seeing errors like: galaxy.jobs.runners.drmaa WARNING 2013-05-02 17:07:51,991 Job output not returned from cluster: [Errno 2] No such file or directory\ : '/mnt/galaxyData/tmp/job_working_directory/002/2066/2066.drmec' - In this example, the /mnt/galaxyData/tmp/job_working_directory/002/2066 folder and /mnt/galaxyData/tmp/job_working_directory/002/2066/2066.drmec files do not exist. Any suggestions? Seems like this might be some type of resource contention issue, but I'm not sure where to investigate next. Thanks in advance, Dave On Mon, Mar 11, 2013 at 9:04 AM, Brian Lin <Brian.Lin(a)tufts.edu> wrote: > Hi guys, I'm running a galaxy cloudman instance and running the usual > tophat->cufflinks->cuffdiff workflow from RNAseq data. > I am using a m2.4xlarge as a master node, and autoscaling from 0-4 workers > of the m2.xlarge type. > I have gotten the error: An error occurred running this job: *Job output > not returned from cluster* > when running fasta groomer, tophat, and now cufflinks. > Following up troubleshooting from other people in the mailing list, I have > set a new line in universe_wsgi.ini of retry_job_output_collection=30 > Unfortunately, this does not seem to have fixed the problem. > The stdout is blank, and stderr gives Job output not returned from cluster > > Under manage jobs in the admin panel, it lists 4 out of the 6 jobs as > currently running. What is confusing is that of the 4 "running," one has > already returned the error in the user dataset panel and yet is still > listed as running. > > From the SGE log, I see these errors: > > > 03/11/2013 14:32:52|worker|ip-10-159-47-223|W|job 42.1 failed on host ip-10-30-130-84.ec2.internal before writing exit_status because: shepherd exited with exit status 19: before writing exit_status > 03/11/2013 14:32:52|worker|ip-10-159-47-223|W|job 43.1 failed on host ip-10-30-130-84.ec2.internal before writing exit_status because: shepherd exited with exit status 19: before writing exit_status > 03/11/2013 14:39:41|worker|ip-10-159-47-223|E|adminhost "ip-10-30-130-84.ec2.internal" already exists > 03/11/2013 14:40:07|worker|ip-10-159-47-223|E|exechost "ip-10-30-130-84.ec2.internal" already exists > 03/11/2013 14:50:54|worker|ip-10-159-47-223|E|adminhost "ip-10-30-130-84.ec2.internal" already exists > 03/11/2013 14:50:55|worker|ip-10-159-47-223|E|exechost "ip-10-30-130-84.ec2.internal" already exists > > > Does anyone have any idea how to solve this error? It has removed my > ability to use workflows completely and I still have not been able to run a > single analysis to completion due to it. > > Thanks for any insight anyone provide! > > Brian > -- > Brian Lin > contact(a)brian-lin.com > brian.lin(a)tufts.edu > > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ >

1 1