- galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] [galaxy-user] Amazon EC2: An error occurred running this job: Job output not returned from cluster
by Brian Lin 04 May '13

04 May '13

I am unfortunately not suuuper sure how I fixed this issue, as I was doing some pretty bad troubleshooting techniques and changing a ton of things at once. There are two things I may be able to suggest. When you create the amazon instance, there is an option to use optimized EBS storage. There is a possibility that that option combined with the retry option>30 in my first email are able to solve the issue. If you haven't already, it would be worth it to try just adding that extra line, commit changes and restart the service afterwards. I noticed the first two times I tried modifying the ini file, the changes were not committed, so that may contribute to it not working. Good luck, Brian -- Brian Lin contact(a)brian-lin.com brian.lin(a)tufts.edu On Fri, May 3, 2013 at 4:33 PM, Dave Lin <dave(a)verdematics.com> wrote: > Dear Galaxy-Dev > > I was hoping to see if anybody had any suggestions to resolve this error. > > To summarize, I'm using cloudman/Amazon EC2. > I typically batch analyze 20-100 data sets against a workflow. (launched > serially using bioblend script) > > I'm consistently seeing the following error "An error occurred running > this job: Job output not returned from cluster" when I launch a large > number of samples. > > If I analyze the same data sets/workflow, but launch 5 at a time, the > analysis proceeds smoothly. > > Any pointers would be appreciated. > > Thanks > Dave > > > > > On Thu, May 2, 2013 at 1:51 PM, Dave Lin <dave(a)verdematics.com> wrote: > >> I am getting similar errors as Brian reported back in March. (Note, we >> appear to have the same last name, but no relation) >> >> An error occurred with this dataset: *Job output not returned from >> cluster* >> * >> * >> - Running on Cloudman with 5-6 nodes. (xlarge) >> - The error seems to occur consistently when I launch multiple workflows >> in batch (using bioblend) >> - Probably not relevant, but is failing on a BWA step. >> - I am able to run successfully the same workflow against one of the >> datasets that failed in batch. >> - Change-set is from Feb 8, 2013. 8794:1c7174911392. Stable branch. Prior >> to that, I was running different galaxy instances using changesets from >> last year and never ran into this problem. >> - I'm seeing errors like: >> galaxy.jobs.runners.drmaa WARNING 2013-05-02 17:07:51,991 Job output not >> returned from cluster: [Errno 2] No such file or directory\ >> : '/mnt/galaxyData/tmp/job_working_directory/002/2066/2066.drmec' >> - In this example, the /mnt/galaxyData/tmp/job_working_directory/002/2066 >> folder and /mnt/galaxyData/tmp/job_working_directory/002/2066/2066.drmec >> files do not exist. >> >> >> Any suggestions? Seems like this might be some type of resource >> contention issue, but I'm not sure where to investigate next. >> >> Thanks in advance, >> Dave >> >> >> >> On Mon, Mar 11, 2013 at 9:04 AM, Brian Lin <Brian.Lin(a)tufts.edu> wrote: >> >>> Hi guys, I'm running a galaxy cloudman instance and running the usual >>> tophat->cufflinks->cuffdiff workflow from RNAseq data. >>> I am using a m2.4xlarge as a master node, and autoscaling from 0-4 >>> workers of the m2.xlarge type. >>> I have gotten the error: An error occurred running this job: *Job >>> output not returned from cluster* >>> when running fasta groomer, tophat, and now cufflinks. >>> Following up troubleshooting from other people in the mailing list, I >>> have set a new line in universe_wsgi.ini of retry_job_output_collection=30 >>> Unfortunately, this does not seem to have fixed the problem. >>> The stdout is blank, and stderr gives Job output not returned from >>> cluster >>> >>> Under manage jobs in the admin panel, it lists 4 out of the 6 jobs as >>> currently running. What is confusing is that of the 4 "running," one has >>> already returned the error in the user dataset panel and yet is still >>> listed as running. >>> >>> From the SGE log, I see these errors: >>> >>> >>> 03/11/2013 14:32:52|worker|ip-10-159-47-223|W|job 42.1 failed on host ip-10-30-130-84.ec2.internal before writing exit_status because: shepherd exited with exit status 19: before writing exit_status >>> 03/11/2013 14:32:52|worker|ip-10-159-47-223|W|job 43.1 failed on host ip-10-30-130-84.ec2.internal before writing exit_status because: shepherd exited with exit status 19: before writing exit_status >>> 03/11/2013 14:39:41|worker|ip-10-159-47-223|E|adminhost "ip-10-30-130-84.ec2.internal" already exists >>> 03/11/2013 14:40:07|worker|ip-10-159-47-223|E|exechost "ip-10-30-130-84.ec2.internal" already exists >>> 03/11/2013 14:50:54|worker|ip-10-159-47-223|E|adminhost "ip-10-30-130-84.ec2.internal" already exists >>> 03/11/2013 14:50:55|worker|ip-10-159-47-223|E|exechost "ip-10-30-130-84.ec2.internal" already exists >>> >>> >>> Does anyone have any idea how to solve this error? It has removed my >>> ability to use workflows completely and I still have not been able to run a >>> single analysis to completion due to it. >>> >>> Thanks for any insight anyone provide! >>> >>> Brian >>> -- >>> Brian Lin >>> contact(a)brian-lin.com >>> brian.lin(a)tufts.edu >>> >>> ___________________________________________________________ >>> The Galaxy User list should be used for the discussion of >>> Galaxy analysis and other features on the public server >>> at usegalaxy.org. Please keep all replies on the list by >>> using "reply all" in your mail client. For discussion of >>> local Galaxy instances and the Galaxy source code, please >>> use the Galaxy Development list: >>> >>> http://lists.bx.psu.edu/listinfo/galaxy-dev >>> >>> To manage your subscriptions to this and other Galaxy lists, >>> please use the interface at: >>> >>> http://lists.bx.psu.edu/ >>> >> >> >

1 0

Re: [galaxy-dev] [galaxy-user] Amazon EC2: An error occurred running this job: Job output not returned from cluster
by Dave Lin 03 May '13

03 May '13

I am getting similar errors as Brian reported back in March. (Note, we appear to have the same last name, but no relation) An error occurred with this dataset: *Job output not returned from cluster* * * - Running on Cloudman with 5-6 nodes. (xlarge) - The error seems to occur consistently when I launch multiple workflows in batch (using bioblend) - Probably not relevant, but is failing on a BWA step. - I am able to run successfully the same workflow against one of the datasets that failed in batch. - Change-set is from Feb 8, 2013. 8794:1c7174911392. Stable branch. Prior to that, I was running different galaxy instances using changesets from last year and never ran into this problem. - I'm seeing errors like: galaxy.jobs.runners.drmaa WARNING 2013-05-02 17:07:51,991 Job output not returned from cluster: [Errno 2] No such file or directory\ : '/mnt/galaxyData/tmp/job_working_directory/002/2066/2066.drmec' - In this example, the /mnt/galaxyData/tmp/job_working_directory/002/2066 folder and /mnt/galaxyData/tmp/job_working_directory/002/2066/2066.drmec files do not exist. Any suggestions? Seems like this might be some type of resource contention issue, but I'm not sure where to investigate next. Thanks in advance, Dave On Mon, Mar 11, 2013 at 9:04 AM, Brian Lin <Brian.Lin(a)tufts.edu> wrote: > Hi guys, I'm running a galaxy cloudman instance and running the usual > tophat->cufflinks->cuffdiff workflow from RNAseq data. > I am using a m2.4xlarge as a master node, and autoscaling from 0-4 workers > of the m2.xlarge type. > I have gotten the error: An error occurred running this job: *Job output > not returned from cluster* > when running fasta groomer, tophat, and now cufflinks. > Following up troubleshooting from other people in the mailing list, I have > set a new line in universe_wsgi.ini of retry_job_output_collection=30 > Unfortunately, this does not seem to have fixed the problem. > The stdout is blank, and stderr gives Job output not returned from cluster > > Under manage jobs in the admin panel, it lists 4 out of the 6 jobs as > currently running. What is confusing is that of the 4 "running," one has > already returned the error in the user dataset panel and yet is still > listed as running. > > From the SGE log, I see these errors: > > > 03/11/2013 14:32:52|worker|ip-10-159-47-223|W|job 42.1 failed on host ip-10-30-130-84.ec2.internal before writing exit_status because: shepherd exited with exit status 19: before writing exit_status > 03/11/2013 14:32:52|worker|ip-10-159-47-223|W|job 43.1 failed on host ip-10-30-130-84.ec2.internal before writing exit_status because: shepherd exited with exit status 19: before writing exit_status > 03/11/2013 14:39:41|worker|ip-10-159-47-223|E|adminhost "ip-10-30-130-84.ec2.internal" already exists > 03/11/2013 14:40:07|worker|ip-10-159-47-223|E|exechost "ip-10-30-130-84.ec2.internal" already exists > 03/11/2013 14:50:54|worker|ip-10-159-47-223|E|adminhost "ip-10-30-130-84.ec2.internal" already exists > 03/11/2013 14:50:55|worker|ip-10-159-47-223|E|exechost "ip-10-30-130-84.ec2.internal" already exists > > > Does anyone have any idea how to solve this error? It has removed my > ability to use workflows completely and I still have not been able to run a > single analysis to completion due to it. > > Thanks for any insight anyone provide! > > Brian > -- > Brian Lin > contact(a)brian-lin.com > brian.lin(a)tufts.edu > > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ >

1 1

Load balancing and job configuration
by Olivier Inizan 03 May '13

03 May '13

Dear galaxy-dev-list, I am trying to use new-style job configuration whith a load-balancing on 2 web servers and 2 job hanlders. I have configured load balancing as follows (universe_wsgi.ini): # Configuration of the internal HTTP server. [server:web0] use = egg:Paste#http port = 8083 host = 127.0.0.1 use_threadpool = true threadpool_workers = 7 [server:web1] use = egg:Paste#http port = 8082 host = 127.0.0.1 use_threadpool = true threadpool_workers = 7 [server:manager] use = egg:Paste#http port = 8079 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler0] use = egg:Paste#http port = 8090 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler1] use = egg:Paste#http port = 8091 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 This configuration works fine (manager.log): galaxy.jobs.manager DEBUG 2013-04-23 12:20:22,901 Starting job handler galaxy.jobs.runners DEBUG 2013-04-23 12:20:22,902 Starting 5 LocalRunner workers galaxy.jobs DEBUG 2013-04-23 12:20:22,904 Loaded job runner 'galaxy.jobs.runners.local:LocalJobRunner' as 'local' galaxy.jobs.runners DEBUG 2013-04-23 12:20:22,909 Starting 3 LWRRunner workers galaxy.jobs DEBUG 2013-04-23 12:20:22,911 Loaded job runner 'galaxy.jobs.runners.lwr:LwrJobRunner' as 'lwr' galaxy.jobs DEBUG 2013-04-23 12:20:22,911 Legacy destination with id 'local:///', url 'local:///' converted, got params: galaxy.jobs.handler DEBUG 2013-04-23 12:20:22,911 Loaded job runners plugins: lwr:local galaxy.jobs.handler INFO 2013-04-23 12:20:22,912 job handler stop queue started galaxy.jobs.handler INFO 2013-04-23 12:20:22,919 job handler queue started To use new-style job configuration I have create the following job_conf.xml: <?xml version="1.0"?>  <job_conf> <plugins> <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner"/> </plugins> <handlers default="handlers">  <handler id="server:web0" tags="handlers"/> <handler id="server:web1" tags="handlers"/> </handlers> <destinations default="local"> <destination id="local" runner="local"/> </destinations> </job_conf> When I restart the instance the manager.log outputs: galaxy.jobs DEBUG 2013-05-02 17:25:28,402 Loading job configuration from ./job_conf.xml galaxy.jobs DEBUG 2013-05-02 17:25:28,402 Read definition for handler 'server:web0' galaxy.jobs DEBUG 2013-05-02 17:25:28,403 Read definition for handler 'server:web1' galaxy.jobs DEBUG 2013-05-02 17:25:28,403 <handlers> default set to child with id or tag 'handlers' galaxy.jobs DEBUG 2013-05-02 17:25:28,403 <destinations> default set to child with id or tag 'local' galaxy.jobs DEBUG 2013-05-02 17:25:28,403 Done loading job configuration So everything seems fine but when I try to run a tool, the corresponding job is in permanent waiting status (gray color in history). I think I have missed something in the configuration process, Thanks for your help, Olivier

2 2

Quick question about new job configuration
by Ganote, Carrie L 03 May '13

03 May '13

Hi folks, First of all, I really love the new job configuration style. It makes it very easy to change swaths of jobs and to streamline the way jobs are submitted to the cluster. I had no issues with the dynamic runner and I like the way that tags interact. My question is for the dynamic job runner; is it possible to send the job to a different job *handler* dynamically? Or is it already given to the default handler at the time that the dynamic function is called? I want to do something like this: def ncbi_blastn_wrapper(job): # Send to a different handler? inp_data = dict( [ ( da.name, da.dataset ) for da in job.input_datasets ] ) inp_data.update( [ ( da.name, da.dataset ) for da in job.input_library_datasets ] ) query_file = inp_data[ "query" ].file_name query_size = os.path.getsize( query_file ) if query_size > 1024 * 1024: handlername = "bighandler" else: handlername = "smallhandler" return JobDestination(runner="pbs", handler=handlername, params={"Resource_List": "24:00:00"}) The way I have it set up now, my handlers are pointing at different clusters with different capabilities, and it would be cool to make the decision which to run at runtime. Thanks a ton! Carrie Ganote

1 0

workflow multiple inputs
by Olivia Doppelt 03 May '13

03 May '13

Dear developpers, Last week we made an update on the default branch of galaxy-dist. Some problems concerning the api were detected and resolved thanks to Dannon Baker by pulling the stable branch. However, a very interesting fonctionality was working well on this default branch, concerning the multiple inputs for workflows. There were two icons next to the input dataset window, a multiple file icon and a link icon to link paired input. Why is this functionnality not included in the stable branch ? isn't stable yet? It is a very useful though. Please keep me informed about that. Cheers, Olivia Doppelt-Azeroual Pasteur Institute, CIB, PARIS

2 1

Re: [galaxy-dev] [Biopython] GFF parsing with biopython
by Brad Chapman 03 May '13

03 May '13

Mic; (moving to galaxy-dev list so folks there can follow, but future questions are more appropriate for the Biopython list only since this isn't a Galaxy question) > I have the following GFF file from a SNAP > > X1 SNAP Einit 2579 2712 -3.221 + . X1-snap.1 [...] > With the code below I have tried to parse the above GFF file The attributes you're missing are parts of the feature, not the SeqRecord itself, which is why you're seeing attribute error. Here's a full example that pulls all of the information from an example line: from BCBio import GFF in_file = "snap.gff" with open(in_file) as in_handle: for rec in GFF.parse(in_handle): feature = rec.features[0] print rec.id print feature.qualifiers["source"][0] print feature.type print feature.location.start print feature.location.end print feature.qualifiers["score"][0] print feature.location.strand print feature.qualifiers.get("X1-snap.1", [None])[0] which outputs: X1 SNAP Einit 2578 2712 -3.221 1 true Hope this helps, Brad

2 1

Problem accessing test tool shed repositories
by Sytchev, Ilya 02 May '13

02 May '13

Hi, I can't install any tools from the test tool shed into the local Galaxy instances. I can see the list of the repositories but nothing happens when I click on any of them. I can still access repositories in the main tool shed. The same thing happens in fresh installations of both galaxy-central and galaxy-dist. Attached screenshot shows a list of Sequence Analysis repositories in the test tool shed (note the lack of buttons). I'd appreciate any help. Thanks, Ilya

2 2

MPileup for cloudman instance
by Iry Witham 02 May '13

02 May '13

Hi All, I have noticed that there is no Mpileup available in either of the tool sheds. Is there a simple way to install it in my instance? Thanks, Iry The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.

1 0

Tests for Workflows on the tool shed?
by Peter Cock 02 May '13

02 May '13

Hello all, I'm looking at how best to incorporate Galaxy workflows into a publication - a standalone *.ga file in supplementary materials would probably work but is static and I fear likely to bit rot. I think posting each workflow (or set of workflows) as a repository on the main Galaxy Tool Shed would be better, as I can continue to update it as needed, and it can be linking to it via the citable URL, e.g. Greg's example on the Test Tool Shed which declares some dependencies: http://testtoolshed.g2.bx.psu.edu/view/greg/heteroplasmy_workflow (This assumes the Tool Shed URLs will be stable, but I'm having to assume that anyway for tool wrappers.) However, my natural next question is how do we define unit tests for these workflows? In the case of a methods paper, some sample data for testing would be reasonable. In the case of a more applied paper, the workflow could perhaps include the actual data used in the paper and reproduce the published results (ambitious but quite possible for moderate sized datasets). I'm guessing this would be a major enhancement - something for Greg and Dave to tackle once the current round of work for testing tools on the Tool Shed is done (grin). Shall I open a Trello issue for this? Regards, Peter

1 0

toolshed down?
by Geert Vandeweyer 02 May '13

02 May '13

hi, I'm getting the following error when trying to reach the main/test toolsheds. Are they down? **** This webpage is not available Chromium's connection attempt to toolshed.g2.bx.psu.edu was rejected. The website may be down, or your network may not be properly configured. **** also from within galaxy, I can't reach it, nor can the galaxy-process itself: galaxy.model.migrate.check INFO 2013-05-02 12:12:45,219 At database version 114 tool_shed.galaxy_install.migrate.check DEBUG 2013-05-02 12:12:45,390 pysqlite>=2 egg successfully loaded for sqlite dialect The URL http://toolshed.g2.bx.psu.edu/repository/get_tool_dependencies?name=bowtie_… raised the exception: <urlopen error [Errno 111] Connection refused> The URL http://toolshed.g2.bx.psu.edu/repository/get_tool_dependencies?name=bowtie_… raised the exception: <urlopen error [Errno 111] Connection refused> The URL http://toolshed.g2.bx.psu.edu/repository/get_tool_dependencies?name=lastz&o… raised the exception: <urlopen error [Errno 111] Connection refused> The URL http://toolshed.g2.bx.psu.edu/repository/get_tool_dependencies?name=lastz_p… raised the exception: <urlopen error [Errno 111] Connection refused> Traceback (most recent call last): File "/galaxy/galaxy-beta/galaxy-dist/lib/galaxy/webapps/galaxy/buildapp.py", line 37, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File "/galaxy/galaxy-beta/galaxy-dist/lib/galaxy/app.py", line 54, in __init__ verify_tools( self, db_url, kwargs.get( 'global_conf', {} ).get( '__file__', None ), self.config.database_engine_options ) File "/galaxy/galaxy-beta/galaxy-dist/lib/tool_shed/galaxy_install/migrate/check.py", line 56, in verify_tools tool_shed_accessible, missing_tool_configs_dict = common_util.check_for_missing_tools( app, tool_panel_configs, latest_tool_migration_script_number ) File "/galaxy/galaxy-beta/galaxy-dist/lib/tool_shed/util/common_util.py", line 66, in check_for_missing_tools return tool_shed_accessible, missing_tool_configs_dict UnboundLocalError: local variable 'missing_tool_configs_dict' referenced before assignment -- Geert Vandeweyer, Ph.D. Department of Medical Genetics University of Antwerp Prins Boudewijnlaan 43 2650 Edegem Belgium Tel: +32 (0)3 275 97 56 E-mail: geert.vandeweyer(a)ua.ac.be http://ua.ac.be/cognitivegenetics http://www.linkedin.com/pub/geert-vandeweyer/26/457/726

2 2