- galaxy-dev - lists.galaxyproject.org

Galaxy Server Processes Dying?
by Dan Sullivan 10 Jul '13

10 Jul '13

Hi, Galaxy Developers, I have what I'm hoping is a fairly simple inquiry for the Galaxy community; basically, our production Galaxy server processes appear to be dying off over time. Our production Galaxy instance implements apache web scaling features so I have a number of server processes, for example my apache Apache configuration has: BalancerMember http://127.0.0.1:8080 BalancerMember http://127.0.0.1:8081 BalancerMember http://127.0.0.1:8082 BalancerMember http://127.0.0.1:8083 BalancerMember http://127.0.0.1:8084 BalancerMember http://127.0.0.1:8085 Nothing unconventional as I understand it. Similarly, my galaxy config has matching [server:ws3], [server:ws2] configuration blocks for each of these processes. When I restart Galaxy, everything is all fine and good. I'll see a server listening on each one of these ports (if I do something like lsof -i TCP -P, for example). What appears to be happening, is that for whatever reason, these server processes seem to die off over time (i.e eventually nothing is listening on ports 8080-8085). This process can take days, and at the time when no servers are available, Apache will begin throwing 503 service unavailable errors. I am fairly confident this process is gradual, for example I just checked now and the Galaxy was still available, however one server had died (the one on TCP port 8082). I do do have a single separate job manager and two job handlers; at this point I believe this problem to be related to the servers only (i.e. the job manager and job handlers do not appear to be crashing). Now, I believe that late last week I might have 'caught' the last server process dying, just by coincidence, although I am not 100% certain. Here is the Traceback as it occurred: galaxy.jobs.runners.pbs DEBUG 2013-07-02 08:47:12,011 (6822/39485.sc01) PBS job state changed from Q to R galaxy.jobs.runners.pbs DEBUG 2013-07-02 08:54:36,565 (6822/39485.sc01) PBS job state changed from R to C galaxy.jobs.runners.pbs DEBUG 2013-07-02 08:54:36,566 (6822/39485.sc01) PBS job has completed successfully galaxy.jobs DEBUG 2013-07-02 08:54:36,685 Tool did not define exit code or stdio handling; checking stderr for success galaxy.datatypes.metadata DEBUG 2013-07-02 08:54:36,812 loading metadata from file for: HistoryDatasetAssociation 6046 galaxy.jobs DEBUG 2013-07-02 08:54:38,153 job 6822 ended galaxy.jobs.runners.pbs DEBUG 2013-07-02 08:54:49,130 (6812/39473.sc01) PBS job state changed from R to E galaxy.jobs.runners.pbs DEBUG 2013-07-02 08:54:52,267 (6812/39473.sc01) PBS job state changed from E to C galaxy.jobs.runners.pbs ERROR 2013-07-02 08:54:52,267 (6812/39473.sc01) PBS job failed: Unknown error: -11 galaxy.jobs.runners ERROR 2013-07-02 08:54:52,267 (unknown) Unhandled exception calling fail_job Traceback (most recent call last): File "/group/galaxy/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 58, in run_next method(arg) File "/group/galaxy/galaxy-dist/lib/galaxy/jobs/runners/pbs.py", line 560, in fail_job if pbs_job_state.stop_job: AttributeError: 'AsynchronousJobState' object has no attribute 'stop_job' Now, I have some questions regarding this issue; 1) It appears to me that although this is a sub-optimal solution, restarting Galaxy solves this problem (i.e. server processes will be listening after restarting Galaxy). Is it possible, or safe, or sane to just restart a single server on a singe port? Ideally I would actually like to fix the problem that is causing my server processes to crash, although I figured it wouldn't hurt to ask this question regardless. 2) Similar to the question above, is it possible to configure Galaxy in a way that server processes re-spawn in a self-service manner (i.e. is this a feature of Galaxy, for example, because server processes dying regularly is either a known issue or expected and tolerable (but undesired) behaivor)? 3) To me, the error messages above aren't very meaningful, other than the Traceback appears to be PBS-related. Would anybody be able comment on the problem above (i.e. have you seen something like this), or comment on Galaxy server processes dying in general? I have done some brief searching of the Galaxy mailing list for server crashes and did not find anything suggesting this is a common problem. 4) I am not 100% confident at this point that the Traceback above is what killed the server process. Does anybody know of a specific string I can search for (a literal) to identify when a server process actually dies? I have done some basic review of log data (our Galaxy server generates lots of logs), and Traceback does not appear to be a valid string to uniquely identify a server crash (they occur too frequently). I currently have logging configured at DEBUG. In case this is relevant, I am using the following change set for Galaxy: > hg parents changeset: 9320:47ddf167c9f1 branch: stable tag: tip user: Nate Coraor <nate(a)bx.psu.edu> date: Wed May 01 09:50:31 2013 -0400 summary: Use Galaxy's ErrorMiddleware since Paste's doesn't return start_response. Fixes downloading tarballs from the Tool Shed when use_debug = false. > I appreciate the time you took in reading my email, and any expertise you could provide in helping me troubleshoot this issue. Dan Sullivan

3 2

datatype display_data function
by Himanshu Gupta 10 Jul '13

10 Jul '13

Hello everyone, I had been using an old galaxy build, downloaded in 2010, on my local server and now I am updating it to latest build. I had made some changes as per our need. Along with the output file I have added a piece of code which was making us able to download the stderr file as well in case. I had made those changes inside the display function of lib/galaxy/webapps/galaxy/controllers/dataset.py file. But galaxy have gone through significant changes. Now the display function of lib/galaxy/webapps/galaxy/controllers/dataset.py calls display_data function. def display(self, trans, dataset_id=None, preview=False, filename=None, to_ext=None, chunk=None, **kwd): ... return data.datatype.display_data(trans, data, preview, filename, to_ext, chunk, **kwd) display_data function is present in lib/galaxy/datatypes/data.py as well as in lib/galaxy/datatypes/tabular.py The code which I had changed in previous galaxy is present in ..../data.py but the display function is calling the display_data function of .../tabular.py always. How will I make it able to call display_data function of ..../data.py Please tell me the solution of the above problem or is there any better way to download some more file like stdout, stderr apart from the regular output file. Any help would be really appreciated. P.S. I am creating stderr and stdout file inside database/user_directory Thanks Himanshu

2 1

Setting up reference genomes for GATK Genotyper
by Marco Ocana 10 Jul '13

10 Jul '13

Hi, I am running an instance of Galaxy that I setup using Cloudman. We are trying to use the GATK Unified Genotyper, but we are missing the reference genomes. I saw in your website instructions on how to setup reference genomes for some other tools, but not for the Unified Genotyper. Where can I find instructions on how to setup reference genomes for this tool? Thanks Marco

1 1

trouble with share string
by Deniz Erezyilmaz 10 Jul '13

10 Jul '13

After 1 year of using the same share string at the launch page * https://biocloudcentral.herokuapp.com/launch, *the program no longer loads my application; I. The instance loads, but my galaxyData folder is empty. I am not sure if the problem has occurred because Galaxy has moved something or whether I have deleted something critical. Any ideas? Thanks

1 0

NFS noac option
by Timothy Carr 10 Jul '13

10 Jul '13

Hi All, I've setup the Galaxy application on a standard NFS mount without the 'noac' option and then the galaxy datasets on a enabled 'noac' NFS mount. I updated the universe configuration to reflect the updated locations. The performance however to the 'noac' mount is really poor, something like 700kb/sec on a 1GB interface. Is this the only option available or have alternatives been recommended to improve the speed ? Regards Timothy Carr University of Cape Town

1 0

remove already selected options from selection menu
by Tim-Christoph Roloff 10 Jul '13

10 Jul '13

Hi, I am working on a tool that allows to filter on a selected set of contrasts. The contrast names are read from the second column of a history item. In order to allow selection of individual filters for each contrast, I am using the <repeat> tag. Now I would like to make sure that the user selects each contrast at max one time. Can I use the <filter ...> to remove the contrasts that have already been chosen from the selection? Or is there a more elegant way? <repeat name="contrasts" title="Contrast" min="1" max="3"> <param name="contrast" type="select" label="Contrast to filter on" multiple="false"> <options from_dataset="edgercont"> <column name="name" index="1"/> <column name="value" index="1"/> </options> </param> <param name="lfc" type="float" label="Linear fold-change" value="2.0" /> <param name="pval" type="float" label="p-value cutoff" value="0.001" /> </repeat> Thanks for your help in advance Tim

2 3

no metadata for tools in toolshed
by Ido Tamir 10 Jul '13

10 Jul '13

Hi, I grabbed some xml files from Hannons website for his fastx toolkit (improved over galaxys default version). Then I created a repository, uploaded the xml files, but the toolshed complains that it could not find the tools for creating metadata in the xml files. Could somebody please tell me what the problem is with these files? They look like valid tool.xml files to me. http://toolshed.g2.bx.psu.edu/view/idot/fastx_toolkit2 http://idot@toolshed.g2.bx.psu.edu/repos/idot/fastx_toolkit2 thank you very much, ido

2 1

Configuring the time of permanently deleting data sets.
by Perez, Ricardo 10 Jul '13

10 Jul '13

Dear all, I am currently trying to find a way to decrease the time the users need to wait to permanently delete their data sets. In galaxy I read the following "Histories that have been deleted for more than a time period specified by the Galaxy administrator(s) may be permanently deleted."; I am wondering where can I configure this option in my local instance of galaxy. I have looked for such an option in the universe_wsgi.ini file but I have not found it. I would greatly appreciate if someone can point me to the right direction to modify this. Thank you all, --Ricardo Perez

3 2

Tool Error: Failure Preparing Job
by Venice Margarette Juanillas 10 Jul '13

10 Jul '13

Hi all, I'm working on a local Galaxy instance and I want to integrate TASSEL's GLM pipeline into it. But when I run the tool, it prompts: "tool error An error occurred running this job:* failure preparing job*" This is how the log looks like: galaxy.jobs.handler INFO 2013-07-08 16:00:47,648 (73) Job dispatched galaxy.jobs.runners.local DEBUG 2013-07-08 16:00:47,735 Local runner: starting job 73 galaxy.jobs.runners.local ERROR 2013-07-08 16:00:47,939 failure running job 73 Traceback (most recent call last): File "/storage/irri-gsl-galaxy/lib/galaxy/jobs/runners/local.py", line 72, in run_job job_wrapper.prepare() File "/storage/irri-gsl-galaxy/lib/galaxy/jobs/__init__.py", line 184, in prepare self.command_line = self.tool.build_command_line( param_dict ) File "/storage/irri-gsl-galaxy/lib/galaxy/tools/__init__.py", line 2435, in build_command_line command_line = fill_template( self.command, context=param_dict ) File "/storage/irri-gsl-galaxy/lib/galaxy/util/template.py", line 9, in fill_template return str( Template( source=template_text, searchList=[context] ) ) UnicodeEncodeError: 'ascii' codec can't encode character u'\u2013' in position 23: ordinal not in range(128) galaxy.datatypes.metadata DEBUG 2013-07-08 16:00:48,206 Cleaning up external metadata files 127.0.0.1 - - [08/Jul/2013:16:00:50 +0800] "GET /api/histories/846c4efa2e1f970a HTTP/1.1" 200 - " http://localhost:8081/history?status=done&show_deleted=False&filename=None&…" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0" I figured the problem is that UnicodeEncodeError but I don't have an idea how to fix this or where to begin. I do hope you can help me. Thanks and Best Regards, Venice Margarette B. Juanillas

2 1

allow users for specifying the number of CPUs
by Keilwagen, Jens 10 Jul '13

10 Jul '13

Hi guys, we are currently trying to integrate CLC Bios short read assembler into our local Galaxy version. So far everything went smoothly. However, the command line tool allows for specifying the number of CPU that should be used for computation. We are wondering, whether it is also possible to enable Galaxy users also to specify the number of CPUs. Thanks for any help. Best regards, Jens

3 2