galaxy webserver jobs on a cluster
by Laure QUINTRIC
Hello,
We are facing some difficulties making the link between galaxy webserver
and our cluster.
As galaxy jobs are launched interactively with the user connected on
galaxy webserver on this webserver, how is it possible to use PBS qsub
command installed on the cluster to perform the job ?
Does PBS qsub has to be installed on the webserver also ?
In Galaxy documentation, there're the lines below :
"The underlying method for execution is “pluggable”. Currently jobs can
be executed on the same machine where the GALAXY instance is running, or
dispatched to a computational cluster using a standard queue manager
(portable batch system [PBS] is used at http://main.g2.bx.psu.edu),
however other dispatch strategies can be implemented and plugged in easily"
Can you give more information about these other dispatch strategies ?
Thanks for your help.
Laure
--
Laure QUINTRIC
Ifremer - IDM/RIC
Technopôle Brest-Iroise
29280 Plouzané
Tel: 02 98 22 49 11 - Fax : 02 98 22 46 47
E-mail : Laure.Quintric(a)ifremer.fr
10 years, 9 months
Upload file button
by Zhe Chen
Hi,
I am creating a tool to convert format. The tool will read from the
original file, convert the format, then output.
Should I put a "Browser" button like the example tool "Get Data -> Upload
File" or I should put a simple input textbox, let the user paste the input
file path?
You can see the original tool here:
http://hiv-dev.lanl.gov:8081/content/sequence/FORMAT_CONVERSION/form.html
Simply speaking, how can I implement the "Upload File" button. I try to add
<param name="filePath" type="file" size="30" label="Or upload your
file:" ajax-upload="true"/>
to the xml file of the tool. The UI looks right, but when I click
"execute", nothing happens.
Thanks
10 years, 9 months
Problem offloading upload with nginx
by Dave Walton
Good Morning,
I have a galaxy instance at our institution, and we are trying to make the file upload with nginx work.
It appears that when I do a file download, it is getting handled by nginx (running top I see an nginx process appear, do work and disappear when the file is done downloading). However, when I try to do an upload, I see additional python processes appear in top, and I see the upload.py script go by in my runner0.log file, but I don't see an nginx process appear in top. Also, when I go to run several large uploads in a row, and then attempt to run some other job (like a sort) it blocks while waiting for the files to upload. This leads me to believe the upload is running in galaxy, not nginx.
I am also running "watch ls -ltr upload_store" from galaxy-dist/database/tmp while the upload is occurring and I never see anything written to the upload temp directory.
We are running galaxy on a 4 cpu server, which is running SUSE Linux Enterprise Server 11 (x86_64) for an os.
I have galaxy configured to run 3 webapps, and 1 job runner.
We have nginx installed and built with the nginx_upload_module (detail below)
jaxgalaxydev01:~ # /usr/local/nginx/sbin/nginx -V
nginx version: nginx/0.8.53
built by gcc 4.3.2 [gcc-4_3-branch revision 141291] (SUSE Linux)
configure arguments: --prefix=/usr/local/nginx-0.8.53 --add-module=/usr/local/src/nginx_upload_module-2.2.0
In both my universe_wsgi.webapp.ini and universe_wsgi.runner.ini I have the following lines defined in my [app:main] section:
# Directive for using nginx to do file downloads
nginx_x_accel_redirect_base = /_x_accel_redirect
# Directives to allow nginx to deal with file uploads
nginx_upload_store = database/tmp/upload_store
nginx_upload_path = /_upload
Below is the entire contents of my nginx.conf file:
worker_processes 1;
events {
worker_connections 1024;
}
user galaxy;
http {
include mime.types;
default_type application/octet-stream;
# Added for compression and caching. According to Galaxy wiki page:
# http://bitbucket.org/galaxy/galaxy-central/wiki/Config/nginxProxy
# This will decrease downlod and page load times for clients
gzip on;
gzip_http_version 1.1;
gzip_vary on;
gzip_comp_level 4;
gzip_proxied any;
gzip_types text/plain text/css application/x-javascript text/xml application/xml text/javascript application/json;
gzip_buffers 16 8k;
gzip_disable "MSIE [1-6].(?!.*SV1)";
sendfile on;
keepalive_timeout 65;
# Added to support proxying to galaxy server
upstream galaxy_app {
server localhost:8080;
server localhost:8081;
server localhost:8082;
}
server {
listen 80;
server_name localhost;
# Line added for galaxy support
client_max_body_size 10G;
# Added to allow nginx to handle file uploads
location /_upload {
upload_store /galaxydata/galaxy-setup/galaxy-dist/database/tmp/upload_store;
upload_pass_form_field "";
upload_set_form_field "__${upload_field_name}__is_composite" "true";
upload_set_form_field "__${upload_field_name}__keys" "name path";
upload_set_form_field "${upload_field_name}_name" "$upload_file_name";
upload_set_form_field "${upload_field_name}_path" "$upload_tmp_path";
upload_pass_args on;
upload_pass /_upload_done;
}
location /_upload_done {
set $dst /tool_runner/index;
if ($args ~ nginx_redir=([^&]+)) {
set $dst $1;
}
rewrite "" $dst;
}
# added to allow nginx to handle file downloads
location /_x_accel_redirect/ {
internal;
alias /;
}
# The following series of "locations" are for off-loading static
# content from the galaxy server.
location /static {
alias /galaxydata/galaxy-setup/galaxy-dist/static;
# This will decrease downlod and page load times for clients
expires 24h;
}
location /static/style {
alias /galaxydata/galaxy-setup/galaxy-dist/static/june_2007_style/blue;
}
location /static/scripts {
alias /galaxydata/galaxy-setup/galaxy-dist/static/scripts/packed;
# This will decrease downlod and page load times for clients
expires 24h;
}
location /favicon.ico {
alias /galaxydata/galaxy-setup/galaxy-dist/static/favicon.ico;
}
location /robots.txt {
alias /galaxydata/galaxy-setup/galaxy-dist/static/robots.txt;
}
# End of static content
location / {
proxy_pass http://galaxy_app;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
I have restarted both my galaxy server and my nginx server to confirm that the configurations above have been picked up.
I would be interested in:
1) any other suggestions to confirm where the upload is occurring (assuming I'm mistaken about it running in galaxy).
2) any insight as to how I might have my server(s) misconfigured, in order to get upload working in nginx.
Any assistance or suggestions you could provide would be greatly appreciated.
Thanks,
Dave
10 years, 9 months
Search tools button
by Zhe Chen
Hi,
On the left tool list panel of Galaxy website. On the top, there is a
"options" button, when I click it, it shows a "Search Tools". How can I
get it on my own local Galaxy server? On my local one, when I click
"Options" button, the "Search Tools" button does not show up.
Thanks
10 years, 9 months
timeout for running a tool?
by Kostas Karasavvas
Hi all!
Is there such a timeout setting? I use different parameters (which take much
longer) and I get an empty dataset (no errors in stdout/err and paster.log).
Normally, for smaller runs, the tool works fine.
A quick look in the conf files seems to indicate that the answer to the
above is no.
Any ideas/suggestions?
Thank you,
Kostas
10 years, 9 months
free little cloud for a year anyone?
by Ross
In case you haven't seen this, from November 1 http://aws.amazon.com/free/
New customers only. Must provide a valid CC and you'll pay for all use over
the free tier limits - 12 months of :
AWS Free Usage Tier (Per Month):
- 750 hours of Amazon EC2 <http://aws.amazon.com/ec2> Linux Micro
Instance usage (613 MB of memory and 32-bit and 64-bit platform support) –
enough hours to run continuously each month*
- 750 hours of an Elastic Load
Balancer<http://aws.amazon.com/elasticloadbalancing/> plus
15 GB data processing*
- 10 GB of Amazon Elastic Block Storage <http://aws.amazon.com/ebs>, plus
1 million I/Os, 1 GB of snapshot storage, 10,000 snapshot Get Requests and
1,000 snapshot Put Requests*
- 5 GB of Amazon S3 storage <http://aws.amazon.com/s3>, 20,000 Get
Requests, and 2,000 Put Requests*
- 30 GB per of internet data transfer (15 GB of data transfer “in” and 15
GB of data transfer “out” across all services except Amazon CloudFront)*
- 25 Amazon SimpleDB <http://aws.amazon.com/simpledb> Machine Hours and 1
GB of Storage**
- 100,000 Requests of Amazon Simple Queue Service<http://aws.amazon.com/sqs>
**
- 100,000 Requests, 100,000 HTTP notifications and 1,000 email
notifications for Amazon Simple Notification
Service<http://aws.amazon.com/sns>
**
In addition to these services, the AWS Management
Console<http://aws.amazon.com/console> is
available at no charge to help you build and manage your application on AWS.
10 years, 9 months
Log rotation tips
by Ry4an Brase
I set up logrotate on our production galaxy instance here, but after
moving the file Galaxy (or more likely paste I imagine) doesn't re-open
the log file. Is there some post-rotate action I can set in my
logrotate command to let galaxy know that I pulled the logging rug out
from under it? On apache this would be a SIGHUP, but some daemons use
SIGUSRs of various flavors.
How are others handling logs that grow without bounds?
--
Ry4an Brase 612-626-6575
Software Developer Application Development
University of Minnesota Supercomputing Institute http://www.msi.umn.edu
10 years, 9 months
quick question on building tool's interface...
by Kostas Karasavvas
Hi all,
When building a tool's interface would it be possible to have a different
<help></help> tag for different <param type="select"> options of a
'conditional' tag?
The options change the tool parameters significantly and having a big <help>
tag that includes explanation of all parameters of all the options would be
a bit confusing.
Thank you,
Kostas
PS. I had some issues with an email alias. Sorry if you got duplicates!
10 years, 9 months
More on variable multiple output files: identification
by Al Simons
Thanks to the list's help, I have my multiple output files. The next problem is that the displayed descriptions all have the identical text as the "master" (first output) file. I need to know which is which. The data in each file that my tool is indistinguishable, so I can't label the data internally. I need to precisely track which resultant dataset corresponds to which of the program's output streams. I can't even dead-reckon, because the datasets appear to be randomly ordered with respect to either the file's creation time or its original name.
It would be _really_ nice if the output name extracted from the file name could be included in (or just be) the dataset name displayed in the history. That would give the program control of the displayed name, which would be extremely helpful.
I see a tantalizing clue at the bottom of the ToolsMultipleOutput page, to reference "__collected_datasets__['primary'][name]." in a post processing hook. However, I can't find any reference to post processing hooks in the wiki or on Google. I'd welcome a pointer to any documentation about post-processing. Is there a documentation section that I'm overlooking?
Thanks for the support,
-Al
10 years, 9 months
integer or float parameters - regression on trunk?
by Peter
Hi all,
I've found what I consider to be a regression from Daniel Blankenberg's change:
> http://bitbucket.org/galaxy/galaxy-central/changeset/1f4d6b25a84e
>
> Fix for Integer/FloatToolParameter.get_html_field() when 'value' is provided
> as an integer/float. Fixes an issue seen when saving workflows: If an integer
> or float tool parameter is changed to a value of 0 or 0.0 and saved, the form
> field would be redisplayed using the default tool value; and not the value that
> is now saved in the database.
If have a tool parameter like this:
<param name="dummy" type="integer" value="" label="Test integer with
empty default"/>
Prior to Dan's change, the HTML would show with the value as specified
(empty string). After Dan's change, the HTML page for the tool loads with
"0" shown, which is not what I wanted (we can get that with value="0").
I think the old functionality is more useful - couple it with a
validator and this
forces the user to pick a value rather than always having a valid default.
However, what I really want to include optional="true" value="" and allow the
value to be an integer or the empty string. In this use-case the HTML should
start with the given default shown (empty string as value). See also:
http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-October/003455.html
Does that make sense?
Regards,
Peter
10 years, 9 months
