October 2010 - galaxy-dev - lists.galaxyproject.org

galaxy webserver jobs on a cluster
by Laure QUINTRIC 27 Oct '10

27 Oct '10

Hello, We are facing some difficulties making the link between galaxy webserver and our cluster. As galaxy jobs are launched interactively with the user connected on galaxy webserver on this webserver, how is it possible to use PBS qsub command installed on the cluster to perform the job ? Does PBS qsub has to be installed on the webserver also ? In Galaxy documentation, there're the lines below : "The underlying method for execution is “pluggable”. Currently jobs can be executed on the same machine where the GALAXY instance is running, or dispatched to a computational cluster using a standard queue manager (portable batch system [PBS] is used at http://main.g2.bx.psu.edu) however other dispatch strategies can be implemented and plugged in easily" Can you give more information about these other dispatch strategies ? Thanks for your help. Laure -- Laure QUINTRIC Ifremer - IDM/RIC Technopôle Brest-Iroise 29280 Plouzané Tel: 02 98 22 49 11 - Fax : 02 98 22 46 47 E-mail : Laure.Quintric(a)ifremer.fr

2 1

Upload file button
by Zhe Chen 27 Oct '10

27 Oct '10

Hi, I am creating a tool to convert format. The tool will read from the original file, convert the format, then output. Should I put a "Browser" button like the example tool "Get Data -> Upload File" or I should put a simple input textbox, let the user paste the input file path? You can see the original tool here: http://hiv-dev.lanl.gov:8081/content/sequence/FORMAT_CONVERSION/form.html Simply speaking, how can I implement the "Upload File" button. I try to add <param name="filePath" type="file" size="30" label="Or upload your file:" ajax-upload="true"/> to the xml file of the tool. The UI looks right, but when I click "execute", nothing happens. Thanks

2 1

Problem offloading upload with nginx
by Dave Walton 25 Oct '10

25 Oct '10

Good Morning, I have a galaxy instance at our institution, and we are trying to make the file upload with nginx work. It appears that when I do a file download, it is getting handled by nginx (running top I see an nginx process appear, do work and disappear when the file is done downloading). However, when I try to do an upload, I see additional python processes appear in top, and I see the upload.py script go by in my runner0.log file, but I don't see an nginx process appear in top. Also, when I go to run several large uploads in a row, and then attempt to run some other job (like a sort) it blocks while waiting for the files to upload. This leads me to believe the upload is running in galaxy, not nginx. I am also running "watch ls -ltr upload_store" from galaxy-dist/database/tmp while the upload is occurring and I never see anything written to the upload temp directory. We are running galaxy on a 4 cpu server, which is running SUSE Linux Enterprise Server 11 (x86_64) for an os. I have galaxy configured to run 3 webapps, and 1 job runner. We have nginx installed and built with the nginx_upload_module (detail below) jaxgalaxydev01:~ # /usr/local/nginx/sbin/nginx -V nginx version: nginx/0.8.53 built by gcc 4.3.2 [gcc-4_3-branch revision 141291] (SUSE Linux) configure arguments: --prefix=/usr/local/nginx-0.8.53 --add-module=/usr/local/src/nginx_upload_module-2.2.0 In both my universe_wsgi.webapp.ini and universe_wsgi.runner.ini I have the following lines defined in my [app:main] section: # Directive for using nginx to do file downloads nginx_x_accel_redirect_base = /_x_accel_redirect # Directives to allow nginx to deal with file uploads nginx_upload_store = database/tmp/upload_store nginx_upload_path = /_upload Below is the entire contents of my nginx.conf file: worker_processes 1; events { worker_connections 1024; } user galaxy; http { include mime.types; default_type application/octet-stream; # Added for compression and caching. According to Galaxy wiki page: # http://bitbucket.org/galaxy/galaxy-central/wiki/Config/nginxProxy # This will decrease downlod and page load times for clients gzip on; gzip_http_version 1.1; gzip_vary on; gzip_comp_level 4; gzip_proxied any; gzip_types text/plain text/css application/x-javascript text/xml application/xml text/javascript application/json; gzip_buffers 16 8k; gzip_disable "MSIE [1-6].(?!.*SV1)"; sendfile on; keepalive_timeout 65; # Added to support proxying to galaxy server upstream galaxy_app { server localhost:8080; server localhost:8081; server localhost:8082; } server { listen 80; server_name localhost; # Line added for galaxy support client_max_body_size 10G; # Added to allow nginx to handle file uploads location /_upload { upload_store /galaxydata/galaxy-setup/galaxy-dist/database/tmp/upload_store; upload_pass_form_field ""; upload_set_form_field "__${upload_field_name}__is_composite" "true"; upload_set_form_field "__${upload_field_name}__keys" "name path"; upload_set_form_field "${upload_field_name}_name" "$upload_file_name"; upload_set_form_field "${upload_field_name}_path" "$upload_tmp_path"; upload_pass_args on; upload_pass /_upload_done; } location /_upload_done { set $dst /tool_runner/index; if ($args ~ nginx_redir=([^&]+)) { set $dst $1; } rewrite "" $dst; } # added to allow nginx to handle file downloads location /_x_accel_redirect/ { internal; alias /; } # The following series of "locations" are for off-loading static # content from the galaxy server. location /static { alias /galaxydata/galaxy-setup/galaxy-dist/static; # This will decrease downlod and page load times for clients expires 24h; } location /static/style { alias /galaxydata/galaxy-setup/galaxy-dist/static/june_2007_style/blue; } location /static/scripts { alias /galaxydata/galaxy-setup/galaxy-dist/static/scripts/packed; # This will decrease downlod and page load times for clients expires 24h; } location /favicon.ico { alias /galaxydata/galaxy-setup/galaxy-dist/static/favicon.ico; } location /robots.txt { alias /galaxydata/galaxy-setup/galaxy-dist/static/robots.txt; } # End of static content location / { proxy_pass http://galaxy_app; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } # redirect server error pages to the static page /50x.html # error_page 500 502 503 504 /50x.html; location = /50x.html { root html; } } } I have restarted both my galaxy server and my nginx server to confirm that the configurations above have been picked up. I would be interested in: 1) any other suggestions to confirm where the upload is occurring (assuming I'm mistaken about it running in galaxy). 2) any insight as to how I might have my server(s) misconfigured, in order to get upload working in nginx. Any assistance or suggestions you could provide would be greatly appreciated. Thanks, Dave

2 8

Search tools button
by Zhe Chen 25 Oct '10

25 Oct '10

Hi, On the left tool list panel of Galaxy website. On the top, there is a "options" button, when I click it, it shows a "Search Tools". How can I get it on my own local Galaxy server? On my local one, when I click "Options" button, the "Search Tools" button does not show up. Thanks

4 5

timeout for running a tool?
by Kostas Karasavvas 25 Oct '10

25 Oct '10

Hi all! Is there such a timeout setting? I use different parameters (which take much longer) and I get an empty dataset (no errors in stdout/err and paster.log). Normally, for smaller runs, the tool works fine. A quick look in the conf files seems to indicate that the answer to the above is no. Any ideas/suggestions? Thank you, Kostas

2 2

free little cloud for a year anyone?
by Ross 24 Oct '10

24 Oct '10

In case you haven't seen this, from November 1 http://aws.amazon.com/free/ New customers only. Must provide a valid CC and you'll pay for all use over the free tier limits - 12 months of : AWS Free Usage Tier (Per Month): - 750 hours of Amazon EC2 <http://aws.amazon.com/ec2> Linux Micro Instance usage (613 MB of memory and 32-bit and 64-bit platform support) – enough hours to run continuously each month* - 750 hours of an Elastic Load Balancer<http://aws.amazon.com/elasticloadbalancing/> plus 15 GB data processing* - 10 GB of Amazon Elastic Block Storage <http://aws.amazon.com/ebs>, plus 1 million I/Os, 1 GB of snapshot storage, 10,000 snapshot Get Requests and 1,000 snapshot Put Requests* - 5 GB of Amazon S3 storage <http://aws.amazon.com/s3>, 20,000 Get Requests, and 2,000 Put Requests* - 30 GB per of internet data transfer (15 GB of data transfer “in” and 15 GB of data transfer “out” across all services except Amazon CloudFront)* - 25 Amazon SimpleDB <http://aws.amazon.com/simpledb> Machine Hours and 1 GB of Storage** - 100,000 Requests of Amazon Simple Queue Service<http://aws.amazon.com/sqs> ** - 100,000 Requests, 100,000 HTTP notifications and 1,000 email notifications for Amazon Simple Notification Service<http://aws.amazon.com/sns> ** In addition to these services, the AWS Management Console<http://aws.amazon.com/console> is available at no charge to help you build and manage your application on AWS.

1 0

Log rotation tips
by Ry4an Brase 22 Oct '10

22 Oct '10

I set up logrotate on our production galaxy instance here, but after moving the file Galaxy (or more likely paste I imagine) doesn't re-open the log file. Is there some post-rotate action I can set in my logrotate command to let galaxy know that I pulled the logging rug out from under it? On apache this would be a SIGHUP, but some daemons use SIGUSRs of various flavors. How are others handling logs that grow without bounds? -- Ry4an Brase 612-626-6575 Software Developer Application Development University of Minnesota Supercomputing Institute http://www.msi.umn.edu

2 3

quick question on building tool's interface...
by Kostas Karasavvas 22 Oct '10

22 Oct '10

Hi all, When building a tool's interface would it be possible to have a different <help></help> tag for different <param type="select"> options of a 'conditional' tag? The options change the tool parameters significantly and having a big <help> tag that includes explanation of all parameters of all the options would be a bit confusing. Thank you, Kostas PS. I had some issues with an email alias. Sorry if you got duplicates!

3 2

More on variable multiple output files: identification
by Al Simons 22 Oct '10

22 Oct '10

Thanks to the list's help, I have my multiple output files. The next problem is that the displayed descriptions all have the identical text as the "master" (first output) file. I need to know which is which. The data in each file that my tool is indistinguishable, so I can't label the data internally. I need to precisely track which resultant dataset corresponds to which of the program's output streams. I can't even dead-reckon, because the datasets appear to be randomly ordered with respect to either the file's creation time or its original name. It would be _really_ nice if the output name extracted from the file name could be included in (or just be) the dataset name displayed in the history. That would give the program control of the displayed name, which would be extremely helpful. I see a tantalizing clue at the bottom of the ToolsMultipleOutput page, to reference "__collected_datasets__['primary'][name]." in a post processing hook. However, I can't find any reference to post processing hooks in the wiki or on Google. I'd welcome a pointer to any documentation about post-processing. Is there a documentation section that I'm overlooking? Thanks for the support, -Al

3 5

integer or float parameters - regression on trunk?
by Peter 22 Oct '10

22 Oct '10

Hi all, I've found what I consider to be a regression from Daniel Blankenberg's change: > http://bitbucket.org/galaxy/galaxy-central/changeset/1f4d6b25a84e > > Fix for Integer/FloatToolParameter.get_html_field() when 'value' is provided > as an integer/float. Fixes an issue seen when saving workflows: If an integer > or float tool parameter is changed to a value of 0 or 0.0 and saved, the form > field would be redisplayed using the default tool value; and not the value that > is now saved in the database. If have a tool parameter like this: <param name="dummy" type="integer" value="" label="Test integer with empty default"/> Prior to Dan's change, the HTML would show with the value as specified (empty string). After Dan's change, the HTML page for the tool loads with "0" shown, which is not what I wanted (we can get that with value="0"). I think the old functionality is more useful - couple it with a validator and this forces the user to pick a value rather than always having a valid default. However, what I really want to include optional="true" value="" and allow the value to be an integer or the empty string. In this use-case the HTML should start with the given default shown (empty string as value). See also: http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-October/003455.html Does that make sense? Regards, Peter

2 3