October 2011 - galaxy-dev - lists.galaxyproject.org

hardware for local galaxy install
by Jeremy Goecks 12 Oct '11

12 Oct '11

David, Please direct questions about local installations to the galaxy-dev mailing list (cc'd); there are a large number of experienced people of this list that can likely help with this and similar questions. > Just a quick question about the compute infrastructure of the setup running the public instance of Galaxy. Our Galaxy copy is on our High Performance Cluster here at X and they are currently canvassing users about the next upgrade of the HPC. This cluster serves the needs of all the university academics here and so we are sharing time with physicists, aerodynamics engineers etc. However, the HPC board of directors tell me that they see biological computing problems as the next big growth area for them - especially now we have a copy of Galaxy. What they would like to know is what kind of mix of new nodes would be most useful to us. They have about X million pounds to spend (about X million dollars) and the question as I understand it is determining the right mix of expensive high memory nodes (e.g. with 1 or 2TB RAM on board) and cheaper lower memory nodes (e.g. with 32G ram on board or less even). The problem for me is that I have no idea what to suggest (I am not that compute savvy). So any advice from your experiences (or that of the Galaxy team) would be most helpful. Your hardware will depend on what types of analyses you plan to support/run. For instance, assembly jobs (e.g. ABySS, Velvet, Trinity) require high-memory nodes, and read mapping jobs (e.g. BWA, Bowtie, Tophat) are best run on high CPU nodes. Also, creating indices (often used in visualization) such as bigWig and bigBed require large amounts of memory as well. You'll want to do some research on the tools that you plan to run most often and determine the best hardware for them. Nate Coraor's presentation at GCC 2011 has some useful information about setting up a production Galaxy as well as details about our public instance: http://wiki.g2.bx.psu.edu/Events/GCC2011 In short, we use multiple clusters to distribute resource-intensive jobs appropriately. Also, you may want to search the galaxy-dev mailing list as there's been discussion about hardware specs for a Galaxy instance previously: http://gmod.827538.n3.nabble.com/Galaxy-Development-f815885.html Nate and others, please feel free to chime in with additional information. Good luck, J.

2 1

scatterplot broken
by David Hoover 12 Oct '11

12 Oct '11

Hi, The scatterplot tool is broken in the distributed version. It works on the main Galaxy server, but on our local installation it gives this error: An error occurred running this job:Warning messages: 1: In min(x) : no non-missing arguments to min; returning Inf 2: In max(x) : no non-missing arguments to max; returning -Inf 3: In min(x) : no non-missing arguments to min; returning Inf 4: In max(x) : no non-missing arguments to max; re Any ideas? David Hoover Helix Systems Staff SCB/CIT/NIH

1 1

Re: [galaxy-dev] trinity on galaxy
by Jeremy Goecks 12 Oct '11

12 Oct '11

David, I'm cc'ing the galaxy-dev mailing list; let's keep this thread on the list for archival and community purposes. > >> I had a look through the Trinity.pl script and found that the default butterfly >> heap is set to 1G. >> >> I've changed this to 2G. Would you re-run the trinity job and let me know if this helps. Rather than mucking with the script, It's possible to change the Butterfly memory size using command line parameters: --- Butterfly options: # # --run_butterfly :executes butterfly commands. Do not set this if you want to spawn them on a computing grid. # # --bfly_opts :parameters to pass through to butterfly (see butterfly documentation). # # --bflyHeapSpace :java heap space setting for butterfly (default: 1000M) => yields command java -Xmx1000M -jar Butterfly.jar ... $bfly_opts --- This can be done in the Trinity wrapper at <galaxy-home>/tools/ngs_rna/trinity_all.xml ; there are already some Butterfly options set in the wrapper, so it should be easy to add more. > We see the following at the end of the Trinity logs - I don't know what it means but I thought you might... > number of good components: 1 > number of reads found = 1253 (from total of 1254) which came from 1252 pairs > ## Read PathPair results: 1251 singletons, num pairs: 1, num pairs discarded: 0 > total number of components = 1 > working on subcomponent 0 > method: combineSimilarPathsThatEndAtV(-1) with 1 paths > method: combineSimilarPathsThatEndAtV(2) with 1 paths > method: combineSimilarPathsThatEndAtV(1248) with 1 paths > method: combineSimilarPathsThatEndAtV(1223) with 1 paths > method: combineSimilarPathsThatEndAtV(-2) with 2 paths > > Path Illustration: > > ==== PATH: [-1, 2, 1248, -2] > === Read: [[-1, 2, 1248], []] count: 2 > = Read: [[2], []] count: 1030 > == Read: [[2, 1248], []] count: 23 > === Read: [[2, 1248, -2], []] count: 2 > = Read: [[1248], []] count: 197 > > > Final path reported: comp12043_c0_seq1 len=1658 ~FPKM=29.7 path=[2:0-1220 1248:1221-1657] > FinalPath: [-1, 2, 1248, -2] with support <1254, 1254> > total number of paths reported = 1 from 1 components > Done > SUCCESS[53682]: java -Xmx1G -jar /gpfs/cluster/isys/galaxy/Galaxy/trinityrnaseq_r2011-08-20/Butterfly/Butterfly.jar -N 25506261 -L 300 -F 300 -C chrysalis/RawComps.6/comp12043 -V 10 --stderr > SUCCESS[53680]: java -Xmx1G -jar /gpfs/cluster/isys/galaxy/Galaxy/trinityrnaseq_r2011-08-20/Butterfly/Butterfly.jar -N 25506261 -L 300 -F 300 -C chrysalis/RawComps.24/comp48425 -V 10 --stderr > SUCCESS[53681]: java -Xmx1G -jar /gpfs/cluster/isys/galaxy/Galaxy/trinityrnaseq_r2011-08-20/Butterfly/Butterfly.jar -N 25506261 -L 300 -F 300 -C chrysalis/RawComps.8/comp16349 -V 10 --stderr > > > Sorry, 1 of 53682 jobs failed. Overall, this looks very good - almost all the jobs succeeded. > Failed commands written to file: failed_cmds.27222.txt > > Error, cmd: /gpfs/cluster/isys/galaxy/Galaxy/trinityrnaseq_r2011-08-20/util/cmd_process_forker.pl -c /gpfs/cluster/isys/galaxy/Galaxy/galaxy-dist/database/job_working_directory/679/trinity_out_dir/chrysalis/butterfly_commands.adj --CPU 4 --shuffle died with ret 256 at /gpfs/cluster/isys/galaxy/Galaxy/trinityrnaseq_r2011-08-20/Trinity.pl line 571. This is a bit cryptic to me. You'll want to determine what error code 256 means and see whether it may be a memory issue; you may also want to email the Trinity mailing list and add for help there. Good luck, J.

2 1

cloudman on private cloud
by Rodrigo García Herrera 12 Oct '11

12 Oct '11

Hello! If I setup a private cloud infrastructure with ubuntu, can I setup galaxy cloudman there? Thanks! Rodrigo

2 1

Possible to browse local filesystem
by alessandro albiero 11 Oct '11

11 Oct '11

Dear all, I have a problem with galaxy and I have already tried to write on ther user list.... My problem concern the upload on galaxy of new files. We are setting up a galaxy istance on our server (HPC Cluster with torque and PBS) for internal use. Our problem is that we manage only NGS data (SOLiD and Illumina Data) that are already stored on the server. We tried to set up the user directory in the universe_wgsi.ini file to allow to any user tu upload their local files, but adding new file and re-uploading the dir we found duplicated files (it seems that galaxy re-upload every time all the files ignoring if some are already present). This method is OK for us but how can we avoid the file duplications ? In a presentation found on web (blankenberg_pittsburg_2001), we saw that there is the possibility to automatically upload the sequenced reads on Galaxy (and let them available to all our user). Where can we find information for doing this ? Thank in advance All the best Alessandro -- ****************************** Alessandro Albiero Bioinformatics Lab CRIBI-Padua University Viale G. Colombo, 3 35121 Padova-Italy http://genomics.cribi.unipd.it http://genomics.cribi.unipd.it/index.php/User:Albiero Phone. +39. 049 827-6165 Fax. +39. 049 827-6159 ********************************

2 1

Downloading large files from local galaxy
by Roy Weckiewicz 08 Oct '11

08 Oct '11

Hi, We have a local galaxy instance running on mac osx connected to a mysql database. We are having an issue downloading files which are > 2Gb. I've read posts on this mailing list that mention that there is a browser limit on downloading files of this size, which is fine -- a command-line method would be okay. Furthermore, I've read some solutions that mention using either wget or curl along with the link from the 'disk icon' in the history to download the data. We've tried both curl and wget, which seem to work fine with files < 2Gb, but again fails with the larger files. Wget will tend to time-out, and attempts to retry over and over (wget 'address-from-galaxy-history') and curl will hang forever as well (curl -O 'address-from-galaxy-history'). Attempts have been made to both download the data from the host computer to itself, and from the host to a remote computer. Are we perhaps missing some critical configuration setting? We would like to stay away from a ftp solution if at all possible. Is anyone using a simple method to retrieve these large files? -Roy W. -- Roy Weckiewicz Texas A&M University

3 3

Accept-Ranges: bytes for
by Kip Bodi 07 Oct '11

07 Oct '11

Hi all, I have a local galaxy instance, proxied by Apache, and am trying to get the 'display at UCSC' link to work. I've followed all of the instructions and am nearly there, but now I'm getting this response from UCSC when it tries to load the track: "Byte-range request was ignored by server. Expected Partial Content 206." etc. I think the error is because the PasteWSGIServer isn't allowing byte-range requests. When I look at the header, "Accept-Ranges: bytes" is not present: [galaxy@galaxy galaxy_dist]$ curl --head localhost HTTP/1.1 200 OK Date: Fri, 07 Oct 2011 22:23:49 GMT Server: PasteWSGIServer/0.5 Python/2.6.6 content-type: text/html; charset=UTF-8 Set-Cookie: galaxysession=c6ca0ddb55be603a4e98b4f13a4db8bfb8635f0aa576697114d40b13a8355efe2cf7a075ef81d769; expires=Thu, 05-Jan-2012 17:23:49 GMT; Max-Age=7776000; Path=/; Version=1 Vary: Accept-Encoding Connection: close Whereas on another server, "Accept-Ranges: bytes" appears in the header: [galaxy@galaxy galaxy_dist]$ curl --head apollo.med.tufts.edu HTTP/1.1 403 Forbidden Date: Fri, 07 Oct 2011 22:22:42 GMT Server: Apache/2.2.21 (Fedora) *Accept-Ranges: bytes* Content-Length: 4609 Connection: close Content-Type: text/html; charset=UTF-8 I'm running RHEL6 with the latest version of galaxy-dist. I've tried manually enabling 'accept-ranges' but haven't had any luck. Any suggestions? Can I enable it somehow in universe_wsgi.ini? Best, -Kip

1 0

Datafiles for the Gencode partition tool
by Harald Dahle 07 Oct '11

07 Oct '11

Hello, I've been trying to configure the Gencode Partition tool on our galaxy installation, but have not been able to find any information about what sort of data it expects to find, and my best attempts at guessing still lead to errors being produced. Can anyone help me with a few pointers to the correct datafiles/format of the files? Thanks in advance! Best regards, Harald Dahle

1 0

CloudMan on Eucalyptus
by Andrey Tovchigrechko 06 Oct '11

06 Oct '11

We want to use CloudMan to provision and control a cluster with our own VM image (with Galaxy front-end as well). Our plan was to make the VMs for both our internal Eucalyptus cloud as well as EC2. Will CloudMan work with Eucalyptus? If not, how much would our effort have be in order to extend the source code to make it work? I am seeing in the repo an empty file https://bitbucket.org/galaxy/cloudman/src/1440e0147c85/cm/clouds/eucalyptus… Thanks, Andrey

2 1

bug report submission bug
by Oleksandr Moskalenko 06 Oct '11

06 Oct '11

Someone shared a history with me for the purpose of debugging their Galaxy problem in our local Galaxy instance. I set the "error_email_to" to our bug reporting email address, but I can't submit a bug report using the "bug" icon in the history for datasets in an error state. The traceback is below. The job failure was a missing tool by the way. Has anyone run into this before? 10.244.xxx.xxx - - [03/Oct/2011:16:02:12 -0400] "POST /dataset/report_error HTTP/1.1" 500 - "http://galaxy.hpc.ufl.edu/datase t/errors?id=767" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari /534.48.3" Error - <type 'exceptions.IndexError'>: list index out of range URL: http://galaxy.hpc.ufl.edu/dataset/report_error File '/galaxy/prod/galaxy/eggs/Paste-1.6-py2.6.egg/paste/exceptions/errormiddleware.py', line 143 in __call__ app_iter = self.application(environ, start_response) File '/galaxy/prod/galaxy/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/galaxy/prod/galaxy/lib/galaxy/web/framework/middleware/remoteuser.py', line 111 in __call__ return self.app( environ, start_response ) File '/galaxy/prod/galaxy/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/galaxy/prod/galaxy/lib/galaxy/web/framework/base.py', line 145 in __call__ body = method( trans, **kwargs ) File '/galaxy/prod/galaxy/lib/galaxy/web/controllers/dataset.py', line 171 in report_error job = hda.creating_job_associations[0].job IndexError: list index out of range

1 1