A dataset's set_meta() is done as part of the job, so if you are not running jobs on a cluster, set_meta() will be run locally as well, which is certainly chewing up cpu on your server.  If running externally, set_meta() will run on the cluster when the user does anything in the "Edit Attributes" page that call set_meta(), including "Auto-detect".

As soon as I get a chance, I'll look at enhancing set_meta() to check if "set_metadata_externally" is True for those data types that take significant processing, and if jobs are running locally, metadata will be set differently.


On Jan 8, 2010, at 12:25 PM, Assaf Gordon wrote:

It is set to "False", but my galaxy runs jobs locally, not on a cluster...
(at least, not directly through the SGE Runner).

Does this work with local-runner too (i.e. starting a new process to set the metadata) ?
Also, does the "external" method works when the use changes the type in the "Edit Attributes" page ?



Greg Von Kuster wrote, On 01/08/2010 10:54 AM:
Hello Assaf,

Is your instance configured to set metadata externally ( on your cluster
nodes )?  If not, in your universe_wsgi.ini file, add the following to
the [app:main] section:

set_metadata_externally = True


On Jan 6, 2010, at 5:13 PM, Assaf Gordon wrote:

Hello all,

Continuing the search for slowness in my local Galaxy server (see

The datatypes/sequence.py file is also scanning and parsing entire
files when creating a new FASTA/FASTQ file.
It's nice and fun and informative for small files, but with a 2.7GB
FASTA file - the python process stays at 100% CPU for a long long
time, causing everything else to be very slow.

The offending code is at sequence.py, method "set_meta", lines 30-39.

I think Illumina expects 25x coverage of the human genome in a single
run by the end of the year - this will roughly translates to 8 FASTQ
files of more than 8GB each => FASTA files of 4GB each... Galaxy will
not be able to just casually scan these files.

-gordon


_______________________________________________
galaxy-dev mailing list

Greg Von Kuster
Galaxy Development Team
greg@bx.psu.edu





Greg Von Kuster
Galaxy Development Team
greg@bx.psu.edu