January 2012 - galaxy-dev - lists.galaxyproject.org

running functional tests on a galaxy server
by Fields, Christopher J 20 Jan '12

20 Jan '12

I have been following the basic protocol for running remote tests on a Galaxy server (in this case, our local one) using: http://wiki.g2.bx.psu.edu/Admin/Running%20Tests >From reading this, I assume the tests are to be run from the galaxy root directory, like so: GALAXY_TEST_EXTERNAL=1 GALAXY_TEST_HOST=my.local.galaxy.url GALAXY_TEST_PORT=8080 \ ./nosetests.sh However, no 'nosetests.sh' script exists there. Should we be using ./scripts/nosetests.py? ./scripts/functional_tests.py? Any clarification on this would be very helpful, we're just searching for a way to run tests in as automated a way as possible using the server configuration. chris Chris Fields Senior Research Scientist National Center for Supercomputing Applications Institute for Genomic Biology University of Illinois at Urbana-Champaign

3 5

January 20, 2012 Galaxy Development News Brief
by Jennifer Jackson 20 Jan '12

20 Jan '12

January 20, 2012 Galaxy Development News Brief Highlights: * New Object Store data integration layer introduced * RNA-seq updates: TopHat to 1.4.0 and Cufflinks, CuffDiff, CuffCompare to 1.3.0. * Tool Shed installation features and community tool additions * Trackster performance upgrades and new drag & drop composite track creation Mercurial pull: hg pull -u -r 63bc46cc73b7 Complete News Brief http://wiki.g2.bx.psu.edu/DevNewsBriefs/2012_01_20 Release history http://wiki.g2.bx.psu.edu/DevNewsBriefs/ Install/Administration Help http://getgalaxy.org Support http://galaxyproject.org/wiki/Support Thanks for using Galaxy!

1 0

job quotas
by Ivan Merelli 20 Jan '12

20 Jan '12

Hi, is it possible to restrict the number of concurrent jobs for a single user in a local instance of galaxy? I see that in the public site this feature is implemented, but I don't find documentation about how to implement this locally. I think it's not possible through pbs because all users submit jobs as the defauls galaxy user. Cheers, Ivan

2 1

.gz extension gets stripped off when uploading via data library Upload via filesystem paths but not via Get Data -> Upload File
by Leandro Hermida 20 Jan '12

20 Jan '12

Hello, We've created a new binary datatype for .fastq.gz files following the same methodology as the BAM files since we don't want our fasta.gz files to be gunzipped. I added the appropriate code in upload.py to make sure of this. This new datatype and extension successfully does not gunzip our files. But when we upload it into a data library via the data library "Upload via filesystem paths" it for some reason automatically strips the .gz part out. When we take the same .fastq.gz file and upload it via Get Data -> Upload File it works fine, nothing is stripped from file name. Where is it doing this and how can prevent from stripping the .gz via the data library menus? thanks, Leandro

3 4

Re: [galaxy-dev] Galaxy Hang after DrmCommunicationException
by Edward Kirton 19 Jan '12

19 Jan '12

excellent, thanks for the correction! On Wed, Jan 18, 2012 at 8:59 AM, Shantanu Pavgi <pavgi(a)uab.edu> wrote: > > Ed, > > I think you may want to add job with DrmCommunicationException error > back into the watched_jobs list. > {{{ > except ( drmaa.DrmCommunicationException ), e: > # Catch drmaa communication exception, log a warning message and > # continue to monitor jobs. > log.warning("(%s/%s) Couldn't communicate with the cluster scheduler to > check job status." %( galaxy_job_id, job_id )) > # keep old job state > new_watched.append( drm_job_state ) > continue > }}} > > Here is a patch and related discussion on it: > https://bitbucket.org/galaxy/galaxy-central/pull-request/23/catch-drmcommun… > > > -- > Shantanu > > > On Jan 13, 2012, at 2:11 PM, Edward Kirton wrote: > > i had seen the job process die with this error: > > if state != old_state: > UnboundLocalError: local variable 'state' referenced before assignment > > since the DRM timeout is an intermittent error, i'm not absolutely > positive i have nailed it (a bit more time will tell -- i'm keeping an eye > on it with debug log messages), but it seems so. i intended to share this > as a patched clone when i became sure but when i read your email i just > sent what i had immediately. let us know if that seems to solve the > problem for you, so we'll have some confirmation. > > glad to help, > ed > > p.s. i have another patch for "output not returned from cluster" errors > that i'm also still validating, due to NFS race since we don't have inode > metadata caching turned off as the galaxy developers suggest. > > On Fri, Jan 13, 2012 at 8:06 AM, Ann Black <annblack(a)eng.uiowa.edu> wrote: > >> Thanks so much! I have applied the fix to our env. >> >> In looking over the logic, there was an existing exception block that >> would have caught the communication exception generically – but the job >> moved (in this scenario erroneously) the job into a failure workflow. I >> would like to understand what ended up hanging galaxy – so it must be >> related to a valid job being moved into failure state ? Did you follow it >> down the rabbit hole by any chance to see what caused the hang in your env >> ? >> >> Thanks again, >> >> Ann >> >> From: Edward Kirton <eskirton(a)lbl.gov> >> Date: Thu, 12 Jan 2012 13:00:27 -0800 >> To: Ann Black <annblack(a)eng.uiowa.edu> >> Cc: "galaxy-dev(a)lists.bx.psu.edu" <galaxy-dev(a)lists.bx.psu.edu> >> Subject: Re: [galaxy-dev] Galaxy Hang after DrmCommunicationException >> >> sometimes the scheduler can't keep up with all the work in it's 15sec >> cycle, so it doesn't respond to some messages. here's a fix i've been >> trying that seems to work. >> >> in lib/galaxy/jobs/runners/drmaa.py: >> >> def check_watched_items( self ): >> """ >> Called by the monitor thread to look at each watched job and deal >> with state changes. >> """ >> new_watched = [] >> for drm_job_state in self.watched: >> job_id = drm_job_state.job_id >> galaxy_job_id = drm_job_state.job_wrapper.job_id >> old_state = drm_job_state.old_state >> try: >> state = self.ds.jobStatus( job_id ) >> # InternalException was reported to be necessary on some >> DRMs, but >> # this could cause failures to be detected as completion! >> Please >> # report if you experience problems with this. >> except ( drmaa.InvalidJobException, drmaa.InternalException >> ), e: >> # we should only get here if an orphaned job was put into >> the queue at app star >> tup >> log.debug("(%s/%s) job left DRM queue with following >> message: %s" % ( galaxy_jo >> b_id, job_id, e ) ) >> self.work_queue.put( ( 'finish', drm_job_state ) ) >> continue >> # BEGIN DRM TIMEOUT: Don't fail on scheduler communication >> error (probably just too busy) >> except ( drmaa.DrmCommunicationException ), e: >> log.warning("(%s/%s) DRM Communication Exception" % ( >> galaxy_job_id, job_id >> )) >> continue >> # END DRM TIMEOUT >> >> On Wed, Jan 11, 2012 at 9:18 AM, Ann Black <annblack(a)eng.uiowa.edu>wrote: >> >>> Good Morning galaxy group! >>> >>> I was hoping that someone might have some ideas on a problem we have >>> experienced a handful of times running galaxy on our local cluster. >>> >>> Occasionally we experience some communication timeouts between out >>> cluster head node and a compute node which will self heal. However, this in >>> turn will hang galaxy. Below you will see output from our galaxy log file. >>> When the ERROR happens (which is not often) it consistently seems to hang >>> galaxy. We have to kill it off and restart it. We are running galaxy as a >>> single PID at this time (we are still just testing it out, etc) and it is >>> running on our head node (which we plan to move off of in the future). >>> >>> galaxy.jobs.runners.drmaa DEBUG 2012-01-10 19:19:58,800 (1654/698075) >>> state change: job is running >>> galaxy.jobs.runners.drmaa ERROR 2012-01-10 20:57:47,021 (1654/698075) >>> Unable to check job status >>> Traceback (most recent call last): >>> File "/data/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 236, >>> in check_watched_items >>> state = self.ds.jobStatus( job_id ) >>> File "/data/galaxy-dist/eggs/drmaa-0.4b3-py2.7.egg/drmaa/__init__.py", >>> line 522, in jobStatus >>> _h.c(_w.drmaa_job_ps, jobName, _ct.byref(status)) >>> File "/data/galaxy-dist/eggs/drmaa-0.4b3-py2.7.egg/drmaa/helpers.py", >>> line 213, in c >>> return f(*(args + (error_buffer, sizeof(error_buffer)))) >>> File "/data/galaxy-dist/eggs/drmaa-0.4b3-py2.7.egg/drmaa/errors.py", >>> line 90, in error_check >>> raise _ERRORS[code-1]("code %s: %s" % (code, error_buffer.value)) >>> DrmCommunicationException: code 2: failed receiving gdi request response >>> for mid=24442 (got syncron message receive timeout error). >>> galaxy.jobs.runners.drmaa WARNING 2012-01-10 20:58:05,090 (1654/698075) >>> job will now be errored >>> galaxy.jobs.runners.drmaa DEBUG 2012-01-10 20:59:06,396 (1654/698075) >>> User killed running job, but error encountered removing from DRM queue: >>> code 2: failed receiving gdi request response for mid=24444 (got syncron >>> message receive timeout error). >>> galaxy.datatypes.metadata DEBUG 2012-01-10 20:59:06,896 Cleaning up >>> external metadata files >>> galaxy.datatypes.metadata DEBUG 2012-01-10 20:59:06,947 Failed to >>> cleanup MetadataTempFile temp files from >>> database/tmp/metadata_out_HistoryDatasetAssociation_2913_ZUTgBy: No JSON >>> object could be decoded: line 1 column 0 (char 0) >>> galaxy.datatypes.metadata DEBUG 2012-01-10 20:59:09,640 Cleaning up >>> external metadata files >>> galaxy.jobs INFO 2012-01-10 20:59:09,697 job 1656 unable to run: one or >>> more inputs in error state >>> galaxy.datatypes.metadata DEBUG 2012-01-10 20:59:10,121 Cleaning up >>> external metadata files >>> galaxy.jobs INFO 2012-01-10 20:59:10,159 job 1655 unable to run: one or >>> more inputs in error state >>> galaxy.datatypes.metadata DEBUG 2012-01-10 20:59:12,076 Cleaning up >>> external metadata files >>> galaxy.jobs INFO 2012-01-10 20:59:12,126 job 1657 unable to run: one or >>> more inputs in error state >>> galaxy.datatypes.metadata DEBUG 2012-01-10 20:59:13,601 Cleaning up >>> external metadata files >>> galaxy.jobs INFO 2012-01-10 20:59:13,650 job 1658 unable to run: one or >>> more inputs in error state >>> >>> >>> Has anyone else experienced this or have some ideas on how we can >>> further debug to figure out why galaxy hangs? >>> >>> Thanks much! >>> >>> Ann Black-Ziegelbein >>> >>> ___________________________________________________________ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>> >>> http://lists.bx.psu.edu/ >>> >> >> > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ > > >

2 2

Quota will not decrease with permanent delete
by Mary Anne Alliegro 19 Jan '12

19 Jan '12

Hi Galaxy Users, I have permanently deleted numerous files. My usage % has decreased, but this is NOT reflected in my Gb report (upper right)- it remains the same. Am I missing some phantom trash bin? If not, will Galaxy recalculate my Gb usage so that I may proceed with my project? Thank you, Mary Anne

5 6

how to set FTP quota?
by Leon Mei 19 Jan '12

19 Jan '12

Dear Galaxy team, We just installed ProFTPd (the same as Galaxy Main) on our Galaxy server. Now we would like to implement a FTP quota. I am wondering how Galaxy main handles this. Could you give us some hints? And does the disk quota counting script include the FTP directory? Thanks very much, Leon -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - https://wiki.nbic.nl/index.php/Next_Generation_Sequencing Skype: leon_mei Mobile: +31 6 41709231

3 2

Is it possible to pass runtime values to workflows through the API?
by Carlos Borroto 19 Jan '12

19 Jan '12

Hi, I was wondering if is possible to pass runtime values to workflows thought the API? I want to run the same workflow for several inputs, but there are values other than the inputs that I need to change for each run. Is it possible to leave these values as 'To be set at runtime' and set them through the API? Also, I see 'Step Actions' do not support the option of 'To be set at runtime', could this be included in a future release, in other words, can this be a feature request? Thanks, Carlos

2 3

Error download composite dataset from history
by Louise-Amélie Schmitt 19 Jan '12

19 Jan '12

Hello! I'm running into an error when trying to download a composite dataset from the history with the floppy disk icon. Here is the error message in the logs: galaxy.web.controllers.dataset ERROR 2012-01-18 16:32:10,324 Unable to remove temporary library download archive and directory Traceback (most recent call last): File "/g/funcgen/galaxy-dev/lib/galaxy/web/controllers/dataset.py", line 285, in archive_composite_dataset os.rmdir( tmpd ) OSError: [Errno 39] Directory not empty: './database/tmp/tmpkRv28I' Then I tried replacing the os.rmdir by a shutil.rmtree to see if it would solve the issue, and here is what I got: galaxy.web.controllers.dataset ERROR 2012-01-18 16:34:30,239 Unable to remove temporary library download archive and directory Traceback (most recent call last): File "/g/funcgen/galaxy-dev/lib/galaxy/web/controllers/dataset.py", line 286, in archive_composite_dataset shutil.rmtree( tmpd ) File "/g/steinmetz/collaboration/software/CentOS5/opt/Python-2.6.5/lib/python2.6/shutil.py", line 221, in rmtree onerror(os.remove, fullname, sys.exc_info()) File "/g/steinmetz/collaboration/software/CentOS5/opt/Python-2.6.5/lib/python2.6/shutil.py", line 219, in rmtree os.remove(fullname) OSError: [Errno 16] Device or resource busy: './database/tmp/tmpwpsShD/.nfs000000000289ccea00000138' But when I check the contents of the direcory, there is no hidden .nfs-something file. Does anyone have an idea why this could occur? Both our testing instance and our production instance have the same issue, the first being on galaxy-central and the second on galaxy-dist. Thanks, L-A

2 4

problems with location of filter bar in trackster
by Hans-Rudolf Hotz 19 Jan '12

19 Jan '12

Hi I am still in the process of upgrading all our Galaxy servers to the current changeset ("b258de1e6cea", Nov. 18) and now I am running into troubles with the visualization: Depending on the location of my mouse, the filter bar and the other extra buttons jump between two positions, which makes it impossible to use, see Screenshot_1.png and Screenshot_2.png I was looking for relevant code changes, which were committed after Nov. 18. I found changeset "856aac70b018" ("Trackster: refactor action icon creation and initialization" from Nov. 23) for ~/static/scripts/trackster.js Adding this change fixes the problem somehow, but I lose the original icons (for 'set display mode', 'Hide/Show track content', etc) and the bar is now below the track, see Screenshot_3.png Just using the ~/static/scripts/trackster.js from 'central' doesn't solve the problem either. Are there any extra steps I can do in addition to changeset "856aac70b018" in order to solve this problem (except for waiting/using for the next release) Thank you very much for your help Hans

2 4