- galaxy-dev - lists.galaxyproject.org

workflow startup issues
by Geert Vandeweyer 22 May '13

22 May '13

Hi, After the latest upgrade (april), I'm having issues when starting (batch) workfows. Sometimes, it seems that step dependencies are not stored correctly, leading to some steps being started too soon, without prior datasets being completed. Specifically, for some workflows, some of the later steps start immediately. The issue seems to be random: in batch workflow submission, some run flawlessly, others show the issue, at different steps. I use one handler in the job_conf to exclude race conditions and shared variable clashes. I suspect there is an issue with fetching or storing the steps to/from the database (a remote mysql database in my case). On the mysql-side there are no errors in the log files, and the galaxy-logs don't show clear issues either. Does the job new job handler check for failed database connections? If not, could somebody point me to the code that performs these steps, I'd like to add a check & retry routine to see if this improves the situation. best, Geert -- Geert Vandeweyer, Ph.D. Department of Medical Genetics University of Antwerp Prins Boudewijnlaan 43 2650 Edegem Belgium Tel: +32 (0)3 275 97 56 E-mail: geert.vandeweyer(a)ua.ac.be http://ua.ac.be/cognitivegenetics http://www.linkedin.com/pub/geert-vandeweyer/26/457/726

2 5

Re: [galaxy-dev] Irregular failures on cluster (pbs_python?)
by Fields, Christopher J 21 May '13

21 May '13

On May 21, 2013, at 2:13 PM, Christopher Fields <cjfields(a)illinois.edu> wrote: > On May 20, 2013, at 3:08 PM, Nate Coraor <nate(a)bx.psu.edu> wrote: >> ... >> Hi Chris, >> >> We're disconnecting under all normal conditions and most error conditions - it looks like only a few conditions would not properly disconnect: >> >> - If pbs_submit() fails 5 times in a row >> - If an exception is raised anywhere in the queue_job() method after pbs_connect() is called >> - If the call to pbs_statjob() in check_all_jobs() or check_single_job() raises an exception (if that's even possible) >> >> For the exceptions, you would see that an exception was caught in the log file, so you should be able to determine if this is happening. >> >> For the pbs_submit() case, you'd see the message "All attempts to submit job failed". >> >> You may want to move the call to pbs_connect() in queue_job() so that it occurs immediately prior to the call to pbs_submit() and see if that makes a difference. The reason we connect so early on is to avoid writing out the job's files if the PBS server doesn't exist anyway. > > Yes, seeing the pbs_submit() case. Lowering the # handlers does seem to help, but we still seem to run into this after Sorry, saw a unicorn. Meant, 'we still seem to run into this after a period of time on the cluster'. Coffee… :P chris

1 0

Irregular failures on cluster (pbs_python?)
by Fields, Christopher J 21 May '13

21 May '13

I've been seeing this error popping up quite a bit recently (we're using Torque 3.0.5), which is giving a general 'cluster failure' Galaxy error on jobs: galaxy.jobs.runners.pbs WARNING 2013-05-19 18:56:44,073 (10588) pbs_submit failed (try 1/5), PBS error 15033: No free connections This just recently started springing up after we had been using the cluster for over a year, not sure why it would start acting up now. This appears to be related to a bug/defect with pbs_python, which doesn't seem to have been fixed yet (I posted a query whether this has been addressed): https://oss.trac.surfsara.nl/pbs_python/ticket/29 Restarting helps, but are there any other recommended workarounds? Is the only solution recompiling Torque with NCONNECTS? chris

2 4

changing the order of display in the left pane
by vijayalakshmi 21 May '13

21 May '13

Hi all, I am trying to deploy Galaxy in my local server with some modifications. But when I change the order of <section> inside <toolbox> in "*tool_conf.xml *" file , it's NOT taking effect. Should I modify some other files to change the order of display in the left pane?? Thanks, VJ.

2 2

How to create BAM indices on cluster upon upload
by Anthonius deBoer 21 May '13

21 May '13

Hi, I am uploading a lot of data to our galaxy system, and it seems that the index creation is always happening on the server instance of the galaxy server and NOT on the cluster. This seems to be particularly true for BAM files, where the index creation is being done on the galaxy server and it is done SERIALLY at that....This means that uploading 100s of BAM files can sometimes take hour upon hours... I have a cluster with 500 cores so I was wondering how I can get GALAXY to create these indices on the cluster in a parallel fashion, in order to speed up the process.... It seems that the UPLOAD process IS kicked off on the server, but then the index creation for BAM files is done afterwards on the galaxy server itself... Any help would be welcome... Thon

1 0

Cloudman / galaxyIndicies
by Rob Leclerc 21 May '13

21 May '13

We'd like to trim the the galaxyIndicies/genome directory to human and blastdb. As we may have several CM instances, we would like to share this filesystem across all instances. I would like to simply copy these data to a new volume and mount this filesystem with all our CM instances. Do I just need to rename specified filesystem in pd.yaml to ourGalaxyIndicies and then mount our CMs to /mnt/ourGalaxyIndicies/, or are their other locations I need to modify as well? Are the size and snap_id required? Cheers! Rob Rob Leclerc, PhD <http://www.linkedin.com/in/robleclerc> <https://twitter.com/#!/robleclerc> P: (US) +1-(917)-873-3037 P: (Shanghai) +86-1-(861)-612-5469 Personal Email: rob.leclerc(a)aya.yale.edu

2 1

Cloudman - missing security rules in wiki page
by Joachim Jacob | VIB | 21 May '13

21 May '13

Hi all, I have tried to fire up an Cloud instance of Galaxy following the wiki page: http://wiki.galaxyproject.org/CloudMan However, two attempts failed (Cloudman interface did not come up - I could SSH into the machine). Next, I have tried the CloudLaunch interface (added the link to the wiki page), and tried that way. It succeeded. It seems that the Security group settings differ between what CloudLaunch sets, and the instructions written on the wiki page. In wiki: Ports Protocol Source GalaxyWeb 0-65535 tcp sg-1b495b72 20-21 tcp 0.0.0.0/0 22 tcp 0.0.0.0/0 80 tcp 0.0.0.0/0 30000-30100 tcp 0.0.0.0/0 42284 tcp 0.0.0.0/0 Set by CloudLaunch: Ports Protocol Source CloudMan 1-65535 tcp sg-61ba790a 1-65535 udp sg-61ba790a -1 icmp sg-61ba790a 20-21 tcp 0.0.0.0/0 22 tcp 0.0.0.0/0 80 tcp 0.0.0.0/0 30000-30100 tcp 0.0.0.0/0 42284 tcp 0.0.0.0/0 Since I did not test manually launching the cloud instance with these settings, I post them here. If anybody has tested these settings, perhaps the wiki page can be updated? Cheers, Joachim -- Joachim Jacob Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib

2 1

Error in installing Galaxy.
by sridhar srinivasan 21 May '13

21 May '13

Hi , I am getting error during Installing galaxy locally. Traceback (most recent call last): File "/illumina/apps/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/buildapp.py", line 35, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File "/illumina/apps/galaxy/galaxy-dist/lib/galaxy/app.py", line 51, in __init__ create_or_verify_database( db_url, kwargs.get( 'global_conf', {} ).get( '__file__', None ), self.config.database_engine_options, app=self ) File "/illumina/apps/galaxy/galaxy-dist/lib/galaxy/model/migrate/check.py", line 50, in create_or_verify_database dataset_table = Table( "dataset", meta, autoload=True ) File "/illumina/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/schema.py", line 108, in __call__ return type.__call__(self, name, metadata, *args, **kwargs) File "/illumina/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/schema.py", line 236, in __init__ _bind_or_error(metadata).reflecttable(self, include_columns=include_columns) File "/illumina/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 1261, in reflecttable conn = self.contextual_connect() File "/illumina/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 1229, in contextual_connect return self.Connection(self, self.pool.connect(), close_with_result=close_with_result, **kwargs) File "/illumina/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/pool.py", line 142, in connect return _ConnectionFairy(self).checkout() File "/illumina/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/pool.py", line 304, in __init__ rec = self._connection_record = pool.get() File "/illumina/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/pool.py", line 161, in get return self.do_get() File "/illumina/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/pool.py", line 639, in do_get con = self.create_connection() File "/illumina/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/pool.py", line 122, in create_connection return _ConnectionRecord(self) File "/illumina/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/pool.py", line 198, in __init__ self.connection = self.__connect() File "/illumina/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/pool.py", line 261, in __connect connection = self.__pool._creator() File "/illumina/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/strategies.py", line 80, in connect raise exc.DBAPIError.instance(None, None, e) OperationalError: (OperationalError) FATAL: Ident authentication failed for user "galaxy" None None I created a user galaxy to install galaxy locally. Thanks in Advance. Sridhar

4 10

how to set GridEngine parameters for tools
by Andreas Kuntzagk 21 May '13

21 May '13

Hi, I recently updated our local galaxy instance first time after a few month. Afterwards I noticed that the way to setup job runners has changed from single lines to an extra XML-file. I now started to write a job_conf.xml but I don't understand where I put the parameters for GridEngine. I already read http://wiki.galaxyproject.org/Admin/Config/Jobs but seem to have missed this info. For example in the old style I have lines like velveth = drmaa://-l h_vmem=25G/ In the new style I would write " ... <destinations> <destination id="cluster" runner="drmaa"/> </destinations> <tools> <tool id="velveth" destination="cluster"/> </tools> ... " But where do I put the "-l h_vmem"? regards, Andreas -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich

2 2

Test Tool Shed - Installation Error "None" for repository dependencies
by Peter Cock 21 May '13

21 May '13

Hi Greg & Dave, I'm trying to use the new inter-repository dependency mechanism without explicitly naming the tool shed (i.e. assume the repository is in the same ToolShed), as discussed here: http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-May/014736.html My blast2go wrapper is currently listed under "Latest revision: installation errors" yet there is no clear error shown: http://testtoolshed.g2.bx.psu.edu/view/peterjc/blast2go/9ef0d4ecb9d5 Revision: 10:9ef0d4ecb9d5 Automated tool test results Tool test results Automated test environment *Time tested:* ~ 8 hours ago*System:* Linux 3.5.0-21-generic*Architecture:* x86_64*Python version:* 2.7.3*Galaxy revision:* 9796:2b456e221582*Galaxy database version:* 115*Tool shed revision:**Tool shed database version:**Tool shed mercurial version:* Installation errors* - no functional tests were run for any tools in this changeset revision* Repository dependencies Tool shedNameOwnerChangeset revisiontesttoolshed.g2.bx.psu.edublast2go peterjc9ef0d4ecb9d5ErrorNone Do you see this too? Have I hit a problem with the error capturing? Thanks, Peter

1 0