- galaxy-dev - lists.galaxyproject.org

Tool Tests failing with "Exception: History in error state."
by Peter Cock 04 Feb '15

04 Feb '15

Retitling thread, On Thu, Nov 6, 2014 at 11:08 AM, Peter Cock <p.j.a.cock(a)googlemail.com> wrote: > Thanks Dave, > > The good news is yes, the tests are running again on the > Test Tool Shed (although not the main Tool Shed yet), and > many of my tools now have successful test results from > last night. > > e.g. My new basic mummer tool which now has a full set > of dependency packages thanks to Bjoern: > https://testtoolshed.g2.bx.psu.edu/view/peterjc/mummer > > The bad news is there are many unexpected failures with: > "Exception: History in error state." > > I'm sure you'll learn more once you look over the logs, > > Thank you, > > Peter Right now the mummer example given above is passing, but I am still getting this error on some of my tools, e.g. https://testtoolshed.g2.bx.psu.edu/view/peterjc/blast2go tested 2015-01-25 11:21:12 https://testtoolshed.g2.bx.psu.edu/view/peterjc/get_orfs_or_cdss tested 2015-01-25 10:52:03 (Many of my other tools are also currently listed with different unexpected failures, some of which include the "Exception: History in error state." traceback.) Interestingly, I've just got what appears to be the same exception from a TravisCI test run (following a trivial change to README files): https://travis-ci.org/peterjc/galaxy_blast/builds/48893916 e.g. (one of many similar failing tests in this log) ====================================================================== ERROR: test_tool_000000 (functional.test_toolbox.TestForTool_blastxml_to_top_descr) BLAST top hit descriptions ( blastxml_to_top_descr ) > Test-1 ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/travis/build/peterjc/galaxy_blast/galaxy-central-master/test/functional/test_toolbox.py", line 268, in test_tool self.do_it( td ) File "/home/travis/build/peterjc/galaxy_blast/galaxy-central-master/test/functional/test_toolbox.py", line 37, in do_it stage_data_in_history( galaxy_interactor, testdef.test_data(), test_history, shed_tool_id ) File "/home/travis/build/peterjc/galaxy_blast/galaxy-central-master/test/base/interactor.py", line 42, in stage_data_in_history upload_wait() File "/home/travis/build/peterjc/galaxy_blast/galaxy-central-master/test/base/interactor.py", line 279, in wait while not self.__history_ready( history_id ): File "/home/travis/build/peterjc/galaxy_blast/galaxy-central-master/test/base/interactor.py", line 297, in __history_ready return self._state_ready( state, error_msg="History in error state." ) File "/home/travis/build/peterjc/galaxy_blast/galaxy-central-master/test/base/interactor.py", line 356, in _state_ready raise Exception( error_msg ) Exception: History in error state. I do not recall seeing this problem locally when using my development instllation of galaxy-central. The TravisCI failure suggests this could be a stochastic resource contention (I can re-run the TravisCI build and see if this works second time round?), which is more prone to fail under the Tool Shed framework? Peter

3 6

Interested in hosting GCC2016?
by Dave Clements 03 Feb '15

03 Feb '15

Hello all, The Galaxy Community Conference has been held annually since 2010. This year's event (GCC2015) <http://gcc2015.tsl.ac.uk/> is being hosted by The Sainsbury Lab <http://tsl.ac.uk/> in Norwich, UK. *We are now seeking proposals from any organization interested in hosting GCC2016.* GCC draws 200+ participants from data-intensive life science research. Participants come from around the world, from all career stages, and do research spanning the tree of life. Universities, hospitals and medical schools, research organizations, and industry are all represented, including the largest and most influential organizations in the world. What do we need to host GCC, you ask? In approximately decreasing order of importance: - *Enthusiasm to plan and organize a 3+ day event for more than 200 people.* - Space for 4-6 parallel Training Day sessions, each able to accommodate from 25 to 75 participants. - Central meeting space with capacity for 250-300 people. - Affordability (or access to sponsorship funds; there's a reason we've always held GCC in academic settings) - Nearby space for breakouts, poster sessions and sponsors. - Nearby space for lunch, coffee breaks. - Good wifi for all events (that's 200+ people and their devices). - Space for a hackathon for 2 days before the event. - Easy to get to by air, from within the States, and from overseas - Nearby, affordable housing, or easy walking distance or easy public transport from lodging options to conference facilities - Close proximity to a pub and other social hubs. Hosts get copious (and enthusiastic) organizational support from Galaxy's Outreach Team (who have been helping to organize GCC since 2011), and even more copious gratitude (even *adulation!)* from the Galaxy Community. If you are interested in possibly hosting GCC2016 next summer then please contact Galaxy Outreach <outreach(a)galaxyproject.org>. Our goal is to announce the site at this year's conference in July. Hoping to work with you in 2016! Dave Clements and the Galaxy Team <https://wiki.galaxyproject.org/GalaxyTeam> -- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ https://wiki.galaxyproject.org/

1 0

Questions about upgrading a toolshed instance
by Misharl Monsoor 02 Feb '15

02 Feb '15

Hi everybody, In our lab, we have a Galaxy instance with runs also a toolshed. Recently, we have upgraded our Galaxy instance (August 11 2014, "latest_2014.08.11 ") to the version January 13 2015, "latest_2015.01.13" . My first question is to know if the upgrade of the Galaxy instance is sufficient to upgrade the toolshed instance? If no, is there a procedure to upgrade the toolshed instance separately (a script to upgrade the toolshed database or other toolshed config files for example)? Thank you very much by advance, Bests regards, Misharl.

2 1

concatenate_datasets docker image can't create output
by Jeltje van Baren 02 Feb '15

02 Feb '15

Hi, I'm following instructions at https://github.com/apetkau/galaxy-hackathon-2014. When I try to run the concatenate-datasets on two input files, two odd things happen. First, TWO nearly identical commands are generated in the History panel, only differing in their output filename. In paster.log, only the second one shows up. Second, the program fails, with this info in the History: An error occurred with this dataset: Galaxy slots passed through contain as 1 /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh: line 2: can't create /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_14.dat This happens for both commands in the History panel, only the output filename in the other error is dataset_15.dat Oddly, the dataset_14.dat and dataset_15.dat are both created during this command, they just end up empty. Paster.log: galaxy.jobs.runners DEBUG 2015-01-23 11:08:35,973 (10) command is: docker inspect busybox:ubuntu-14.04 > /dev/null 2>&1 [ $? -ne 0 ] && docker pull busybox:ubuntu-14.04 > /dev/null 2>&1 docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:ro -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:ro -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:rw -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:rw -w /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10 --net none busybox:ubuntu-14.04 /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh; return_code=$?; if [ -f /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file ] ; then cp /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_15.dat ; fi; sh -c "exit $return_code" galaxy.jobs.runners.local DEBUG 2015-01-23 11:08:36,040 (10) executing job script: /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/galaxy_10.sh galaxy.jobs DEBUG 2015-01-23 11:08:36,091 (10) Persisting job destination (destination id: docker_local) What could be happening here, and how do I fix it? Thanks, -Jeltje

2 8

Galaxy installation inside a secure environment
by Abdulrahman Azab 02 Feb '15

02 Feb '15

??Hi Galaxy people, At the University of Oslo, we have an infrastructure for storage and computation on sensitive data, called TSD (presented in the attached image). from inside, the infrastructure is logically divided into projects. Each project has a set of shared VMs (among its users) which are running over a file-system (HNAS). The shared project VMS has access to a computational SLURM cluster which is installed on another file-system (Colossus). There are directories in colossus which are mounted on HNAS for each project in order to allow each project users to push files into colossus and run jobs on the cluster. Users of a particular project can access the project main VMs through user VMs (one for each user). Each project VMs (both shared and user VMs) are in a separate subnet. All of this is inside the TSD. Now to access the TSD from outside, there is a complex authentication mechanism where a user can access his/her own VM. And to transfer data from/to the TSD is another complex story. The important thing is that there is NO internet access in or out.? There are two issues here: 1 - What we need to do is to install one Galaxy VM inside each project area, so that it is accessible by all project users. But we cannot use mercurial to access your distribution server. We can though install a bitbucket server inside the TSD and have the code-base there, so that It can be accessed by all project VMs, but I'm not sure what is the procedure here. 2- We are very concerned about the issue of regularly updating Galaxy instances in projects to the recent release. In many cases it causes many problems, e.g. tool versioning conflicts. So we have the idea of installing each of our tools together with all of its dependencies in a separate docker container, and run those as images on each Galaxy project VM. Is this possible and tested? Should this permanently solve the upgrading problem, or do you suggested another alternative? Thank you, Yours sincerely, Abdulrahman Azab Head engineer, ELIXIR.NO / The Genomic HyperBrowser team Department of Informatics, University of Oslo, Boks 1072 Blindern, NO-0316 OSLO, Norway Email: azab(a)ifi.uio.no, Cell-phone: +47 46797339 ---- Senior Lecturer in Computer Engineering Faculty of Engineering, University of Mansoura, 35516-Mansoura, Egypt Email: abdulrahman.azab(a)mans.edu.eg</owa/abdulrahman.azab(a)mans.edu.eg>

2 1

Pulsar on a remote (SGE) cluster
by Joseph Brent Greer 30 Jan '15

30 Jan '15

Hi all, We have Pulsar working in a synchronous fashion using RESTful services on our local network. Now, we're trying to use Pulsar on a remote (SGE) cluster and we will not have the ability to mount a shared file system. What is the best way to use Galaxy to queue jobs? If Pulsar is the answer, how do we defer to the cluster's queue and have Pulsar wait on it? How should Pulsar notify Galaxy when multiple jobs are done running? We've looked through the documentation and listservs and haven't found anything directly related. Any help or suggestions would be great. Thanks, Joe Greer Proteomics Center of Excellence Northwestern University

2 1

condor jobs
by Shrum, Donald C 30 Jan '15

30 Jan '15

Hi all, I've configured one of our tools to submit jobs to our condor cluster. I can see the job is routed to the condor runner: ==> handler4.log <== galaxy.jobs.handler DEBUG 2015-01-30 09:14:58,092 (508) Dispatching to condor runner galaxy.jobs DEBUG 2015-01-30 09:14:58,204 (508) Persisting job destination (destination id: condor) I can see that indeed the job is submitted to the condor cluster: [root@galaxy galaxy-dist]# condor_q -- Submitter: galaxy.local : <10.177.61.90:55265> : galaxy.local ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 21.0 galaxy 1/30 09:15 0+00:00:02 R 0 0.0 galaxy_509.sh The job begins to run: ==> handler4.log <== galaxy.jobs.runners.condor DEBUG 2015-01-30 09:15:03,827 (508/20) job is now running galaxy.jobs.runners.condor DEBUG 2015-01-30 09:15:05,183 (508/20) job has completed Galaxy is almost immediately removing the job working directory: Here is a snippet of the errors: ==> handler4.log <== galaxy.jobs.runners DEBUG 2015-01-30 09:15:06,372 (508/20) Unable to cleanup /panfs/storage.local/opt/galaxy-dist/database/pbs/galaxy_508.ec: [Errno 2] No such file or directory: '/panfs/storage.local/opt/galaxy-dist/database/pbs/galaxy_508.ec' galaxy.jobs DEBUG 2015-01-30 09:15:06,816 setting dataset state to ERROR galaxy.datatypes.metadata DEBUG 2015-01-30 09:15:06,996 Failed to cleanup MetadataTempFile temp files from /panfs/storage.local/galaxy-data/job_working_directory/000/508/metadata_out_HistoryDatasetAssociation_717_zrzVqh: No JSON object could be decoded Is it possible galaxy is attempting to query condor and see if the job is running, not finding anything and deciding that the job is not running and bailing out? I've reconstructed the process step by step using the logs but I have not been able to see exactly where the condor_submit command is shown so I can try to submit the same job manually. Does anyone have a suggestion for debugging this? Thanks, Don Florida State University Research Computing Center

2 1

data sharing without duplication?
by Fernandez Edgar 30 Jan '15

30 Jan '15

Good morning Gents, I hope everyone is well. I was wondering if it's possible to share data on Galaxy without all the users downloading the data to their history. Let me explain, teachers will be sharing data with their students. And the students will need to work with that specific data. However, I would like them to be able to read that data without replicating it on their history. Is that possible? Cordialement / Regards, Edgar Fernandez System Administrator (Linux) Direction Générale des Technologies de l'Information et de la Communication * Bur. : 1-514-343-6111 poste 16568 Université de Montréal PAVILLON ROGER-GAUDRY, bureau X-218

3 2

Nothing being tested on Test and main Tool Shed?
by Peter Cock 30 Jan '15

30 Jan '15

Hello all, I am currently hoping to review the automated test results for some repositories which I have recently updated, in one case for dependency handling, the other functional changes: https://testtoolshed.g2.bx.psu.edu/view/peterjc/mummer https://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus These have not yet been tested. On further investigation of a sample of my other tools, it appears none of them have been tested on the Test Tool Shed since 2014-09-15, e.g. https://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename https://testtoolshed.g2.bx.psu.edu/view/peterjc/sample_seqs https://testtoolshed.g2.bx.psu.edu/view/peterjc/samtools_idxstats https://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3 https://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod Similarly, some of my tools on the Main Tool Shed appear not to have been tested since 2014-09-21, e.g. https://toolshed.g2.bx.psu.edu/view/peterjc/seq_rename https://toolshed.g2.bx.psu.edu/view/peterjc/effectivet3 https://toolshed.g2.bx.psu.edu/view/peterjc/clinod or 2014-10-27, https://toolshed.g2.bx.psu.edu/view/peterjc/sample_seqs https://toolshed.g2.bx.psu.edu/view/peterjc/samtools_idxstats Is there a known problem with the automated tool testing (previously every second night) on the Tool Sheds? Or have you had to further reduce the testing cycle? Testing less frequently seems fine, say fortnightly, if this can be supplemented by testing updated tools everynight. That would give Tool Authors prompt feedback on their updates, but also catch regressions where changes in Galaxy break a previously working tool. Regards, Peter

4 17

Galactic News!!! February 2015 Edition
by Dave Clements 30 Jan '15

30 Jan '15

1 0