October 2013 - galaxy-dev - lists.galaxyproject.org

GA file format documentation
by Philip Mabon 07 Oct '13

07 Oct '13

Is there any documentation on how exported workflow with '.ga' extension are structured? We are looking for the ability to create our workflows without using the workflow editor thru the web portal. Thanks Takadonet

2 3

error checking active jobs : KeyError: 'destination'
by Ganote, Carrie L 07 Oct '13

07 Oct '13

Hi List, I'm having this same issue - if I dispatch a job to dynamic job runner, and it then hands the job to pbs, then it will flip out if I restart the server. In my case, though, there is no error in the job state - it is sitting in the Torque queue where it should be. As far as I can tell, galaxy successfully reloads the job back into the history. But it falters on recovering this destination for pbs. If I delete the queued jobs, it quiets down, but this is not a sustainable solution - I need to be able to reboot the server without all of my users' cluster jobs causing faults every second. Log: galaxy.jobs.runners.pbs DEBUG 2013-10-04 15:50:32,454 Set default PBS server to m1.mason.indiana.edu galaxy.jobs.runners DEBUG 2013-10-04 15:50:32,455 Starting 4 PBSRunner workers galaxy.jobs DEBUG 2013-10-04 15:50:32,459 Loaded job runner 'galaxy.jobs.runners.pbs:PBSJobRunner' as 'pbs' galaxy.jobs.handler DEBUG 2013-10-04 15:50:32,459 Loaded job runners plugins: local:pbs galaxy.jobs.handler INFO 2013-10-04 15:50:32,460 job handler stop queue started galaxy.jobs DEBUG 2013-10-04 15:50:32,478 (514) Working directory for job is: /N/dc/projects/galaxy/job_working_directory/000/514 galaxy.jobs.handler DEBUG 2013-10-04 15:50:32,478 recovering job 514 in pbs runner galaxy.jobs WARNING 2013-10-04 15:50:32,478 (514) Job runner URLs are deprecated, use destinations instead. galaxy.jobs.runners.pbs DEBUG 2013-10-04 15:50:32,479 (514/176938.m1.mason) is still in PBS queued state, adding to the PBS queue galaxy.jobs.handler INFO 2013-10-04 15:50:32,485 job handler queue started ... galaxy.jobs.runners ERROR 2013-10-04 15:50:33,456 Unhandled exception checking active jobs Traceback (most recent call last): File "/N/hd03/galaxy/Mason/galaxy-ncgas/lib/galaxy/jobs/runners/__init__.py", line 362, in monitor self.check_watched_items() File "/N/hd03/galaxy/Mason/galaxy-ncgas/lib/galaxy/jobs/runners/pbs.py", line 385, in check_watched_items ( failures, statuses ) = self.check_all_jobs() File "/N/hd03/galaxy/Mason/galaxy-ncgas/lib/galaxy/jobs/runners/pbs.py", line 466, in check_all_jobs pbs_server_name = self.__get_pbs_server(pbs_job_state.job_destination.params) File "/N/hd03/galaxy/Mason/galaxy-ncgas/lib/galaxy/jobs/runners/pbs.py", line 222, in __get_pbs_server return job_destination_params['destination'].split('@')[-1] KeyError: 'destination' Thanks so much for wisdom on this. Sincerely, Carrie Ganote

1 0

fastx_toolkit not recognizing fastq files
by UMD Bioinformatics 07 Oct '13

07 Oct '13

Hello, I've installed Galaxy on two separate machines. None of the fastx tools recognize my fastq files. They are not appearing in the drop down list on Galaxy for input options. I'm getting the same behavior on both machines. I've used the tool_shed to install fastx on one machine and I've done it manually on the other. fastx_tools work perfectly from the command line. Cheers Ian

2 2

GA file format documentation
by Philip Mabon 07 Oct '13

07 Oct '13

Is there any documentation on how exported workflow with '.ga' extension are structured? We are looking for the ability to create our workflows without using the workflow editor thru the web portal. Thanks

1 0

Re: [galaxy-dev] Functional tests broken, NameError: global name 'db_path' is not defined
by UMD Bioinformatics 07 Oct '13

07 Oct '13

Hello all, I'm having this same issue with Functional Test being broken. I've switched between 'stable' and 'default' and it doesn't seem to fix the issue. Its possible that I'm not switching properly? This is my first local install of Galaxy so I'm new to the structure, detailed help would be very appreciated. Cheers Ian $ sudo hg update default 0 files updated, 0 files merged, 0 files removed, 0 files unresolved $ hg heads | more changeset: 10421:a477486bf18e branch: stable tag: tip user: Nate Coraor <nate(a)bx.psu.edu> date: Thu Sep 26 11:02:58 2013 -0400 summary: Bugfix for tool-to-destination mapping, tool ids are lowercased but the mapping id was not lowercased. changeset: 10411:c42567f43aa7 user: greg date: Mon Aug 19 13:19:56 2013 -0400 summary: Filter invalid objects when generating the list of repository_dependencies objects that are associated with a tool shed repository installed into Galaxy. $ sudo hg update stable 194 files updated, 0 files merged, 48 files removed, 0 files unresolved $ hg heads | more changeset: 10421:a477486bf18e branch: stable tag: tip user: Nate Coraor <nate(a)bx.psu.edu> date: Thu Sep 26 11:02:58 2013 -0400 summary: Bugfix for tool-to-destination mapping, tool ids are lowercased but the mapping id was not lowercased. changeset: 10411:c42567f43aa7 user: greg date: Mon Aug 19 13:19:56 2013 -0400 summary: Filter invalid objects when generating the list of repository_dependencies objects that are associated with a tool shed repository installed into Galaxy. > Message: 16 > Date: Thu, 3 Oct 2013 12:06:16 +0100 > From: Peter Cock <p.j.a.cock(a)googlemail.com> > To: Galaxy Dev <galaxy-dev(a)bx.psu.edu> > Subject: [galaxy-dev] Functional tests broken, NameError: global name > 'db_path' is not defined > Message-ID: > <CAKVJ-_63+TJrLT1=CyxJeFpk1Z=6w9i=MkbaHdc-=dSFFCK+eg(a)mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1 > > Hi all, > > Something has broken on the functional test framework. > > Attempting to run the tests fails, as does just listing the IDs: > > $ ./run_functional_tests.sh --list > /mnt/galaxy/galaxy-central/eggs/twill-0.9-py2.6.egg/twill/other_packages/_mechanize_dist/_auth.py:14: > DeprecationWarning: the md5 module is deprecated; use hashlib instead > import re, base64, urlparse, posixpath, md5, sha, sys, copy > /mnt/galaxy/galaxy-central/eggs/twill-0.9-py2.6.egg/twill/other_packages/_mechanize_dist/_auth.py:14: > DeprecationWarning: the sha module is deprecated; use the hashlib > module instead > import re, base64, urlparse, posixpath, md5, sha, sys, copy > /mnt/galaxy/galaxy-central/eggs/pysam-0.4.2_kanwei_b10f6e722e9a-py2.6-linux-x86_64-ucs4.egg/pysam/__init__.py:1: > RuntimeWarning: __builtin__.file size changed, may indicate binary > incompatibility > from csamtools import * > Traceback (most recent call last): > File "./scripts/functional_tests.py", line 457, in <module> > sys.exit( main() ) > File "./scripts/functional_tests.py", line 246, in main > database_connection = 'sqlite:///' + os.path.join( db_path, > 'universe.sqlite' ) > NameError: global name 'db_path' is not defined > 'run_functional_tests.sh help' for help > > > $ hg branch > stable > [galaxy@ppserver galaxy-central]$ hg heads | more > changeset: 11805:a3c018044538 > tag: tip > user: greg > date: Wed Oct 02 16:44:48 2013 -0400 > summary: Eliminate the problematic "Select one" option from select fields. > > changeset: 11761:a477486bf18e > branch: stable > parent: 11689:4b1c7ad464ea > user: Nate Coraor <nate(a)bx.psu.edu> > date: Thu Sep 26 11:02:58 2013 -0400 > summary: Bugfix for tool-to-destination mapping, tool ids are > lowercased but the mapping id was not lowercased. > > ... > > > Regards, > > Peter > > > ------------------------------ > > Message: 17 > Date: Thu, 03 Oct 2013 08:53:47 -0400 > From: Dave Bouvier <dave(a)bx.psu.edu> > To: Peter Cock <p.j.a.cock(a)googlemail.com>, Galaxy Dev > <galaxy-dev(a)bx.psu.edu> > Subject: Re: [galaxy-dev] Functional tests broken, NameError: global > name 'db_path' is not defined > Message-ID: <524D68DB.5060803(a)bx.psu.edu> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Peter, > > This bug was fixed on the default branch in 10427:33092bc258d7, or > whichever changeset that corresponds to in your repository. The relevant > changelog entry, if you want to find and apply it to your repository, is > as follows: > > changeset: 10427:33092bc258d7 > user: Carl Eberhard <carlfeberhard(a)gmail.com> > date: Wed Aug 21 10:00:09 2013 -0700 > summary: Functional tests: fix sqlite db_path var; fix api test for > hda update with nonsense key > > > --Dave B. > > On 10/03/2013 07:06 AM, Peter Cock wrote: >> Hi all, >> >> Something has broken on the functional test framework. >> >> Attempting to run the tests fails, as does just listing the IDs: >> >> $ ./run_functional_tests.sh --list >> /mnt/galaxy/galaxy-central/eggs/twill-0.9-py2.6.egg/twill/other_packages/_mechanize_dist/_auth.py:14: >> DeprecationWarning: the md5 module is deprecated; use hashlib instead >> import re, base64, urlparse, posixpath, md5, sha, sys, copy >> /mnt/galaxy/galaxy-central/eggs/twill-0.9-py2.6.egg/twill/other_packages/_mechanize_dist/_auth.py:14: >> DeprecationWarning: the sha module is deprecated; use the hashlib >> module instead >> import re, base64, urlparse, posixpath, md5, sha, sys, copy >> /mnt/galaxy/galaxy-central/eggs/pysam-0.4.2_kanwei_b10f6e722e9a-py2.6-linux-x86_64-ucs4.egg/pysam/__init__.py:1: >> RuntimeWarning: __builtin__.file size changed, may indicate binary >> incompatibility >> from csamtools import * >> Traceback (most recent call last): >> File "./scripts/functional_tests.py", line 457, in <module> >> sys.exit( main() ) >> File "./scripts/functional_tests.py", line 246, in main >> database_connection = 'sqlite:///' + os.path.join( db_path, >> 'universe.sqlite' ) >> NameError: global name 'db_path' is not defined >> 'run_functional_tests.sh help' for help >> >> >> $ hg branch >> stable >> [galaxy@ppserver galaxy-central]$ hg heads | more >> changeset: 11805:a3c018044538 >> tag: tip >> user: greg >> date: Wed Oct 02 16:44:48 2013 -0400 >> summary: Eliminate the problematic "Select one" option from select fields. >> >> changeset: 11761:a477486bf18e >> branch: stable >> parent: 11689:4b1c7ad464ea >> user: Nate Coraor <nate(a)bx.psu.edu> >> date: Thu Sep 26 11:02:58 2013 -0400 >> summary: Bugfix for tool-to-destination mapping, tool ids are >> lowercased but the mapping id was not lowercased. >> >> ... >> >> >> Regards, >> >> Peter >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ >> > > > ------------------------------ > > Message: 18 > Date: Thu, 3 Oct 2013 14:41:17 +0100 > From: Peter Cock <p.j.a.cock(a)googlemail.com> > To: Dave Bouvier <dave(a)bx.psu.edu> > Cc: Galaxy Dev <galaxy-dev(a)bx.psu.edu> > Subject: Re: [galaxy-dev] Functional tests broken, NameError: global > name 'db_path' is not defined > Message-ID: > <CAKVJ-_4rHBrLM_zK__cxWRJkwWChR3KhQ3mGgrkNsv06awG19w(a)mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1 > > On Thu, Oct 3, 2013 at 1:53 PM, Dave Bouvier <dave(a)bx.psu.edu> wrote: >> Peter, >> >> This bug was fixed on the default branch in 10427:33092bc258d7, or whichever >> changeset that corresponds to in your repository. The relevant changelog >> entry, if you want to find and apply it to your repository, is as follows: >> >> changeset: 10427:33092bc258d7 >> user: Carl Eberhard <carlfeberhard(a)gmail.com> >> date: Wed Aug 21 10:00:09 2013 -0700 >> summary: Functional tests: fix sqlite db_path var; fix api test for hda >> update with nonsense key >> >> >> --Dave B. > > Thanks - from the date that ought to be on my development system > already, but I can check more specifically now. > > Peter > > > > > Message: 21 > Date: Thu, 3 Oct 2013 15:33:30 +0100 > From: Peter Cock <p.j.a.cock(a)googlemail.com> > To: Dave Bouvier <dave(a)bx.psu.edu> > Cc: Galaxy Dev <galaxy-dev(a)bx.psu.edu> > Subject: Re: [galaxy-dev] Functional tests broken, NameError: global > name 'db_path' is not defined > Message-ID: > <CAKVJ-_4-qw=Kz+5HstT6fHvVUbORbfx1DvsDpX3eiAOTYPgGFA(a)mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1 > > On Thu, Oct 3, 2013 at 1:53 PM, Dave Bouvier <dave(a)bx.psu.edu> wrote: >> Peter, >> >> This bug was fixed on the default branch in 10427:33092bc258d7, or whichever >> changeset that corresponds to in your repository. The relevant changelog >> entry, if you want to find and apply it to your repository, is as follows: >> >> changeset: 10427:33092bc258d7 >> user: Carl Eberhard <carlfeberhard(a)gmail.com> >> date: Wed Aug 21 10:00:09 2013 -0700 >> summary: Functional tests: fix sqlite db_path var; fix api test for hda >> update with nonsense key >> > > Switching to the default branch did indeed fix it - so this needs > to be added to the stable branch (not there yet): > https://bitbucket.org/galaxy/galaxy-central/src/a477486bf18eafdd14dd7ba1e91… > > Thanks, > > Peter > >

1 0

Intermittent workflow errors with local galaxy running postgresql 9.3
by Christensen, Todd 05 Oct '13

05 Oct '13

Hi All, We have been having issues with workflows generating intermittent errors using sqlite. We recently converted to postgresql 9.3 in an attempt to resolve the issue as suggested in prior posts. Despite switching to postgresql we continue to have these errors. Does anyone have any additional suggestions that may resolve this issue? Thanks, Todd Traceback (most recent call last): File "/opt/paml/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 122, in prepare_job job_wrapper.runner_command_line = self.build_command_line( job_wrapper, include_metadata=include_metadata, include_work_dir_outputs=include_work_dir_outputs ) File "/opt/paml/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 172, in build_command_line work_dir_outputs = self.get_work_dir_outputs( job_wrapper ) File "/opt/paml/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 222, in get_work_dir_outputs for dataset in dataset_assoc.dataset.dataset.history_associations + dataset_assoc.dataset.dataset.library_associations: File "/opt/paml/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/attributes.py", line 168, in __get__ return self.impl.get(instance_state(instance),dict_) File "/opt/paml/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/attributes.py", line 453, in get value = self.callable_(state, passive) File "/opt/paml/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/strategies.py", line 508, in _load_for_state return self._emit_lazyload(session, state, ident_key) File "/opt/paml/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/strategies.py", line 574, in _emit_lazyload result = q.all() File "/opt/paml/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 2115, in all return list(self) File "/opt/paml/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 2227, in __iter__ return self._execute_and_instances(context) File "/opt/paml/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File "/opt/paml/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1449, in execute params) File "/opt/paml/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement compiled_sql, distilled_params File "/opt/paml/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context context) File "/opt/paml/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1691, in _execute_context context) File "/opt/paml/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/default.py", line 331, in do_execute cursor.execute(statement, parameters) OperationalError: (OperationalError) database is locked u'SELECT library_dataset_dataset_association.id AS library_dataset_dataset_association_id, PAML EMAIL DISCLAIMER: Information contained in this message may be privileged and confidential. If the reader of this message is not the intended recipient, be notified that any dissemination, distribution or copying of this communication is strictly prohibited. If this communication is received in error, please notify the sender immediately by replying to the message and deleting from your computer. Thank you

2 1

multiple input files
by Ulf Schaefer 04 Oct '13

04 Oct '13

Dear all We frequently find ourselves in situations where a tool needs to be run with a lot of input files. For example, run the GATK UnifiedGenotyper with easily dozens of bam files. Using the <repeat> in this case requires quite a bit of clicking. Is there a more conventient way fo doing this? Maybe similar to the multi-file-select that is possible for workflow inputs? I saw some older discussions on this or similar issues, but I am a bit lost what the current official stable proposed solution for this is. Thanks for your help Ulf ************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of Public Health England, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. http://www.gov.uk/PHE **************************************************************************

4 8

Re: [galaxy-dev] [galaxy-user] slow Galaxy server
by Peter Cock 03 Oct '13

03 Oct '13

Hello all, One of the users on our local galaxy has reported this error, trying IE9, Firefox and Chrome on Windows: "An error occurred while getting updates from the server. Please contact a Galaxy administrator if the problem persists." Meanwhile the server works fine for me. Are there any known causes of this error (other than heavy load which does not seem to be the case here)? Thanks, Peter On Fri, Aug 2, 2013 at 5:49 PM, Politz, Samuel M. <spolitz(a)wpi.edu> wrote: > This is a followup to my previous post on this topic. The exact message (in > red) that I see after a long delay in running a tool is “An error occurred > while getting updates from the server. Please contact a Galaxy administrator > if the problem persists.” I have seen this message twice while waiting for > two unrelated tools to run. There is no green’bug’ icon so I could not > report this to an administrator. Another main server user reported this > error message to the user mailing list as well. The response to their post > was to wait a few minutes. Well, in this case I have been waiting > overnight. > > > > Best regards, > > Sam Politz > > > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/

1 0

Functional tests broken, NameError: global name 'db_path' is not defined
by Peter Cock 03 Oct '13

03 Oct '13

Hi all, Something has broken on the functional test framework. Attempting to run the tests fails, as does just listing the IDs: $ ./run_functional_tests.sh --list /mnt/galaxy/galaxy-central/eggs/twill-0.9-py2.6.egg/twill/other_packages/_mechanize_dist/_auth.py:14: DeprecationWarning: the md5 module is deprecated; use hashlib instead import re, base64, urlparse, posixpath, md5, sha, sys, copy /mnt/galaxy/galaxy-central/eggs/twill-0.9-py2.6.egg/twill/other_packages/_mechanize_dist/_auth.py:14: DeprecationWarning: the sha module is deprecated; use the hashlib module instead import re, base64, urlparse, posixpath, md5, sha, sys, copy /mnt/galaxy/galaxy-central/eggs/pysam-0.4.2_kanwei_b10f6e722e9a-py2.6-linux-x86_64-ucs4.egg/pysam/__init__.py:1: RuntimeWarning: __builtin__.file size changed, may indicate binary incompatibility from csamtools import * Traceback (most recent call last): File "./scripts/functional_tests.py", line 457, in <module> sys.exit( main() ) File "./scripts/functional_tests.py", line 246, in main database_connection = 'sqlite:///' + os.path.join( db_path, 'universe.sqlite' ) NameError: global name 'db_path' is not defined 'run_functional_tests.sh help' for help $ hg branch stable [galaxy@ppserver galaxy-central]$ hg heads | more changeset: 11805:a3c018044538 tag: tip user: greg date: Wed Oct 02 16:44:48 2013 -0400 summary: Eliminate the problematic "Select one" option from select fields. changeset: 11761:a477486bf18e branch: stable parent: 11689:4b1c7ad464ea user: Nate Coraor <nate(a)bx.psu.edu> date: Thu Sep 26 11:02:58 2013 -0400 summary: Bugfix for tool-to-destination mapping, tool ids are lowercased but the mapping id was not lowercased. ... Regards, Peter

2 3

Please save these dates! GCC2014: June 30 - July 2
by Dave Clements 02 Oct '13

02 Oct '13

Hello all, The 2014 Galaxy Community Conference (GCC2014)<http://wiki.galaxyproject.org/Events/GCC2014> has been scheduled for June 30 through July 2, at the Homewood Campus<http://webapps.jhu.edu/jhuniverse/information_about_hopkins/campuses/homewo…> of Johns Hopkins University <http://jhu.edu/>, in Baltimore, Maryland<http://visitors.baltimorecity.gov/>, United States. Galaxy Community Conferences are an opportunity to participate in presentations, discussions, poster sessions, lightning talks and breakouts, all about high-throughput biology and the tools that support it. The conference will also includes a*Training Day* offering in-depth topic coverage, across several concurrent sessions. See the GCC2013 web site<http://wiki.galaxyproject.org/Events/GCC2013> for an idea of what happens at a Galaxy Community Conference. See you next summer, GCC2014 Organizing Committee<http://wiki.galaxyproject.org/Events/GCC2014/Organizers> -- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ http://wiki.galaxyproject.org/

1 0