Hi Thon, On Jul 11, 2012, at 3:38 PM, Anthonius deBoer wrote:

My Galaxy version is the most recent, but I had cloned this from an earlier version and have made lots of changes over the months...

This could pose problems unless you are sure that your code is in a functional state after you pull the latest revision from Galaxy central.

I'm a little confused by the fact that the tool_shed tools seem to be defined BOTH by the various XML files AND the database....

The tools themselves are not "defined by the database", but metadata about the tools id stored in the database. For each repository you install from the tool shed, you will have 1 row in the tol_shed_repository table in your database. This table has a column named metadata that stores this information about the repository, including its contents (tools, datatypes, workflows, etc).

I have on occasions manually edited the XML file which I guess screws up the database...

Possibly. If you install a repository from the tool shed that includes a tool config, changing your local version of the tool config could pose problems if you get updates for your installed repository from the tool shed. If you don't do this, however, there should be no problem caused by changing your local version of the tool config.

it is bad practice to have two different places that can easily get out of sync I would think...Either stick with the XML paradigm OR move everything into a database...Doing both is bound to get out of sync....

The way the tool shed components are architected do not result in this "syncing issues". You do not have to use the tool shed at all. Based on your statements that you like to change your versions of the tool configs and Galaxy code, you can protect your version of things by not running the tool migration scripts at all. Your Galaxy instance will still work as you expect.

I don't really want to dive into the database since I don't know the schema and not that good at SQL to begin with, but maybe a script that would sync stuff would be an idea...

Thanks

On Jul 11, 2012, at 12:29 PM, Greg Von Kuster <greg@bx.psu.edu> wrote:

...

Hi Thon,

What revision of Galaxy are you running? Have you made multiple attempts at the migration? Are you running a postgres database? How many records do you have in your tool_dependency table in your database, and what is the status of each record? Have you made any changes to the Galaxy code base in your environment? Any additional information that you can send will help track down the problem in your environment.

Thanks!

Greg Von Kuster

On Jul 11, 2012, at 2:46 PM, Anthonius deBoer wrote:

...

I'm getting this error when I try to run the tool migration:

sh ./scripts/migrate_tools/0003_tools.sh install_dependencies No handlers could be found for logger "docutils" Repositories will be installed into configured tool_path location ../shed_tools Adding new row (or updating an existing row) for repository 'freebayes' in the tool_shed_repository table. Traceback (most recent call last): File "./scripts/migrate_tools/migrate_tools.py", line 21, in <module> app = MigrateToolsApplication( sys.argv[ 1 ] ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/migrate/common.py", line 150, in __init__ install_dependencies=install_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/install_manager.py", line 37, in __init__ self.install_repository( repository_elem, install_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/install_manager.py", line 258, in install_repository install_dependencies=install_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/install_manager.py", line 174, in handle_repository_contents tool_dependencies=tool_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/util/shed_util.py", line 1231, in handle_tool_dependencies tool_dependency = install_package( app, elem, tool_shed_repository, tool_dependencies=tool_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/tool_dependencies/install_util.py", line 52, in install_package install_dir = get_tool_dependency_install_dir( app, tool_shed_repository, package_name, package_version ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/tool_dependencies/install_util.py", line 42, in get_tool_dependency_install_dir repository.installed_changeset_revision ) ) File "/home/tdeboer/code/galaxy_env/lib/python2.6/posixpath.py", line 67, in join elif path == '' or path.endswith('/'): AttributeError: 'NoneType' object has no attribute 'endswith'

Any ideas?

Thanks

Thon ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

Hi John, On Jul 11, 2012, at 4:53 PM, John Chilton wrote:

On Wed, Jul 11, 2012 at 3:25 PM, Greg Von Kuster <greg@bx.psu.edu> wrote:

...

The way the tool shed components are architected do not result in this "syncing issues". You do not have to use the tool shed at all. Based on your statements that you like to change your versions of the tool configs and Galaxy code, you can protect your version of things by not running the tool migration scripts at all. Your Galaxy instance will still work as you expect.

That is not accurate, right?

This is, in fact, accurate.

You are removing the files from the repository, if we merge with you they will disappear also.

That is correct, but my point was that you are not required to use the migration scripts that automatically install the tools that have been migrated out of the Galaxy code base and into the tool shed. These scripts state that you can skip the installation process and simply make a second attempt to start your Galaxy server. Galaxy will start up with the second attempt. If you have made proprietary changes to the tool files, mercurial will ask you if you want to delete the altered file (since it was deleted in the update) or keep the version with your changes. You get to make the choice. If you did not make any proprietary changes to the file, it will get removed automatically with the update. However, you can download the tool from the tool shed as a compressed tarball, uncompress it and store it in the same place that you had it before it was removed from the distribution. Using this approach will not create any database tables, and you will not have to use the migration scripts to install the repository to get access to the migrated tools.

You are also removing lines from tool_conf.xml.sample so if people have hg cp'd that file it will also cause the tools to be lost on auto merges.

You should not hg cp sample files as they can always change with updates. You should simply cp them so you'll have control over your own version of the file.

The periodic merges local instances have to do with galaxy-dist is already a complicated process. It usually takes me an hour or two.

So I don't think you should advocate people diverging even more from galaxy's core code base.

I'm certainly not advocating that policy. I think you simply misunderstood my recommendations to Thon.

Especially, considering during your exchange with me last month, you had advocated pulling changes right from the public bitbucket repository directly into a production code base.

I guess you misunderstood what I was saying as an option for you. I just felt that since you have a test environment and a production environment, you could go through the update process to your test environment, pulling directly from bitbucket. After any issues are resolved, you could use the same process to update your production environment. In stating this, I was not advocating anything - I was simply relaying to you the process we use here at Penn State for our public Galaxy environment, and letting you know that this approach has worked well for us for years. If it doesn't work in your environment, that ok...

-John

...

I don't really want to dive into the database since I don't know the schema and not that good at SQL to begin with, but maybe a script that would sync stuff would be an idea...

Thanks

On Jul 11, 2012, at 12:29 PM, Greg Von Kuster <greg@bx.psu.edu> wrote:

Hi Thon,

What revision of Galaxy are you running? Have you made multiple attempts at the migration? Are you running a postgres database? How many records do you have in your tool_dependency table in your database, and what is the status of each record? Have you made any changes to the Galaxy code base in your environment? Any additional information that you can send will help track down the problem in your environment.

Thanks!

Greg Von Kuster

On Jul 11, 2012, at 2:46 PM, Anthonius deBoer wrote:

...

I'm getting this error when I try to run the tool migration:

sh ./scripts/migrate_tools/0003_tools.sh install_dependencies No handlers could be found for logger "docutils" Repositories will be installed into configured tool_path location ../shed_tools Adding new row (or updating an existing row) for repository 'freebayes' in the tool_shed_repository table. Traceback (most recent call last): File "./scripts/migrate_tools/migrate_tools.py", line 21, in <module> app = MigrateToolsApplication( sys.argv[ 1 ] ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/migrate/common.py", line 150, in __init__ install_dependencies=install_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/install_manager.py", line 37, in __init__ self.install_repository( repository_elem, install_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/install_manager.py", line 258, in install_repository install_dependencies=install_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/install_manager.py", line 174, in handle_repository_contents tool_dependencies=tool_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/util/shed_util.py", line 1231, in handle_tool_dependencies tool_dependency = install_package( app, elem, tool_shed_repository, tool_dependencies=tool_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/tool_dependencies/install_util.py", line 52, in install_package install_dir = get_tool_dependency_install_dir( app, tool_shed_repository, package_name, package_version ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/tool_dependencies/install_util.py", line 42, in get_tool_dependency_install_dir repository.installed_changeset_revision ) ) File "/home/tdeboer/code/galaxy_env/lib/python2.6/posixpath.py", line 67, in join elif path == '' or path.endswith('/'): AttributeError: 'NoneType' object has no attribute 'endswith'

Any ideas?

Thanks

Thon ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

I would not advise doing this. Based on your environment, I recommend that you do not attempt to install tools from the tool shed as doing so may pose problems for the same tools that you have altered in your environment and that you do not want to get "out of sync". On Jul 11, 2012, at 4:09 PM, Anthonius deBoer wrote:

If I drop the following two tables, would those be reconstructed again correctly if I had the XML files in order?

tool_version_association and tool_version

Thanks

Thon

On Jul 11, 2012, at 12:29 PM, Greg Von Kuster <greg@bx.psu.edu> wrote:

...

Hi Thon,

What revision of Galaxy are you running? Have you made multiple attempts at the migration? Are you running a postgres database? How many records do you have in your tool_dependency table in your database, and what is the status of each record? Have you made any changes to the Galaxy code base in your environment? Any additional information that you can send will help track down the problem in your environment.

Thanks!

Greg Von Kuster

On Jul 11, 2012, at 2:46 PM, Anthonius deBoer wrote:

...

I'm getting this error when I try to run the tool migration:

sh ./scripts/migrate_tools/0003_tools.sh install_dependencies No handlers could be found for logger "docutils" Repositories will be installed into configured tool_path location ../shed_tools Adding new row (or updating an existing row) for repository 'freebayes' in the tool_shed_repository table. Traceback (most recent call last): File "./scripts/migrate_tools/migrate_tools.py", line 21, in <module> app = MigrateToolsApplication( sys.argv[ 1 ] ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/migrate/common.py", line 150, in __init__ install_dependencies=install_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/install_manager.py", line 37, in __init__ self.install_repository( repository_elem, install_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/install_manager.py", line 258, in install_repository install_dependencies=install_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/install_manager.py", line 174, in handle_repository_contents tool_dependencies=tool_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/util/shed_util.py", line 1231, in handle_tool_dependencies tool_dependency = install_package( app, elem, tool_shed_repository, tool_dependencies=tool_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/tool_dependencies/install_util.py", line 52, in install_package install_dir = get_tool_dependency_install_dir( app, tool_shed_repository, package_name, package_version ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/tool_dependencies/install_util.py", line 42, in get_tool_dependency_install_dir repository.installed_changeset_revision ) ) File "/home/tdeboer/code/galaxy_env/lib/python2.6/posixpath.py", line 67, in join elif path == '' or path.endswith('/'): AttributeError: 'NoneType' object has no attribute 'endswith'

Any ideas?

Thanks

Thon ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/