Hello Will, On Jun 17, 2014, at 6:06 PM, Will Holtz <wholtz@lygos.com> wrote:

It looks like Dave fixed this issue in 4c58aa19a3d7. Thank you!

However I am still having import issues. I am now getting the message: "Archive of repository package_bowtie_0_12_7 owned by devteam Import failed: repository owner devteam does not have an account in this Tool Shed." This is on a local toolshed running 9b78595ec11 where I am performing the input from an admin account. I'm guessing the issue is that I have 'use_remote_user=True' for LDAP authentication and that means that a devteam account cannot be automatically created to allow this capsule to be added without modification.

Sorry you're still running into problems. No regular development or testing is done using the use_remote_user setting due to resource limitations.

Perhaps on import of a capsule (by an administrator) they could be given the option of preserving the existing owners or re-assigning ownership to an existing user (defaulting to self)?

This would be non-trivial, and probably would introduce fragility into the process. However, I believe there is a solution (see my next comment) although I haven't tested it with the use_remote_user setting.

Of course, what I really want is inter-toolshed dependancies. Maybe I'm missing something, but I'm finding it quite painful just to get a tool development environment setup that makes use of any existing repositories.

There was some work done in this area in the June 2, 2014 release. Here is some information for more easily setting up a local development Tool Shed that use new features introduced in the release. Hopefully this will help. Greg Von Kuster Bootstrapping a New Development Tool Shed The June 2, 2014 release introduces the ability to easily bootstrap a new development Tool Shed to prepare it for importing a repository capsule whose contained repositories can be used as the foundation for developing new Galaxy tools and other utilities. This development Tool Shed can be treated as a component in a multi-step process that simplifies and streamlines Galaxy tool development and validation in the local Tool Shed and moving the validated repositories into the test or main public Galaxy Tool Sheds. Tool Shed framework enhancements included in the June 2, 2014 release support this overall process, which will be explained fully in a future article. Here we’ll restrict our discussion to highlights of the enhancements. Several files are included in a new directory named ~/lib/tool_shed/scripts/api/bootstrap_from_toolshed. The file named user_info.xml.sample should be copied to a file with the same name, but eliminating the .sample extension (i.e., user_info.xml). The information in this file is used to automatically create a new “admin” user account in your local development Tool Shed. This should be the account you use in the test and main public Galaxy Tool Sheds if you plan to export your work from your development Tool Shed and import it into one or both of the public Tool Sheds. If you plan to use this new bootstrapping process, make sure your local development Tool Shed environment is pristine: The hgweb.config file must be empty or missing (it will automatically get created if it doesn’t exist) and the configured location for repositories must be an empty directory. The configured database must be new and not yet migrated. Make sure the ~/lib/tool_shed/scripts/api/bootstrap_from_toolshed/user_info.xml file contains the desired account information. The ~/run_tool_shed.sh script, used for starting up a Tool Shed, has been enhanced to enable this bootstrapping process by using a new -bootstrap_from_tool_shed flag. Here’s an example. %sh run_tool_shed.sh -bootstrap_from_tool_shed http://toolshed.g2.bx.psu.edu The above example will initialize a local development Tool Shed (here we’ll assume its URL is http://localhost:9009) by bootstrapping from the main public Galaxy Tool Shed. The bootstrapping process will perform the following actions in the order listed. Ensure the Tool Shed environment is pristine (e.g., empty hgweb.config file and new database that has not been migrated). Copy all .sample files configured in run_tool_shed.sh. Run the database migration process. Execute the script ~/lib/tool_shed/scripts/bootstrap_tool_shed/create_user_with_api_key.py ~/tool_shed_wsgi.ini to create a new user and an associated API key using the information defined in ~/lib/tool_shed/scripts/bootstrap_tool_shed/user_info.xml. Automatically modify the ~/tool_shed_wsgi.ini file to configure the above user as an admin_user for the Tool Shed. Start the Tool Shed web server. Execute the script ~/lib/tool_shed/scripts/api/create_users.py -a <api_key> -f http://toolshed.g2.bx.psu.edu -t http://localhost:9009. Execute the script ~/lib/tool_shed/scripts/api/create_categories.py -a <api_key> -f http://toolshed.g2.bx.psu./edu -t http://localhost:9009. Executing the run_tool_shed.sh script using this new flag as shown in the example above will create a new local development Tool Shed with an admin user automatically created and configured and the Tool Shed populated with all of the users and categories from the main public Galaxy Tool Shed. This simplifies the process of exporting a repository capsule from the main public Tool Shed and importing it into the local development Tool Shed for further development.

thank you for your help,

-Will

On Wed, Jun 11, 2014 at 12:40 PM, Will Holtz <wholtz@lygos.com> wrote: I am now able to export capsules from the main/test toolsheds -- thanks Dave! When attempting to import these capsules into my local toolshed (latest_2014.06.02 for changeset fb68af9a775a) I receive the following error:

URL: https://galaxy.lygos.com:99/repository/import_capsule File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/debug/prints.py', line 106 in __call__ environ, self.app) File '/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/wsgilib.py', line 543 in intercept_output app_iter = application(environ, replacement_start_response) File '/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-dist/lib/galaxy/webapps/tool_shed/framework/middleware/remoteuser.py', line 74 in __call__ return self.app( environ, start_response ) File '/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 190 in handle_request body = method( trans, **kwargs ) File '/home/galaxy/galaxy-dist/lib/galaxy/webapps/tool_shed/controllers/repository.py', line 1992 in import_capsule import_util.check_status_and_reset_downloadable( trans, import_results_tups ) File '/home/galaxy/galaxy-dist/lib/tool_shed/util/import_util.py', line 34 in check_status_and_reset_downloadable tip_changeset_revision = repository.tip( trans.app ) AttributeError: 'NoneType' object has no attribute 'tip'

I have seen the same behavior for capsules based on the following repositories from the test toolshed: package_biopython_1_62, package_vienna_rna_2_1, and package_bowtie_0_12_7. I am logged in as an admin user for the import process.

thanks, -Will

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

Thanks for the corrections Will, and I'm glad this worked for you. Greg Von Kuster On Jun 18, 2014, at 12:22 PM, Will Holtz <wholtz@lygos.com> wrote:

Hi Greg,

I used the toolshed bootstrapping script and was able to get capsules from testtoolshed to import. Here are a couple of notes that you may find interesting.

1) In a few places you reference the directory ~/lib/tool_shed/scripts/api/bootstrap_from_toolshed/ and these should be changed to ~/lib/tool_shed/scripts/bootstrap_from_toolshed/ 2) I was worried that 'use_remote_user=True' was going to negatively impact the bootstrap process, as accounts are being created that don't exist in LDAP. However, there didn't appear to be any such negative interaction. In user_info.xml I used my email address that maps to my LDAP login (wholtz@lygos.com), a long gibberish password, and wholtz for my username.

Thank you for your help Greg.

-Will

On Tue, Jun 17, 2014 at 7:51 PM, Greg Von Kuster <greg@bx.psu.edu> wrote: Hello Will,

On Jun 17, 2014, at 6:06 PM, Will Holtz <wholtz@lygos.com> wrote:

...

It looks like Dave fixed this issue in 4c58aa19a3d7. Thank you!

However I am still having import issues. I am now getting the message: "Archive of repository package_bowtie_0_12_7 owned by devteam Import failed: repository owner devteam does not have an account in this Tool Shed." This is on a local toolshed running 9b78595ec11 where I am performing the input from an admin account. I'm guessing the issue is that I have 'use_remote_user=True' for LDAP authentication and that means that a devteam account cannot be automatically created to allow this capsule to be added without modification.

Sorry you're still running into problems. No regular development or testing is done using the use_remote_user setting due to resource limitations.

...

Perhaps on import of a capsule (by an administrator) they could be given the option of preserving the existing owners or re-assigning ownership to an existing user (defaulting to self)?

This would be non-trivial, and probably would introduce fragility into the process. However, I believe there is a solution (see my next comment) although I haven't tested it with the use_remote_user setting.

...

Of course, what I really want is inter-toolshed dependancies. Maybe I'm missing something, but I'm finding it quite painful just to get a tool development environment setup that makes use of any existing repositories.

There was some work done in this area in the June 2, 2014 release. Here is some information for more easily setting up a local development Tool Shed that use new features introduced in the release. Hopefully this will help.

Greg Von Kuster Bootstrapping a New Development Tool Shed

The June 2, 2014 release introduces the ability to easily bootstrap a new development Tool Shed to prepare it for importing a repository capsule whose contained repositories can be used as the foundation for developing new Galaxy tools and other utilities. This development Tool Shed can be treated as a component in a multi-step process that simplifies and streamlines Galaxy tool development and validation in the local Tool Shed and moving the validated repositories into the test or main public Galaxy Tool Sheds. Tool Shed framework enhancements included in the June 2, 2014 release support this overall process, which will be explained fully in a future article. Here we’ll restrict our discussion to highlights of the enhancements.

Several files are included in a new directory named ~/lib/tool_shed/scripts/api/bootstrap_from_toolshed. The file named user_info.xml.sample should be copied to a file with the same name, but eliminating the .sample extension (i.e., user_info.xml). The information in this file is used to automatically create a new “admin” user account in your local development Tool Shed. This should be the account you use in the test and main public Galaxy Tool Sheds if you plan to export your work from your development Tool Shed and import it into one or both of the public Tool Sheds.

If you plan to use this new bootstrapping process, make sure your local development Tool Shed environment is pristine:

The hgweb.config file must be empty or missing (it will automatically get created if it doesn’t exist) and the configured location for repositories must be an empty directory. The configured database must be new and not yet migrated. Make sure the ~/lib/tool_shed/scripts/api/bootstrap_from_toolshed/user_info.xml file contains the desired account information. The ~/run_tool_shed.sh script, used for starting up a Tool Shed, has been enhanced to enable this bootstrapping process by using a new -bootstrap_from_tool_shed flag. Here’s an example.

%sh run_tool_shed.sh -bootstrap_from_tool_shed http://toolshed.g2.bx.psu.edu The above example will initialize a local development Tool Shed (here we’ll assume its URL is http://localhost:9009) by bootstrapping from the main public Galaxy Tool Shed. The bootstrapping process will perform the following actions in the order listed.

Ensure the Tool Shed environment is pristine (e.g., empty hgweb.config file and new database that has not been migrated). Copy all .sample files configured in run_tool_shed.sh. Run the database migration process. Execute the script ~/lib/tool_shed/scripts/bootstrap_tool_shed/create_user_with_api_key.py ~/tool_shed_wsgi.ini to create a new user and an associated API key using the information defined in ~/lib/tool_shed/scripts/bootstrap_tool_shed/user_info.xml. Automatically modify the ~/tool_shed_wsgi.ini file to configure the above user as an admin_user for the Tool Shed. Start the Tool Shed web server. Execute the script ~/lib/tool_shed/scripts/api/create_users.py -a <api_key> -f http://toolshed.g2.bx.psu.edu -t http://localhost:9009. Execute the script ~/lib/tool_shed/scripts/api/create_categories.py -a <api_key> -f http://toolshed.g2.bx.psu./edu -t http://localhost:9009. Executing the run_tool_shed.sh script using this new flag as shown in the example above will create a new local development Tool Shed with an admin user automatically created and configured and the Tool Shed populated with all of the users and categories from the main public Galaxy Tool Shed. This simplifies the process of exporting a repository capsule from the main public Tool Shed and importing it into the local development Tool Shed for further development.

...

thank you for your help,

-Will

On Wed, Jun 11, 2014 at 12:40 PM, Will Holtz <wholtz@lygos.com> wrote: I am now able to export capsules from the main/test toolsheds -- thanks Dave! When attempting to import these capsules into my local toolshed (latest_2014.06.02 for changeset fb68af9a775a) I receive the following error:

URL: https://galaxy.lygos.com:99/repository/import_capsule File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/debug/prints.py', line 106 in __call__ environ, self.app) File '/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/wsgilib.py', line 543 in intercept_output app_iter = application(environ, replacement_start_response) File '/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-dist/lib/galaxy/webapps/tool_shed/framework/middleware/remoteuser.py', line 74 in __call__ return self.app( environ, start_response ) File '/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 190 in handle_request body = method( trans, **kwargs ) File '/home/galaxy/galaxy-dist/lib/galaxy/webapps/tool_shed/controllers/repository.py', line 1992 in import_capsule import_util.check_status_and_reset_downloadable( trans, import_results_tups ) File '/home/galaxy/galaxy-dist/lib/tool_shed/util/import_util.py', line 34 in check_status_and_reset_downloadable tip_changeset_revision = repository.tip( trans.app ) AttributeError: 'NoneType' object has no attribute 'tip'

I have seen the same behavior for capsules based on the following repositories from the test toolshed: package_biopython_1_62, package_vienna_rna_2_1, and package_bowtie_0_12_7. I am logged in as an admin user for the import process.

thanks, -Will

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/