- galaxy-dev - lists.galaxyproject.org

reducing overhead for quick jobs
by Robert Baertsch 30 Mar '13

30 Mar '13

Hi All, After changing the sleep in manager.py from 5 seconds to 0.2 seconds, the job start is very snappy. I did not touch the various sleep(10)s that don't seem to be relevant. I'm testing how long it takes to sort a short bed file in galaxy. After the fix, the problem seems to be at the end of the process. In the timeline below the gap between the job ending on the server and the client getting the update is 5 seconds. Is that a built in sleep or something else? Can I turn off metadata for tools that don't generate it? BTW: thousands of seconds are on in the manager and handler logs and should be turned on in the main.log. Rather than 2 round trips to the server. I think we need a special class of tools that run in a single round trip (like the eyeball viz). The max run time for this class of jobs should be no more than a few seconds. Is that is the roadmap somewhere? -Robert Baertsch UC Santa Cruz https://medbook.ucsc.edu RUN1 - 9 seconds (1 sec to start, 0.2 sec to run , 2 sec metadata, 5.5 secs to update client) 12:08:12.? main: log job on main thread 12:08:12.8 manager : galaxy.jobs.manager DEBUG 2013-03-30 12:08:12,867 (2308) Job assigned to handler 'handler2' 12:08:13.26 handler: start handler 12:08:13.35 handler: job dispatched 12:08:13,44 handler: job runner started 12:08:13,72 handler: python tool started 12:08:13,91 handler: python tool finished 12:08:13,98 handler: set_metadata.sh started 12:08:15,70 handler: metadata ended 12:08:15,85 handler: job Ended 12:08:21 main: /api/histories - update on client RUN2 - 9 seconds (1.8 sec to start, 0.2 sec to run, 2 sec metadata, 4.5 sec to update client) 12:23:00.? main: log job on main thread 12:23:00,60 manager : galaxy.jobs.manager DEBUG 2013-03-30 12:08:12,867 (2308) Job assigned to handler 'handler2' 12:23:01,63 handler: start handler 12:23:01,72 handler: job dispatched 12:23:01,81 handler: job runner started 12:23:02,44 handler: python tool started 12:23:02,64 handler: python tool finished 12:23:02,72 handler: set_metadata.sh started 12:23:04,64 handler: metadata ended 12:23:04,71 handler: job Ended 12:23:09 main: /api/histories - update on client

1 0

Freesurfer Files MGZ Extension Stripped
by Bill White 30 Mar '13

30 Mar '13

I am having trouble with Galaxy removing the .mgz file extension from Freesurfer files I upload. I understand .gz gets stripped for gzipped files, but the pattern matching algorithm for detecting the extension must be treating the '.' as 'any character' rather than a literal '.' character. I am left with .m files. Anyone else have this problem? The .mgz file is not a compressed format. -- Bill C. White, MS Research Associate Programmer University of Tulsa Tandy School of Computer Science http://insilico.utulsa.edu/

1 0

Re: [galaxy-dev] inable to login to testtoolshed login because of Server Error: AttributeError: 'GalaxySession' object has no attribute 'current_history'
by Dave Bouvier 29 Mar '13

29 Mar '13

Jorrit, Thank you for the additional information, following those steps I was able to reproduce the error and will be looking into the issue. --Dave B. On 3/28/13 18:04:48.000, jorrit poelen wrote: > Hey Dave, > > I just reproduced this issue by: > > 1. clear all cookies (reset browser) > 2. login to testtoolshed.g2.bx.psu.edu > 3. login succeeds > 4. logout > 5. logout succeeds > 6. (re-)login to testtoolshed.g2.bx.psu.edu > 7. login fails with Server Error (for stacktrace, see below). > > thx, > -jorrit > > URL: http://testtoolshed.g2.bx.psu.edu/user/login?use_panels=True > File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Paste-1.7.5.1-py2.7.egg/paste/exceptions/errormiddleware.py', line 144 in __call__ > app_iter = self.application(environ, sr_checker) > File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Paste-1.7.5.1-py2.7.egg/paste/debug/prints.py', line 106 in __call__ > environ, self.app) > File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Paste-1.7.5.1-py2.7.egg/paste/wsgilib.py', line 543 in intercept_output > app_iter = application(environ, replacement_start_response) > File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', line 84 in __call__ > return self.application(environ, start_response) > File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', line 633 in __call__ > return self.application(environ, start_response) > File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/framework/base.py', line 128 in __call__ > return self.handle_request( environ, start_response ) > File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/framework/base.py', line 184 in handle_request > body = method( trans, **kwargs ) > File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/webapps/galaxy/controllers/user.py', line 466 in login > active_view="user" ) > File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/framework/__init__.py', line 863 in fill_template > return self.fill_template_mako( filename, **kwargs ) > File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/framework/__init__.py', line 874 in fill_template_mako > return template.render( **data ) > File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Mako-0.4.1-py2.7.egg/mako/template.py', line 296 in render > return runtime._render(self, self.callable_, args, data) > File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line 660 in _render > **_kwargs_for_callable(callable_, data)) > File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line 692 in _render_context > _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) > File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line 718 in _exec_template > callable_(context, *args, **kwargs) > File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/database/compiled_templates/community/base/base_panels.mako.py', line 85 in render_body > __M_writer(unicode(self.masthead())) > File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/database/compiled_templates/community/webapps/galaxy/base_panels.mako.py', line 209 in render_masthead > tab( "lab", "Lab", None, menu_options=menu_options, visible=( trans.user and ( trans.user.requests or trans.app.security_agent.get_accessible_request_types( trans, trans.user ) ) ) ) > AttributeError: 'User' object has no attribute 'requests' > > On Mar 28, 2013, at 2:22 PM, Dave Bouvier wrote: > >> Jorrit, >> >> While I investigate the root cause of this, have you tried clearing all cookies for testtoolshed.g2.bx.psu.edu? >> >> --Dave B. >> >> On 3/28/13 17:14:55.000, jorrit poelen wrote: >>> Hey y'all galaxiers, >>> >>> Today, I tried to login using the testtoolshed web ui. To make sure that it wasn't just isolated to an existing account, I created a new account, and still I encountered the Server Error attached below. This error is not shown when entering an invalid password. I was able to login and push hg changes a couple of days ago. >>> >>> Was the test toolshed recently upgraded? >>> >>> Please let me know what I can do to login to testtoolshed at http://testtoolshed.g2.bx.psu.edu >>> >>> thx, >>> -jorrit >>> >>> PS Please find attached stack trace on login with valid password through toolshed web ui at around 2pm PST. >>> >>> URL: http://testtoolshed.g2.bx.psu.edu/user/login?use_panels=True >>> File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Paste-1.7.5.1-py2.7.egg/paste/exceptions/errormiddleware.py', line 144 in __call__ >>> app_iter = self.application(environ, sr_checker) >>> File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Paste-1.7.5.1-py2.7.egg/paste/debug/prints.py', line 106 in __call__ >>> environ, self.app) >>> File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Paste-1.7.5.1-py2.7.egg/paste/wsgilib.py', line 543 in intercept_output >>> app_iter = application(environ, replacement_start_response) >>> File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', line 84 in __call__ >>> return self.application(environ, start_response) >>> File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', line 633 in __call__ >>> return self.application(environ, start_response) >>> File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/framework/base.py', line 128 in __call__ >>> return self.handle_request( environ, start_response ) >>> File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/framework/base.py', line 184 in handle_request >>> body = method( trans, **kwargs ) >>> File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/webapps/galaxy/controllers/user.py', line 466 in login >>> active_view="user" ) >>> File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/framework/__init__.py', line 863 in fill_template >>> return self.fill_template_mako( filename, **kwargs ) >>> File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/framework/__init__.py', line 874 in fill_template_mako >>> return template.render( **data ) >>> File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Mako-0.4.1-py2.7.egg/mako/template.py', line 296 in render >>> return runtime._render(self, self.callable_, args, data) >>> File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line 660 in _render >>> **_kwargs_for_callable(callable_, data)) >>> File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line 692 in _render_context >>> _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) >>> File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line 718 in _exec_template >>> callable_(context, *args, **kwargs) >>> File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/database/compiled_templates/community/base/base_panels.mako.py', line 85 in render_body >>> __M_writer(unicode(self.masthead())) >>> File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/database/compiled_templates/community/webapps/galaxy/base_panels.mako.py', line 337 in render_masthead >>> usage = trans.app.quota_agent.get_usage( trans=trans ) >>> File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/quota/__init__.py', line 26 in get_usage >>> history = trans.history >>> File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/framework/__init__.py', line 741 in get_history >>> history = self.galaxy_session.current_history >>> AttributeError: 'GalaxySession' object has no attribute 'current_history' >>> >>> >>> >>> ___________________________________________________________ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>> http://lists.bx.psu.edu/ >>> >>> To search Galaxy mailing lists use the unified search at: >>> http://galaxyproject.org/search/mailinglists/ >>> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ > >

1 1

Re: [galaxy-dev] Exporting histories fails: no space left on device
by Jeremy Goecks 29 Mar '13

29 Mar '13

Please keep all replies on-list so that everyone can contribute. Someone more knowledgeable about systems than me suggests that lsof(8) and/or /proc/<galaxy server pid>/fd should yield some clues as to what file is being written to. Good luck, J. On Mar 25, 2013, at 10:01 AM, Joachim Jacob | VIB | wrote: > Hi, > > About the exporting of history, which fails: > 1. the preparation seems to work fine: meaning: choosing 'Export this history' in the History menu leads to a URL that reports initially that the export is still in progress. > > 2. when the export is finished, and I click the download link, the root partition fills and the browser displays "Error reading from remote server". A folder ccpp-2013-03-25-14:51:15-27045.new is created in the directory /var/spool/abrt, which fills the root partition. > > The handler reports in its log: > """ > galaxy.jobs DEBUG 2013-03-25 14:38:33,322 (8318) Working directory for job is: /mnt/galaxydb/job_working_directory/008/8318 > galaxy.jobs.handler DEBUG 2013-03-25 14:38:33,322 dispatching job 8318 to local runner > galaxy.jobs.handler INFO 2013-03-25 14:38:33,368 (8318) Job dispatched > galaxy.jobs.runners.local DEBUG 2013-03-25 14:38:33,432 Local runner: starting job 8318 > galaxy.jobs.runners.local DEBUG 2013-03-25 14:38:33,572 executing: python /home/galaxy/galaxy-dist/lib/galaxy/tools/imp_exp/export_history.py -G /mnt/galaxytemp/tmpHAEokb/tmpQM6g_R /mnt/galaxytemp/tmpHAEokb/tmpeg7bYF /mnt/galaxytemp/tmpHAEokb/tmpPXJ245 /mnt/galaxydb/files/013/dataset_13993.dat > galaxy.jobs.runners.local DEBUG 2013-03-25 14:41:29,420 execution finished: python /home/galaxy/galaxy-dist/lib/galaxy/tools/imp_exp/export_history.py -G /mnt/galaxytemp/tmpHAEokb/tmpQM6g_R /mnt/galaxytemp/tmpHAEokb/tmpeg7bYF /mnt/galaxytemp/tmpHAEokb/tmpPXJ245 /mnt/galaxydb/files/013/dataset_13993.dat > galaxy.jobs DEBUG 2013-03-25 14:41:29,476 Tool did not define exit code or stdio handling; checking stderr for success > galaxy.tools DEBUG 2013-03-25 14:41:29,530 Error opening galaxy.json file: [Errno 2] No such file or directory: '/mnt/galaxydb/job_working_directory/008/8318/galaxy.json' > galaxy.jobs DEBUG 2013-03-25 14:41:29,555 job 8318 ended > """ > > The system reports: > """ > Mar 25 14:51:26 galaxy abrt[16805]: Write error: No space left on device > Mar 25 14:51:27 galaxy abrt[16805]: Error writing '/var/spool/abrt/ccpp-2013-03-25-14:51:15-27045.new/coredump' > """ > > > Thanks, > Joachim > > > Joachim Jacob > > Rijvisschestraat 120, 9052 Zwijnaarde > Tel: +32 9 244.66.34 > Bioinformatics Training and Services (BITS) > http://www.bits.vib.be > @bitsatvib > > > > On Tue 19 Mar 2013 11:22:27 PM CET, Jeremy Goecks wrote: >> I'm unable to reproduce this behavior using a clean version of galaxy-dist. The code (export_history.py) doesn't create any temporary files and appears to write directly to the output file, so it seems unlikely that Galaxy is writing anything to the root directory. >> >> Can you provide the name of any file that Galaxy appears to be writing to outside of <galaxy-home>? What about watching the job output file/export file to see if that's increasingly in size and causing the out-of-space error? >> >> Best, >> J. >> >> On Mar 19, 2013, at 10:56 AM, Joachim Jacob | VIB | wrote: >> >>> Hi all, >>> >>> >>> Exporting histories fails on our server: " Reason: *Error reading from remote server". >>> >>> *When looking at the logs and the system: >>> tail /var/log/messages >>> Mar 19 15:52:47 galaxy abrt[25605]: Write error: No space left on device >>> Mar 19 15:52:49 galaxy abrt[25605]: Error writing '/var/spool/abrt/ccpp-2013-03-19-15:52:37-13394.new/coredump' >>> >>> So I watched my system when I repeated the export, and saw that Galaxy fills up the root directory (/), instead of any temporary directory. >>> >>> Somebody has an idea where to adjust this setting, so the export function uses any temporary directory? >>> >>> >>> Thanks, >>> Joachim >>> >>> -- >>> Joachim Jacob >>> >>> Rijvisschestraat 120, 9052 Zwijnaarde >>> Tel: +32 9 244.66.34 >>> Bioinformatics Training and Services (BITS) >>> http://www.bits.vib.be >>> @bitsatvib >>> >>> ___________________________________________________________ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>> >>> http://lists.bx.psu.edu/ >> >> >>

3 6

Re: [galaxy-dev] galaxy not picking up updates from test toolshed
by Dave Bouvier 29 Mar '13

29 Mar '13

Jorrit, I've not been able to reproduce the first scenario using a local Galaxy instance against the test tool shed. Could you tell me which revision of Galaxy you're running when you encounter this behavior? I do see that the revision bb20766df19c that you were initially trying to update is immediately followed by another downloadable revision, in which case Galaxy will (correctly) not find any updates for it, due to the reproducibility concerns I mentioned in my previous email. --Dave B. On 3/28/13 17:58:10.000, jorrit poelen wrote: > Hey Dave, > > I took some time to try to reproduce the behavior you described using a local test tool shed. > > As I understand your previous email, the first scenario (below) outcome of using my local test toolshed is expected, but the second scenario is not. Also, the first scenario outcome using the psu test toolshed seems different compared to running it against my local test tool shed. I was unable to test the second scenario, because of a apparent (recently introduced) bug in the psu test tool shed login. > > What am I missing? > > Thanks much for you patience, > -jorrit > > ------- > First scenario - Change README / update > -------- > Steps: > 1. install repository A from local test tool shed B into local galaxy instance C > 2. clone respository A from local test tool into local dev environment > **3. apply change to README** > 4. hg commit / push changes to local test tool shed B > 5. verify that test tool shed received changes by checking content of README and change set hash. > 6. in galaxy instance C > admin > manage installed tool shed repositories > repository A > get updates > 7. galaxy instance C web page says that updates have been detected and applied > **8. verify that README of repo A in galaxy instance C has been changed.** > > Outcome: > local tool shed instance test - update detected, changes were applied to local galaxy instance C > > psu tool shed test - update *not* detected, changes were *not* applied to local galaxy instance C (see initial post) > > ------ > Second scenario - Bump tool version / update > ------ > Steps: same as first scenario, except for > **3. apply change to tool version 0.1 --> 0.11 ** > **8. verify that tool version 0.1 has changed to 0.11 of repo A in galaxy instance C has been changed.** > > Outcome: > local tool shed instance test - update detected, changes were applied: a new version of tool is installed in Galaxy instance. > > psu tool shed test - unable to test because I could not login to psu test toolshed > > > > > > > > On Mar 28, 2013, at 2:19 PM, Dave Bouvier wrote: > >> Jorrit, >> >> I may have been a bit unclear in my previous response. Allow me to illustrate two scenarios: >> >> First scenario: >> >> 1. A repository is created containing a BWA tool with version 0.5.9 >> 2. The repository is installed to Galaxy. >> 3. New README file is uploaded, tool version is *not* changed. >> >> Second scenario: >> >> 1. A repository is created containing a BWA tool with version 0.5.9 >> 2. The repository is installed to Galaxy. >> 3. New README file is uploaded, and updated BWA tool is uploaded with version 0.6 >> >> In the first scenario, Galaxy will find that the repository has been updated, and apply the update. >> >> In the second scenario, Galaxy will not find that the repository has been updated, but will allow you to install both repositories, the one containing version 0.5.9 and the one containing version 0.6 of the BWA tool. >> >> The reason for this behavior is that if a tool has been updated to produce different output, the tool version should be updated, which leads to a separate installable changeset revision being generated. This way, any analyses run on the old tool version can be re-run and produce the same output, but any new analyses can run on the new version of the tool, with whatever new features have been added. This ensures reproducibility throughout the version lineage of the tool, while still allowing Galaxy users to use the most recent version of an available tool for future analyses. >> >> --Dave B. >> >> On 3/28/13 17:03:10.000, jorrit poelen wrote: >>> Hey Dave, >>> >>> Thanks for pointing me to the wiki that helped demystify the hg changeset <--> repository metadata connection for me. >>> >>> I'll make sure to change the version numbers in the tool xml definitions whenever a change needs to be make available to galaxy instances through the toolshed. So, even for an update in the README file, I'll make sure to change some version to make the change available. This feels a little clunky for me now, but I am sure there's a good reason to make it work as it does now. >>> >>> I very much appreciate your prompt response and I'll make sure to dig around the wiki a little longer before posting a question next time ; ) >>> >>> -jorrit >>> >>> On Mar 26, 2013, at 10:41 AM, Dave Bouvier wrote: >>> >>>> Jorrit, >>>> >>>> First, allow me to clarify a point of naming: >>>> >>>> It's not quite right to say "tools" in this situation, as tools are only one of several utilities that a tool shed repository can contain, along with datatypes, workflows, repository dependencies, and so on. This is an important distinction, because some repositories can (and do) contain no tools, and only datatypes and/or other Galaxy utilities. >>>> >>>> In your process description for the current behavior, the tool shed will not report updates to a repository if the tool version has changed. For more details on how and why this happens, see the following wiki page: >>>> >>>> http://wiki.galaxyproject.org/RepositoryRevisions#Adding_additional_change_… >>>> >>>> Where the behavior is detailed in these sections: >>>> >>>> Adding additional change sets to the initial change set in a repository >>>> • What is repository metadata? >>>> • What does it mean when repository metadata is associated with a change set in the repository change log? >>>> • Adding a change set that does not generate a new repository metadata record >>>> • Adding a change set that generates a new repository metadata record >>>> • More examples of installable repository change set revisions >>>> >>>> >>>> --Dave B. >>>> >>>> On 3/26/13 12:45:41.000, jorrit poelen wrote: >>>>> Hey Dave, >>>>> >>>>> Thanks against for your quick response! >>>>> >>>>> I checked my universe_wsgi.ini file and can confirm that the enabled_tool_shed_check was *commented*. >>>>> >>>>> I've uncommented the settings and changed them to: >>>>> >>>>> enable_tool_shed_check = True >>>>> hours_between_check = 1 >>>>> >>>>> Then I followed the steps outlined in my initial email and the observed the same (unexpected) behavior. >>>>> >>>>> Perhaps I am describing a feature request rather than a bug: My expectation is that whenever I explicitly request to update a tool, a immediate check is performed against the relevant tool shed. >>>>> >>>>> To make sure that I correctly understand the existing tool update feature, please confirm that the following describes the currently expected tool update behavior: >>>>> >>>>> 0. admin of galaxy instance Z enables the feature to check for tool updates in toolsheds every hour >>>>> 1. tool X revision 1.0 is installed from tool shed Y into galaxy instance Z by tool developer using hg push >>>>> 2. tool X is current revision is updated to revision 1.1 in tool shed Y >>>>> 3. immediately after, admin of galaxy instance Z attempts to (manually) upgrade tool X to version 1.1 using the admin web interface >>>>> 4. galaxy instance Z reports that no new version of tool X are available, even though a new version exists in the toolshed >>>>> 5. galaxy instance Z detects version 1.1 of tool X at most one hour after the update was made >>>>> 6. admin of galaxy instance Z tries again >>>>> >>>>> Thanks for your patience and efforts, >>>>> >>>>> -jorrit >>>>> >>>>> On Mar 26, 2013, at 8:19 AM, Dave Bouvier wrote: >>>>> >>>>>> Jorrit, >>>>>> >>>>>> Just to eliminate one possible cause of this behavior, could you confirm that your local Galaxy instance's universe_wsgi.ini file's enable_tool_shed_check setting is uncommented and set to True, and the hours_between_check is also uncommented and set to a value between 1 and 12? >>>>>> >>>>>> --Dave B. >>>>>> >>>>>> On 3/25/13 21:26:07.000, jorrit poelen wrote: >>>>>>> Hey y'all, >>>>>>> >>>>>>> I noticed that my local galaxy instance doens't pickup latest updates from test toolshed. >>>>>>> >>>>>>> Steps to reproduce: >>>>>>> >>>>>>> 1. install obotools from test toolshed using hg clone http://jorrit@testtoolshed.g2.bx.psu.edu/repos/jorrit/obotools , revision 45:bb20766df19c >>>>>>> >>>>>>> 2. hg push new change of obotools of galaxy tool to testtoolshed , creating revision 46:532808477800 >>>>>>> >>>>>>> 3. update obotools in galaxy admin >>>>>>> >>>>>>> expected result: galaxy admin detects new version and give an option to update. >>>>>>> >>>>>>> actual result: galaxy admin does not detect new version. >>>>>>> >>>>>>> I've attached some galaxy logging below. One thing I noticed that the reported commit time in testtoolshed seems 7 hours off - just minutes after I pushed the change, testtoolshed says the change was commit "~ 7 hours ago". >>>>>>> >>>>>>> Workaround I am using to do update is to deactivate and remove the tool all together, and re-install straight from test-toolshed. What should I do to make the tool version update work? I am probably missing something. >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> -jorrit >>>>>>> >>>>>>> >>>>>>> Galaxy logging on looking for updates in test toolshed: >>>>>>> --- >>>>>>> >>>>>>> 24.130.122.217 - - [26/Mar/2013:01:12:46 +0000] "GET /admin_toolshed/browse_repositories HTTP/1.1" 200 - "http://galaxy:7474/admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10" >>>>>>> 24.130.122.217 - - [26/Mar/2013:01:12:51 +0000] "GET /admin_toolshed/browse_repositories?operation=get+updates&id=a799d38679e985db HTTP/1.1" 302 - "http://galaxy:7474/admin_toolshed/browse_repositories" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10" >>>>>>> 24.130.122.217 - - [26/Mar/2013:01:12:51 +0000] "GET /admin_toolshed/check_for_updates?id=a799d38679e985db HTTP/1.1" 302 - "http://galaxy:7474/admin_toolshed/browse_repositories" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10" >>>>>>> 24.130.122.217 - - [26/Mar/2013:01:12:52 +0000] "GET /admin_toolshed/update_to_changeset_revision?tool_shed_url=http://testtoolshed.g2.bx.psu.edu/&name=obotools&owner=jorrit&changeset_revision=bb20766df19c&latest_changeset_revision=bb20766df19c&latest_ctx_rev=45 HTTP/1.1" 302 - "http://galaxy:7474/admin_toolshed/browse_repositories" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10" >>>>>>> 24.130.122.217 - - [26/Mar/2013:01:12:52 +0000] "GET /admin_toolshed/manage_repository?status=done&message=The+installed+repository+named+%27obotools%27+is+current%2C+there+are+no+updates+available.++&id=a799d38679e985db HTTP/1.1" 200 - "http://galaxy:7474/admin_toolshed/browse_repositories" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10" >>>>>>> ___________________________________________________________ >>>>>>> Please keep all replies on the list by using "reply all" >>>>>>> in your mail client. To manage your subscriptions to this >>>>>>> and other Galaxy lists, please use the interface at: >>>>>>> http://lists.bx.psu.edu/ >>>>>>> >>>>>>> To search Galaxy mailing lists use the unified search at: >>>>>>> http://galaxyproject.org/search/mailinglists/ >>>>>>> >>>>>> ___________________________________________________________ >>>>>> Please keep all replies on the list by using "reply all" >>>>>> in your mail client. To manage your subscriptions to this >>>>>> and other Galaxy lists, please use the interface at: >>>>>> http://lists.bx.psu.edu/ >>>>>> >>>>>> To search Galaxy mailing lists use the unified search at: >>>>>> http://galaxyproject.org/search/mailinglists/ >>>>> >>>>> >>> >>> > >

1 0

list of tool versions and galaxy ami?
by Nikhil Joshi 28 Mar '13

28 Mar '13

Hi all, Is there a canonical list of current tool versions that are running in the public galaxy? Is there a way to get that list? Also, I was hoping that there might be a galaxy ami that I could use in the amazon cloud.... and by that I mean NOT a cloudman instance. I am looking for an instance that runs galaxy directly. Does that exist? - Nik. -- Nikhil Joshi Bioinformatics Analyst/Programmer UC Davis Bioinformatics Core http://bioinformatics.ucdavis.edu/ najoshi -at- ucdavis -dot- edu 530.752.2698 (w)

1 0

inable to login to testtoolshed login because of Server Error: AttributeError: 'GalaxySession' object has no attribute 'current_history'
by jorrit poelen 28 Mar '13

28 Mar '13

Hey y'all galaxiers, Today, I tried to login using the testtoolshed web ui. To make sure that it wasn't just isolated to an existing account, I created a new account, and still I encountered the Server Error attached below. This error is not shown when entering an invalid password. I was able to login and push hg changes a couple of days ago. Was the test toolshed recently upgraded? Please let me know what I can do to login to testtoolshed at http://testtoolshed.g2.bx.psu.edu thx, -jorrit PS Please find attached stack trace on login with valid password through toolshed web ui at around 2pm PST. URL: http://testtoolshed.g2.bx.psu.edu/user/login?use_panels=True File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Paste-1.7.5.1-py2.7.egg/paste/exceptions/errormiddleware.py', line 144 in __call__ app_iter = self.application(environ, sr_checker) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Paste-1.7.5.1-py2.7.egg/paste/debug/prints.py', line 106 in __call__ environ, self.app) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Paste-1.7.5.1-py2.7.egg/paste/wsgilib.py', line 543 in intercept_output app_iter = application(environ, replacement_start_response) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/framework/base.py', line 128 in __call__ return self.handle_request( environ, start_response ) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/framework/base.py', line 184 in handle_request body = method( trans, **kwargs ) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/webapps/galaxy/controllers/user.py', line 466 in login active_view="user" ) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/framework/__init__.py', line 863 in fill_template return self.fill_template_mako( filename, **kwargs ) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/framework/__init__.py', line 874 in fill_template_mako return template.render( **data ) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Mako-0.4.1-py2.7.egg/mako/template.py', line 296 in render return runtime._render(self, self.callable_, args, data) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line 660 in _render **_kwargs_for_callable(callable_, data)) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line 692 in _render_context _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line 718 in _exec_template callable_(context, *args, **kwargs) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/database/compiled_templates/community/base/base_panels.mako.py', line 85 in render_body __M_writer(unicode(self.masthead())) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/database/compiled_templates/community/webapps/galaxy/base_panels.mako.py', line 337 in render_masthead usage = trans.app.quota_agent.get_usage( trans=trans ) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/quota/__init__.py', line 26 in get_usage history = trans.history File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/framework/__init__.py', line 741 in get_history history = self.galaxy_session.current_history AttributeError: 'GalaxySession' object has no attribute 'current_history'

2 1

Re: [galaxy-dev] galaxy not picking up updates from test toolshed
by Dave Bouvier 28 Mar '13

28 Mar '13

Jorrit, I may have been a bit unclear in my previous response. Allow me to illustrate two scenarios: First scenario: 1. A repository is created containing a BWA tool with version 0.5.9 2. The repository is installed to Galaxy. 3. New README file is uploaded, tool version is *not* changed. Second scenario: 1. A repository is created containing a BWA tool with version 0.5.9 2. The repository is installed to Galaxy. 3. New README file is uploaded, and updated BWA tool is uploaded with version 0.6 In the first scenario, Galaxy will find that the repository has been updated, and apply the update. In the second scenario, Galaxy will not find that the repository has been updated, but will allow you to install both repositories, the one containing version 0.5.9 and the one containing version 0.6 of the BWA tool. The reason for this behavior is that if a tool has been updated to produce different output, the tool version should be updated, which leads to a separate installable changeset revision being generated. This way, any analyses run on the old tool version can be re-run and produce the same output, but any new analyses can run on the new version of the tool, with whatever new features have been added. This ensures reproducibility throughout the version lineage of the tool, while still allowing Galaxy users to use the most recent version of an available tool for future analyses. --Dave B. On 3/28/13 17:03:10.000, jorrit poelen wrote: > Hey Dave, > > Thanks for pointing me to the wiki that helped demystify the hg changeset <--> repository metadata connection for me. > > I'll make sure to change the version numbers in the tool xml definitions whenever a change needs to be make available to galaxy instances through the toolshed. So, even for an update in the README file, I'll make sure to change some version to make the change available. This feels a little clunky for me now, but I am sure there's a good reason to make it work as it does now. > > I very much appreciate your prompt response and I'll make sure to dig around the wiki a little longer before posting a question next time ; ) > > -jorrit > > On Mar 26, 2013, at 10:41 AM, Dave Bouvier wrote: > >> Jorrit, >> >> First, allow me to clarify a point of naming: >> >> It's not quite right to say "tools" in this situation, as tools are only one of several utilities that a tool shed repository can contain, along with datatypes, workflows, repository dependencies, and so on. This is an important distinction, because some repositories can (and do) contain no tools, and only datatypes and/or other Galaxy utilities. >> >> In your process description for the current behavior, the tool shed will not report updates to a repository if the tool version has changed. For more details on how and why this happens, see the following wiki page: >> >> http://wiki.galaxyproject.org/RepositoryRevisions#Adding_additional_change_… >> >> Where the behavior is detailed in these sections: >> >> Adding additional change sets to the initial change set in a repository >> • What is repository metadata? >> • What does it mean when repository metadata is associated with a change set in the repository change log? >> • Adding a change set that does not generate a new repository metadata record >> • Adding a change set that generates a new repository metadata record >> • More examples of installable repository change set revisions >> >> >> --Dave B. >> >> On 3/26/13 12:45:41.000, jorrit poelen wrote: >>> Hey Dave, >>> >>> Thanks against for your quick response! >>> >>> I checked my universe_wsgi.ini file and can confirm that the enabled_tool_shed_check was *commented*. >>> >>> I've uncommented the settings and changed them to: >>> >>> enable_tool_shed_check = True >>> hours_between_check = 1 >>> >>> Then I followed the steps outlined in my initial email and the observed the same (unexpected) behavior. >>> >>> Perhaps I am describing a feature request rather than a bug: My expectation is that whenever I explicitly request to update a tool, a immediate check is performed against the relevant tool shed. >>> >>> To make sure that I correctly understand the existing tool update feature, please confirm that the following describes the currently expected tool update behavior: >>> >>> 0. admin of galaxy instance Z enables the feature to check for tool updates in toolsheds every hour >>> 1. tool X revision 1.0 is installed from tool shed Y into galaxy instance Z by tool developer using hg push >>> 2. tool X is current revision is updated to revision 1.1 in tool shed Y >>> 3. immediately after, admin of galaxy instance Z attempts to (manually) upgrade tool X to version 1.1 using the admin web interface >>> 4. galaxy instance Z reports that no new version of tool X are available, even though a new version exists in the toolshed >>> 5. galaxy instance Z detects version 1.1 of tool X at most one hour after the update was made >>> 6. admin of galaxy instance Z tries again >>> >>> Thanks for your patience and efforts, >>> >>> -jorrit >>> >>> On Mar 26, 2013, at 8:19 AM, Dave Bouvier wrote: >>> >>>> Jorrit, >>>> >>>> Just to eliminate one possible cause of this behavior, could you confirm that your local Galaxy instance's universe_wsgi.ini file's enable_tool_shed_check setting is uncommented and set to True, and the hours_between_check is also uncommented and set to a value between 1 and 12? >>>> >>>> --Dave B. >>>> >>>> On 3/25/13 21:26:07.000, jorrit poelen wrote: >>>>> Hey y'all, >>>>> >>>>> I noticed that my local galaxy instance doens't pickup latest updates from test toolshed. >>>>> >>>>> Steps to reproduce: >>>>> >>>>> 1. install obotools from test toolshed using hg clone http://jorrit@testtoolshed.g2.bx.psu.edu/repos/jorrit/obotools , revision 45:bb20766df19c >>>>> >>>>> 2. hg push new change of obotools of galaxy tool to testtoolshed , creating revision 46:532808477800 >>>>> >>>>> 3. update obotools in galaxy admin >>>>> >>>>> expected result: galaxy admin detects new version and give an option to update. >>>>> >>>>> actual result: galaxy admin does not detect new version. >>>>> >>>>> I've attached some galaxy logging below. One thing I noticed that the reported commit time in testtoolshed seems 7 hours off - just minutes after I pushed the change, testtoolshed says the change was commit "~ 7 hours ago". >>>>> >>>>> Workaround I am using to do update is to deactivate and remove the tool all together, and re-install straight from test-toolshed. What should I do to make the tool version update work? I am probably missing something. >>>>> >>>>> Thanks! >>>>> >>>>> -jorrit >>>>> >>>>> >>>>> Galaxy logging on looking for updates in test toolshed: >>>>> --- >>>>> >>>>> 24.130.122.217 - - [26/Mar/2013:01:12:46 +0000] "GET /admin_toolshed/browse_repositories HTTP/1.1" 200 - "http://galaxy:7474/admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10" >>>>> 24.130.122.217 - - [26/Mar/2013:01:12:51 +0000] "GET /admin_toolshed/browse_repositories?operation=get+updates&id=a799d38679e985db HTTP/1.1" 302 - "http://galaxy:7474/admin_toolshed/browse_repositories" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10" >>>>> 24.130.122.217 - - [26/Mar/2013:01:12:51 +0000] "GET /admin_toolshed/check_for_updates?id=a799d38679e985db HTTP/1.1" 302 - "http://galaxy:7474/admin_toolshed/browse_repositories" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10" >>>>> 24.130.122.217 - - [26/Mar/2013:01:12:52 +0000] "GET /admin_toolshed/update_to_changeset_revision?tool_shed_url=http://testtoolshed.g2.bx.psu.edu/&name=obotools&owner=jorrit&changeset_revision=bb20766df19c&latest_changeset_revision=bb20766df19c&latest_ctx_rev=45 HTTP/1.1" 302 - "http://galaxy:7474/admin_toolshed/browse_repositories" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10" >>>>> 24.130.122.217 - - [26/Mar/2013:01:12:52 +0000] "GET /admin_toolshed/manage_repository?status=done&message=The+installed+repository+named+%27obotools%27+is+current%2C+there+are+no+updates+available.++&id=a799d38679e985db HTTP/1.1" 200 - "http://galaxy:7474/admin_toolshed/browse_repositories" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10" >>>>> ___________________________________________________________ >>>>> Please keep all replies on the list by using "reply all" >>>>> in your mail client. To manage your subscriptions to this >>>>> and other Galaxy lists, please use the interface at: >>>>> http://lists.bx.psu.edu/ >>>>> >>>>> To search Galaxy mailing lists use the unified search at: >>>>> http://galaxyproject.org/search/mailinglists/ >>>>> >>>> ___________________________________________________________ >>>> Please keep all replies on the list by using "reply all" >>>> in your mail client. To manage your subscriptions to this >>>> and other Galaxy lists, please use the interface at: >>>> http://lists.bx.psu.edu/ >>>> >>>> To search Galaxy mailing lists use the unified search at: >>>> http://galaxyproject.org/search/mailinglists/ >>> >>> > >

1 0

upgrading cufflinks version to 2.0.2
by Karen Chait 28 Mar '13

28 Mar '13

Hello, I am trying to upgrade cufflinks version from 1.3.0 to 2.0.2. I changed the directory in the PATH to version 2.0.2 but when I run cuffdiff on the galaxy it still uses the version 1.3.0. Do I need to change the code of the wrapper? Another question is how to I enable both version of cufflinks to appear and run on the galaxy? Thanks, Karen

2 1

Toolshed - Installation of static files
by Lance Parsons 28 Mar '13

28 Mar '13

For one of my tool wrappers (htseq-count) in the toolshed, I point to an image file in the Help section. At the moment, I'm including the image in the repository as static/imagescount_modes.png and pointing to it in the wrapper.xml file with ".. image:: /static/images/count_modes.png". However, the toolshed installation doesn't copy this file to the static directory of the galaxy install (which doesn't seem like a good idea anyway). Is there any way for me to include a static file and then point to that file in the help when the tool is installed automatically via the toolshed.. Thanks. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University

2 4