On Thu, Sep 2, 2010 at 12:35 PM, Hans-Rudolf Hotz hrh@fmi.ch wrote:

Peter

...

I've just started looking at installing a local copy of Galaxy, and have registered with the mailing list.

First one quick question - is there an easy way to search the Galaxy wiki (or documentation in general)? Perhaps my questions are already discussed if I knew where to look...

Have a look at

http://bitbucket.org/galaxy/galaxy-central/wiki/SiteMap

and

http://bitbucket.org/galaxy/galaxy-central/wiki/SiteIndex

Helpful, but not quite what I was looking for. Maybe searching the wiki pages for a project is a missing feature in bitbucket?

...

I understand that to install Galaxy locally we just clone the mercurial repository (either galaxy-central or galaxy-dist) and then use hg to periodically update this to the latest code. i.e. There are no formal releases with a version number that come as a tar ball or similar.

not really a version number, but I recommend to follow the "News Briefs"

http://bitbucket.org/galaxy/galaxy-central/wiki/Features/DevNewsBriefs

That's useful. Does this also get sent out by email, or on a news server with RSS/Atom feeds? I don't want to have to manually check the wiki every month.

...

Naturally, once I have a copy of Galaxy, I will be making some changes to it - in particular configuration changes, for example to use PostgreSQL.

Is the the workflow described here the recommend approach?

http://bytebucket.org/galaxy/galaxy-central/wiki/DevConf2010/galaxy_devconf_... (I found this presentation via a Google search)

Have a look at:

http://bitbucket.org/galaxy/galaxy-central/wiki/Config/ProductionServer

I've seen that and will be working my way though it in detail. However, this doesn't mention using hg to update an existing installation, specifically how to handle merging the upstream changes with any local changes.

...

If I understand it, the idea is that we rename tool_conf.xml.sample to tool_conf.xml (and remove it from the .hgignore file) so that any changes made to the sample configuration will be merged into our local configuration?

Alternatively, we could just take a copy and leave tool_conf.xml in the ignore file. We would then have to manually apply any changes made to the tool_conf.xml.sample on a case by case basis.

Also do you recommend I make my own changes on a branch?

Depends on how much do you want to change?

For us, adding our own tool is the big thing about Galaxy and we are pretty happy with the default Galaxy distro. Of course we have a few customizations, but they are added very quickly to a new 'release' by hand.

Could you clarify how you update your Galaxy install via hg, specifically regarding preserving local configuration changes and extra tools?

I expect our changes to be just to configuration files, and the addition of some new in-house local tools/workflows. It would seem sensible to track changes to any local tools (e.g. the XML files defining how Galaxy calls them) within the same hg repository. Given I don't plan to make any changes to Galaxy itself, this sort of thing shouldn't cause much trouble for hg merges (I hope).

Thanks,

Peter

On Thu, Sep 2, 2010 at 3:10 PM, Hans-Rudolf Hotz hrh@fmi.ch wrote:

Peter

...

...

not really a version number, but I recommend to follow the "News Briefs"

http://bitbucket.org/galaxy/galaxy-central/wiki/Features/DevNewsBriefs

That's useful. Does this also get sent out by email, or on a news server with RSS/Atom feeds? I don't want to have to manually check the wiki every month.

yes, you will get an e-mail via [galaxy-dev], for example see:

http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-August/003173.html

Lovely :)

...

...

For us, adding our own tool is the big thing about Galaxy and we are pretty happy with the default Galaxy distro. Of course we have a few customizations, but they are added very quickly to a new 'release' by hand.

Could you clarify how you update your Galaxy install via hg, specifically regarding preserving local configuration changes and extra tools?

In our case, we update our production server every three month with "hg pull" (last time we did "hg pull -u -r  -u - 837aabec314e" , pulling from http://bitbucket.org/galaxy/galaxy-dist/).

Local customizations in file like 'lib/galaxy/datatypes/registry.py' or 'lib/galaxy/datatypes/sequence.py' are nicely merged.

"conflicts", which might happen in files like 'static/welcome.html' can be resolved during the upgrade process.

Your own tools and tool description files are not touched. I would put them in a separate folder, just to be on the safe side

And at the end I manually fix the file universe_wsgi.ini

Do you check in any of your local changes into the local hg repository?

I recommend, start playing with galaxy, add your own tools, do an update a few weeks later, and you will see: It is really not rocket science. The galaxy developers do a great job providing a stable and simple system.

I'm sure I'll manage - its just this repository based distribution/install system which runs in situ is a bit unusual for me (plus I don't yet know much about using hg/mercurial).

Peter

On Mon, Sep 6, 2010 at 7:43 PM, Ry4an Brase ry4an+galaxy@msi.umn.edu wrote:

On Thu, Sep 02, 2010 at 12:39:54PM -0400, Nate Coraor wrote:

...

I haven't actually done this since our local modifications are committed to the Galaxy codebase ;).  However, it does make sense and as far as I know, UMN does this for their production Galaxy server.  Perhaps Ry4an or others there can comment on whether this process still works correctly for them.

We do commit all of our local changes and have done renames on our config files.  We maintain a code-flow like this:

galaxy-dist -> umn-galaxy-dev -> umn-galaxy-staging -> umn-galaxy-prod

where "->" could be read as "was cloned by".

So initially, we had just umn-galaxy-dev, where we did the renames of all of the config files, and then made local changes of our own.  That, in turn was cloned by staging, and so on.  In staging we have configuration options more suitable for a production server (debugging off, etc.) and those were commited in umn-galaxy-staging, but never pushed back to umn-galaxy-dev.

Mercurial does a pretty great job of merging, so when we do a 'hg pull' in umn-galaxy-dev to get the latest from galaxy-dist we mostly get clean merges on everything, and when we don't the history in umn-galaxy-dev let's me go back and see what the local-modification-developers were thinking.  The only changesets that exist in staging that don't exist in dev are configuration settings (including different disk paths and database connection strings) and those always apply cleanly.  Similarly the move from staging to prod just tweaks some settings, provides our welcome.html, and alters the database connection strings.

It might sound like over kill, but it's near-zero hassle, provides us with great local change tracking (who, what, when, and why) and provides us with an easy way to get local changes from dev to prod without the developers having access to prod.

Thanks for that detailed reply - I wasn't ignoring you, I've just been away a few days. I will see how I get on with this, but I foresee a number of pitfalls, first I need to learn the basics of hg.

Having given ownership of the repository and its files to a non-login galaxy user, I am having trouble getting hg pull to work - it says "Not trusting file /opt/galaxy-dist/.hg/hgrc from untrusted user galaxy, group galaxy_admin)" even though I am a member of the group galaxy_admin. The initial work around I found online was to create a ~/.hgrc file containing the following:

[trusted] users = * groups = *

After that, I was able to run:

sudo hg pull -u -r f09915c8da94

and it updated the code to the 8 September 2010 announcement, and seems to have merged my local (not yet checked in) changes fine. I doubt this is the best solution though.

Given the advice from the other thread is to create the galaxy user account as a normal login account, do you normally run "hg pull" etc as the galaxy user?

Thanks,

Peter

On Thu, Sep 9, 2010 at 2:56 PM, Ry4an Brase ry4an+galaxy@msi.umn.edu wrote:

...

The initial work around I found online was to create a ~/.hgrc file containing the following:

[trusted] users = * groups = *

Just remove the users= line and change groups from '*' to 'galaxy_admin', and you have a fine solution. ... Creating a trust block in your ~/.hgrc was a fine solution, just tweak it to trust as narrowly as possible and you're good to go.

Understood. Of course, any other developers/admins here will have to do the same in their ~/.hgrc but that isn't a great hardship as long as I document it.

I would, however, recommend committing before doing the update.  If you commit you know that no matter how poorly the update goes you can always 'hg update' back to exactly where you were.  Whereas, if you 'pull -u' (pull with automatic update) there's no way for you to revert if you don't like what happened.

I will try that in future.

...

Given the advice from the other thread is to create the galaxy user account as a normal login account, do you normally run "hg pull" etc as the galaxy user?

I run them as me and then chmod the working dir files over, but only because I have our galaxy user set to have no login shell.  Either works.

Oh good. Our administrator did not seem happy about making the galaxy user a login account - nice to know others are already using galaxy with a no login shell account.

Thanks,

Peter