Question Regarding fetch_eggs &c. vs pip
Hi Galaxy Dev, I've been looking at the setup scripts for Galaxy to try to understand a problem I recently had provisioning a Galaxy server. I will readily admit that I have not read all of the relevant code top-to-bottom, but I have at least skimmed all of it and read much of it. Sorry if these questions are answered somewhere in Trello, the Wiki, or somewhere else, but I was not able to find answers in any public locations. As a small bit of probably irrelevant context: I'm working with the Globus Genomics group on the DevOps side of things. We're using Chef. I've only just started working with the group in the past couple of weeks (so my expertise with Galaxy itself is limited to nonexistent). First, to describe the problem: We want to provision a server running Galaxy without explicitly wrapping it in a virtualenv. Unless I missed something, that means that it's using system python. When we use pip to install a package from a git repository before running the setup scripts, fetch_eggs fails saying it failed to fetch WebError 0.8a If we install the same package from git with `pip install --egg ...` we get a hunky-dory system where everything seems to work. As far as I can tell, there is no reason that this should be the case. Sure, putting the git source directly into site-packages might cause issues upon installation, but EggNotFetchable exceptions should only be thrown if the egg actually can't be pulled down from eggs.g2.bx.psu.edu , right? I don't feel comfortable trying to make further progress on my provisioning scripts without knowing why this is happening. I'd hate to be bitten by this later on in the process. Yes, the package in question may have poor behavior (likely it does), but that doesn't change the fact that the error is totally misleading. Furthermore, it doesn't appear that this poor behavior impedes me from doing a pip install of the WebError package or any other packages from PyPI. In case someone else wants to test to replicate, this is the command being used: pip install --egg git+ https://github.com/globusonline/python-nexus-client@599f04edef6b72569b7a5b27... problems occur if you omit `--egg`. Second, a question about the rationale for Galaxy's egg handling: Why is all of this wrapped up in these scripts in the first place? I understand that pip might not be present on every platform, and I don't mean to question a decision to support systems without it. However, as detailed below, Galaxy does not support any platforms which are incapable of running pip. Furthermore, pip is being pushed by the Python maintainers over easy_install, so it's not like there isn't a clear choice in terms of which one to support. Perhaps most importantly, there don't appear to be any clear-cut options to do the following, which I would consider a more ordinary workflow: - Run a galaxy script (like check_eggs) to generate a list of packages from eggs.g2.bx.psu.edu for platform (redirect output to requirements.txt or similar) - `pip install -r requirements.txt` The above could be part of a galaxy provisioning script, rather than exposed to the administrator. That also makes it significantly easier to control and manage the virtualenv in which we run Galaxy, since we don't have to worry about egg-related logic that we don't control and we know that the virtualenv's bin dir will be earlier in the PATH than the system pip's dir. Yes, I said above that our setup is presently using system python -- switching to a virtualenv is one of the many items on my to-do list. In fact, I would expect that the default, desired setup for Galaxy would be to put it in a virtualenv, rather than using system python, and to use pip, rather than fetch_eggs.py and company. When I look at the logic being used here, especially at https://bitbucket.org/galaxy/galaxy-central/src/f0ae870b22e9/lib/galaxy/eggs... , it looks like a solution built exclusively for platforms on which pip is not installed. According to the wiki, Galaxy support only goes back as far as 2.6, and get-pip.py supports 2.6, so there is no way of building a Galaxy server on a platform that can't also have pip installed. Adding pip to the requirements for Galaxy would not be particularly onerous, and may simplify things significantly (no need to bundle get-pip.py or similar). As a last note about the misleading error from the fetch_eggs script, telling me that WebError is "NotFetchable". I probably wouldn't have much of a complaint about this if the error had been more on target and I hadn't felt the need to do things like patch lib/galaxy/eggs to print a stacktrace. For example, if the script detected that there was a source installed package which was getting underfoot, it should have alerted me or even suggested installing my various packages in egg format. That kind of error detection is hard to maintain and hard to keep accurate since the Galaxy team's priority is to build Galaxy, not a package manager. This, I suppose, circles back to an earlier question: why isn't Galaxy using a python package manager to... manage its packages? At the very least something along the lines of `./scripts/common_setup.sh --use-pip` should be added. It doesn't seem like it would be that hard to implement -- but I feel like my lack of knowledge of Galaxy disqualifies me from building the changeset reliably. Of course, if no one objects, I will readily do so anyway (when I have the time) as a proof of concept. Barring even that improvement, the Wiki page at https://wiki.galaxyproject.org/Admin/Config/Eggs should definitely be updated to include some note on why Galaxy has this complex logic for fetching eggs instead of using pip. A quick tl;dr and summary: - I don't have extensive experience with Galaxy, so I may not know what I'm talking about. - fetch_eggs.py can be made to raise EggNotFetchable by doing a pip install from a VCS without using `--egg`. This is a bug in fetch_eggs.py - All supported platforms for Galaxy support pip - Galaxy should have an option to use pip to download its packages over https from eggs.g2.bx.psu.edu - Galaxy should probably default to using pip when it's available, since its failure modes are significantly better than a home-brewed package manager -- this also leads to good behavior in virtualenvs. Best regards, and many thanks for your time and attention, -Stephen
Hi Stephen, I'll try to reply as in-depth as I can. On Wed, Feb 4, 2015 at 1:41 PM, Stephen Rosen <sirosen@uchicago.edu> wrote:
Hi Galaxy Dev,
I've been looking at the setup scripts for Galaxy to try to understand a problem I recently had provisioning a Galaxy server. I will readily admit that I have not read all of the relevant code top-to-bottom, but I have at least skimmed all of it and read much of it. Sorry if these questions are answered somewhere in Trello, the Wiki, or somewhere else, but I was not able to find answers in any public locations.
As a small bit of probably irrelevant context: I'm working with the Globus Genomics group on the DevOps side of things. We're using Chef. I've only just started working with the group in the past couple of weeks (so my expertise with Galaxy itself is limited to nonexistent).
First, to describe the problem:
We want to provision a server running Galaxy without explicitly wrapping it in a virtualenv. Unless I missed something, that means that it's using system python. When we use pip to install a package from a git repository before running the setup scripts, fetch_eggs fails saying it failed to fetch WebError 0.8a If we install the same package from git with `pip install --egg ...` we get a hunky-dory system where everything seems to work.
A virtualenv is itself just a wrapper around whichever python binary was used to create it. I'd still suggest using a virtualenv created with the system python unless you have a really strong reason not to. In fact, I'm working on Galaxy process management and a command line tool for that management that will automatically create and use a virtualenv going forward. I'm a bit confused at what's happening here - you mention installing a package from a git repository with pip, but then refer to Galaxy's fetch_eggs(.py) script, which doesn't use pip or git.
As far as I can tell, there is no reason that this should be the case. Sure, putting the git source directly into site-packages might cause issues upon installation, but EggNotFetchable exceptions should only be thrown if the egg actually can't be pulled down from eggs.g2.bx.psu.edu , right?
EggNotFetchable can be thrown if you happen to be using a platform for which we do not provide eggs, although those are fairly uncommon. Right now we should cover x86/x86_64 Linux and any flavor of Intel OS X after 10.5.
I don't feel comfortable trying to make further progress on my provisioning scripts without knowing why this is happening. I'd hate to be bitten by this later on in the process.
Yes, the package in question may have poor behavior (likely it does), but that doesn't change the fact that the error is totally misleading. Furthermore, it doesn't appear that this poor behavior impedes me from doing a pip install of the WebError package or any other packages from PyPI.
In case someone else wants to test to replicate, this is the command being used: pip install --egg git+ https://github.com/globusonline/python-nexus-client@599f04edef6b72569b7a5b27... problems occur if you omit `--egg`.
None of this process is using Galaxy's egg handling, so I am not sure where the EggNotFetchable is coming from. What command are you running to get an EggNotFetchable error.
Second, a question about the rationale for Galaxy's egg handling:
Why is all of this wrapped up in these scripts in the first place? I understand that pip might not be present on every platform, and I don't mean to question a decision to support systems without it. However, as detailed below, Galaxy does not support any platforms which are incapable of running pip.
This isn't the case - Galaxy does not use pip to install the framework dependencies at all. Some tool dependencies installed from the Tool Shed do use pip, but that's entirely separate from the dependencies of the Galaxy application. The `scripts/scramble.py` script can be used to automatically build eggs on platforms which we do not prebuild eggs for. If this is necessary, `scripts/fetch_eggs.py` should tell you.
Furthermore, pip is being pushed by the Python maintainers over easy_install, so it's not like there isn't a clear choice in terms of which one to support.
Perhaps most importantly, there don't appear to be any clear-cut options to do the following, which I would consider a more ordinary workflow:
- Run a galaxy script (like check_eggs) to generate a list of packages from eggs.g2.bx.psu.edu for platform (redirect output to requirements.txt or similar) - `pip install -r requirements.txt`
This is exactly what `scripts/check_eggs.py` and `scripts/fetch_eggs.py` do. There are 3 reasons for the way we handle eggs in Galaxy: 1. Galaxy has a huge (and ever-growing) list of dependent python modules with C extensions. If we did not prebuild and distribute eggs for these, the initial setup to get Galaxy running would be long and problematic. Some people who download Galaxy to develop tools may not even have compilers installed, let alone the multitude of -dev or -devel packages that aren't part of a default Debian or RHEL installation that would be required to build all of these packages from source. One of the things that I feel makes Galaxy so accessible is that you can start using it immediately after you clone the source. So that ability to clone and start and have it work as reliably (and quickly) as possible is a high priority. 2. Galaxy started using eggs in 2005 or 2006. At this time, everything used distutils. pkg_resources came around, which soon brought setuptools and easy_install. After this came distutils2, pip and finally, these days, wheels. Our need for binary dependency packaging predated almost all of these (in fact, most packages in these days didn't even install .egg-info, which was the only reliable way to know what version of a module you were using) and as each new iteration of packaging/management came along it was never clear that any of them had "won" (and in fact, most of them lost). On top of this, the Python packaging folks have known for years that Python's platform detection for binary compatibility is broken[1]. While I was assured it'd be fixed soon, even with a complete reimplementation of Python packaging (wheels), they still haven't even made an effort to fix this problem[2]. In fact, binary wheels for Linux are explicitly not allowed on PyPI because of this. 3. We tightly control the versions of all of our dependencies, which is not always possible with pip if you aren't also controlling the source of your packages. [1] https://mail.python.org/pipermail/distutils-sig/2010-January/015345.html [2] http://lucumr.pocoo.org/2014/1/27/python-on-wheels/ The above could be part of a galaxy provisioning script, rather than
exposed to the administrator.
That also makes it significantly easier to control and manage the virtualenv in which we run Galaxy, since we don't have to worry about egg-related logic that we don't control and we know that the virtualenv's bin dir will be earlier in the PATH than the system pip's dir. Yes, I said above that our setup is presently using system python -- switching to a virtualenv is one of the many items on my to-do list. In fact, I would expect that the default, desired setup for Galaxy would be to put it in a virtualenv, rather than using system python, and to use pip, rather than fetch_eggs.py and company.
A virtualenv is indeed the strongly preferred setup as I mentioned above. However, Galaxy does not install its eggs to the virtualenv. The virtualenv is there to avoid conflicts with things in the default python's site-packages/dist-packages. Galaxy's eggs are installed to (by default) the `eggs/` directory in the Galaxy source. However, the problem, as I now see it, is that you are trying to install all of Galaxy's dependencies, even at their correct versions, using pip, rather than letting Galaxy handle its eggs as it does. This is not going to work, Galaxy is going to insist on using its eggs.
When I look at the logic being used here, especially at https://bitbucket.org/galaxy/galaxy-central/src/f0ae870b22e9/lib/galaxy/eggs... , it looks like a solution built exclusively for platforms on which pip is not installed. According to the wiki, Galaxy support only goes back as far as 2.6, and get-pip.py supports 2.6, so there is no way of building a Galaxy server on a platform that can't also have pip installed. Adding pip to the requirements for Galaxy would not be particularly onerous, and may simplify things significantly (no need to bundle get-pip.py or similar).
As mentioned above, we don't use or depend on pip, so it's not required. That said, a lot of our egg fetching logic could likely be replaced with pip (this code predates pip by a few years). And the eggs could probably be replaced with prebuilt wheels. However, even if we did use pip/wheels, we'd need to install them from our own repository, and it'd still require modifications for binary platform incompatibilities. The egg handling code we have now works pretty reliably, so I am not sure there is a whole lot to be gained by changing it until Python finally figures out how to handle binary compatibility properly.
As a last note about the misleading error from the fetch_eggs script, telling me that WebError is "NotFetchable". I probably wouldn't have much of a complaint about this if the error had been more on target and I hadn't felt the need to do things like patch lib/galaxy/eggs to print a stacktrace. For example, if the script detected that there was a source installed package which was getting underfoot, it should have alerted me or even suggested installing my various packages in egg format.
This is a bug - it is supposed to explain that there is a version conflict. If you have a stack trace, please send it along.
That kind of error detection is hard to maintain and hard to keep accurate since the Galaxy team's priority is to build Galaxy, not a package manager. This, I suppose, circles back to an earlier question: why isn't Galaxy using a python package manager to... manage its packages?
At the very least something along the lines of `./scripts/common_setup.sh --use-pip` should be added. It doesn't seem like it would be that hard to implement -- but I feel like my lack of knowledge of Galaxy disqualifies me from building the changeset reliably. Of course, if no one objects, I will readily do so anyway (when I have the time) as a proof of concept.
There are modifications to a few of the dependencies, such as psycopg2. When psycopg2 is scrambled, the scrambling process fetches and compiles a bit of PostgreSQL's libpq and then statically links to it to provide a standalone egg that does not depend on the user having installed libpq on their system. So a pip-based system that allowed building from source would need to account for this. --nate
Barring even that improvement, the Wiki page at https://wiki.galaxyproject.org/Admin/Config/Eggs should definitely be updated to include some note on why Galaxy has this complex logic for fetching eggs instead of using pip.
A quick tl;dr and summary: - I don't have extensive experience with Galaxy, so I may not know what I'm talking about. - fetch_eggs.py can be made to raise EggNotFetchable by doing a pip install from a VCS without using `--egg`. This is a bug in fetch_eggs.py - All supported platforms for Galaxy support pip - Galaxy should have an option to use pip to download its packages over https from eggs.g2.bx.psu.edu - Galaxy should probably default to using pip when it's available, since its failure modes are significantly better than a home-brewed package manager -- this also leads to good behavior in virtualenvs.
Best regards, and many thanks for your time and attention, -Stephen
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Nate, Thanks so much for helping me out with this. It seems that I've miscommunicated what I'm doing a little bit. I'm installing a package from that git repo using pip separately from running the galaxy setup script. That is, I want to install a python package and I want to setup galaxy as two separate (and ideally independent) steps in my script. I have no desire to do anything particularly esoteric or clever with Galaxy's eggs or to install them myself. Somehow, doing a pip install of my desired package is breaking the later run of fetch_eggs.py If I understand you correctly, fetch_eggs.py can hit conflicts with site-packages, but is misreporting them as EggNotFetchable. I am aware that we're using a fork of the Galaxy source ( https://bitbucket.org/galaxycloud/galaxy/src ), so it may be that we have an outdated of fetch_eggs.py which carries a bug that has since been patched. Sorry for not mentioning it earlier -- it slipped my mind that we're probably trailing behind the modern Galaxy default head. If I put in the extra legwork now to wrap Galaxy in a virtualenv, this issue will presumably disappear, but that assumes that I don't want to add any packages to the python used for the uwsgi process. I'll have to ask folks on the Globus Genomics team whether or not that is needed -- I certainly hope not, since it means that there is some packaging conflict we need to resolve. The only reason I haven't done so already is time constraint -- I've been trying to get this install scripting done as quickly as possible without compromising anything truly needful. All the same, I'll take some time tomorrow to provision a fresh server and generate the stacktrace for you. I'll also take some time to test with the latest Galaxy source, to see if I get different behavior. In the best case, this bug no longer exists for the modern source, but in the worst case it bears the attention. I don't have any sense of the reliability of fetch_eggs.py and company -- my experience has been of a particular bug that presented on my first contact with these scripts, so naturally I have a bias against them. That said, I think the Python community may have finally settled on a package manager, even if we can't seem to agree on a package format or tooling surrounding it. That's just an opinion though -- I have no blog post from Guido to back it up. I was not aware of the UCS2 vs. UCS4 issue -- thanks very much for the citations, very helpful in understanding the problem space. Thanks, -Stephen On Wed, Feb 4, 2015 at 5:41 PM, Nate Coraor <nate@bx.psu.edu> wrote:
Hi Stephen,
I'll try to reply as in-depth as I can.
On Wed, Feb 4, 2015 at 1:41 PM, Stephen Rosen <sirosen@uchicago.edu> wrote:
Hi Galaxy Dev,
I've been looking at the setup scripts for Galaxy to try to understand a problem I recently had provisioning a Galaxy server. I will readily admit that I have not read all of the relevant code top-to-bottom, but I have at least skimmed all of it and read much of it. Sorry if these questions are answered somewhere in Trello, the Wiki, or somewhere else, but I was not able to find answers in any public locations.
As a small bit of probably irrelevant context: I'm working with the Globus Genomics group on the DevOps side of things. We're using Chef. I've only just started working with the group in the past couple of weeks (so my expertise with Galaxy itself is limited to nonexistent).
First, to describe the problem:
We want to provision a server running Galaxy without explicitly wrapping it in a virtualenv. Unless I missed something, that means that it's using system python. When we use pip to install a package from a git repository before running the setup scripts, fetch_eggs fails saying it failed to fetch WebError 0.8a If we install the same package from git with `pip install --egg ...` we get a hunky-dory system where everything seems to work.
A virtualenv is itself just a wrapper around whichever python binary was used to create it. I'd still suggest using a virtualenv created with the system python unless you have a really strong reason not to. In fact, I'm working on Galaxy process management and a command line tool for that management that will automatically create and use a virtualenv going forward.
I'm a bit confused at what's happening here - you mention installing a package from a git repository with pip, but then refer to Galaxy's fetch_eggs(.py) script, which doesn't use pip or git.
As far as I can tell, there is no reason that this should be the case. Sure, putting the git source directly into site-packages might cause issues upon installation, but EggNotFetchable exceptions should only be thrown if the egg actually can't be pulled down from eggs.g2.bx.psu.edu , right?
EggNotFetchable can be thrown if you happen to be using a platform for which we do not provide eggs, although those are fairly uncommon. Right now we should cover x86/x86_64 Linux and any flavor of Intel OS X after 10.5.
I don't feel comfortable trying to make further progress on my provisioning scripts without knowing why this is happening. I'd hate to be bitten by this later on in the process.
Yes, the package in question may have poor behavior (likely it does), but that doesn't change the fact that the error is totally misleading. Furthermore, it doesn't appear that this poor behavior impedes me from doing a pip install of the WebError package or any other packages from PyPI.
In case someone else wants to test to replicate, this is the command being used: pip install --egg git+ https://github.com/globusonline/python-nexus-client@599f04edef6b72569b7a5b27... problems occur if you omit `--egg`.
None of this process is using Galaxy's egg handling, so I am not sure where the EggNotFetchable is coming from. What command are you running to get an EggNotFetchable error.
Second, a question about the rationale for Galaxy's egg handling:
Why is all of this wrapped up in these scripts in the first place? I understand that pip might not be present on every platform, and I don't mean to question a decision to support systems without it. However, as detailed below, Galaxy does not support any platforms which are incapable of running pip.
This isn't the case - Galaxy does not use pip to install the framework dependencies at all. Some tool dependencies installed from the Tool Shed do use pip, but that's entirely separate from the dependencies of the Galaxy application.
The `scripts/scramble.py` script can be used to automatically build eggs on platforms which we do not prebuild eggs for. If this is necessary, `scripts/fetch_eggs.py` should tell you.
Furthermore, pip is being pushed by the Python maintainers over easy_install, so it's not like there isn't a clear choice in terms of which one to support.
Perhaps most importantly, there don't appear to be any clear-cut options to do the following, which I would consider a more ordinary workflow:
- Run a galaxy script (like check_eggs) to generate a list of packages from eggs.g2.bx.psu.edu for platform (redirect output to requirements.txt or similar) - `pip install -r requirements.txt`
This is exactly what `scripts/check_eggs.py` and `scripts/fetch_eggs.py` do.
There are 3 reasons for the way we handle eggs in Galaxy:
1. Galaxy has a huge (and ever-growing) list of dependent python modules with C extensions. If we did not prebuild and distribute eggs for these, the initial setup to get Galaxy running would be long and problematic. Some people who download Galaxy to develop tools may not even have compilers installed, let alone the multitude of -dev or -devel packages that aren't part of a default Debian or RHEL installation that would be required to build all of these packages from source. One of the things that I feel makes Galaxy so accessible is that you can start using it immediately after you clone the source. So that ability to clone and start and have it work as reliably (and quickly) as possible is a high priority.
2. Galaxy started using eggs in 2005 or 2006. At this time, everything used distutils. pkg_resources came around, which soon brought setuptools and easy_install. After this came distutils2, pip and finally, these days, wheels. Our need for binary dependency packaging predated almost all of these (in fact, most packages in these days didn't even install .egg-info, which was the only reliable way to know what version of a module you were using) and as each new iteration of packaging/management came along it was never clear that any of them had "won" (and in fact, most of them lost). On top of this, the Python packaging folks have known for years that Python's platform detection for binary compatibility is broken[1]. While I was assured it'd be fixed soon, even with a complete reimplementation of Python packaging (wheels), they still haven't even made an effort to fix this problem[2]. In fact, binary wheels for Linux are explicitly not allowed on PyPI because of this.
3. We tightly control the versions of all of our dependencies, which is not always possible with pip if you aren't also controlling the source of your packages.
[1] https://mail.python.org/pipermail/distutils-sig/2010-January/015345.html [2] http://lucumr.pocoo.org/2014/1/27/python-on-wheels/
The above could be part of a galaxy provisioning script, rather than
exposed to the administrator.
That also makes it significantly easier to control and manage the virtualenv in which we run Galaxy, since we don't have to worry about egg-related logic that we don't control and we know that the virtualenv's bin dir will be earlier in the PATH than the system pip's dir. Yes, I said above that our setup is presently using system python -- switching to a virtualenv is one of the many items on my to-do list. In fact, I would expect that the default, desired setup for Galaxy would be to put it in a virtualenv, rather than using system python, and to use pip, rather than fetch_eggs.py and company.
A virtualenv is indeed the strongly preferred setup as I mentioned above. However, Galaxy does not install its eggs to the virtualenv. The virtualenv is there to avoid conflicts with things in the default python's site-packages/dist-packages. Galaxy's eggs are installed to (by default) the `eggs/` directory in the Galaxy source.
However, the problem, as I now see it, is that you are trying to install all of Galaxy's dependencies, even at their correct versions, using pip, rather than letting Galaxy handle its eggs as it does. This is not going to work, Galaxy is going to insist on using its eggs.
When I look at the logic being used here, especially at https://bitbucket.org/galaxy/galaxy-central/src/f0ae870b22e9/lib/galaxy/eggs... , it looks like a solution built exclusively for platforms on which pip is not installed. According to the wiki, Galaxy support only goes back as far as 2.6, and get-pip.py supports 2.6, so there is no way of building a Galaxy server on a platform that can't also have pip installed. Adding pip to the requirements for Galaxy would not be particularly onerous, and may simplify things significantly (no need to bundle get-pip.py or similar).
As mentioned above, we don't use or depend on pip, so it's not required. That said, a lot of our egg fetching logic could likely be replaced with pip (this code predates pip by a few years). And the eggs could probably be replaced with prebuilt wheels. However, even if we did use pip/wheels, we'd need to install them from our own repository, and it'd still require modifications for binary platform incompatibilities. The egg handling code we have now works pretty reliably, so I am not sure there is a whole lot to be gained by changing it until Python finally figures out how to handle binary compatibility properly.
As a last note about the misleading error from the fetch_eggs script, telling me that WebError is "NotFetchable". I probably wouldn't have much of a complaint about this if the error had been more on target and I hadn't felt the need to do things like patch lib/galaxy/eggs to print a stacktrace. For example, if the script detected that there was a source installed package which was getting underfoot, it should have alerted me or even suggested installing my various packages in egg format.
This is a bug - it is supposed to explain that there is a version conflict. If you have a stack trace, please send it along.
That kind of error detection is hard to maintain and hard to keep accurate since the Galaxy team's priority is to build Galaxy, not a package manager. This, I suppose, circles back to an earlier question: why isn't Galaxy using a python package manager to... manage its packages?
At the very least something along the lines of `./scripts/common_setup.sh --use-pip` should be added. It doesn't seem like it would be that hard to implement -- but I feel like my lack of knowledge of Galaxy disqualifies me from building the changeset reliably. Of course, if no one objects, I will readily do so anyway (when I have the time) as a proof of concept.
There are modifications to a few of the dependencies, such as psycopg2. When psycopg2 is scrambled, the scrambling process fetches and compiles a bit of PostgreSQL's libpq and then statically links to it to provide a standalone egg that does not depend on the user having installed libpq on their system. So a pip-based system that allowed building from source would need to account for this.
--nate
Barring even that improvement, the Wiki page at https://wiki.galaxyproject.org/Admin/Config/Eggs should definitely be updated to include some note on why Galaxy has this complex logic for fetching eggs instead of using pip.
A quick tl;dr and summary: - I don't have extensive experience with Galaxy, so I may not know what I'm talking about. - fetch_eggs.py can be made to raise EggNotFetchable by doing a pip install from a VCS without using `--egg`. This is a bug in fetch_eggs.py - All supported platforms for Galaxy support pip - Galaxy should have an option to use pip to download its packages over https from eggs.g2.bx.psu.edu - Galaxy should probably default to using pip when it's available, since its failure modes are significantly better than a home-brewed package manager -- this also leads to good behavior in virtualenvs.
Best regards, and many thanks for your time and attention, -Stephen
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi guys, the latest sync of your fork with Galaxy stable was on 2013-02-08, according to: https://bitbucket.org/galaxycloud/galaxy/commits/all?page=10 This is the diff of fetch_eggs.py in between your version and galaxy-central: https://gist.github.com/martenson/f5bc887b1721732991e0 It doesn't look like there is something that may be affecting the problem. M. On Wed Feb 04 2015 at 8:31:08 PM Stephen Rosen <sirosen@uchicago.edu> wrote:
Hi Nate,
Thanks so much for helping me out with this.
It seems that I've miscommunicated what I'm doing a little bit. I'm installing a package from that git repo using pip separately from running the galaxy setup script. That is, I want to install a python package and I want to setup galaxy as two separate (and ideally independent) steps in my script. I have no desire to do anything particularly esoteric or clever with Galaxy's eggs or to install them myself. Somehow, doing a pip install of my desired package is breaking the later run of fetch_eggs.py
If I understand you correctly, fetch_eggs.py can hit conflicts with site-packages, but is misreporting them as EggNotFetchable. I am aware that we're using a fork of the Galaxy source ( https://bitbucket.org/galaxycloud/galaxy/src ), so it may be that we have an outdated of fetch_eggs.py which carries a bug that has since been patched. Sorry for not mentioning it earlier -- it slipped my mind that we're probably trailing behind the modern Galaxy default head.
If I put in the extra legwork now to wrap Galaxy in a virtualenv, this issue will presumably disappear, but that assumes that I don't want to add any packages to the python used for the uwsgi process. I'll have to ask folks on the Globus Genomics team whether or not that is needed -- I certainly hope not, since it means that there is some packaging conflict we need to resolve. The only reason I haven't done so already is time constraint -- I've been trying to get this install scripting done as quickly as possible without compromising anything truly needful.
All the same, I'll take some time tomorrow to provision a fresh server and generate the stacktrace for you. I'll also take some time to test with the latest Galaxy source, to see if I get different behavior. In the best case, this bug no longer exists for the modern source, but in the worst case it bears the attention.
I don't have any sense of the reliability of fetch_eggs.py and company -- my experience has been of a particular bug that presented on my first contact with these scripts, so naturally I have a bias against them. That said, I think the Python community may have finally settled on a package manager, even if we can't seem to agree on a package format or tooling surrounding it. That's just an opinion though -- I have no blog post from Guido to back it up.
I was not aware of the UCS2 vs. UCS4 issue -- thanks very much for the citations, very helpful in understanding the problem space.
Thanks, -Stephen
On Wed, Feb 4, 2015 at 5:41 PM, Nate Coraor <nate@bx.psu.edu> wrote:
Hi Stephen,
I'll try to reply as in-depth as I can.
On Wed, Feb 4, 2015 at 1:41 PM, Stephen Rosen <sirosen@uchicago.edu> wrote:
Hi Galaxy Dev,
I've been looking at the setup scripts for Galaxy to try to understand a problem I recently had provisioning a Galaxy server. I will readily admit that I have not read all of the relevant code top-to-bottom, but I have at least skimmed all of it and read much of it. Sorry if these questions are answered somewhere in Trello, the Wiki, or somewhere else, but I was not able to find answers in any public locations.
As a small bit of probably irrelevant context: I'm working with the Globus Genomics group on the DevOps side of things. We're using Chef. I've only just started working with the group in the past couple of weeks (so my expertise with Galaxy itself is limited to nonexistent).
First, to describe the problem:
We want to provision a server running Galaxy without explicitly wrapping it in a virtualenv. Unless I missed something, that means that it's using system python. When we use pip to install a package from a git repository before running the setup scripts, fetch_eggs fails saying it failed to fetch WebError 0.8a If we install the same package from git with `pip install --egg ...` we get a hunky-dory system where everything seems to work.
A virtualenv is itself just a wrapper around whichever python binary was used to create it. I'd still suggest using a virtualenv created with the system python unless you have a really strong reason not to. In fact, I'm working on Galaxy process management and a command line tool for that management that will automatically create and use a virtualenv going forward.
I'm a bit confused at what's happening here - you mention installing a package from a git repository with pip, but then refer to Galaxy's fetch_eggs(.py) script, which doesn't use pip or git.
As far as I can tell, there is no reason that this should be the case. Sure, putting the git source directly into site-packages might cause issues upon installation, but EggNotFetchable exceptions should only be thrown if the egg actually can't be pulled down from eggs.g2.bx.psu.edu , right?
EggNotFetchable can be thrown if you happen to be using a platform for which we do not provide eggs, although those are fairly uncommon. Right now we should cover x86/x86_64 Linux and any flavor of Intel OS X after 10.5.
I don't feel comfortable trying to make further progress on my provisioning scripts without knowing why this is happening. I'd hate to be bitten by this later on in the process.
Yes, the package in question may have poor behavior (likely it does), but that doesn't change the fact that the error is totally misleading. Furthermore, it doesn't appear that this poor behavior impedes me from doing a pip install of the WebError package or any other packages from PyPI.
In case someone else wants to test to replicate, this is the command being used: pip install --egg git+ https://github.com/globusonline/python-nexus-client@599f04edef6b72569b7a5b27... problems occur if you omit `--egg`.
None of this process is using Galaxy's egg handling, so I am not sure where the EggNotFetchable is coming from. What command are you running to get an EggNotFetchable error.
Second, a question about the rationale for Galaxy's egg handling:
Why is all of this wrapped up in these scripts in the first place? I understand that pip might not be present on every platform, and I don't mean to question a decision to support systems without it. However, as detailed below, Galaxy does not support any platforms which are incapable of running pip.
This isn't the case - Galaxy does not use pip to install the framework dependencies at all. Some tool dependencies installed from the Tool Shed do use pip, but that's entirely separate from the dependencies of the Galaxy application.
The `scripts/scramble.py` script can be used to automatically build eggs on platforms which we do not prebuild eggs for. If this is necessary, `scripts/fetch_eggs.py` should tell you.
Furthermore, pip is being pushed by the Python maintainers over easy_install, so it's not like there isn't a clear choice in terms of which one to support.
Perhaps most importantly, there don't appear to be any clear-cut options to do the following, which I would consider a more ordinary workflow:
- Run a galaxy script (like check_eggs) to generate a list of packages from eggs.g2.bx.psu.edu for platform (redirect output to requirements.txt or similar) - `pip install -r requirements.txt`
This is exactly what `scripts/check_eggs.py` and `scripts/fetch_eggs.py` do.
There are 3 reasons for the way we handle eggs in Galaxy:
1. Galaxy has a huge (and ever-growing) list of dependent python modules with C extensions. If we did not prebuild and distribute eggs for these, the initial setup to get Galaxy running would be long and problematic. Some people who download Galaxy to develop tools may not even have compilers installed, let alone the multitude of -dev or -devel packages that aren't part of a default Debian or RHEL installation that would be required to build all of these packages from source. One of the things that I feel makes Galaxy so accessible is that you can start using it immediately after you clone the source. So that ability to clone and start and have it work as reliably (and quickly) as possible is a high priority.
2. Galaxy started using eggs in 2005 or 2006. At this time, everything used distutils. pkg_resources came around, which soon brought setuptools and easy_install. After this came distutils2, pip and finally, these days, wheels. Our need for binary dependency packaging predated almost all of these (in fact, most packages in these days didn't even install .egg-info, which was the only reliable way to know what version of a module you were using) and as each new iteration of packaging/management came along it was never clear that any of them had "won" (and in fact, most of them lost). On top of this, the Python packaging folks have known for years that Python's platform detection for binary compatibility is broken[1]. While I was assured it'd be fixed soon, even with a complete reimplementation of Python packaging (wheels), they still haven't even made an effort to fix this problem[2]. In fact, binary wheels for Linux are explicitly not allowed on PyPI because of this.
3. We tightly control the versions of all of our dependencies, which is not always possible with pip if you aren't also controlling the source of your packages.
[1] https://mail.python.org/pipermail/distutils-sig/2010-January/015345.html [2] http://lucumr.pocoo.org/2014/1/27/python-on-wheels/
The above could be part of a galaxy provisioning script, rather than
exposed to the administrator.
That also makes it significantly easier to control and manage the virtualenv in which we run Galaxy, since we don't have to worry about egg-related logic that we don't control and we know that the virtualenv's bin dir will be earlier in the PATH than the system pip's dir. Yes, I said above that our setup is presently using system python -- switching to a virtualenv is one of the many items on my to-do list. In fact, I would expect that the default, desired setup for Galaxy would be to put it in a virtualenv, rather than using system python, and to use pip, rather than fetch_eggs.py and company.
A virtualenv is indeed the strongly preferred setup as I mentioned above. However, Galaxy does not install its eggs to the virtualenv. The virtualenv is there to avoid conflicts with things in the default python's site-packages/dist-packages. Galaxy's eggs are installed to (by default) the `eggs/` directory in the Galaxy source.
However, the problem, as I now see it, is that you are trying to install all of Galaxy's dependencies, even at their correct versions, using pip, rather than letting Galaxy handle its eggs as it does. This is not going to work, Galaxy is going to insist on using its eggs.
When I look at the logic being used here, especially at https://bitbucket.org/galaxy/galaxy-central/src/f0ae870b22e9/lib/galaxy/eggs... , it looks like a solution built exclusively for platforms on which pip is not installed. According to the wiki, Galaxy support only goes back as far as 2.6, and get-pip.py supports 2.6, so there is no way of building a Galaxy server on a platform that can't also have pip installed. Adding pip to the requirements for Galaxy would not be particularly onerous, and may simplify things significantly (no need to bundle get-pip.py or similar).
As mentioned above, we don't use or depend on pip, so it's not required. That said, a lot of our egg fetching logic could likely be replaced with pip (this code predates pip by a few years). And the eggs could probably be replaced with prebuilt wheels. However, even if we did use pip/wheels, we'd need to install them from our own repository, and it'd still require modifications for binary platform incompatibilities. The egg handling code we have now works pretty reliably, so I am not sure there is a whole lot to be gained by changing it until Python finally figures out how to handle binary compatibility properly.
As a last note about the misleading error from the fetch_eggs script, telling me that WebError is "NotFetchable". I probably wouldn't have much of a complaint about this if the error had been more on target and I hadn't felt the need to do things like patch lib/galaxy/eggs to print a stacktrace. For example, if the script detected that there was a source installed package which was getting underfoot, it should have alerted me or even suggested installing my various packages in egg format.
This is a bug - it is supposed to explain that there is a version conflict. If you have a stack trace, please send it along.
That kind of error detection is hard to maintain and hard to keep accurate since the Galaxy team's priority is to build Galaxy, not a package manager. This, I suppose, circles back to an earlier question: why isn't Galaxy using a python package manager to... manage its packages?
At the very least something along the lines of `./scripts/common_setup.sh --use-pip` should be added. It doesn't seem like it would be that hard to implement -- but I feel like my lack of knowledge of Galaxy disqualifies me from building the changeset reliably. Of course, if no one objects, I will readily do so anyway (when I have the time) as a proof of concept.
There are modifications to a few of the dependencies, such as psycopg2. When psycopg2 is scrambled, the scrambling process fetches and compiles a bit of PostgreSQL's libpq and then statically links to it to provide a standalone egg that does not depend on the user having installed libpq on their system. So a pip-based system that allowed building from source would need to account for this.
--nate
Barring even that improvement, the Wiki page at https://wiki.galaxyproject.org/Admin/Config/Eggs should definitely be updated to include some note on why Galaxy has this complex logic for fetching eggs instead of using pip.
A quick tl;dr and summary: - I don't have extensive experience with Galaxy, so I may not know what I'm talking about. - fetch_eggs.py can be made to raise EggNotFetchable by doing a pip install from a VCS without using `--egg`. This is a bug in fetch_eggs.py - All supported platforms for Galaxy support pip - Galaxy should have an option to use pip to download its packages over https from eggs.g2.bx.psu.edu - Galaxy should probably default to using pip when it's available, since its failure modes are significantly better than a home-brewed package manager -- this also leads to good behavior in virtualenvs.
Best regards, and many thanks for your time and attention, -Stephen
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Stephen, On Wed, Feb 4, 2015 at 8:30 PM, Stephen Rosen <sirosen@uchicago.edu> wrote:
Hi Nate,
Thanks so much for helping me out with this.
It seems that I've miscommunicated what I'm doing a little bit. I'm installing a package from that git repo using pip separately from running the galaxy setup script. That is, I want to install a python package and I want to setup galaxy as two separate (and ideally independent) steps in my script. I have no desire to do anything particularly esoteric or clever with Galaxy's eggs or to install them myself. Somehow, doing a pip install of my desired package is breaking the later run of fetch_eggs.py
If I understand you correctly, fetch_eggs.py can hit conflicts with site-packages, but is misreporting them as EggNotFetchable. I am aware that we're using a fork of the Galaxy source ( https://bitbucket.org/galaxycloud/galaxy/src ), so it may be that we have an outdated of fetch_eggs.py which carries a bug that has since been patched. Sorry for not mentioning it earlier -- it slipped my mind that we're probably trailing behind the modern Galaxy default head.
If I put in the extra legwork now to wrap Galaxy in a virtualenv, this issue will presumably disappear, but that assumes that I don't want to add any packages to the python used for the uwsgi process.
You can add additional packages to this virtualenv, although there could be problems if you install different versions of the same packages that Galaxy has as dependencies. For what it's worth, you should not need to install additional packages in Galaxy's virtualenv, and tools can be configured to use a different python or virtualenv than the one in which Galaxy is started.
I'll have to ask folks on the Globus Genomics team whether or not that is needed -- I certainly hope not, since it means that there is some packaging conflict we need to resolve. The only reason I haven't done so already is time constraint -- I've been trying to get this install scripting done as quickly as possible without compromising anything truly needful.
All the same, I'll take some time tomorrow to provision a fresh server and generate the stacktrace for you. I'll also take some time to test with the latest Galaxy source, to see if I get different behavior. In the best case, this bug no longer exists for the modern source, but in the worst case it bears the attention.
Thanks for the help with debugging.
I don't have any sense of the reliability of fetch_eggs.py and company -- my experience has been of a particular bug that presented on my first contact with these scripts, so naturally I have a bias against them. That said, I think the Python community may have finally settled on a package manager, even if we can't seem to agree on a package format or tooling surrounding it. That's just an opinion though -- I have no blog post from Guido to back it up.
I hope we'll see the necessary improvements within the next year, but I think anything we do now will essentially reinvent the wheel (no pun intended) with regards to what will (hopefully) be officially implemented to fix the problem. --nate
I was not aware of the UCS2 vs. UCS4 issue -- thanks very much for the citations, very helpful in understanding the problem space.
Thanks, -Stephen
On Wed, Feb 4, 2015 at 5:41 PM, Nate Coraor <nate@bx.psu.edu> wrote:
Hi Stephen,
I'll try to reply as in-depth as I can.
On Wed, Feb 4, 2015 at 1:41 PM, Stephen Rosen <sirosen@uchicago.edu> wrote:
Hi Galaxy Dev,
I've been looking at the setup scripts for Galaxy to try to understand a problem I recently had provisioning a Galaxy server. I will readily admit that I have not read all of the relevant code top-to-bottom, but I have at least skimmed all of it and read much of it. Sorry if these questions are answered somewhere in Trello, the Wiki, or somewhere else, but I was not able to find answers in any public locations.
As a small bit of probably irrelevant context: I'm working with the Globus Genomics group on the DevOps side of things. We're using Chef. I've only just started working with the group in the past couple of weeks (so my expertise with Galaxy itself is limited to nonexistent).
First, to describe the problem:
We want to provision a server running Galaxy without explicitly wrapping it in a virtualenv. Unless I missed something, that means that it's using system python. When we use pip to install a package from a git repository before running the setup scripts, fetch_eggs fails saying it failed to fetch WebError 0.8a If we install the same package from git with `pip install --egg ...` we get a hunky-dory system where everything seems to work.
A virtualenv is itself just a wrapper around whichever python binary was used to create it. I'd still suggest using a virtualenv created with the system python unless you have a really strong reason not to. In fact, I'm working on Galaxy process management and a command line tool for that management that will automatically create and use a virtualenv going forward.
I'm a bit confused at what's happening here - you mention installing a package from a git repository with pip, but then refer to Galaxy's fetch_eggs(.py) script, which doesn't use pip or git.
As far as I can tell, there is no reason that this should be the case. Sure, putting the git source directly into site-packages might cause issues upon installation, but EggNotFetchable exceptions should only be thrown if the egg actually can't be pulled down from eggs.g2.bx.psu.edu , right?
EggNotFetchable can be thrown if you happen to be using a platform for which we do not provide eggs, although those are fairly uncommon. Right now we should cover x86/x86_64 Linux and any flavor of Intel OS X after 10.5.
I don't feel comfortable trying to make further progress on my provisioning scripts without knowing why this is happening. I'd hate to be bitten by this later on in the process.
Yes, the package in question may have poor behavior (likely it does), but that doesn't change the fact that the error is totally misleading. Furthermore, it doesn't appear that this poor behavior impedes me from doing a pip install of the WebError package or any other packages from PyPI.
In case someone else wants to test to replicate, this is the command being used: pip install --egg git+ https://github.com/globusonline/python-nexus-client@599f04edef6b72569b7a5b27... problems occur if you omit `--egg`.
None of this process is using Galaxy's egg handling, so I am not sure where the EggNotFetchable is coming from. What command are you running to get an EggNotFetchable error.
Second, a question about the rationale for Galaxy's egg handling:
Why is all of this wrapped up in these scripts in the first place? I understand that pip might not be present on every platform, and I don't mean to question a decision to support systems without it. However, as detailed below, Galaxy does not support any platforms which are incapable of running pip.
This isn't the case - Galaxy does not use pip to install the framework dependencies at all. Some tool dependencies installed from the Tool Shed do use pip, but that's entirely separate from the dependencies of the Galaxy application.
The `scripts/scramble.py` script can be used to automatically build eggs on platforms which we do not prebuild eggs for. If this is necessary, `scripts/fetch_eggs.py` should tell you.
Furthermore, pip is being pushed by the Python maintainers over easy_install, so it's not like there isn't a clear choice in terms of which one to support.
Perhaps most importantly, there don't appear to be any clear-cut options to do the following, which I would consider a more ordinary workflow:
- Run a galaxy script (like check_eggs) to generate a list of packages from eggs.g2.bx.psu.edu for platform (redirect output to requirements.txt or similar) - `pip install -r requirements.txt`
This is exactly what `scripts/check_eggs.py` and `scripts/fetch_eggs.py` do.
There are 3 reasons for the way we handle eggs in Galaxy:
1. Galaxy has a huge (and ever-growing) list of dependent python modules with C extensions. If we did not prebuild and distribute eggs for these, the initial setup to get Galaxy running would be long and problematic. Some people who download Galaxy to develop tools may not even have compilers installed, let alone the multitude of -dev or -devel packages that aren't part of a default Debian or RHEL installation that would be required to build all of these packages from source. One of the things that I feel makes Galaxy so accessible is that you can start using it immediately after you clone the source. So that ability to clone and start and have it work as reliably (and quickly) as possible is a high priority.
2. Galaxy started using eggs in 2005 or 2006. At this time, everything used distutils. pkg_resources came around, which soon brought setuptools and easy_install. After this came distutils2, pip and finally, these days, wheels. Our need for binary dependency packaging predated almost all of these (in fact, most packages in these days didn't even install .egg-info, which was the only reliable way to know what version of a module you were using) and as each new iteration of packaging/management came along it was never clear that any of them had "won" (and in fact, most of them lost). On top of this, the Python packaging folks have known for years that Python's platform detection for binary compatibility is broken[1]. While I was assured it'd be fixed soon, even with a complete reimplementation of Python packaging (wheels), they still haven't even made an effort to fix this problem[2]. In fact, binary wheels for Linux are explicitly not allowed on PyPI because of this.
3. We tightly control the versions of all of our dependencies, which is not always possible with pip if you aren't also controlling the source of your packages.
[1] https://mail.python.org/pipermail/distutils-sig/2010-January/015345.html [2] http://lucumr.pocoo.org/2014/1/27/python-on-wheels/
The above could be part of a galaxy provisioning script, rather than
exposed to the administrator.
That also makes it significantly easier to control and manage the virtualenv in which we run Galaxy, since we don't have to worry about egg-related logic that we don't control and we know that the virtualenv's bin dir will be earlier in the PATH than the system pip's dir. Yes, I said above that our setup is presently using system python -- switching to a virtualenv is one of the many items on my to-do list. In fact, I would expect that the default, desired setup for Galaxy would be to put it in a virtualenv, rather than using system python, and to use pip, rather than fetch_eggs.py and company.
A virtualenv is indeed the strongly preferred setup as I mentioned above. However, Galaxy does not install its eggs to the virtualenv. The virtualenv is there to avoid conflicts with things in the default python's site-packages/dist-packages. Galaxy's eggs are installed to (by default) the `eggs/` directory in the Galaxy source.
However, the problem, as I now see it, is that you are trying to install all of Galaxy's dependencies, even at their correct versions, using pip, rather than letting Galaxy handle its eggs as it does. This is not going to work, Galaxy is going to insist on using its eggs.
When I look at the logic being used here, especially at https://bitbucket.org/galaxy/galaxy-central/src/f0ae870b22e9/lib/galaxy/eggs... , it looks like a solution built exclusively for platforms on which pip is not installed. According to the wiki, Galaxy support only goes back as far as 2.6, and get-pip.py supports 2.6, so there is no way of building a Galaxy server on a platform that can't also have pip installed. Adding pip to the requirements for Galaxy would not be particularly onerous, and may simplify things significantly (no need to bundle get-pip.py or similar).
As mentioned above, we don't use or depend on pip, so it's not required. That said, a lot of our egg fetching logic could likely be replaced with pip (this code predates pip by a few years). And the eggs could probably be replaced with prebuilt wheels. However, even if we did use pip/wheels, we'd need to install them from our own repository, and it'd still require modifications for binary platform incompatibilities. The egg handling code we have now works pretty reliably, so I am not sure there is a whole lot to be gained by changing it until Python finally figures out how to handle binary compatibility properly.
As a last note about the misleading error from the fetch_eggs script, telling me that WebError is "NotFetchable". I probably wouldn't have much of a complaint about this if the error had been more on target and I hadn't felt the need to do things like patch lib/galaxy/eggs to print a stacktrace. For example, if the script detected that there was a source installed package which was getting underfoot, it should have alerted me or even suggested installing my various packages in egg format.
This is a bug - it is supposed to explain that there is a version conflict. If you have a stack trace, please send it along.
That kind of error detection is hard to maintain and hard to keep accurate since the Galaxy team's priority is to build Galaxy, not a package manager. This, I suppose, circles back to an earlier question: why isn't Galaxy using a python package manager to... manage its packages?
At the very least something along the lines of `./scripts/common_setup.sh --use-pip` should be added. It doesn't seem like it would be that hard to implement -- but I feel like my lack of knowledge of Galaxy disqualifies me from building the changeset reliably. Of course, if no one objects, I will readily do so anyway (when I have the time) as a proof of concept.
There are modifications to a few of the dependencies, such as psycopg2. When psycopg2 is scrambled, the scrambling process fetches and compiles a bit of PostgreSQL's libpq and then statically links to it to provide a standalone egg that does not depend on the user having installed libpq on their system. So a pip-based system that allowed building from source would need to account for this.
--nate
Barring even that improvement, the Wiki page at https://wiki.galaxyproject.org/Admin/Config/Eggs should definitely be updated to include some note on why Galaxy has this complex logic for fetching eggs instead of using pip.
A quick tl;dr and summary: - I don't have extensive experience with Galaxy, so I may not know what I'm talking about. - fetch_eggs.py can be made to raise EggNotFetchable by doing a pip install from a VCS without using `--egg`. This is a bug in fetch_eggs.py - All supported platforms for Galaxy support pip - Galaxy should have an option to use pip to download its packages over https from eggs.g2.bx.psu.edu - Galaxy should probably default to using pip when it's available, since its failure modes are significantly better than a home-brewed package manager -- this also leads to good behavior in virtualenvs.
Best regards, and many thanks for your time and attention, -Stephen
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
It took me a couple of extra days to get the time to test this, sorry about the delay. The initial run: WebError 0.8a couldn't be downloaded automatically. You can try building it by hand with: python scripts/scramble.py -e WebError Fetch failed. When I patch a traceback.print_exc() into fetch_eggs.py , this is what I see: Some eggs are out of date, attempting to fetch... Warning: sqlalchemy (a dependent egg of sqlalchemy-migrate) cannot be fetched Traceback (most recent call last): File "./scripts/fetch_eggs.py", line 36, in <module> c.resolve() # Only fetch eggs required by the config File "/opt/galaxy/lib/galaxy/eggs/__init__.py", line 349, in resolve raise EggNotFetchable( missing ) EggNotFetchable: <unprintable EggNotFetchable object> WebError 0.8a couldn't be downloaded automatically. You can try building it by hand with: python scripts/scramble.py -e WebError Fetch failed. If I look into lib/galaxy/eggs , that line is here: https://bitbucket.org/galaxycloud/galaxy/src/e7f887cdf229aaf3b44977ecbf24130... In the current galaxy source, that is here: https://bitbucket.org/galaxy/galaxy-central/src/1b09e60f84db859ae49bed71c6f3... As a bit of an aside, I really do have to install this other package in the same python that galaxy uses, since it's one of the major parts of our fork -- not part of our custom tools. I wouldn't consider this a normal use case of course, but I'll bet that many people will install galaxy with system python, which can lead to similar circumstances. Please let me know if I can provide any other useful information on what I'm seeing. Again, thanks for looking into this with me, -Stephen On Thu, Feb 5, 2015 at 8:19 AM, Nate Coraor <nate@bx.psu.edu> wrote:
Hi Stephen,
On Wed, Feb 4, 2015 at 8:30 PM, Stephen Rosen <sirosen@uchicago.edu> wrote:
Hi Nate,
Thanks so much for helping me out with this.
It seems that I've miscommunicated what I'm doing a little bit. I'm installing a package from that git repo using pip separately from running the galaxy setup script. That is, I want to install a python package and I want to setup galaxy as two separate (and ideally independent) steps in my script. I have no desire to do anything particularly esoteric or clever with Galaxy's eggs or to install them myself. Somehow, doing a pip install of my desired package is breaking the later run of fetch_eggs.py
If I understand you correctly, fetch_eggs.py can hit conflicts with site-packages, but is misreporting them as EggNotFetchable. I am aware that we're using a fork of the Galaxy source ( https://bitbucket.org/galaxycloud/galaxy/src ), so it may be that we have an outdated of fetch_eggs.py which carries a bug that has since been patched. Sorry for not mentioning it earlier -- it slipped my mind that we're probably trailing behind the modern Galaxy default head.
If I put in the extra legwork now to wrap Galaxy in a virtualenv, this issue will presumably disappear, but that assumes that I don't want to add any packages to the python used for the uwsgi process.
You can add additional packages to this virtualenv, although there could be problems if you install different versions of the same packages that Galaxy has as dependencies. For what it's worth, you should not need to install additional packages in Galaxy's virtualenv, and tools can be configured to use a different python or virtualenv than the one in which Galaxy is started.
I'll have to ask folks on the Globus Genomics team whether or not that is needed -- I certainly hope not, since it means that there is some packaging conflict we need to resolve. The only reason I haven't done so already is time constraint -- I've been trying to get this install scripting done as quickly as possible without compromising anything truly needful.
All the same, I'll take some time tomorrow to provision a fresh server and generate the stacktrace for you. I'll also take some time to test with the latest Galaxy source, to see if I get different behavior. In the best case, this bug no longer exists for the modern source, but in the worst case it bears the attention.
Thanks for the help with debugging.
I don't have any sense of the reliability of fetch_eggs.py and company -- my experience has been of a particular bug that presented on my first contact with these scripts, so naturally I have a bias against them. That said, I think the Python community may have finally settled on a package manager, even if we can't seem to agree on a package format or tooling surrounding it. That's just an opinion though -- I have no blog post from Guido to back it up.
I hope we'll see the necessary improvements within the next year, but I think anything we do now will essentially reinvent the wheel (no pun intended) with regards to what will (hopefully) be officially implemented to fix the problem.
--nate
I was not aware of the UCS2 vs. UCS4 issue -- thanks very much for the citations, very helpful in understanding the problem space.
Thanks, -Stephen
On Wed, Feb 4, 2015 at 5:41 PM, Nate Coraor <nate@bx.psu.edu> wrote:
Hi Stephen,
I'll try to reply as in-depth as I can.
On Wed, Feb 4, 2015 at 1:41 PM, Stephen Rosen <sirosen@uchicago.edu> wrote:
Hi Galaxy Dev,
I've been looking at the setup scripts for Galaxy to try to understand a problem I recently had provisioning a Galaxy server. I will readily admit that I have not read all of the relevant code top-to-bottom, but I have at least skimmed all of it and read much of it. Sorry if these questions are answered somewhere in Trello, the Wiki, or somewhere else, but I was not able to find answers in any public locations.
As a small bit of probably irrelevant context: I'm working with the Globus Genomics group on the DevOps side of things. We're using Chef. I've only just started working with the group in the past couple of weeks (so my expertise with Galaxy itself is limited to nonexistent).
First, to describe the problem:
We want to provision a server running Galaxy without explicitly wrapping it in a virtualenv. Unless I missed something, that means that it's using system python. When we use pip to install a package from a git repository before running the setup scripts, fetch_eggs fails saying it failed to fetch WebError 0.8a If we install the same package from git with `pip install --egg ...` we get a hunky-dory system where everything seems to work.
A virtualenv is itself just a wrapper around whichever python binary was used to create it. I'd still suggest using a virtualenv created with the system python unless you have a really strong reason not to. In fact, I'm working on Galaxy process management and a command line tool for that management that will automatically create and use a virtualenv going forward.
I'm a bit confused at what's happening here - you mention installing a package from a git repository with pip, but then refer to Galaxy's fetch_eggs(.py) script, which doesn't use pip or git.
As far as I can tell, there is no reason that this should be the case. Sure, putting the git source directly into site-packages might cause issues upon installation, but EggNotFetchable exceptions should only be thrown if the egg actually can't be pulled down from eggs.g2.bx.psu.edu , right?
EggNotFetchable can be thrown if you happen to be using a platform for which we do not provide eggs, although those are fairly uncommon. Right now we should cover x86/x86_64 Linux and any flavor of Intel OS X after 10.5.
I don't feel comfortable trying to make further progress on my provisioning scripts without knowing why this is happening. I'd hate to be bitten by this later on in the process.
Yes, the package in question may have poor behavior (likely it does), but that doesn't change the fact that the error is totally misleading. Furthermore, it doesn't appear that this poor behavior impedes me from doing a pip install of the WebError package or any other packages from PyPI.
In case someone else wants to test to replicate, this is the command being used: pip install --egg git+ https://github.com/globusonline/python-nexus-client@599f04edef6b72569b7a5b27... problems occur if you omit `--egg`.
None of this process is using Galaxy's egg handling, so I am not sure where the EggNotFetchable is coming from. What command are you running to get an EggNotFetchable error.
Second, a question about the rationale for Galaxy's egg handling:
Why is all of this wrapped up in these scripts in the first place? I understand that pip might not be present on every platform, and I don't mean to question a decision to support systems without it. However, as detailed below, Galaxy does not support any platforms which are incapable of running pip.
This isn't the case - Galaxy does not use pip to install the framework dependencies at all. Some tool dependencies installed from the Tool Shed do use pip, but that's entirely separate from the dependencies of the Galaxy application.
The `scripts/scramble.py` script can be used to automatically build eggs on platforms which we do not prebuild eggs for. If this is necessary, `scripts/fetch_eggs.py` should tell you.
Furthermore, pip is being pushed by the Python maintainers over easy_install, so it's not like there isn't a clear choice in terms of which one to support.
Perhaps most importantly, there don't appear to be any clear-cut options to do the following, which I would consider a more ordinary workflow:
- Run a galaxy script (like check_eggs) to generate a list of packages from eggs.g2.bx.psu.edu for platform (redirect output to requirements.txt or similar) - `pip install -r requirements.txt`
This is exactly what `scripts/check_eggs.py` and `scripts/fetch_eggs.py` do.
There are 3 reasons for the way we handle eggs in Galaxy:
1. Galaxy has a huge (and ever-growing) list of dependent python modules with C extensions. If we did not prebuild and distribute eggs for these, the initial setup to get Galaxy running would be long and problematic. Some people who download Galaxy to develop tools may not even have compilers installed, let alone the multitude of -dev or -devel packages that aren't part of a default Debian or RHEL installation that would be required to build all of these packages from source. One of the things that I feel makes Galaxy so accessible is that you can start using it immediately after you clone the source. So that ability to clone and start and have it work as reliably (and quickly) as possible is a high priority.
2. Galaxy started using eggs in 2005 or 2006. At this time, everything used distutils. pkg_resources came around, which soon brought setuptools and easy_install. After this came distutils2, pip and finally, these days, wheels. Our need for binary dependency packaging predated almost all of these (in fact, most packages in these days didn't even install .egg-info, which was the only reliable way to know what version of a module you were using) and as each new iteration of packaging/management came along it was never clear that any of them had "won" (and in fact, most of them lost). On top of this, the Python packaging folks have known for years that Python's platform detection for binary compatibility is broken[1]. While I was assured it'd be fixed soon, even with a complete reimplementation of Python packaging (wheels), they still haven't even made an effort to fix this problem[2]. In fact, binary wheels for Linux are explicitly not allowed on PyPI because of this.
3. We tightly control the versions of all of our dependencies, which is not always possible with pip if you aren't also controlling the source of your packages.
[1] https://mail.python.org/pipermail/distutils-sig/2010-January/015345.html [2] http://lucumr.pocoo.org/2014/1/27/python-on-wheels/
The above could be part of a galaxy provisioning script, rather than
exposed to the administrator.
That also makes it significantly easier to control and manage the virtualenv in which we run Galaxy, since we don't have to worry about egg-related logic that we don't control and we know that the virtualenv's bin dir will be earlier in the PATH than the system pip's dir. Yes, I said above that our setup is presently using system python -- switching to a virtualenv is one of the many items on my to-do list. In fact, I would expect that the default, desired setup for Galaxy would be to put it in a virtualenv, rather than using system python, and to use pip, rather than fetch_eggs.py and company.
A virtualenv is indeed the strongly preferred setup as I mentioned above. However, Galaxy does not install its eggs to the virtualenv. The virtualenv is there to avoid conflicts with things in the default python's site-packages/dist-packages. Galaxy's eggs are installed to (by default) the `eggs/` directory in the Galaxy source.
However, the problem, as I now see it, is that you are trying to install all of Galaxy's dependencies, even at their correct versions, using pip, rather than letting Galaxy handle its eggs as it does. This is not going to work, Galaxy is going to insist on using its eggs.
When I look at the logic being used here, especially at https://bitbucket.org/galaxy/galaxy-central/src/f0ae870b22e9/lib/galaxy/eggs... , it looks like a solution built exclusively for platforms on which pip is not installed. According to the wiki, Galaxy support only goes back as far as 2.6, and get-pip.py supports 2.6, so there is no way of building a Galaxy server on a platform that can't also have pip installed. Adding pip to the requirements for Galaxy would not be particularly onerous, and may simplify things significantly (no need to bundle get-pip.py or similar).
As mentioned above, we don't use or depend on pip, so it's not required. That said, a lot of our egg fetching logic could likely be replaced with pip (this code predates pip by a few years). And the eggs could probably be replaced with prebuilt wheels. However, even if we did use pip/wheels, we'd need to install them from our own repository, and it'd still require modifications for binary platform incompatibilities. The egg handling code we have now works pretty reliably, so I am not sure there is a whole lot to be gained by changing it until Python finally figures out how to handle binary compatibility properly.
As a last note about the misleading error from the fetch_eggs script, telling me that WebError is "NotFetchable". I probably wouldn't have much of a complaint about this if the error had been more on target and I hadn't felt the need to do things like patch lib/galaxy/eggs to print a stacktrace. For example, if the script detected that there was a source installed package which was getting underfoot, it should have alerted me or even suggested installing my various packages in egg format.
This is a bug - it is supposed to explain that there is a version conflict. If you have a stack trace, please send it along.
That kind of error detection is hard to maintain and hard to keep accurate since the Galaxy team's priority is to build Galaxy, not a package manager. This, I suppose, circles back to an earlier question: why isn't Galaxy using a python package manager to... manage its packages?
At the very least something along the lines of `./scripts/common_setup.sh --use-pip` should be added. It doesn't seem like it would be that hard to implement -- but I feel like my lack of knowledge of Galaxy disqualifies me from building the changeset reliably. Of course, if no one objects, I will readily do so anyway (when I have the time) as a proof of concept.
There are modifications to a few of the dependencies, such as psycopg2. When psycopg2 is scrambled, the scrambling process fetches and compiles a bit of PostgreSQL's libpq and then statically links to it to provide a standalone egg that does not depend on the user having installed libpq on their system. So a pip-based system that allowed building from source would need to account for this.
--nate
Barring even that improvement, the Wiki page at https://wiki.galaxyproject.org/Admin/Config/Eggs should definitely be updated to include some note on why Galaxy has this complex logic for fetching eggs instead of using pip.
A quick tl;dr and summary: - I don't have extensive experience with Galaxy, so I may not know what I'm talking about. - fetch_eggs.py can be made to raise EggNotFetchable by doing a pip install from a VCS without using `--egg`. This is a bug in fetch_eggs.py - All supported platforms for Galaxy support pip - Galaxy should have an option to use pip to download its packages over https from eggs.g2.bx.psu.edu - Galaxy should probably default to using pip when it's available, since its failure modes are significantly better than a home-brewed package manager -- this also leads to good behavior in virtualenvs.
Best regards, and many thanks for your time and attention, -Stephen
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Stephen and Nate, it seems that the problem with the WebError egg is now hitting also BioBlend TravisCI testing: https://travis-ci.org/galaxyproject/bioblend/builds/50391748 Both for Python 2.6 and 2.7, scripts/common_startup.sh fails with: "WebError 0.8a couldn't be downloaded automatically. You can try building it by hand with: python scripts/scramble.py -c config/galaxy.ini.sample -e WebError Fetch failed." Please note that this has been working fine for 6 months (last TravisCI build 19 days ago) and is still working on my laptops. Nicola Il 08.02.2015 04:35 Stephen Rosen ha scritto: > It took me a couple of extra days to get the time to test this, sorry about the delay. > > The initial run: > > WebError 0.8a couldn't be downloaded automatically. You can try > building it by hand with: > python scripts/scramble.py -e WebError > Fetch failed. > > When I patch a traceback.print_exc() into fetch_eggs.py , this is what I see: > > Some eggs are out of date, attempting to fetch... > Warning: sqlalchemy (a dependent egg of sqlalchemy-migrate) cannot be fetched > Traceback (most recent call last): > File "./scripts/fetch_eggs.py", line 36, in > c.resolve() # Only fetch eggs required by the config > File "/opt/galaxy/lib/galaxy/eggs/__init__.py", line 349, in resolve > raise EggNotFetchable( missing ) > EggNotFetchable: > WebError 0.8a couldn't be downloaded automatically. You can try > building it by hand with: > python scripts/scramble.py -e WebError > Fetch failed. > > If I look into lib/galaxy/eggs , that line is here: > https://bitbucket.org/galaxycloud/galaxy/src/e7f887cdf229aaf3b44977ecbf24130ddc8467e3/lib/galaxy/eggs/__init__.py?at=default#cl-349 [14] > In the current galaxy source, that is here: > https://bitbucket.org/galaxy/galaxy-central/src/1b09e60f84db859ae49bed71c6f3fd838af612f5/lib/galaxy/eggs/__init__.py?at=default#cl-351 [15] > > As a bit of an aside, I really do have to install this other package in the same python that galaxy uses, since it's one of the major parts of our fork -- not part of our custom tools. > I wouldn't consider this a normal use case of course, but I'll bet that many people will install galaxy with system python, which can lead to similar circumstances. > > Please let me know if I can provide any other useful information on what I'm seeing. > > Again, thanks for looking into this with me, > -Stephen > > On Thu, Feb 5, 2015 at 8:19 AM, Nate Coraor wrote: > >> Hi Stephen, >> >> On Wed, Feb 4, 2015 at 8:30 PM, Stephen Rosen wrote: >> >>> Hi Nate, >>> >>> Thanks so much for helping me out with this. >>> >>> It seems that I've miscommunicated what I'm doing a little bit. I'm installing a package from that git repo using pip separately from running the galaxy setup script. That is, I want to install a python package and I want to setup galaxy as two separate (and ideally independent) steps in my script. >>> I have no desire to do anything particularly esoteric or clever with Galaxy's eggs or to install them myself. Somehow, doing a pip install of my desired package is breaking the later run of fetch_eggs.py >>> >>> If I understand you correctly, fetch_eggs.py can hit conflicts with site-packages, but is misreporting them as EggNotFetchable. >>> I am aware that we're using a fork of the Galaxy source ( https://bitbucket.org/galaxycloud/galaxy/src [1] ), so it may be that we have an outdated of fetch_eggs.py which carries a bug that has since been patched. >>> Sorry for not mentioning it earlier -- it slipped my mind that we're probably trailing behind the modern Galaxy default head. >>> >>> If I put in the extra legwork now to wrap Galaxy in a virtualenv, this issue will presumably disappear, but that assumes that I don't want to add any packages to the python used for the uwsgi process. >> >> You can add additional packages to this virtualenv, although there could be problems if you install different versions of the same packages that Galaxy has as dependencies. For what it's worth, you should not need to install additional packages in Galaxy's virtualenv, and tools can be configured to use a different python or virtualenv than the one in which Galaxy is started. >> >>> I'll have to ask folks on the Globus Genomics team whether or not that is needed -- I certainly hope not, since it means that there is some packaging conflict we need to resolve. >>> The only reason I haven't done so already is time constraint -- I've been trying to get this install scripting done as quickly as possible without compromising anything truly needful. >>> >>> All the same, I'll take some time tomorrow to provision a fresh server and generate the stacktrace for you. >>> I'll also take some time to test with the latest Galaxy source, to see if I get different behavior. >>> In the best case, this bug no longer exists for the modern source, but in the worst case it bears the attention. >> >> Thanks for the help with debugging. >> >> I don't have any sense of the reliability of fetch_eggs.py and company -- my experience has been of a particular bug that presented on my first contact with thes >> >>> finally settled on a package manager, even if we can't seem to agree on a package format or tooling surrounding it. >>> That's just an opinion though -- I have no blog post from Guido to back it up. >>> I hope we'll see the necessary improvements within the next year, but I think anything we do now will essentially reinvent the wheel (no pun intended) with regards to what will (hopefully) be officially implemented to fix the problem. >>> --nate >> >> I was not aware of the UCS2 vs. UCS4 issue -- thanks very much for the citations, very helpful in understanding the problem space. >> Thanks, >> -Stephen >> >> On Wed, Feb 4, 2015 at 5:41 PM, Nate Coraor c solid; padding-left: 1ex;"> >>> Hi Stephen, >>> I'll try to reply as in-depth as I can. >>> >>> On Wed, Feb 4, 2015 at 1:41 PM, Stephen Rosen wrote: >>> >>>> Hi Galaxy Dev, >>>> >>>> I've been looking at the setup scripts for Galaxy to try to understand a problem I recently had provisioning a Galaxy server. I will readily admit that I have not read all of the relevant code top-to-bottom, but I have at least skimmed all of it and read much of it. >>>> Sorry if these questions are answered somewhere in Trello, the Wiki, or somewhere else, but I was not able to find answers in any public locations. >>>> >>>> As a small bit of probably irrelevant context: >>>> I'm working with the Globus Genomics group on the DevOps side of things. >>>> We're using Chef. >>>> I've only just started working with the group in the past couple of weeks (so my expertise with Galaxy itself is limited to nonexistent). >>>> >>>> First, to describe the problem: >>>> >>>> We want to provision a server running Galaxy without explicitly wrapping it in a virtualenv. >>>> Unless I missed something, that means that it's using system python. When we use pip to install a package from a git repository before running the setup scripts, fetch_eggs fails saying it failed to fetch WebError 0.8a If we install the same package from git with `pip install --egg ...` we get a hunky-dory system where everything seems to work. >>> >>> A virtualenv is itself just a wrapper around whichever python binary was used to create it. I'd still suggest using a virtualenv created with the system python unless you have a really strong reason not to. In fact, I'm working on Galaxy process management and a command line tool for that management that will automatically create and use a virtualenv going forward. >>> I'm a bit confused at what's happening here - you mention installing a package from a git repository with pip, but then refer to Galaxy's fetch_eggs(.py) script, which doesn't use pip or git. >>> >>>> As far as I can tell, there is no reason that this should be the case. Sure, putting the git source directly into site-packages might cause issues upon installation, but EggNotFetchable exceptions should only be thrown if the egg actually can't be pulled down from eggs.g2.bx.psu.edu [2] , right? >>> >>> EggNotFetchable can be thrown if you happen to be using a platform for which we do not provide eggs, although those are fairly uncommon. Right now we should cover x86/x86_64 Linux and any flavor of Intel OS X after 10.5. >>> >>> I don't feel comfortable trying to make further progress on my provisioning scripts without knowing why this is happening. >>> I'd hate to be bitten by this later on in the process >>> >>>> br />Furthermore, it doesn't appear that this poor behavior impedes me from doing a pip install of the WebError package or any other packages from PyPI. >>>> >>>> In case someone else wants to test to replicate, this is the command being used: >>>> pip install --egg git+https://github.com/globusonline/python-nexus-client@599f04edef6b72569b7a5b272b0b847dcda3ea99#egg=nexus-client [3] >>>> problems occur if you omit `--egg`. >>>> None of this process is using Galaxy's egg handling, so I am not sure where the EggNotFetchable is coming from. What command are you running to get an EggNotFetchable error. >>> ltr"> >>> >>> Second, a question about the rationale for Galaxy's egg handling: >>> >>> Why is all of this wrapped up in these scripts in the first place? >>> I understand that pip might not be present on every platform, and I don't mean to question a decision to support systems without it. >>> However, as detailed below, Galaxy does not support any platforms which are incapable of running pip. >>> This isn't the case - Galaxy does not us >>> >>>> of the Galaxy application. >>>> The `scripts/scramble.py` script can be used to automatically build eggs on platforms which we do not prebuild eggs for. If this is necessary, `scripts/fetch_eggs.py` should tell you. >>>> >>>> Furthermore, pip is being pushed >>> n maintainers over easy_install, so it's not like there isn't a clear choice in terms of which one to support. >>> Perhaps most importantly, there don't appear to be any clear-cut options to do the following, which I would consider a more ordinary workflow: >>> >>> - Run a galaxy script (like check_eggs) to generate a list of packages from eggs.g2.bx.psu.edu [9] for platform (redirect output to requirements.txt or similar) >>> - `pip install -r requirements.txt` >>> This is exactly what `scripts/check_eggs.py` and `scripts/fetch_eggs.py` do. >>> There are 3 reasons for the way we handle eggs in Galaxy: >>> 1. Galaxy has a huge (and ever-growing) list of dependent python modules with C extensions. If we did not prebuild and distribute eggs for these, the initial setup to get Galaxy running would be long and problematic. Some people who download Galaxy to develop tools may not even h >>> >>>> ll of these packages from source. One of the things that I feel makes Galaxy so accessible is that you can start using it immediately after you clone the source. So that ability to clone and start and have it work as reliably (and quickly) as possible is a high priority. >>>> 2. Galaxy started using eggs in 2005 or 2006. At this time, everything used distutils. pkg_resources came around, which soon brought setuptools and easy_install. After this came distutils2, pip and finally, these days, wheels. Our need for binary dependency packaging predated almost all of these (in fact, most packages in these days >>> nstall .egg-info, which was the only reliable way to know what version of a module you were using) and as each new iteration of packaging/management came along it was never clear that any of them had "won" (and in fact, most of them lost). On top of this, the Python packaging folks have known for years that Python's platform detection for binary compatibility is broken[1]. While I was assured it'd be fixed soon, even with a complete reimplementation of Python packaging (wheels), they still haven't even made an effort to fix this problem[2]. In fact, binary wheels for Linux are explicitly not allowed on PyPI because of this. >>> >>> 3. We tightly control the versions of all of our dependencies, which is not always possible with pip if you aren't also controlling the source of your packages. >>> [1] https://mail.python.org/pipermail/distutils-sig/2010-January/015345.html [10] >>> [2] http://lucumr.pocoo.org/2014/1/27/python-on-wheels/ [11] >>> >>> The above could be part of a galaxy provisioning script, rather than exposed to the administrator. >>> That also makes it significantly easier to control and manage the virtualenv in which we run Galaxy, since we don't have to worry about egg-related logic that we don't control and we know that the virtualenv's bin dir will be earlier in the PATH than the system pip's dir. >>> Yes, I said above that our setup is presently using system python -- switching to a virtualenv is one of the many items on my to-do list. >>> In fact, I would expect that the default, desired setup for Galaxy would be to put it in a virtualenv, rather than using system python, and to use pip, rather than fetch_eggs.py and company. >>> A virtualenv is indeed the strongly preferred setup as I mentioned above. However, Galaxy does not install its eggs to the virtualenv. The virtualenv is there to avoid conflicts with things in the default python's site-packages/dist-packages. Galaxy's eggs are installed to (by default) the `eggs/` directory in the Galaxy source. >>> However, the problem, as I now see it, is that you are trying to install all of Galaxy's dependencies, even at their correct versions, using pip, rather than letting Galaxy handle its eggs as it does. This is not going to work, Galaxy is going to insist on using its eggs. >>> "> >>>> When I look at the logic being used here, especially at https://bitbucket.org/galaxy/galaxy-central/src/f0ae870b22e9/lib/galaxy/eggs/?at=default [4] , it looks like a solution built exclusively for platforms on which pip is not installed. >>>> According to the wiki, Galaxy support only goes back as far as 2.6, and get-pip.py supports 2.6, so there is no way of building a Galaxy server on a platform that can't also have pip installed. >>>> Adding pip to the requirements for Galaxy would not be particularly onerous, and may simplify things significantly (no need to bundle get-pip.py or similar). >>>> As mentioned >>> n't use or depend on pip, so it's not required. That said, a lot of our egg fetching logic could likely be replaced with pip (this code predates pip by a few years). And the eggs could probably be replaced with prebuilt wheels. However, even if we did use pip/wheels, we'd need to install them from our own repository, and it'd still require modifications for binary platform incompatibilities. The egg handling code we have now works pretty reliably, so I am not sure there is a whole lot to be gained by changing it until Python finally figures out how to handle binary compatibility properly. >>> >>> As a la >>> >>>> ut this if the error had been more on target and I hadn't felt the need to do things like patch lib/galaxy/eggs to print a stacktrace. >>>> For example, if the script detected that there was a source installed package which was getting underfoot, it should have alerted me or even suggested installing my various packages in egg format. >>>> This is a bug - it is supposed to explain that there is a version conflict. If you have a stack trace, please send it along. >>>> >>>> That kind of error detection is hard t >>> d hard to keep accurate since the Galaxy team's priority is to build Galaxy, not a package manager. >>> This, I suppose, circles back to an earlier question: why isn't Galaxy using a python package manager to... manage its packages? >>> >>> At the very least something along the lines of `./scripts/common_setup.sh --use-pip` should be added. >>> It doesn't seem like it would be that hard to implement -- but I feel like my lack of knowledge of Galaxy disqualifies me from building the changeset reliably. >>> Of course, if no one objects, I will readily do so anyway (when I have the time) as a proof of concept. >>> There are modifications to a few of the dependencies, such as psycopg2. When psycopg2 is scrambled, the scrambling process fetches and compiles a bit of PostgreSQL's libpq and then statically links to it to provide a standalone egg that does not depend on t >>> >>>> Barring even that improvement, the Wiki page at https://wiki.galaxyproject.org/Admin/Config/Eggs [5] should definitely be updated to include some note on why Galaxy has this complex logic for fetching eggs instead of using pip. >>>> >>>> A quick tl;dr and summary: >>> I don't have extensive experience with Galaxy, so I may not know what I'm talking about. >>> - fetch_eggs.py can be made to raise EggNotFetchable by doing a pip install from a VCS without using `--egg`. This is a bug in fetch_eggs.py >>> - All supported platforms for Galaxy support pip >>> - Galaxy should have an option to use pip to download its packages over https from eggs.g2.bx.psu.edu [12] >>> - Galaxy should probably default to using pip when it's available, since its failure modes >>> >>>> many thanks for your time and attention, >>>> -Stephen ___________________________________________________________ >>>> Please keep all replies on the list by using "reply all" >>>> in your mail client. To manage your subscriptions to this >>>> and other Galaxy lists, please use the interface at: >>>> https://lists.galaxyproject.org/ [6] >>>> >>>> To search Galaxy mailing lists use the unified search at: >>>> http://galaxyproject.org/search/mailinglists/ [7] >>> >>>> Connetti gratis il mondo con la nuova indoona: hai la chat, le chiamate, le video chiamate e persino le chiamate di gruppo. E chiami gratis anche i numeri fissi e mobili nel mondo! Scarica subito l’app Vai su https://www.indoona.com/
I believe I've fixed this in 16583:5b2f1eab2819, if you want to give it another shot. External version of pkg_resources (and not ours in lib/) is what was causing the weird egg fetching errors. Newer versions of pkg_resources create a mangled distribution string for some eggs with nonstandard version identifiers. On Thu Feb 12 2015 at 6:44:29 AM Nicola Soranzo <nsoranzo@tiscali.it> wrote:
Hi Stephen and Nate, it seems that the problem with the WebError egg is now hitting also BioBlend TravisCI testing:
https://travis-ci.org/galaxyproject/bioblend/builds/50391748
Both for Python 2.6 and 2.7, scripts/common_startup.sh fails with:
"WebError 0.8a couldn't be downloaded automatically. You can try building it by hand with:
python scripts/scramble.py -c config/galaxy.ini.sample -e WebError Fetch failed."
Please note that this has been working fine for 6 months (last TravisCI build 19 days ago) and is still working on my laptops.
Nicola
Il 08.02.2015 04:35 Stephen Rosen ha scritto:
It took me a couple of extra days to get the time to test this, sorry about the delay.
The initial run:
WebError 0.8a couldn't be downloaded automatically. You can try building it by hand with: python scripts/scramble.py -e WebError Fetch failed.
When I patch a traceback.print_exc() into fetch_eggs.py , this is what I see:
Some eggs are out of date, attempting to fetch... Warning: sqlalchemy (a dependent egg of sqlalchemy-migrate) cannot be fetched Traceback (most recent call last): File "./scripts/fetch_eggs.py", line 36, in
c.resolve() # Only fetch eggs required by the config File "/opt/galaxy/lib/galaxy/eggs/__init__.py", line 349, in resolve raise EggNotFetchable( missing ) EggNotFetchable:
WebError 0.8a couldn't be downloaded automatically. You can try building it by hand with: python scripts/scramble.py -e WebError Fetch failed.
If I look into lib/galaxy/eggs , that line is here:
https://bitbucket.org/galaxycloud/galaxy/src/e7f887cdf229aaf3b44977ecbf24130... In the current galaxy source, that is here:
https://bitbucket.org/galaxy/galaxy-central/src/1b09e60f84db859ae49bed71c6f3...
As a bit of an aside, I really do have to install this other package in the same python that galaxy uses, since it's one of the major parts of our fork -- not part of our custom tools. I wouldn't consider this a normal use case of course, but I'll bet that many people will install galaxy with system python, which can lead to similar circumstances.
Please let me know if I can provide any other useful information on what I'm seeing.
Again, thanks for looking into this with me, -Stephen
On Thu, Feb 5, 2015 at 8:19 AM, Nate Coraor <nate@bx.psu.edu> wrote:
Hi Stephen,
On Wed, Feb 4, 2015 at 8:30 PM, Stephen Rosen <sirosen@uchicago.edu> wrote:
Hi Nate,
Thanks so much for helping me out with this.
It seems that I've miscommunicated what I'm doing a little bit. I'm installing a package from that git repo using pip separately from running the galaxy setup script. That is, I want to install a python package and I want to setup galaxy as two separate (and ideally independent) steps in my script. I have no desire to do anything particularly esoteric or clever with Galaxy's eggs or to install them myself. Somehow, doing a pip install of my desired package is breaking the later run of fetch_eggs.py
If I understand you correctly, fetch_eggs.py can hit conflicts with site-packages, but is misreporting them as EggNotFetchable. I am aware that we're using a fork of the Galaxy source ( https://bitbucket.org/galaxycloud/galaxy/src ), so it may be that we have an outdated of fetch_eggs.py which carries a bug that has since been patched. Sorry for not mentioning it earlier -- it slipped my mind that we're probably trailing behind the modern Galaxy default head.
If I put in the extra legwork now to wrap Galaxy in a virtualenv, this issue will presumably disappear, but that assumes that I don't want to add any packages to the python used for the uwsgi process.
You can add additional packages to this virtualenv, although there could be problems if you install different versions of the same packages that Galaxy has as dependencies. For what it's worth, you should not need to install additional packages in Galaxy's virtualenv, and tools can be configured to use a different python or virtualenv than the one in which Galaxy is started.
I'll have to ask folks on the Globus Genomics team whether or not that is needed -- I certainly hope not, since it means that there is some packaging conflict we need to resolve. The only reason I haven't done so already is time constraint -- I've been trying to get this install scripting done as quickly as possible without compromising anything truly needful.
All the same, I'll take some time tomorrow to provision a fresh server and generate the stacktrace for you. I'll also take some time to test with the latest Galaxy source, to see if I get different behavior. In the best case, this bug no longer exists for the modern source, but in the worst case it bears the attention.
Thanks for the help with debugging.
I don't have any sense of the reliability of fetch_eggs.py and company -- my experience has been of a particular bug that presented on my first contact with these scripts, so naturally I have a bias against them. That said, I think the Python community may have finally settled on a package manager, even if we can't seem to agree on a package format or tooling surrounding it. That's just an opinion though -- I have no blog post from Guido to back it up.
I hope we'll see the necessary improvements within the next year, but I think anything we do now will essentially reinvent the wheel (no pun intended) with regards to what will (hopefully) be officially implemented to fix the problem. --nate
I was not aware of the UCS2 vs. UCS4 issue -- thanks very much for the citations, very helpful in understanding the problem space. Thanks, -Stephen
On Wed, Feb 4, 2015 at 5:41 PM, Nate Coraor <nate@bx.psu.edu> wrote:
Hi Stephen, I'll try to reply as in-depth as I can.
On Wed, Feb 4, 2015 at 1:41 PM, Stephen Rosen <sirosen@uchicago.edu> wrote:
Hi Galaxy Dev,
I've been looking at the setup scripts for Galaxy to try to understand a problem I recently had provisioning a Galaxy server. I will readily admit that I have not read all of the relevant code top-to-bottom, but I have at least skimmed all of it and read much of it. Sorry if these questions are answered somewhere in Trello, the Wiki, or somewhere else, but I was not able to find answers in any public locations.
As a small bit of probably irrelevant context: I'm working with the Globus Genomics group on the DevOps side of things. We're using Chef. I've only just started working with the group in the past couple of weeks (so my expertise with Galaxy itself is limited to nonexistent).
First, to describe the problem:
We want to provision a server running Galaxy without explicitly wrapping it in a virtualenv. Unless I missed something, that means that it's using system python. When we use pip to install a package from a git repository before running the setup scripts, fetch_eggs fails saying it failed to fetch WebError 0.8a If we install the same package from git with `pip install --egg ...` we get a hunky-dory system where everything seems to work.
A virtualenv is itself just a wrapper around whichever python binary was used to create it. I'd still suggest using a virtualenv created with the system python unless you have a really strong reason not to. In fact, I'm working on Galaxy process management and a command line tool for that management that will automatically create and use a virtualenv going forward. I'm a bit confused at what's happening here - you mention installing a package from a git repository with pip, but then refer to Galaxy's fetch_eggs(.py) script, which doesn't use pip or git.
As far as I can tell, there is no reason that this should be the case. Sure, putting the git source directly into site-packages might cause issues upon installation, but EggNotFetchable exceptions should only be thrown if the egg actually can't be pulled down from eggs.g2.bx.psu.edu , right?
EggNotFetchable can be thrown if you happen to be using a platform for which we do not provide eggs, although those are fairly uncommon. Right now we should cover x86/x86_64 Linux and any flavor of Intel OS X after 10.5.
I don't feel comfortable trying to make further progress on my provisioning scripts without knowing why this is happening. I'd hate to be bitten by this later on in the process.
Yes, the package in question may have poor behavior (likely it does), but that doesn't change the fact that the error is totally misleading. Furthermore, it doesn't appear that this poor behavior impedes me from doing a pip install of the WebError package or any other packages from PyPI.
In case someone else wants to test to replicate, this is the command being used: pip install --egg git+ https://github.com/globusonline/python-nexus-client@599f04edef6b72569b7a5b27... problems occur if you omit `--egg`.
None of this process is using Galaxy's egg handling, so I am not sure where the EggNotFetchable is coming from. What command are you running to get an EggNotFetchable error.
Second, a question about the rationale for Galaxy's egg handling:
Why is all of this wrapped up in these scripts in the first place? I understand that pip might not be present on every platform, and I don't mean to question a decision to support systems without it. However, as detailed below, Galaxy does not support any platforms which are incapable of running pip.
This isn't the case - Galaxy does not use pip to install the framework dependencies at all. Some tool dependencies installed from the Tool Shed do use pip, but that's entirely separate from the dependencies of the Galaxy application. The `scripts/scramble.py` script can be used to automatically build eggs on platforms which we do not prebuild eggs for. If this is necessary, `scripts/fetch_eggs.py` should tell you.
Furthermore, pip is being pushed by the Python maintainers over easy_install, so it's not like there isn't a clear choice in terms of which one to support. Perhaps most importantly, there don't appear to be any clear-cut options to do the following, which I would consider a more ordinary workflow:
- Run a galaxy script (like check_eggs) to generate a list of packages from eggs.g2.bx.psu.edu for platform (redirect output to requirements.txt or similar) - `pip install -r requirements.txt`
This is exactly what `scripts/check_eggs.py` and `scripts/fetch_eggs.py` do. There are 3 reasons for the way we handle eggs in Galaxy: 1. Galaxy has a huge (and ever-growing) list of dependent python modules with C extensions. If we did not prebuild and distribute eggs for these, the initial setup to get Galaxy running would be long and problematic. Some people who download Galaxy to develop tools may not even have compilers installed, let alone the multitude of -dev or -devel packages that aren't part of a default Debian or RHEL installation that would be required to build all of these packages from source. One of the things that I feel makes Galaxy so accessible is that you can start using it immediately after you clone the source. So that ability to clone and start and have it work as reliably (and quickly) as possible is a high priority. 2. Galaxy started using eggs in 2005 or 2006. At this time, everything used distutils. pkg_resources came around, which soon brought setuptools and easy_install. After this came distutils2, pip and finally, these days, wheels. Our need for binary dependency packaging predated almost all of these (in fact, most packages in these days didn't even install .egg-info, which was the only reliable way to know what version of a module you were using) and as each new iteration of packaging/management came along it was never clear that any of them had "won" (and in fact, most of them lost). On top of this, the Python packaging folks have known for years that Python's platform detection for binary compatibility is broken[1]. While I was assured it'd be fixed soon, even with a complete reimplementation of Python packaging (wheels), they still haven't even made an effort to fix this problem[2]. In fact, binary wheels for Linux are explicitly not allowed on PyPI because of this.
3. We tightly control the versions of all of our dependencies, which is not always possible with pip if you aren't also controlling the source of your packages. [1] https://mail.python.org/pipermail/distutils-sig/2010-January/015345.html [2] http://lucumr.pocoo.org/2014/1/27/python-on-wheels/
The above could be part of a galaxy provisioning script, rather than exposed to the administrator. That also makes it significantly easier to control and manage the virtualenv in which we run Galaxy, since we don't have to worry about egg-related logic that we don't control and we know that the virtualenv's bin dir will be earlier in the PATH than the system pip's dir. Yes, I said above that our setup is presently using system python -- switching to a virtualenv is one of the many items on my to-do list. In fact, I would expect that the default, desired setup for Galaxy would be to put it in a virtualenv, rather than using system python, and to use pip, rather than fetch_eggs.py and company.
A virtualenv is indeed the strongly preferred setup as I mentioned above. However, Galaxy does not install its eggs to the virtualenv. The virtualenv is there to avoid conflicts with things in the default python's site-packages/dist-packages. Galaxy's eggs are installed to (by default) the `eggs/` directory in the Galaxy source. However, the problem, as I now see it, is that you are trying to install all of Galaxy's dependencies, even at their correct versions, using pip, rather than letting Galaxy handle its eggs as it does. This is not going to work, Galaxy is going to insist on using its eggs.
When I look at the logic being used here, especially at https://bitbucket.org/galaxy/galaxy-central/src/f0ae870b22e9/lib/galaxy/eggs... , it looks like a solution built exclusively for platforms on which pip is not installed. According to the wiki, Galaxy support only goes back as far as 2.6, and get-pip.py supports 2.6, so there is no way of building a Galaxy server on a platform that can't also have pip installed. Adding pip to the requirements for Galaxy would not be particularly onerous, and may simplify things significantly (no need to bundle get-pip.py or similar).
As mentioned above, we don't use or depend on pip, so it's not required. That said, a lot of our egg fetching logic could likely be replaced with pip (this code predates pip by a few years). And the eggs could probably be replaced with prebuilt wheels. However, even if we did use pip/wheels, we'd need to install them from our own repository, and it'd still require modifications for binary platform incompatibilities. The egg handling code we have now works pretty reliably, so I am not sure there is a whole lot to be gained by changing it until Python finally figures out how to handle binary compatibility properly.
As a last note about the misleading error from the fetch_eggs script, telling me that WebError is "NotFetchable". I probably wouldn't have much of a complaint about this if the error had been more on target and I hadn't felt the need to do things like patch lib/galaxy/eggs to print a stacktrace. For example, if the script detected that there was a source installed package which was getting underfoot, it should have alerted me or even suggested installing my various packages in egg format.
This is a bug - it is supposed to explain that there is a version conflict. If you have a stack trace, please send it along.
That kind of error detection is hard to maintain and hard to keep accurate since the Galaxy team's priority is to build Galaxy, not a package manager. This, I suppose, circles back to an earlier question: why isn't Galaxy using a python package manager to... manage its packages?
At the very least something along the lines of `./scripts/common_setup.sh --use-pip` should be added. It doesn't seem like it would be that hard to implement -- but I feel like my lack of knowledge of Galaxy disqualifies me from building the changeset reliably. Of course, if no one objects, I will readily do so anyway (when I have the time) as a proof of concept.
There are modifications to a few of the dependencies, such as psycopg2. When psycopg2 is scrambled, the scrambling process fetches and compiles a bit of PostgreSQL's libpq and then statically links to it to provide a standalone egg that does not depend on the user having installed libpq on their system. So a pip-based system that allowed building from source would need to account for this. --nate
Barring even that improvement, the Wiki page at https://wiki.galaxyproject.org/Admin/Config/Eggs should definitely be updated to include some note on why Galaxy has this complex logic for fetching eggs instead of using pip.
A quick tl;dr and summary: - I don't have extensive experience with Galaxy, so I may not know what I'm talking about. - fetch_eggs.py can be made to raise EggNotFetchable by doing a pip install from a VCS without using `--egg`. This is a bug in fetch_eggs.py - All supported platforms for Galaxy support pip - Galaxy should have an option to use pip to download its packages over https from eggs.g2.bx.psu.edu - Galaxy should probably default to using pip when it's available, since its failure modes are significantly better than a home-brewed package manager -- this also leads to good behavior in virtualenvs.
Best regards, and many thanks for your time and attention, -Stephen ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Connetti gratis il mondo con la nuova indoona: hai la chat, le chiamate, le video chiamate e persino le chiamate di gruppo. E chiami gratis anche i numeri fissi e mobili nel mondo! Scarica subito l’app Vai su https://www.indoona.com/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Yes, the TravisCI build passed again. Thanks a lot Dannon! Should this commit go also to stable branch? Nicola ---- Dannon Baker ha scritto ----
I believe I've fixed this in 16583:5b2f1eab2819, if you want to give it another shot. External version of pkg_resources (and not ours in lib/) is what was causing the weird egg fetching errors. Newer versions of pkg_resources create a mangled distribution string for some eggs with nonstandard version identifiers.
On Thu Feb 12 2015 at 6:44:29 AM Nicola Soranzo <nsoranzo@tiscali.it> wrote:
Hi Stephen and Nate, it seems that the problem with the WebError egg is now hitting also BioBlend TravisCI testing:
https://travis-ci.org/galaxyproject/bioblend/builds/50391748
Both for Python 2.6 and 2.7, scripts/common_startup.sh fails with:
"WebError 0.8a couldn't be downloaded automatically. You can try building it by hand with:
python scripts/scramble.py -c config/galaxy.ini.sample -e WebError Fetch failed."
Please note that this has been working fine for 6 months (last TravisCI build 19 days ago) and is still working on my laptops.
Nicola
Il 08.02.2015 04:35 Stephen Rosen ha scritto:
It took me a couple of extra days to get the time to test this, sorry about the delay.
The initial run:
WebError 0.8a couldn't be downloaded automatically. You can try building it by hand with: python scripts/scramble.py -e WebError Fetch failed.
When I patch a traceback.print_exc() into fetch_eggs.py , this is what I see:
Some eggs are out of date, attempting to fetch... Warning: sqlalchemy (a dependent egg of sqlalchemy-migrate) cannot be fetched Traceback (most recent call last): File "./scripts/fetch_eggs.py", line 36, in
c.resolve() # Only fetch eggs required by the config File "/opt/galaxy/lib/galaxy/eggs/__init__.py", line 349, in resolve raise EggNotFetchable( missing ) EggNotFetchable:
WebError 0.8a couldn't be downloaded automatically. You can try building it by hand with: python scripts/scramble.py -e WebError Fetch failed.
If I look into lib/galaxy/eggs , that line is here: https://bitbucket.org/galaxycloud/galaxy/src/e7f887cdf229aaf3b44977ecbf24130...
In the current galaxy source, that is here: https://bitbucket.org/galaxy/galaxy-central/src/1b09e60f84db859ae49bed71c6f3...
As a bit of an aside, I really do have to install this other package in the same python that galaxy uses, since it's one of the major parts of our fork -- not part of our custom tools.
I wouldn't consider this a normal use case of course, but I'll bet that many people will install galaxy with system python, which can lead to similar circumstances.
Please let me know if I can provide any other useful information on what I'm seeing.
Again, thanks for looking into this with me,
-Stephen
On Thu, Feb 5, 2015 at 8:19 AM, Nate Coraor <nate@bx.psu.edu> wrote:
Hi Stephen,
On Wed, Feb 4, 2015 at 8:30 PM, Stephen Rosen <sirosen@uchicago.edu> wrote:
Hi Nate,
Thanks so much for helping me out with this.
It seems that I've miscommunicated what I'm doing a little bit.
I'm installing a package from that git repo using pip separately from running the galaxy setup script.
That is, I want to install a python package and I want to setup galaxy as two separate (and ideally independent) steps in my script.
I have no desire to do anything particularly esoteric or clever with Galaxy's eggs or to install them myself.
Somehow, doing a pip install of my desired package is breaking the later run of fetch_eggs.py
If I understand you correctly, fetch_eggs.py can hit conflicts with site-packages, but is misreporting them as EggNotFetchable.
I am aware that we're using a fork of the Galaxy source ( https://bitbucket.org/galaxycloud/galaxy/src ), so it may be that we have an outdated of fetch_eggs.py which carries a bug that has since been patched.
Sorry for not mentioning it earlier -- it slipped my mind that we're probably trailing behind the modern Galaxy default head.
If I put in the extra legwork now to wrap Galaxy in a virtualenv, this issue will presumably disappear, but that assumes that I don't want to add any packages to the python used for the uwsgi process.
You can add additional packages to this virtualenv, although there could be problems if you install different versions of the same packages that Galaxy has as dependencies. For what it's worth, you should not need to install additional packages in Galaxy's virtualenv, and tools can be configured to use a different python or virtualenv than the one in which Galaxy is started.
I'll have to ask folks on the Globus Genomics team whether or not that is needed -- I certainly hope not, since it means that there is some packaging conflict we need to resolve.
The only reason I haven't done so already is time constraint -- I've been trying to get this install scripting done as quickly as possible without compromising anything truly needful.
All the same, I'll take some time tomorrow to provision a fresh server and generate the stacktrace for you.
I'll also take some time to test with the latest Galaxy source, to see if I get different behavior.
In the best case, this bug no longer exists for the modern source, but in the worst case it bears the attention.
Thanks for the help with debugging.
I don't have any sense of the reliability of fetch_eggs.py and company -- my experience has been of a particular bug that presented on my first contact with these scripts, so naturally I have a bias against them.
That said, I think the Python community may have finally settled on a package manager, even if we can't seem to agree on a package format or tooling surrounding it.
That's just an opinion though -- I have no blog post from Guido to back it up.
I hope we'll see the necessary improvements within the next year, but I think anything we do now will essentially reinvent the wheel (no pun intended) with regards to what will (hopefully) be officially implemented to fix the problem.
--nate
I was not aware of the UCS2 vs. UCS4 issue -- thanks very much for the citations, very helpful in understanding the problem space.
Thanks, -Stephen
On Wed, Feb 4, 2015 at 5:41 PM, Nate Coraor <nate@bx.psu.edu> wrote:
Hi Stephen,
I'll try to reply as in-depth as I can.
On Wed, Feb 4, 2015 at 1:41 PM, Stephen Rosen <sirosen@uchicago.edu> wrote:
Hi Galaxy Dev,
I've been looking at the setup scripts for Galaxy to try to understand a problem I recently had provisioning a Galaxy server.
I will readily admit that I have not read all of the relevant code top-to-bottom, but I have at least skimmed all of it and read much of it.
Sorry if these questions are answered somewhere in Trello, the Wiki, or somewhere else, but I was not able to find answers in any public locations.
As a small bit of probably irrelevant context: I'm working with the Globus Genomics group on the DevOps side of things. We're using Chef. I've only just started working with the group in the past couple of weeks (so my expertise with Galaxy itself is limited to nonexistent).
First, to describe the problem:
We want to provision a server running Galaxy without explicitly wrapping it in a virtualenv. Unless I missed something, that means that it's using system python.
When we use pip to install a package from a git repository before running the setup scripts, fetch_eggs fails saying it failed to fetch WebError 0.8a
If we install the same package from git with `pip install --egg ...` we get a hunky-dory system where everything seems to work.
A virtualenv is itself just a wrapper around whichever python binary was used to create it. I'd still suggest using a virtualenv created with the system python unless you have a really strong reason not to. In fact, I'm working on Galaxy process management and a command line tool for that management that will automatically create and use a virtualenv going forward.
I'm a bit confused at what's happening here - you mention installing a package from a git repository with pip, but then refer to Galaxy's fetch_eggs(.py) script, which doesn't use pip or git.
As far as I can tell, there is no reason that this should be the case.
Sure, putting the git source directly into site-packages might cause issues upon installation, but EggNotFetchable exceptions should only be thrown if the egg actually can't be pulled down from eggs.g2.bx.psu.edu , right?
EggNotFetchable can be thrown if you happen to be using a platform for which we do not provide eggs, although those are fairly uncommon. Right now we should cover x86/x86_64 Linux and any flavor of Intel OS X after 10.5.
I don't feel comfortable trying to make further progress on my provisioning scripts without knowing why this is happening. I'd hate to be bitten by this later on in the process.
Yes, the package in question may have poor behavior (likely it does), but that doesn't change the fact that the error is totally misleading. Furthermore, it doesn't appear that this poor behavior impedes me from doing a pip install of the WebError package or any other packages from PyPI.
In case someone else wants to test to replicate, this is the command being used:
pip install --egg git+https://github.com/globusonline/python-nexus-client@599f04edef6b72569b7a5b27...
problems occur if you omit `--egg`.
None of this process is using Galaxy's egg handling, so I am not sure where the EggNotFetchable is coming from. What command are you running to get an EggNotFetchable error.
Second, a question about the rationale for Galaxy's egg handling:
Why is all of this wrapped up in these scripts in the first place?
I understand that pip might not be present on every platform, and I don't mean to question a decision to support systems without it.
However, as detailed below, Galaxy does not support any platforms which are incapable of running pip.
This isn't the case - Galaxy does not use pip to install the framework dependencies at all. Some tool dependencies installed from the Tool Shed do use pip, but that's entirely separate from the dependencies of the Galaxy application.
The `scripts/scramble.py` script can be used to automatically build eggs on platforms which we do not prebuild eggs for. If this is necessary, `scripts/fetch_eggs.py` should tell you.
Furthermore, pip is being pushed by the Python maintainers over easy_install, so it's not like there isn't a clear choice in terms of which one to support.
Perhaps most importantly, there don't appear to be any clear-cut options to do the following, which I would consider a more ordinary workflow:
- Run a galaxy script (like check_eggs) to generate a list of packages from eggs.g2.bx.psu.edu for platform (redirect output to requirements.txt or similar)
- `pip install -r requirements.txt`
This is exactly what `scripts/check_eggs.py` and `scripts/fetch_eggs.py` do.
There are 3 reasons for the way we handle eggs in Galaxy:
1. Galaxy has a huge (and ever-growing) list of dependent python modules with C extensions. If we did not prebuild and distribute eggs for these, the initial setup to get Galaxy running would be long and problematic. Some people who download Galaxy to develop tools may not even have compilers installed, let alone the multitude of -dev or -devel packages that aren't part of a default Debian or RHEL installation that would be required to build all of these packages from source. One of the things that I feel makes Galaxy so accessible is that you can start using it immediately after you clone the source. So that ability to clone and start and have it work as reliably (and quickly) as possible is a high priority.
2. Galaxy started using eggs in 2005 or 2006. At this time, everything used distutils. pkg_resources came around, which soon brought setuptools and easy_install. After this came distutils2, pip and finally, these days, wheels. Our need for binary dependency packaging predated almost all of these (in fact, most packages in these days didn't even install .egg-info, which was the only reliable way to know what version of a module you were using) and as each new iteration of packaging/management came along it was never clear that any of them had "won" (and in fact, most of them lost). On top of this, the Python packaging folks have known for years that Python's platform detection for binary compatibility is broken[1]. While I was assured it'd be fixed soon, even with a complete reimplementation of Python packaging (wheels), they still haven't even made an effort to fix this problem[2]. In fact, binary wheels for Linux are explicitly not allowed on PyPI because of this.
3. We tightly control the versions of all of our dependencies, which is not always possible with pip if you aren't also controlling the source of your packages.
[1] https://mail.python.org/pipermail/distutils-sig/2010-January/015345.html
[2] http://lucumr.pocoo.org/2014/1/27/python-on-wheels/
The above could be part of a galaxy provisioning script, rather than exposed to the administrator.
That also makes it significantly easier to control and manage the virtualenv in which we run Galaxy, since we don't have to worry about egg-related logic that we don't control and we know that the virtualenv's bin dir will be earlier in the PATH than the system pip's dir.
Yes, I said above that our setup is presently using system python -- switching to a virtualenv is one of the many items on my to-do list.
In fact, I would expect that the default, desired setup for Galaxy would be to put it in a virtualenv, rather than using system python, and to use pip, rather than fetch_eggs.py and company.
A virtualenv is indeed the strongly preferred setup as I mentioned above. However, Galaxy does not install its eggs to the virtualenv. The virtualenv is there to avoid conflicts with things in the default python's site-packages/dist-packages. Galaxy's eggs are installed to (by default) the `eggs/` directory in the Galaxy source.
However, the problem, as I now see it, is that you are trying to install all of Galaxy's dependencies, even at their correct versions, using pip, rather than letting Galaxy handle its eggs as it does. This is not going to work, Galaxy is going to insist on using its eggs.
When I look at the logic being used here, especially at https://bitbucket.org/galaxy/galaxy-central/src/f0ae870b22e9/lib/galaxy/eggs... , it looks like a solution built exclusively for platforms on which pip is not installed.
According to the wiki, Galaxy support only goes back as far as 2.6, and get-pip.py supports 2.6, so there is no way of building a Galaxy server on a platform that can't also have pip installed.
Adding pip to the requirements for Galaxy would not be particularly onerous, and may simplify things significantly (no need to bundle get-pip.py or similar). As mentioned above, we don't use or depend on pip, so it's not required. That said, a lot of our egg fetching logic could likely be replaced with pip (this code predates pip by a few years). And the eggs could probably be replaced with prebuilt wheels. However, even if we did use pip/wheels, we'd need to install them from our own repository, and it'd still require modifications for binary platform incompatibilities. The egg handling code we have now works pretty reliably, so I am not sure there is a whole lot to be gained by changing it until Python finally figures out how to handle binary compatibility properly.
As a last note about the misleading error from the fetch_eggs script, telling me that WebError is "NotFetchable".
I probably wouldn't have much of a complaint about this if the error had been more on target and I hadn't felt the need to do things like patch lib/galaxy/eggs to print a stacktrace.
For example, if the script detected that there was a source installed package which was getting underfoot, it should have alerted me or even suggested installing my various packages in egg format.
This is a bug - it is supposed to explain that there is a version conflict. If you have a stack trace, please send it along.
That kind of error detection is hard to maintain and hard to keep accurate since the Galaxy team's priority is to build Galaxy, not a package manager.
This, I suppose, circles back to an earlier question: why isn't Galaxy using a python package manager to... manage its packages?
At the very least something along the lines of `./scripts/common_setup.sh --use-pip` should be added.
It doesn't seem like it would be that hard to implement -- but I feel like my lack of knowledge of Galaxy disqualifies me from building the changeset reliably.
Of course, if no one objects, I will readily do so anyway (when I have the time) as a proof of concept.
There are modifications to a few of the dependencies, such as psycopg2. When psycopg2 is scrambled, the scrambling process fetches and compiles a bit of PostgreSQL's libpq and then statically links to it to provide a standalone egg that does not depend on the user having installed libpq on their system. So a pip-based system that allowed building from source would need to account for this.
--nate
Barring even that improvement, the Wiki page at https://wiki.galaxyproject.org/Admin/Config/Eggs should definitely be updated to include some note on why Galaxy has this complex logic for fetching eggs instead of using pip.
A quick tl;dr and summary:
- I don't have extensive experience with Galaxy, so I may not know what I'm talking about.
- fetch_eggs.py can be made to raise EggNotFetchable by doing a pip install from a VCS without using `--egg`. This is a bug in fetch_eggs.py
- All supported platforms for Galaxy support pip
- Galaxy should have an option to use pip to download its packages over https from eggs.g2.bx.psu.edu
- Galaxy should probably default to using pip when it's available, since its failure modes are significantly better than a home-brewed package manager -- this also leads to good behavior in virtualenvs.
Best regards, and many thanks for your time and attention,
-Stephen
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Connetti gratis il mondo con la nuova indoona: hai la chat, le chiamate, le video chiamate e persino le chiamate di gruppo. E chiami gratis anche i numeri fissi e mobili nel mondo! Scarica subito l’app Vai su https://www.indoona.com/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
---- Dannon Baker ha scritto ----
I believe I've fixed this in 16583:5b2f1eab2819, if you want to give it another shot. External version of pkg_resources (and not ours in lib/) is what was causing the weird egg fetching errors. Newer versions of pkg_resources create a mangled distribution string for some eggs with nonstandard version identifiers.
On Thu Feb 12 2015 at 6:44:29 AM Nicola Soranzo <nsoranzo@tiscali.it> wrote:
Hi Stephen and Nate, it seems that the problem with the WebError egg is now hitting also BioBlend TravisCI testing:
https://travis-ci.org/galaxyproject/bioblend/builds/50391748
Both for Python 2.6 and 2.7, scripts/common_startup.sh fails with:
"WebError 0.8a couldn't be downloaded automatically. You can try building it by hand with:
python scripts/scramble.py -c config/galaxy.ini.sample -e WebError Fetch failed."
Please note that this has been working fine for 6 months (last TravisCI build 19 days ago) and is still working on my laptops.
Nicola
Il 08.02.2015 04:35 Stephen Rosen ha scritto:
It took me a couple of extra days to get the time to test this, sorry about the delay.
The initial run:
WebError 0.8a couldn't be downloaded automatically. You can try building it by hand with: python scripts/scramble.py -e WebError Fetch failed.
When I patch a traceback.print_exc() into fetch_eggs.py , this is what I see:
Some eggs are out of date, attempting to fetch... Warning: sqlalchemy (a dependent egg of sqlalchemy-migrate) cannot be fetched Traceback (most recent call last): File "./scripts/fetch_eggs.py", line 36, in
c.resolve() # Only fetch eggs required by the config File "/opt/galaxy/lib/galaxy/eggs/__init__.py", line 349, in resolve raise EggNotFetchable( missing ) EggNotFetchable:
WebError 0.8a couldn't be downloaded automatically. You can try building it by hand with: python scripts/scramble.py -e WebError Fetch failed.
If I look into lib/galaxy/eggs , that line is here: https://bitbucket.org/galaxycloud/galaxy/src/e7f887cdf229aaf3b44977ecbf24130...
In the current galaxy source, that is here: https://bitbucket.org/galaxy/galaxy-central/src/1b09e60f84db859ae49bed71c6f3...
As a bit of an aside, I really do have to install this other package in the same python that galaxy uses, since it's one of the major parts of our fork -- not part of our custom tools.
I wouldn't consider this a normal use case of course, but I'll bet that many people will install galaxy with system python, which can lead to similar circumstances.
Please let me know if I can provide any other useful information on what I'm seeing.
Again, thanks for looking into this with me,
-Stephen
On Thu, Feb 5, 2015 at 8:19 AM, Nate Coraor <nate@bx.psu.edu> wrote:
Hi Stephen,
On Wed, Feb 4, 2015 at 8:30 PM, Stephen Rosen <sirosen@uchicago.edu> wrote:
Hi Nate,
Thanks so much for helping me out with this.
It seems that I've miscommunicated what I'm doing a little bit.
I'm installing a package from that git repo using pip separately from running the galaxy setup script.
That is, I want to install a python package and I want to setup galaxy as two separate (and ideally independent) steps in my script.
I have no desire to do anything particularly esoteric or clever with Galaxy's eggs or to install them myself.
Somehow, doing a pip install of my desired package is breaking the later run of fetch_eggs.py
If I understand you correctly, fetch_eggs.py can hit conflicts with site-packages, but is misreporting them as EggNotFetchable.
I am aware that we're using a fork of the Galaxy source ( https://bitbucket.org/galaxycloud/galaxy/src ), so it may be that we have an outdated of fetch_eggs.py which carries a bug that has since been patched.
Sorry for not mentioning it earlier -- it slipped my mind that we're probably trailing behind the modern Galaxy default head.
If I put in the extra legwork now to wrap Galaxy in a virtualenv, this issue will presumably disappear, but that assumes that I don't want to add any packages to the python used for the uwsgi process.
You can add additional packages to this virtualenv, although there could be problems if you install different versions of the same packages that Galaxy has as dependencies. For what it's worth, you should not need to install additional packages in Galaxy's virtualenv, and tools can be configured to use a different python or virtualenv than the one in which Galaxy is started.
I'll have to ask folks on the Globus Genomics team whether or not that is needed -- I certainly hope not, since it means that there is some packaging conflict we need to resolve.
The only reason I haven't done so already is time constraint -- I've been trying to get this install scripting done as quickly as possible without compromising anything truly needful.
All the same, I'll take some time tomorrow to provision a fresh server and generate the stacktrace for you.
I'll also take some time to test with the latest Galaxy source, to see if I get different behavior.
In the best case, this bug no longer exists for the modern source, but in the worst case it bears the attention.
Thanks for the help with debugging.
I don't have any sense of the reliability of fetch_eggs.py and company -- my experience has been of a particular bug that presented on my first contact with these scripts, so naturally I have a bias against them.
That said, I think the Python community may have finally settled on a package manager, even if we can't seem to agree on a package format or tooling surrounding it.
That's just an opinion though -- I have no blog post from Guido to back it up.
I hope we'll see the necessary improvements within the next year, but I think anything we do now will essentially reinvent the wheel (no pun intended) with regards to what will (hopefully) be officially implemented to fix the problem.
--nate
I was not aware of the UCS2 vs. UCS4 issue -- thanks very much for the citations, very helpful in understanding the problem space.
Thanks, -Stephen
On Wed, Feb 4, 2015 at 5:41 PM, Nate Coraor <nate@bx.psu.edu> wrote:
Hi Stephen,
I'll try to reply as in-depth as I can.
On Wed, Feb 4, 2015 at 1:41 PM, Stephen Rosen <sirosen@uchicago.edu> wrote:
Hi Galaxy Dev,
I've been looking at the setup scripts for Galaxy to try to understand a problem I recently had provisioning a Galaxy server.
I will readily admit that I have not read all of the relevant code top-to-bottom, but I have at least skimmed all of it and read much of it.
Sorry if these questions are answered somewhere in Trello, the Wiki, or somewhere else, but I was not able to find answers in any public locations.
As a small bit of probably irrelevant context: I'm working with the Globus Genomics group on the DevOps side of things. We're using Chef. I've only just started working with the group in the past couple of weeks (so my expertise with Galaxy itself is limited to nonexistent).
First, to describe the problem:
We want to provision a server running Galaxy without explicitly wrapping it in a virtualenv. Unless I missed something, that means that it's using system python.
When we use pip to install a package from a git repository before running the setup scripts, fetch_eggs fails saying it failed to fetch WebError 0.8a
If we install the same package from git with `pip install --egg ...` we get a hunky-dory system where everything seems to work.
A virtualenv is itself just a wrapper around whichever python binary was used to create it. I'd still suggest using a virtualenv created with the system python unless you have a really strong reason not to. In fact, I'm working on Galaxy process management and a command line tool for that management that will automatically create and use a virtualenv going forward.
I'm a bit confused at what's happening here - you mention installing a package from a git repository with pip, but then refer to Galaxy's fetch_eggs(.py) script, which doesn't use pip or git.
As far as I can tell, there is no reason that this should be the case.
Sure, putting the git source directly into site-packages might cause issues upon installation, but EggNotFetchable exceptions should only be thrown if the egg actually can't be pulled down from eggs.g2.bx.psu.edu , right?
EggNotFetchable can be thrown if you happen to be using a platform for which we do not provide eggs, although those are fairly uncommon. Right now we should cover x86/x86_64 Linux and any flavor of Intel OS X after 10.5.
I don't feel comfortable trying to make further progress on my provisioning scripts without knowing why this is happening. I'd hate to be bitten by this later on in the process.
Yes, the package in question may have poor behavior (likely it does), but that doesn't change the fact that the error is totally misleading. Furthermore, it doesn't appear that this poor behavior impedes me from doing a pip install of the WebError package or any other packages from PyPI.
In case someone else wants to test to replicate, this is the command being used:
pip install --egg git+https://github.com/globusonline/python-nexus-client@599f04edef6b72569b7a5b27...
problems occur if you omit `--egg`.
None of this process is using Galaxy's egg handling, so I am not sure where the EggNotFetchable is coming from. What command are you running to get an EggNotFetchable error.
Second, a question about the rationale for Galaxy's egg handling:
Why is all of this wrapped up in these scripts in the first place?
I understand that pip might not be present on every platform, and I don't mean to question a decision to support systems without it.
However, as detailed below, Galaxy does not support any platforms which are incapable of running pip.
This isn't the case - Galaxy does not use pip to install the framework dependencies at all. Some tool dependencies installed from the Tool Shed do use pip, but that's entirely separate from the dependencies of the Galaxy application.
The `scripts/scramble.py` script can be used to automatically build eggs on platforms which we do not prebuild eggs for. If this is necessary, `scripts/fetch_eggs.py` should tell you.
Furthermore, pip is being pushed by the Python maintainers over easy_install, so it's not like there isn't a clear choice in terms of which one to support.
Perhaps most importantly, there don't appear to be any clear-cut options to do the following, which I would consider a more ordinary workflow:
- Run a galaxy script (like check_eggs) to generate a list of packages from eggs.g2.bx.psu.edu for platform (redirect output to requirements.txt or similar)
- `pip install -r requirements.txt`
This is exactly what `scripts/check_eggs.py` and `scripts/fetch_eggs.py` do.
There are 3 reasons for the way we handle eggs in Galaxy:
1. Galaxy has a huge (and ever-growing) list of dependent python modules with C extensions. If we did not prebuild and distribute eggs for these, the initial setup to get Galaxy running would be long and problematic. Some people who download Galaxy to develop tools may not even have compilers installed, let alone the multitude of -dev or -devel packages that aren't part of a default Debian or RHEL installation that would be required to build all of these packages from source. One of the things that I feel makes Galaxy so accessible is that you can start using it immediately after you clone the source. So that ability to clone and start and have it work as reliably (and quickly) as possible is a high priority.
2. Galaxy started using eggs in 2005 or 2006. At this time, everything used distutils. pkg_resources came around, which soon brought setuptools and easy_install. After this came distutils2, pip and finally, these days, wheels. Our need for binary dependency packaging predated almost all of these (in fact, most packages in these days didn't even install .egg-info, which was the only reliable way to know what version of a module you were using) and as each new iteration of packaging/management came along it was never clear that any of them had "won" (and in fact, most of them lost). On top of this, the Python packaging folks have known for years that Python's platform detection for binary compatibility is broken[1]. While I was assured it'd be fixed soon, even with a complete reimplementation of Python packaging (wheels), they still haven't even made an effort to fix this problem[2]. In fact, binary wheels for Linux are explicitly not allowed on PyPI because of this.
3. We tightly control the versions of all of our dependencies, which is not always possible with pip if you aren't also controlling the source of your packages.
[1] https://mail.python.org/pipermail/distutils-sig/2010-January/015345.html
[2] http://lucumr.pocoo.org/2014/1/27/python-on-wheels/
The above could be part of a galaxy provisioning script, rather than exposed to the administrator.
That also makes it significantly easier to control and manage the virtualenv in which we run Galaxy, since we don't have to worry about egg-related logic that we don't control and we know that the virtualenv's bin dir will be earlier in the PATH than the system pip's dir.
Yes, I said above that our setup is presently using system python -- switching to a virtualenv is one of the many items on my to-do list.
In fact, I would expect that the default, desired setup for Galaxy would be to put it in a virtualenv, rather than using system python, and to use pip, rather than fetch_eggs.py and company.
A virtualenv is indeed the strongly preferred setup as I mentioned above. However, Galaxy does not install its eggs to the virtualenv. The virtualenv is there to avoid conflicts with things in the default python's site-packages/dist-packages. Galaxy's eggs are installed to (by default) the `eggs/` directory in the Galaxy source.
However, the problem, as I now see it, is that you are trying to install all of Galaxy's dependencies, even at their correct versions, using pip, rather than letting Galaxy handle its eggs as it does. This is not going to work, Galaxy is going to insist on using its eggs.
When I look at the logic being used here, especially at https://bitbucket.org/galaxy/galaxy-central/src/f0ae870b22e9/lib/galaxy/eggs... , it looks like a solution built exclusively for platforms on which pip is not installed.
According to the wiki, Galaxy support only goes back as far as 2.6, and get-pip.py supports 2.6, so there is no way of building a Galaxy server on a platform that can't also have pip installed.
Adding pip to the requirements for Galaxy would not be particularly onerous, and may simplify things significantly (no need to bundle get-pip.py or similar).
As mentioned above, we don't use or depend on pip, so it's not required. That said, a lot of our egg fetching logic could likely be replaced with pip (this code predates pip by a few years). And the eggs could probably be replaced with prebuilt wheels. However, even if we did use pip/wheels, we'd need to install them from our own repository, and it'd still require modifications for binary platform incompatibilities. The egg handling code we have now works pretty reliably, so I am not sure there is a whole lot to be gained by changing it until Python finally figures out how to handle binary compatibility properly.
As a last note about the misleading error from the fetch_eggs script, telling me that WebError is "NotFetchable".
I probably wouldn't have much of a complaint about this if the error had been more on target and I hadn't felt the need to do things like patch lib/galaxy/eggs to print a stacktrace.
For example, if the script detected that there was a source installed package which was getting underfoot, it should have alerted me or even suggested installing my various packages in egg format.
This is a bug - it is supposed to explain that there is a version conflict. If you have a stack trace, please send it along.
That kind of error detection is hard to maintain and hard to keep accurate since the Galaxy team's priority is to build Galaxy, not a package manager.
This, I suppose, circles back to an earlier question: why isn't Galaxy using a python package manager to... manage its packages?
At the very least something along the lines of `./scripts/common_setup.sh --use-pip` should be added.
It doesn't seem like it would be that hard to implement -- but I feel like my lack of knowledge of Galaxy disqualifies me from building the changeset reliably.
Of course, if no one objects, I will readily do so anyway (when I have the time) as a proof of concept.
There are modifications to a few of the dependencies, such as psycopg2. When psycopg2 is scrambled, the scrambling process fetches and compiles a bit of PostgreSQL's libpq and then statically links to it to provide a standalone egg that does not depend on the user having installed libpq on their system. So a pip-based system that allowed building from source would need to account for this.
--nate
Barring even that improvement, the Wiki page at https://wiki.galaxyproject.org/Admin/Config/Eggs should definitely be updated to include some note on why Galaxy has this complex logic for fetching eggs instead of using pip.
A quick tl;dr and summary:
- I don't have extensive experience with Galaxy, so I may not know what I'm talking about.
- fetch_eggs.py can be made to raise EggNotFetchable by doing a pip install from a VCS without using `--egg`. This is a bug in fetch_eggs.py
- All supported platforms for Galaxy support pip
- Galaxy should have an option to use pip to download its packages over https from eggs.g2.bx.psu.edu
- Galaxy should probably default to using pip when it's available, since its failure modes are significantly better than a home-brewed package manager -- this also leads to good behavior in virtualenvs.
Best regards, and many thanks for your time and attention,
-Stephen
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Connetti gratis il mondo con la nuova indoona: hai la chat, le chiamate, le video chiamate e persino le chiamate di gruppo. E chiami gratis anche i numeri fissi e mobili nel mondo! Scarica subito l’app Vai su https://www.indoona.com/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Glad it passed! And, yes, it should. I'd planned on getting it over there today. On Thu, Feb 12, 2015 at 10:59 AM, Nicola Soranzo <nsoranzo@tiscali.it> wrote:
Yes, the TravisCI build passed again. Thanks a lot Dannon!
Should this commit go also to stable branch? Nicola
---- Dannon Baker ha scritto ----
I believe I've fixed this in 16583:5b2f1eab2819, if you want to give it another shot. External version of pkg_resources (and not ours in lib/) is what was causing the weird egg fetching errors. Newer versions of pkg_resources create a mangled distribution string for some eggs with nonstandard version identifiers.
On Thu Feb 12 2015 at 6:44:29 AM Nicola Soranzo <nsoranzo@tiscali.it> wrote:
Hi Stephen and Nate, it seems that the problem with the WebError egg is now hitting also BioBlend TravisCI testing:
https://travis-ci.org/galaxyproject/bioblend/builds/50391748
Both for Python 2.6 and 2.7, scripts/common_startup.sh fails with:
"WebError 0.8a couldn't be downloaded automatically. You can try building it by hand with:
python scripts/scramble.py -c config/galaxy.ini.sample -e WebError Fetch failed."
Please note that this has been working fine for 6 months (last TravisCI build 19 days ago) and is still working on my laptops.
Nicola
Il 08.02.2015 04:35 Stephen Rosen ha scritto:
It took me a couple of extra days to get the time to test this, sorry about the delay.
The initial run:
WebError 0.8a couldn't be downloaded automatically. You can try building it by hand with: python scripts/scramble.py -e WebError Fetch failed.
When I patch a traceback.print_exc() into fetch_eggs.py , this is what I see:
Some eggs are out of date, attempting to fetch... Warning: sqlalchemy (a dependent egg of sqlalchemy-migrate) cannot be fetched Traceback (most recent call last): File "./scripts/fetch_eggs.py", line 36, in
c.resolve() # Only fetch eggs required by the config File "/opt/galaxy/lib/galaxy/eggs/__init__.py", line 349, in resolve raise EggNotFetchable( missing ) EggNotFetchable:
WebError 0.8a couldn't be downloaded automatically. You can try building it by hand with: python scripts/scramble.py -e WebError Fetch failed.
If I look into lib/galaxy/eggs , that line is here:
https://bitbucket.org/galaxycloud/galaxy/src/e7f887cdf229aaf3b44977ecbf24130...
In the current galaxy source, that is here:
https://bitbucket.org/galaxy/galaxy-central/src/1b09e60f84db859ae49bed71c6f3...
As a bit of an aside, I really do have to install this other package in
the same python that galaxy uses, since it's one of the major parts of our fork -- not part of our custom tools.
I wouldn't consider this a normal use case of course, but I'll bet that
many people will install galaxy with system python, which can lead to similar circumstances.
Please let me know if I can provide any other useful information on what
I'm seeing.
Again, thanks for looking into this with me,
-Stephen
On Thu, Feb 5, 2015 at 8:19 AM, Nate Coraor <nate@bx.psu.edu> wrote:
Hi Stephen,
On Wed, Feb 4, 2015 at 8:30 PM, Stephen Rosen <sirosen@uchicago.edu>
wrote:
Hi Nate,
Thanks so much for helping me out with this.
It seems that I've miscommunicated what I'm doing a little bit.
I'm installing a package from that git repo using pip separately from
running the galaxy setup script.
That is, I want to install a python package and I want to setup galaxy
as two separate (and ideally independent) steps in my script.
I have no desire to do anything particularly esoteric or clever with
Galaxy's eggs or to install them myself.
Somehow, doing a pip install of my desired package is breaking the later
run of fetch_eggs.py
If I understand you correctly, fetch_eggs.py can hit conflicts with
site-packages, but is misreporting them as EggNotFetchable.
I am aware that we're using a fork of the Galaxy source (
https://bitbucket.org/galaxycloud/galaxy/src ), so it may be that we have an outdated of fetch_eggs.py which carries a bug that has since been patched.
Sorry for not mentioning it earlier -- it slipped my mind that we're
probably trailing behind the modern Galaxy default head.
If I put in the extra legwork now to wrap Galaxy in a virtualenv, this
issue will presumably disappear, but that assumes that I don't want to add any packages to the python used for the uwsgi process.
You can add additional packages to this virtualenv, although there could
be problems if you install different versions of the same packages that Galaxy has as dependencies. For what it's worth, you should not need to install additional packages in Galaxy's virtualenv, and tools can be configured to use a different python or virtualenv than the one in which Galaxy is started.
I'll have to ask folks on the Globus Genomics team whether or not that
is needed -- I certainly hope not, since it means that there is some packaging conflict we need to resolve.
The only reason I haven't done so already is time constraint -- I've
been trying to get this install scripting done as quickly as possible without compromising anything truly needful.
All the same, I'll take some time tomorrow to provision a fresh server
and generate the stacktrace for you.
I'll also take some time to test with the latest Galaxy source, to see
if I get different behavior.
In the best case, this bug no longer exists for the modern source, but
in the worst case it bears the attention.
Thanks for the help with debugging.
I don't have any sense of the reliability of fetch_eggs.py and company
-- my experience has been of a particular bug that presented on my first contact with these scripts, so naturally I have a bias against them.
That said, I think the Python community may have finally settled on a
package manager, even if we can't seem to agree on a package format or tooling surrounding it.
That's just an opinion though -- I have no blog post from Guido to back
it up.
I hope we'll see the necessary improvements within the next year, but I
think anything we do now will essentially reinvent the wheel (no pun intended) with regards to what will (hopefully) be officially implemented to fix the problem.
--nate
I was not aware of the UCS2 vs. UCS4 issue -- thanks very much for the
citations, very helpful in understanding the problem space.
Thanks, -Stephen
On Wed, Feb 4, 2015 at 5:41 PM, Nate Coraor <nate@bx.psu.edu> wrote:
Hi Stephen,
I'll try to reply as in-depth as I can.
On Wed, Feb 4, 2015 at 1:41 PM, Stephen Rosen <sirosen@uchicago.edu>
wrote:
Hi Galaxy Dev,
I've been looking at the setup scripts for Galaxy to try to understand a
problem I recently had provisioning a Galaxy server.
I will readily admit that I have not read all of the relevant code
top-to-bottom, but I have at least skimmed all of it and read much of it.
Sorry if these questions are answered somewhere in Trello, the Wiki, or
somewhere else, but I was not able to find answers in any public locations.
As a small bit of probably irrelevant context: I'm working with the Globus Genomics group on the DevOps side of things. We're using Chef. I've only just started working with the group in the past couple of
weeks (so my expertise with Galaxy itself is limited to nonexistent).
First, to describe the problem:
We want to provision a server running Galaxy without explicitly wrapping
Unless I missed something, that means that it's using system python.
When we use pip to install a package from a git repository before running the setup scripts, fetch_eggs fails saying it failed to fetch WebError 0.8a
If we install the same package from git with `pip install --egg ...` we get a hunky-dory system where everything seems to work.
A virtualenv is itself just a wrapper around whichever python binary was used to create it. I'd still suggest using a virtualenv created with the system python unless you have a really strong reason not to. In fact, I'm working on Galaxy process management and a command line tool for that management that will automatically create and use a virtualenv going forward.
I'm a bit confused at what's happening here - you mention installing a
it in a virtualenv. package from a git repository with pip, but then refer to Galaxy's fetch_eggs(.py) script, which doesn't use pip or git.
As far as I can tell, there is no reason that this should be the case.
Sure, putting the git source directly into site-packages might cause
issues upon installation, but EggNotFetchable exceptions should only be thrown if the egg actually can't be pulled down from eggs.g2.bx.psu.edu , right?
EggNotFetchable can be thrown if you happen to be using a platform for
which we do not provide eggs, although those are fairly uncommon. Right now we should cover x86/x86_64 Linux and any flavor of Intel OS X after 10.5.
I don't feel comfortable trying to make further progress on my
I'd hate to be bitten by this later on in the process.
Yes, the package in question may have poor behavior (likely it does), but that doesn't change the fact that the error is totally misleading. Furthermore, it doesn't appear that this poor behavior impedes me from doing a pip install of the WebError package or any other packages from PyPI.
In case someone else wants to test to replicate, this is the command being used:
pip install --egg git+ https://github.com/globusonline/python-nexus-client@599f04edef6b72569b7a5b27...
problems occur if you omit `--egg`.
None of this process is using Galaxy's egg handling, so I am not sure where the EggNotFetchable is coming from. What command are you running to get an EggNotFetchable error.
Second, a question about the rationale for Galaxy's egg handling:
Why is all of this wrapped up in these scripts in the first place?
I understand that pip might not be present on every platform, and I don't mean to question a decision to support systems without it.
However, as detailed below, Galaxy does not support any platforms which are incapable of running pip.
This isn't the case - Galaxy does not use pip to install the framework dependencies at all. Some tool dependencies installed from the Tool Shed do use pip, but that's entirely separate from the dependencies of the Galaxy application.
The `scripts/scramble.py` script can be used to automatically build eggs on platforms which we do not prebuild eggs for. If this is necessary, `scripts/fetch_eggs.py` should tell you.
Furthermore, pip is being pushed by the Python maintainers over easy_install, so it's not like there isn't a clear choice in terms of which one to support.
Perhaps most importantly, there don't appear to be any clear-cut options to do the following, which I would consider a more ordinary workflow:
- Run a galaxy script (like check_eggs) to generate a list of packages from eggs.g2.bx.psu.edu for platform (redirect output to requirements.txt or similar)
- `pip install -r requirements.txt`
This is exactly what `scripts/check_eggs.py` and `scripts/fetch_eggs.py` do.
There are 3 reasons for the way we handle eggs in Galaxy:
1. Galaxy has a huge (and ever-growing) list of dependent python modules with C extensions. If we did not prebuild and distribute eggs for these,
provisioning scripts without knowing why this is happening. the initial setup to get Galaxy running would be long and problematic. Some people who download Galaxy to develop tools may not even have compilers installed, let alone the multitude of -dev or -devel packages that aren't part of a default Debian or RHEL installation that would be required to build all of these packages from source. One of the things that I feel makes Galaxy so accessible is that you can start using it immediately after you clone the source. So that ability to clone and start and have it work as reliably (and quickly) as possible is a high priority.
2. Galaxy started using eggs in 2005 or 2006. At this time, everything
used distutils. pkg_resources came around, which soon brought setuptools and easy_install. After this came distutils2, pip and finally, these days, wheels. Our need for binary dependency packaging predated almost all of these (in fact, most packages in these days didn't even install .egg-info, which was the only reliable way to know what version of a module you were using) and as each new iteration of packaging/management came along it was never clear that any of them had "won" (and in fact, most of them lost). On top of this, the Python packaging folks have known for years that Python's platform detection for binary compatibility is broken[1]. While I was assured it'd be fixed soon, even with a complete reimplementation of Python packaging (wheels), they still haven't even made an effort to fix this problem[2]. In fact, binary wheels for Linux are explicitly not allowed on PyPI because of this.
3. We tightly control the versions of all of our dependencies, which is
not always possible with pip if you aren't also controlling the source of your packages.
[1]
https://mail.python.org/pipermail/distutils-sig/2010-January/015345.html
[2] http://lucumr.pocoo.org/2014/1/27/python-on-wheels/
The above could be part of a galaxy provisioning script, rather than
exposed to the administrator.
That also makes it significantly easier to control and manage the
virtualenv in which we run Galaxy, since we don't have to worry about egg-related logic that we don't control and we know that the virtualenv's bin dir will be earlier in the PATH than the system pip's dir.
Yes, I said above that our setup is presently using system python --
switching to a virtualenv is one of the many items on my to-do list.
In fact, I would expect that the default, desired setup for Galaxy would
be to put it in a virtualenv, rather than using system python, and to use pip, rather than fetch_eggs.py and company.
A virtualenv is indeed the strongly preferred setup as I mentioned
above. However, Galaxy does not install its eggs to the virtualenv. The virtualenv is there to avoid conflicts with things in the default python's site-packages/dist-packages. Galaxy's eggs are installed to (by default) the `eggs/` directory in the Galaxy source.
However, the problem, as I now see it, is that you are trying to install
all of Galaxy's dependencies, even at their correct versions, using pip, rather than letting Galaxy handle its eggs as it does. This is not going to work, Galaxy is going to insist on using its eggs.
When I look at the logic being used here, especially at
https://bitbucket.org/galaxy/galaxy-central/src/f0ae870b22e9/lib/galaxy/eggs... , it looks like a solution built exclusively for platforms on which pip is not installed.
According to the wiki, Galaxy support only goes back as far as 2.6, and
get-pip.py supports 2.6, so there is no way of building a Galaxy server on a platform that can't also have pip installed.
Adding pip to the requirements for Galaxy would not be particularly
onerous, and may simplify things significantly (no need to bundle get-pip.py or similar). As mentioned above, we don't use or depend on pip, so it's not required. That said, a lot of our egg fetching logic could likely be replaced with pip (this code predates pip by a few years). And the eggs could probably be replaced with prebuilt wheels. However, even if we did use pip/wheels, we'd need to install them from our own repository, and it'd still require modifications for binary platform incompatibilities. The egg handling code we have now works pretty reliably, so I am not sure there is a whole lot to be gained by changing it until Python finally figures out how to handle binary compatibility properly.
As a last note about the misleading error from the fetch_eggs script,
telling me that WebError is "NotFetchable".
I probably wouldn't have much of a complaint about this if the error had
been more on target and I hadn't felt the need to do things like patch lib/galaxy/eggs to print a stacktrace.
For example, if the script detected that there was a source installed
package which was getting underfoot, it should have alerted me or even suggested installing my various packages in egg format.
This is a bug - it is supposed to explain that there is a version
conflict. If you have a stack trace, please send it along.
That kind of error detection is hard to maintain and hard to keep
accurate since the Galaxy team's priority is to build Galaxy, not a package manager.
This, I suppose, circles back to an earlier question: why isn't Galaxy
using a python package manager to... manage its packages?
At the very least something along the lines of
`./scripts/common_setup.sh --use-pip` should be added.
It doesn't seem like it would be that hard to implement -- but I feel
like my lack of knowledge of Galaxy disqualifies me from building the changeset reliably.
Of course, if no one objects, I will readily do so anyway (when I have
the time) as a proof of concept.
There are modifications to a few of the dependencies, such as psycopg2.
When psycopg2 is scrambled, the scrambling process fetches and compiles a bit of PostgreSQL's libpq and then statically links to it to provide a standalone egg that does not depend on the user having installed libpq on their system. So a pip-based system that allowed building from source would need to account for this.
--nate
Barring even that improvement, the Wiki page at
https://wiki.galaxyproject.org/Admin/Config/Eggs should definitely be updated to include some note on why Galaxy has this complex logic for fetching eggs instead of using pip.
A quick tl;dr and summary:
- I don't have extensive experience with Galaxy, so I may not know what
I'm talking about.
- fetch_eggs.py can be made to raise EggNotFetchable by doing a pip
install from a VCS without using `--egg`. This is a bug in fetch_eggs.py
- All supported platforms for Galaxy support pip
- Galaxy should have an option to use pip to download its packages over
https from eggs.g2.bx.psu.edu
- Galaxy should probably default to using pip when it's available, since
its failure modes are significantly better than a home-brewed package manager -- this also leads to good behavior in virtualenvs.
Best regards, and many thanks for your time and attention,
-Stephen
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Connetti gratis il mondo con la nuova indoona: hai la chat, le chiamate,
le video chiamate e persino le chiamate di gruppo.
E chiami gratis anche i numeri fissi e mobili nel mondo! Scarica subito l’app Vai su https://www.indoona.com/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
---- Dannon Baker ha scritto ----
I believe I've fixed this in 16583:5b2f1eab2819, if you want to give it another shot. External version of pkg_resources (and not ours in lib/) is what was causing the weird egg fetching errors. Newer versions of pkg_resources create a mangled distribution string for some eggs with nonstandard version identifiers.
On Thu Feb 12 2015 at 6:44:29 AM Nicola Soranzo <nsoranzo@tiscali.it> wrote:
Hi Stephen and Nate, it seems that the problem with the WebError egg is now hitting also BioBlend TravisCI testing:
https://travis-ci.org/galaxyproject/bioblend/builds/50391748
Both for Python 2.6 and 2.7, scripts/common_startup.sh fails with:
"WebError 0.8a couldn't be downloaded automatically. You can try building it by hand with:
python scripts/scramble.py -c config/galaxy.ini.sample -e WebError Fetch failed."
Please note that this has been working fine for 6 months (last TravisCI build 19 days ago) and is still working on my laptops.
Nicola
Il 08.02.2015 04:35 Stephen Rosen ha scritto:
It took me a couple of extra days to get the time to test this, sorry about the delay.
The initial run:
WebError 0.8a couldn't be downloaded automatically. You can try building it by hand with: python scripts/scramble.py -e WebError Fetch failed.
When I patch a traceback.print_exc() into fetch_eggs.py , this is what I see:
Some eggs are out of date, attempting to fetch... Warning: sqlalchemy (a dependent egg of sqlalchemy-migrate) cannot be fetched Traceback (most recent call last): File "./scripts/fetch_eggs.py", line 36, in
c.resolve() # Only fetch eggs required by the config File "/opt/galaxy/lib/galaxy/eggs/__init__.py", line 349, in resolve raise EggNotFetchable( missing ) EggNotFetchable:
WebError 0.8a couldn't be downloaded automatically. You can try building it by hand with: python scripts/scramble.py -e WebError Fetch failed.
If I look into lib/galaxy/eggs , that line is here:
https://bitbucket.org/galaxycloud/galaxy/src/e7f887cdf229aaf3b44977ecbf24130... In the current galaxy source, that is here:
https://bitbucket.org/galaxy/galaxy-central/src/1b09e60f84db859ae49bed71c6f3...
As a bit of an aside, I really do have to install this other package in the same python that galaxy uses, since it's one of the major parts of our fork -- not part of our custom tools. I wouldn't consider this a normal use case of course, but I'll bet that many people will install galaxy with system python, which can lead to similar circumstances.
Please let me know if I can provide any other useful information on what I'm seeing.
Again, thanks for looking into this with me, -Stephen
On Thu, Feb 5, 2015 at 8:19 AM, Nate Coraor <nate@bx.psu.edu> wrote:
Hi Stephen,
On Wed, Feb 4, 2015 at 8:30 PM, Stephen Rosen <sirosen@uchicago.edu> wrote:
Hi Nate,
Thanks so much for helping me out with this.
It seems that I've miscommunicated what I'm doing a little bit. I'm installing a package from that git repo using pip separately from running the galaxy setup script. That is, I want to install a python package and I want to setup galaxy as two separate (and ideally independent) steps in my script. I have no desire to do anything particularly esoteric or clever with Galaxy's eggs or to install them myself. Somehow, doing a pip install of my desired package is breaking the later run of fetch_eggs.py
If I understand you correctly, fetch_eggs.py can hit conflicts with site-packages, but is misreporting them as EggNotFetchable. I am aware that we're using a fork of the Galaxy source ( https://bitbucket.org/galaxycloud/galaxy/src ), so it may be that we have an outdated of fetch_eggs.py which carries a bug that has since been patched. Sorry for not mentioning it earlier -- it slipped my mind that we're probably trailing behind the modern Galaxy default head.
If I put in the extra legwork now to wrap Galaxy in a virtualenv, this issue will presumably disappear, but that assumes that I don't want to add any packages to the python used for the uwsgi process.
You can add additional packages to this virtualenv, although there could be problems if you install different versions of the same packages that Galaxy has as dependencies. For what it's worth, you should not need to install additional packages in Galaxy's virtualenv, and tools can be configured to use a different python or virtualenv than the one in which Galaxy is started.
I'll have to ask folks on the Globus Genomics team whether or not that is needed -- I certainly hope not, since it means that there is some packaging conflict we need to resolve. The only reason I haven't done so already is time constraint -- I've been trying to get this install scripting done as quickly as possible without compromising anything truly needful.
All the same, I'll take some time tomorrow to provision a fresh server and generate the stacktrace for you. I'll also take some time to test with the latest Galaxy source, to see if I get different behavior. In the best case, this bug no longer exists for the modern source, but in the worst case it bears the attention.
Thanks for the help with debugging.
I don't have any sense of the reliability of fetch_eggs.py and company -- my experience has been of a particular bug that presented on my first contact with these scripts, so naturally I have a bias against them. That said, I think the Python community may have finally settled on a package manager, even if we can't seem to agree on a package format or tooling surrounding it. That's just an opinion though -- I have no blog post from Guido to back it up.
I hope we'll see the necessary improvements within the next year, but I think anything we do now will essentially reinvent the wheel (no pun intended) with regards to what will (hopefully) be officially implemented to fix the problem. --nate
I was not aware of the UCS2 vs. UCS4 issue -- thanks very much for the citations, very helpful in understanding the problem space. Thanks, -Stephen
On Wed, Feb 4, 2015 at 5:41 PM, Nate Coraor <nate@bx.psu.edu> wrote:
Hi Stephen, I'll try to reply as in-depth as I can.
On Wed, Feb 4, 2015 at 1:41 PM, Stephen Rosen <sirosen@uchicago.edu> wrote:
Hi Galaxy Dev,
I've been looking at the setup scripts for Galaxy to try to understand a problem I recently had provisioning a Galaxy server. I will readily admit that I have not read all of the relevant code top-to-bottom, but I have at least skimmed all of it and read much of it. Sorry if these questions are answered somewhere in Trello, the Wiki, or somewhere else, but I was not able to find answers in any public locations.
As a small bit of probably irrelevant context: I'm working with the Globus Genomics group on the DevOps side of things. We're using Chef. I've only just started working with the group in the past couple of weeks (so my expertise with Galaxy itself is limited to nonexistent).
First, to describe the problem:
We want to provision a server running Galaxy without explicitly wrapping it in a virtualenv. Unless I missed something, that means that it's using system python. When we use pip to install a package from a git repository before running the setup scripts, fetch_eggs fails saying it failed to fetch WebError 0.8a If we install the same package from git with `pip install --egg ...` we get a hunky-dory system where everything seems to work.
A virtualenv is itself just a wrapper around whichever python binary was used to create it. I'd still suggest using a virtualenv created with the system python unless you have a really strong reason not to. In fact, I'm working on Galaxy process management and a command line tool for that management that will automatically create and use a virtualenv going forward. I'm a bit confused at what's happening here - you mention installing a package from a git repository with pip, but then refer to Galaxy's fetch_eggs(.py) script, which doesn't use pip or git.
As far as I can tell, there is no reason that this should be the case. Sure, putting the git source directly into site-packages might cause issues upon installation, but EggNotFetchable exceptions should only be thrown if the egg actually can't be pulled down from eggs.g2.bx.psu.edu , right?
EggNotFetchable can be thrown if you happen to be using a platform for which we do not provide eggs, although those are fairly uncommon. Right now we should cover x86/x86_64 Linux and any flavor of Intel OS X after 10.5.
I don't feel comfortable trying to make further progress on my provisioning scripts without knowing why this is happening. I'd hate to be bitten by this later on in the process.
Yes, the package in question may have poor behavior (likely it does), but that doesn't change the fact that the error is totally misleading. Furthermore, it doesn't appear that this poor behavior impedes me from doing a pip install of the WebError package or any other packages from PyPI.
In case someone else wants to test to replicate, this is the command being used: pip install --egg git+ https://github.com/globusonline/python-nexus-client@599f04edef6b72569b7a5b27... problems occur if you omit `--egg`.
None of this process is using Galaxy's egg handling, so I am not sure where the EggNotFetchable is coming from. What command are you running to get an EggNotFetchable error.
Second, a question about the rationale for Galaxy's egg handling:
Why is all of this wrapped up in these scripts in the first place? I understand that pip might not be present on every platform, and I don't mean to question a decision to support systems without it. However, as detailed below, Galaxy does not support any platforms which are incapable of running pip.
This isn't the case - Galaxy does not use pip to install the framework dependencies at all. Some tool dependencies installed from the Tool Shed do use pip, but that's entirely separate from the dependencies of the Galaxy application. The `scripts/scramble.py` script can be used to automatically build eggs on platforms which we do not prebuild eggs for. If this is necessary, `scripts/fetch_eggs.py` should tell you.
Furthermore, pip is being pushed by the Python maintainers over easy_install, so it's not like there isn't a clear choice in terms of which one to support. Perhaps most importantly, there don't appear to be any clear-cut options to do the following, which I would consider a more ordinary workflow:
- Run a galaxy script (like check_eggs) to generate a list of packages from eggs.g2.bx.psu.edu for platform (redirect output to requirements.txt or similar) - `pip install -r requirements.txt`
This is exactly what `scripts/check_eggs.py` and `scripts/fetch_eggs.py` do. There are 3 reasons for the way we handle eggs in Galaxy: 1. Galaxy has a huge (and ever-growing) list of dependent python modules with C extensions. If we did not prebuild and distribute eggs for these, the initial setup to get Galaxy running would be long and problematic. Some people who download Galaxy to develop tools may not even have compilers installed, let alone the multitude of -dev or -devel packages that aren't part of a default Debian or RHEL installation that would be required to build all of these packages from source. One of the things that I feel makes Galaxy so accessible is that you can start using it immediately after you clone the source. So that ability to clone and start and have it work as reliably (and quickly) as possible is a high priority. 2. Galaxy started using eggs in 2005 or 2006. At this time, everything used distutils. pkg_resources came around, which soon brought setuptools and easy_install. After this came distutils2, pip and finally, these days, wheels. Our need for binary dependency packaging predated almost all of these (in fact, most packages in these days didn't even install .egg-info, which was the only reliable way to know what version of a module you were using) and as each new iteration of packaging/management came along it was never clear that any of them had "won" (and in fact, most of them lost). On top of this, the Python packaging folks have known for years that Python's platform detection for binary compatibility is broken[1]. While I was assured it'd be fixed soon, even with a complete reimplementation of Python packaging (wheels), they still haven't even made an effort to fix this problem[2]. In fact, binary wheels for Linux are explicitly not allowed on PyPI because of this.
3. We tightly control the versions of all of our dependencies, which is not always possible with pip if you aren't also controlling the source of your packages. [1] https://mail.python.org/pipermail/distutils-sig/2010-January/015345.html [2] http://lucumr.pocoo.org/2014/1/27/python-on-wheels/
The above could be part of a galaxy provisioning script, rather than exposed to the administrator. That also makes it significantly easier to control and manage the virtualenv in which we run Galaxy, since we don't have to worry about egg-related logic that we don't control and we know that the virtualenv's bin dir will be earlier in the PATH than the system pip's dir. Yes, I said above that our setup is presently using system python -- switching to a virtualenv is one of the many items on my to-do list. In fact, I would expect that the default, desired setup for Galaxy would be to put it in a virtualenv, rather than using system python, and to use pip, rather than fetch_eggs.py and company.
A virtualenv is indeed the strongly preferred setup as I mentioned above. However, Galaxy does not install its eggs to the virtualenv. The virtualenv is there to avoid conflicts with things in the default python's site-packages/dist-packages. Galaxy's eggs are installed to (by default) the `eggs/` directory in the Galaxy source. However, the problem, as I now see it, is that you are trying to install all of Galaxy's dependencies, even at their correct versions, using pip, rather than letting Galaxy handle its eggs as it does. This is not going to work, Galaxy is going to insist on using its eggs.
When I look at the logic being used here, especially at https://bitbucket.org/galaxy/galaxy-central/src/f0ae870b22e9/lib/galaxy/eggs... , it looks like a solution built exclusively for platforms on which pip is not installed. According to the wiki, Galaxy support only goes back as far as 2.6, and get-pip.py supports 2.6, so there is no way of building a Galaxy server on a platform that can't also have pip installed. Adding pip to the requirements for Galaxy would not be particularly onerous, and may simplify things significantly (no need to bundle get-pip.py or similar).
As mentioned above, we don't use or depend on pip, so it's not required. That said, a lot of our egg fetching logic could likely be replaced with pip (this code predates pip by a few years). And the eggs could probably be replaced with prebuilt wheels. However, even if we did use pip/wheels, we'd need to install them from our own repository, and it'd still require modifications for binary platform incompatibilities. The egg handling code we have now works pretty reliably, so I am not sure there is a whole lot to be gained by changing it until Python finally figures out how to handle binary compatibility properly.
As a last note about the misleading error from the fetch_eggs script, telling me that WebError is "NotFetchable". I probably wouldn't have much of a complaint about this if the error had been more on target and I hadn't felt the need to do things like patch lib/galaxy/eggs to print a stacktrace. For example, if the script detected that there was a source installed package which was getting underfoot, it should have alerted me or even suggested installing my various packages in egg format.
This is a bug - it is supposed to explain that there is a version conflict. If you have a stack trace, please send it along.
That kind of error detection is hard to maintain and hard to keep accurate since the Galaxy team's priority is to build Galaxy, not a package manager. This, I suppose, circles back to an earlier question: why isn't Galaxy using a python package manager to... manage its packages?
At the very least something along the lines of `./scripts/common_setup.sh --use-pip` should be added. It doesn't seem like it would be that hard to implement -- but I feel like my lack of knowledge of Galaxy disqualifies me from building the changeset reliably. Of course, if no one objects, I will readily do so anyway (when I have the time) as a proof of concept.
There are modifications to a few of the dependencies, such as psycopg2. When psycopg2 is scrambled, the scrambling process fetches and compiles a bit of PostgreSQL's libpq and then statically links to it to provide a standalone egg that does not depend on the user having installed libpq on their system. So a pip-based system that allowed building from source would need to account for this. --nate
Barring even that improvement, the Wiki page at https://wiki.galaxyproject.org/Admin/Config/Eggs should definitely be updated to include some note on why Galaxy has this complex logic for fetching eggs instead of using pip.
A quick tl;dr and summary: - I don't have extensive experience with Galaxy, so I may not know what I'm talking about. - fetch_eggs.py can be made to raise EggNotFetchable by doing a pip install from a VCS without using `--egg`. This is a bug in fetch_eggs.py - All supported platforms for Galaxy support pip - Galaxy should have an option to use pip to download its packages over https from eggs.g2.bx.psu.edu - Galaxy should probably default to using pip when it's available, since its failure modes are significantly better than a home-brewed package manager -- this also leads to good behavior in virtualenvs.
Best regards, and many thanks for your time and attention, -Stephen ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Connetti gratis il mondo con la nuova indoona: hai la chat, le chiamate, le video chiamate e persino le chiamate di gruppo. E chiami gratis anche i numeri fissi e mobili nel mondo! Scarica subito l’app Vai su https://www.indoona.com/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (5)
-
Dannon Baker -
Martin Čech -
Nate Coraor -
Nicola Soranzo -
Stephen Rosen