Python modules not found when starting with init system
Hi All, when my local galaxy is started using the init system (systemd on Ubuntu 16.04 in this instance). Some tools will fail, indicating that some python module do not exist. This is the result of uploading a 2KB plain text file: Traceback (most recent call last): File "/home/galaxy/galaxy/tools/data_source/upload.py", line 20, in <module> from galaxy.datatypes import sniff File "/home/galaxy/galaxy/lib/galaxy/datatypes/sniff.py", line 31, in <module> from galaxy.datatypes.binary import Binary File "/home/galaxy/galaxy/lib/galaxy/datatypes/binary.py", line 14, in <module> import pysam ImportError: No module named pysam Traceback (most recent call last): File "/home/galaxy/galaxy/database/jobs_directory/000/39/set_metadata_EI05GD.py", line 1, in <module> from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata() File "/home/galaxy/galaxy/lib/galaxy_ext/metadata/set_metadata.py", line 23, in <module> from sqlalchemy.orm import clear_mappers ImportError: No module named sqlalchemy.orm When the same galaxy install is started directly using the 'run.sh' script in the install directory, everything works fine. I've verified that I can manually import pysam and sqlalchemy.orm if I execute the python interpreter located at <install_dir>/.venv/bin/python directly. Any ideas? I've attached the init script. TL;DR Galaxy cannot import some python modules when started by the init system. No idea why. Galaxy version: 17.05 OS: Ubuntu 16.04 Many Thanks Max -------------------------------------------------------------------- Un o’r 4 prifysgol uchaf yn y DU a’r orau yng Nghymru am fodlonrwydd myfyrwyr. (Arolwg Cenedlaethol y Myfyrwyr 2016) www.aber.ac.uk Top 4 UK university and best in Wales for student satisfaction (National Student Survey 2016) www.aber.ac.uk
Hi, i ran into that too (on debian 8). I'm not that familiar with virtual environments but this is what I found out. It looks like the virtual environment doesn't get activated properly in the shipped ini.d-file (it is the shipped one, right?). If I add ". /path/to/.venv/bin/activate" to it, everything runs just fine. "run.sh" has this line. The init.d-file hasn't. It just calls the .venv- python directly. It seems like that's not fully equivalent to activating .venv and running python in there. I don't really know why. I'm guessing PATH-magic. Hope that helps a bit. Gudrun Amedick -- Gudrun Amedick IT-Systemtechnik UNIVERSITÄT ZU LÜBECK IT-Service-Center Ratzeburger Allee 160 23562 Lübeck Tel +49 451 3101 2035 Fax +49 451 3101 2004 amedick@itsc.uni-luebeck.de www.itsc.uni-luebeck.de Am Dienstag, den 27.06.2017, 12:54 +0100 schrieb Maximilian Friedersdorff:
Hi All,
when my local galaxy is started using the init system (systemd on Ubuntu 16.04 in this instance). Some tools will fail, indicating that some python module do not exist. This is the result of uploading a 2KB plain text file:
Traceback (most recent call last): File "/home/galaxy/galaxy/tools/data_source/upload.py", line 20, in <module> from galaxy.datatypes import sniff File "/home/galaxy/galaxy/lib/galaxy/datatypes/sniff.py", line 31, in <module> from galaxy.datatypes.binary import Binary File "/home/galaxy/galaxy/lib/galaxy/datatypes/binary.py", line 14, in <module> import pysam ImportError: No module named pysam Traceback (most recent call last): File "/home/galaxy/galaxy/database/jobs_directory/000/39/set_metadata_EI05GD.py", line 1, in <module> from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata() File "/home/galaxy/galaxy/lib/galaxy_ext/metadata/set_metadata.py", line 23, in <module> from sqlalchemy.orm import clear_mappers ImportError: No module named sqlalchemy.orm
When the same galaxy install is started directly using the 'run.sh' script in the install directory, everything works fine.
I've verified that I can manually import pysam and sqlalchemy.orm if I execute the python interpreter located at <install_dir>/.venv/bin/python directly.
Any ideas?
I've attached the init script.
TL;DR
Galaxy cannot import some python modules when started by the init system. No idea why.
Galaxy version: 17.05 OS: Ubuntu 16.04
Many Thanks
Max
-------------------------------------------------------------------- Un o’r 4 prifysgol uchaf yn y DU a’r orau yng Nghymru am fodlonrwydd myfyrwyr. (Arolwg Cenedlaethol y Myfyrwyr 2016) www.aber.ac.uk
Top 4 UK university and best in Wales for student satisfaction (National Student Survey 2016) www.aber.ac.uk ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
On Tue, Jun 27, 2017 at 03:41:27PM +0200, Gudrun Amedick wrote:
Hi,
i ran into that too (on debian 8). I'm not that familiar with virtual environments but this is what I found out.
It looks like the virtual environment doesn't get activated properly in the shipped ini.d-file (it is the shipped one, right?). If I add ". /path/to/.venv/bin/activate" to it, everything runs just fine. "run.sh" has this line. The init.d-file hasn't. It just calls the .venv- python directly. It seems like that's not fully equivalent to activating .venv and running python in there. I don't really know why. I'm guessing PATH-magic.
Hope that helps a bit.
Gudrun Amedick
Hi, The thing is that running the python interpreter in the virtual environment _should_ take care of that anyway. The provided debian init script even says as much: "A simple way to activate this virtualenv is to use the python interpreter in <GALAXY_DIR>/.venv". In any case, explicitly activating the virtualenv solves the problem. Thanks for the pointer Gudrun. Maybe a dev can chime in here. Many Thanks Max -------------------------------------------------------------------- Un o’r 4 prifysgol uchaf yn y DU a’r orau yng Nghymru am fodlonrwydd myfyrwyr. (Arolwg Cenedlaethol y Myfyrwyr 2016) www.aber.ac.uk Top 4 UK university and best in Wales for student satisfaction (National Student Survey 2016) www.aber.ac.uk
Hi, I know. To my understaunding of virtual environments, this shouldn't make a difference. As I said, I don't really get it. It looks like Python tries to find pysam outside of the virtual environment for some reason, that's why I'm guessing PATH. I'm curious though, and if someone knows why this solution works, I'd be happy to hear it too. Glad I could help. Gudrun Amedick Gudrun Amedick IT-Systemtechnik UNIVERSITÄT ZU LÜBECK IT-Service-Center Ratzeburger Allee 160 23562 Lübeck Tel +49 451 3101 2035 Fax +49 451 3101 2004 amedick@itsc.uni-luebeck.de www.itsc.uni-luebeck.de Am Mittwoch, den 28.06.2017, 10:59 +0100 schrieb Maximilian Friedersdorff:
On Tue, Jun 27, 2017 at 03:41:27PM +0200, Gudrun Amedick wrote:
Hi,
i ran into that too (on debian 8). I'm not that familiar with virtual environments but this is what I found out.
It looks like the virtual environment doesn't get activated properly in the shipped ini.d-file (it is the shipped one, right?). If I add ". /path/to/.venv/bin/activate" to it, everything runs just fine. "run.sh" has this line. The init.d-file hasn't. It just calls the .venv- python directly. It seems like that's not fully equivalent to activating .venv and running python in there. I don't really know why. I'm guessing PATH-magic.
Hope that helps a bit.
Gudrun Amedick
Hi,
The thing is that running the python interpreter in the virtual environment _should_ take care of that anyway. The provided debian init script even says as much: "A simple way to activate this virtualenv is to use the python interpreter in <GALAXY_DIR>/.venv".
In any case, explicitly activating the virtualenv solves the problem. Thanks for the pointer Gudrun.
Maybe a dev can chime in here.
Many Thanks
Max
-------------------------------------------------------------------- Un o’r 4 prifysgol uchaf yn y DU a’r orau yng Nghymru am fodlonrwydd myfyrwyr. (Arolwg Cenedlaethol y Myfyrwyr 2016) www.aber.ac.uk
Top 4 UK university and best in Wales for student satisfaction (National Student Survey 2016) www.aber.ac.uk
Hi, I've just resolved a similar problem running uwsgi under supervisord, which turned out to be a typo setting the path environment variable in the supervisord config file. Most things worked ok but some things produced errors finding pysam and sqlalchemy (since the system python was being used in preference to the galaxy venv), while jobs would mysteriously fail with a 'file not found' error. Took some tracking down... James -- Dr. James Abbott Lead Bioinformatician Bioinformatician Data Science Group Imperial College London ________________________________ From: galaxy-dev <galaxy-dev-bounces@lists.galaxyproject.org> on behalf of Gudrun Amedick <amedick@itsc.uni-luebeck.de> Sent: Wednesday, June 28, 2017 1:49:44 PM To: Maximilian Friedersdorff Cc: galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Python modules not found when starting with init system Hi, I know. To my understaunding of virtual environments, this shouldn't make a difference. As I said, I don't really get it. It looks like Python tries to find pysam outside of the virtual environment for some reason, that's why I'm guessing PATH. I'm curious though, and if someone knows why this solution works, I'd be happy to hear it too. Glad I could help. Gudrun Amedick Gudrun Amedick IT-Systemtechnik UNIVERSITÄT ZU LÜBECK IT-Service-Center Ratzeburger Allee 160 23562 Lübeck Tel +49 451 3101 2035 Fax +49 451 3101 2004 amedick@itsc.uni-luebeck.de www.itsc.uni-luebeck.de<http://www.itsc.uni-luebeck.de> Am Mittwoch, den 28.06.2017, 10:59 +0100 schrieb Maximilian Friedersdorff:
On Tue, Jun 27, 2017 at 03:41:27PM +0200, Gudrun Amedick wrote:
Hi,
i ran into that too (on debian 8). I'm not that familiar with virtual environments but this is what I found out.
It looks like the virtual environment doesn't get activated properly in the shipped ini.d-file (it is the shipped one, right?). If I add ". /path/to/.venv/bin/activate" to it, everything runs just fine. "run.sh" has this line. The init.d-file hasn't. It just calls the .venv- python directly. It seems like that's not fully equivalent to activating .venv and running python in there. I don't really know why. I'm guessing PATH-magic.
Hope that helps a bit.
Gudrun Amedick
Hi,
The thing is that running the python interpreter in the virtual environment _should_ take care of that anyway. The provided debian init script even says as much: "A simple way to activate this virtualenv is to use the python interpreter in <GALAXY_DIR>/.venv".
In any case, explicitly activating the virtualenv solves the problem. Thanks for the pointer Gudrun.
Maybe a dev can chime in here.
Many Thanks
Max
-------------------------------------------------------------------- Un o’r 4 prifysgol uchaf yn y DU a’r orau yng Nghymru am fodlonrwydd myfyrwyr. (Arolwg Cenedlaethol y Myfyrwyr 2016) www.aber.ac.uk<http://www.aber.ac.uk>
Top 4 UK university and best in Wales for student satisfaction (National Student Survey 2016) www.aber.ac.uk<http://www.aber.ac.uk>
participants (3)
-
Abbott, James -
Gudrun Amedick -
Maximilian Friedersdorff