Packaging tools with language specific dependencies
Hi everyone, Is there any standard or commonly used way to package tools that have language specific dependencies. I know that with Python libraries one can use setup_virtualenv and with Java jars the JAVA_JAR_LIB strategy is used. Is there anything equivalent for R, Perl and Ruby libraries? Thanks Renato
Hi Renato, Am 30.07.2014 um 12:21 schrieb Renato Alves:
Hi everyone,
Is there any standard or commonly used way to package tools that have language specific dependencies.
I know that with Python libraries one can use setup_virtualenv and with Java jars the JAVA_JAR_LIB strategy is used. Is there anything equivalent for R, Perl and Ruby libraries?
please have a look at: https://github.com/bgruening/galaxytools/tree/master/test_repositories If you have any questions I'm happy to help you! Bjoern
Thanks Renato
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Björn, Thanks, this is exactly what I was looking for. Cheers, Renato Quoting Björn Grüning on 30-07-2014 12:04:
Hi Renato,
Am 30.07.2014 um 12:21 schrieb Renato Alves:
Hi everyone,
Is there any standard or commonly used way to package tools that have language specific dependencies.
I know that with Python libraries one can use setup_virtualenv and with Java jars the JAVA_JAR_LIB strategy is used. Is there anything equivalent for R, Perl and Ruby libraries?
please have a look at: https://github.com/bgruening/galaxytools/tree/master/test_repositories
If you have any questions I'm happy to help you! Bjoern
Thanks Renato
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi BJörn, We have run into some problems with the perl environment. This might also affect other tools that rely on a shebang with absolute paths to the interpreter. Details below. During the installation of the perl package (we tried package_perl_5_18 from main and testing toolshed), the cpanm script gets installed but when executed it is invalid/not found. The issue seems to be the length of the shebang in the script which is limited to 80 characters. In cpanm it's: !#/galaxy_dist/tool_dependencies/perl/5.18.1/bgruening/package_perl_5_18/e89824189ec6/bin/perl That is 92 characters long. When launching the script it gets truncated at "...perl_5_18/e898241", causing the command to fail with a "bad interpreter" error. I was able to workaround this problem by editing the script manually and replacing the long shebang by: #!/usr/bin/env perl As it stands it seems relying on the shebang to use the correct interpreter is a problem. Repositories with long (character wise) names, versions or owners will be more likely to suffer from this. So will installations that are not close to the system root. Renato Quoting Björn Grüning on 30-07-2014 12:04:
Hi Renato,
Am 30.07.2014 um 12:21 schrieb Renato Alves:
Hi everyone,
Is there any standard or commonly used way to package tools that have language specific dependencies.
I know that with Python libraries one can use setup_virtualenv and with Java jars the JAVA_JAR_LIB strategy is used. Is there anything equivalent for R, Perl and Ruby libraries?
please have a look at: https://github.com/bgruening/galaxytools/tree/master/test_repositories
If you have any questions I'm happy to help you! Bjoern
Thanks Renato
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Renato, cpanm is not the only problem it seems to be an general perl 'feature'. How annoying :( As I figured from: https://metacpan.org/source/RJBS/perl-5.20.0/Configure#L9162 we can use:-Dstartperl='#!/usr/bin/env perl' to change them via install. Now every perl script from the main perl has the correct shebang. Unfortunately, every other package installed with cpanm or Makefile.PL will have this annoying long shebang. I'm not sure there is a switch in cpan/cpanm to do that. Under cpan there are a few repositories, called change_shebang... so I assume we are not the only ones that have this problem. So I ended up with: <action type="shell_command">sed -i 's|#!$INSTALL_DIR/bin/|#!/usr/bin/env |' $INSTALL_DIR/bin/*</action> I have uploaded it to the testtoolshed and to galaxytools. If this is working for you, also with depending repositories, I will try to add this to into the Tool Shed core to make this sed command implicit for all perl tools. Cheers, Bjoern Am 01.08.2014 um 16:06 schrieb Renato Alves:
Hi BJörn,
We have run into some problems with the perl environment. This might also affect other tools that rely on a shebang with absolute paths to the interpreter. Details below.
During the installation of the perl package (we tried package_perl_5_18 from main and testing toolshed), the cpanm script gets installed but when executed it is invalid/not found.
The issue seems to be the length of the shebang in the script which is limited to 80 characters. In cpanm it's:
!#/galaxy_dist/tool_dependencies/perl/5.18.1/bgruening/package_perl_5_18/e89824189ec6/bin/perl
That is 92 characters long. When launching the script it gets truncated at "...perl_5_18/e898241", causing the command to fail with a "bad interpreter" error.
I was able to workaround this problem by editing the script manually and replacing the long shebang by: #!/usr/bin/env perl
As it stands it seems relying on the shebang to use the correct interpreter is a problem. Repositories with long (character wise) names, versions or owners will be more likely to suffer from this. So will installations that are not close to the system root.
Renato
Quoting Björn Grüning on 30-07-2014 12:04:
Hi Renato,
Am 30.07.2014 um 12:21 schrieb Renato Alves:
Hi everyone,
Is there any standard or commonly used way to package tools that have language specific dependencies.
I know that with Python libraries one can use setup_virtualenv and with Java jars the JAVA_JAR_LIB strategy is used. Is there anything equivalent for R, Perl and Ruby libraries?
please have a look at: https://github.com/bgruening/galaxytools/tree/master/test_repositories
If you have any questions I'm happy to help you! Bjoern
Thanks Renato
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Bjoern, Thanks for the info and the fix. So far we had no issues but it's not extensive testing. I'll let you know if it fails for some reason. In addition to the shebang change we also started using "perl script" instead of simply "script". Since "perl" is looked up in PATH, all else works as expected. It does get a bit tricky if you need to guess where the script is located. "$(which script)" works but it's hacky. Still, it's the only way you can "fix" the shebang without changing it before. PS: There's shutil.which() but only python 3.3 onwards. Cheers, Renato Quoting Björn Grüning on 02-08-2014 15:26:
Hi Renato,
cpanm is not the only problem it seems to be an general perl 'feature'. How annoying :(
As I figured from: https://metacpan.org/source/RJBS/perl-5.20.0/Configure#L9162
we can use:-Dstartperl='#!/usr/bin/env perl' to change them via install. Now every perl script from the main perl has the correct shebang.
Unfortunately, every other package installed with cpanm or Makefile.PL will have this annoying long shebang.
I'm not sure there is a switch in cpan/cpanm to do that. Under cpan there are a few repositories, called change_shebang... so I assume we are not the only ones that have this problem.
So I ended up with:
<action type="shell_command">sed -i 's|#!$INSTALL_DIR/bin/|#!/usr/bin/env |' $INSTALL_DIR/bin/*</action>
I have uploaded it to the testtoolshed and to galaxytools. If this is working for you, also with depending repositories, I will try to add this to into the Tool Shed core to make this sed command implicit for all perl tools.
Cheers, Bjoern
Am 01.08.2014 um 16:06 schrieb Renato Alves:
Hi BJörn,
We have run into some problems with the perl environment. This might also affect other tools that rely on a shebang with absolute paths to the interpreter. Details below.
During the installation of the perl package (we tried package_perl_5_18 from main and testing toolshed), the cpanm script gets installed but when executed it is invalid/not found.
The issue seems to be the length of the shebang in the script which is limited to 80 characters. In cpanm it's:
!#/galaxy_dist/tool_dependencies/perl/5.18.1/bgruening/package_perl_5_18/e89824189ec6/bin/perl
That is 92 characters long. When launching the script it gets truncated at "...perl_5_18/e898241", causing the command to fail with a "bad interpreter" error.
I was able to workaround this problem by editing the script manually and replacing the long shebang by: #!/usr/bin/env perl
As it stands it seems relying on the shebang to use the correct interpreter is a problem. Repositories with long (character wise) names, versions or owners will be more likely to suffer from this. So will installations that are not close to the system root.
Renato
Quoting Björn Grüning on 30-07-2014 12:04:
Hi Renato,
Am 30.07.2014 um 12:21 schrieb Renato Alves:
Hi everyone,
Is there any standard or commonly used way to package tools that have language specific dependencies.
I know that with Python libraries one can use setup_virtualenv and with Java jars the JAVA_JAR_LIB strategy is used. Is there anything equivalent for R, Perl and Ruby libraries?
please have a look at: https://github.com/bgruening/galaxytools/tree/master/test_repositories
If you have any questions I'm happy to help you! Bjoern
Thanks Renato
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Renato, Am 05.08.2014 um 17:34 schrieb Renato Alves:
Hi Bjoern,
Thanks for the info and the fix.
So far we had no issues but it's not extensive testing. I'll let you know if it fails for some reason.
In addition to the shebang change we also started using "perl script" instead of simply "script". Since "perl" is looked up in PATH, all else works as expected.
Can you try to explain this a bit more?
It does get a bit tricky if you need to guess where the script is located. "$(which script)" works but it's hacky. Still, it's the only way you can "fix" the shebang without changing it before.
PS: There's shutil.which() but only python 3.3 onwards.
Anything I can do to support you? I'm right that the shebang modifications work for you? Do you think that should go into Galaxy Core, so that every perl script under /bin/ will get a simple shebang? Thanks, Bjoern
Cheers, Renato
Quoting Björn Grüning on 02-08-2014 15:26:
Hi Renato,
cpanm is not the only problem it seems to be an general perl 'feature'. How annoying :(
As I figured from: https://metacpan.org/source/RJBS/perl-5.20.0/Configure#L9162
we can use:-Dstartperl='#!/usr/bin/env perl' to change them via install. Now every perl script from the main perl has the correct shebang.
Unfortunately, every other package installed with cpanm or Makefile.PL will have this annoying long shebang.
I'm not sure there is a switch in cpan/cpanm to do that. Under cpan there are a few repositories, called change_shebang... so I assume we are not the only ones that have this problem.
So I ended up with:
<action type="shell_command">sed -i 's|#!$INSTALL_DIR/bin/|#!/usr/bin/env |' $INSTALL_DIR/bin/*</action>
I have uploaded it to the testtoolshed and to galaxytools. If this is working for you, also with depending repositories, I will try to add this to into the Tool Shed core to make this sed command implicit for all perl tools.
Cheers, Bjoern
Am 01.08.2014 um 16:06 schrieb Renato Alves:
Hi BJörn,
We have run into some problems with the perl environment. This might also affect other tools that rely on a shebang with absolute paths to the interpreter. Details below.
During the installation of the perl package (we tried package_perl_5_18 from main and testing toolshed), the cpanm script gets installed but when executed it is invalid/not found.
The issue seems to be the length of the shebang in the script which is limited to 80 characters. In cpanm it's:
!#/galaxy_dist/tool_dependencies/perl/5.18.1/bgruening/package_perl_5_18/e89824189ec6/bin/perl
That is 92 characters long. When launching the script it gets truncated at "...perl_5_18/e898241", causing the command to fail with a "bad interpreter" error.
I was able to workaround this problem by editing the script manually and replacing the long shebang by: #!/usr/bin/env perl
As it stands it seems relying on the shebang to use the correct interpreter is a problem. Repositories with long (character wise) names, versions or owners will be more likely to suffer from this. So will installations that are not close to the system root.
Renato
Quoting Björn Grüning on 30-07-2014 12:04:
Hi Renato,
Am 30.07.2014 um 12:21 schrieb Renato Alves:
Hi everyone,
Is there any standard or commonly used way to package tools that have language specific dependencies.
I know that with Python libraries one can use setup_virtualenv and with Java jars the JAVA_JAR_LIB strategy is used. Is there anything equivalent for R, Perl and Ruby libraries?
please have a look at: https://github.com/bgruening/galaxytools/tree/master/test_repositories
If you have any questions I'm happy to help you! Bjoern
Thanks Renato
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (2)
-
Björn Grüning -
Renato Alves