galaxy-commits April 2014

galaxy-commits@lists.galaxyproject.org

1 participants
261 discussions

commit/galaxy-central: davebgx: Merge fix from stable.
by commits-noreply＠bitbucket.org 28 Apr '14

28 Apr '14

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/8f6cba970f9e/ Changeset: 8f6cba970f9e User: davebgx Date: 2014-04-28 19:20:00 Summary: Merge fix from stable. Affected #: 1 file diff -r d10b0cdffb0f8dda2a7cb5344ec0152f3e678818 -r 8f6cba970f9ee28477a4e763218d0cce73cb9a21 templates/admin/tool_shed_repository/install_tool_dependencies_with_update.mako --- a/templates/admin/tool_shed_repository/install_tool_dependencies_with_update.mako +++ b/templates/admin/tool_shed_repository/install_tool_dependencies_with_update.mako @@ -53,16 +53,21 @@ %for key, requirements_dict in tool_dependencies_dict.items(): <% readme_text = None - key_items = key.split( '/' ) - key_name = key_items[ 0 ] - key_version = key_items[ 1 ] - readme_text = requirements_dict.get( 'readme', None ) - install_dir = os.path.join( trans.app.config.tool_dependency_dir, - key_name, - key_version, - repository.owner, - repository.name, - repository.installed_changeset_revision ) + if key == 'set_environment': + key_name = ', '.join( [ environment_variable[ 'name' ] for environment_variable in requirements_dict ] ) + key_version = '' + install_dir = '' + else: + key_items = key.split( '/' ) + key_name = key_items[ 0 ] + key_version = key_items[ 1 ] + readme_text = requirements_dict.get( 'readme', None ) + install_dir = os.path.join( trans.app.config.tool_dependency_dir, + key_name, + key_version, + repository.owner, + repository.name, + repository.installed_changeset_revision ) %> %if not os.path.exists( install_dir ): <tr> Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: davebgx: Fix server error when attempting to update a repository that defines set_environment tool dependency types.
by commits-noreply＠bitbucket.org 28 Apr '14

28 Apr '14

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/61b4a993d39c/ Changeset: 61b4a993d39c Branch: stable User: davebgx Date: 2014-04-23 22:24:11 Summary: Fix server error when attempting to update a repository that defines set_environment tool dependency types. Affected #: 1 file diff -r 3b42725359224832317a066d95dff596f93ab33f -r 61b4a993d39c5341afea857859221a4f5b94334f templates/admin/tool_shed_repository/install_tool_dependencies_with_update.mako --- a/templates/admin/tool_shed_repository/install_tool_dependencies_with_update.mako +++ b/templates/admin/tool_shed_repository/install_tool_dependencies_with_update.mako @@ -56,16 +56,21 @@ %for key, requirements_dict in tool_dependencies_dict.items(): <% readme_text = None - key_items = key.split( '/' ) - key_name = key_items[ 0 ] - key_version = key_items[ 1 ] - readme_text = requirements_dict.get( 'readme', None ) - install_dir = os.path.join( trans.app.config.tool_dependency_dir, - key_name, - key_version, - repository.owner, - repository.name, - repository.installed_changeset_revision ) + if key == 'set_environment': + key_name = ', '.join( [ environment_variable[ 'name' ] for environment_variable in requirements_dict ] ) + key_version = '' + install_dir = '' + else: + key_items = key.split( '/' ) + key_name = key_items[ 0 ] + key_version = key_items[ 1 ] + readme_text = requirements_dict.get( 'readme', None ) + install_dir = os.path.join( trans.app.config.tool_dependency_dir, + key_name, + key_version, + repository.owner, + repository.name, + repository.installed_changeset_revision ) %> %if not os.path.exists( install_dir ): <tr> Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: dan: Fix for DatasetListWrapper.
by commits-noreply＠bitbucket.org 28 Apr '14

28 Apr '14

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/d10b0cdffb0f/ Changeset: d10b0cdffb0f User: dan Date: 2014-04-28 19:15:50 Summary: Fix for DatasetListWrapper. Affected #: 1 file diff -r 28b307e0c08791b3073dd27c0189fdd28083045d -r d10b0cdffb0f8dda2a7cb5344ec0152f3e678818 lib/galaxy/tools/wrappers.py --- a/lib/galaxy/tools/wrappers.py +++ b/lib/galaxy/tools/wrappers.py @@ -215,3 +215,5 @@ return DatasetFilenameWrapper( dataset, **wrapper_kwds ) list.__init__( self, map( to_wrapper, datasets ) ) + def __str__( self ): + return ','.join( map( str, self ) ) Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: 4 new changesets
by commits-noreply＠bitbucket.org 28 Apr '14

28 Apr '14

4 new commits in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/9dd03dd46d04/ Changeset: 9dd03dd46d04 User: jmchilton Date: 2014-04-28 18:48:51 Summary: Create run_tests.sh to replace run_functional_tests.sh Should be backward compatible but features better argument handling - things can appear in other orders (-id x -data_managers is the same as -data_managers -id x for instance), better error messages if arguments are missing, and more standard short argument with single dash or long argument with double dash variants of each argument are available. Affected #: 1 file diff -r 4a6cd1da30d0c4e3f8e0ed7b1c1e6342bed20e44 -r 9dd03dd46d04b83aeb270ebe538d6075facb5379 run_tests.sh --- /dev/null +++ b/run_tests.sh @@ -0,0 +1,149 @@ +#!/bin/sh + +# A good place to look for nose info: http://somethingaboutorange.com/mrl/projects/nose/ +rm -f run_functional_tests.log + +show_help() { +cat <<EOF +'${0##*/}' for testing all the tools in functional directory +'${0##*/} aaa' for testing one test case of 'aaa' ('aaa' is the file name with path) +'${0##*/} -id bbb' for testing one tool with id 'bbb' ('bbb' is the tool id) +'${0##*/} -sid ccc' for testing one section with sid 'ccc' ('ccc' is the string after 'section::') +'${0##*/} -list' for listing all the tool ids +'${0##*/} -toolshed' for running all the test scripts in the ./test/tool_shed/functional directory +'${0##*/} -toolshed testscriptname' for running one test script named testscriptname in the .test/tool_shed/functional directory +'${0##*/} -workflow test.xml' for running a workflow test case as defined by supplied workflow xml test file (experimental) +'${0##*/} -framework' for running through example tool tests testing framework features in test/functional/tools" +'${0##*/} -framework -id toolid' for testing one framework tool (in test/functional/tools/) with id 'toolid' +'${0##*/} -data_managers -id data_manager_id' for testing one Data Manager with id 'data_manager_id' +EOF +} + +show_list() { + python tool_list.py + echo "===========================================================================================================================================" + echo "'${0##*/} -id bbb' for testing one tool with id 'bbb' ('bbb' is the tool id)" + echo "'${0##*/} -sid ccc' for testing one section with sid 'ccc' ('ccc' is the string after 'section::')" +} + +test_script="./scripts/functional_tests.py" +report_file="run_functional_tests.html" + +while : +do + case "$1" in + -h|--help|-\?) + show_help + exit 0 + ;; + -l|-list|--list) + show_list + exit 0 + ;; + -id|--id) + if [ $# -gt 1 ]; then + test_id=$2; + shift 2 + else + echo "--id requires an argument" 1>&2 + exit 1 + fi + ;; + -s|-sid|--sid) + if [ $# -gt 1 ]; then + section_id=$2 + shift 2 + else + echo "--sid requires an argument" 1>&2 + exit 1 + fi + ;; + -t|-toolshed|--toolshed) + test_script="./test/tool_shed/functional_tests.py" + report_file="./test/tool_shed/run_functional_tests.html" + if [ $# -gt 1 ]; then + toolshed_script=$2 + shift 2 + else + toolshed_script="./test/tool_shed/functional" + shift 1 + fi + ;; + -w|-workflow|--workflow) + if [ $# -gt 1 ]; then + workflow_file=$2 + workflow_test=1 + shift 2 + else + echo "--workflow requires an argument" 1>&2 + exit 1 + fi + ;; + -f|-framework|--framework) + framework_test=1; + shift 1 + ;; + -d|-data_managers|--data_managers) + data_managers_test=1; + shift 1 + ;; + -m|-migrated|--migrated) + migrated_test=1; + shift + ;; + -i|-installed|--installed) + installed_test=1; + shift + ;; + -r|--report_file) + if [ $# -gt 1 ]; then + report_file=$2 + shift 2 + else + echo "--report_file requires an argument" 1>&2 + exit 1 + fi + ;; + --) + shift + break + ;; + -*) + echo "invalid option: $1" 1>&2; + show_help + exit 1 + ;; + *) + break; + ;; + esac +done + +if [ -n "$migrated_test" ] ; then + [ -n "$test_id" ] && class=":TestForTool_$test_id" || class="" + extra_args="functional.test_toolbox$class -migrated" +elif [ -n "$installed_test" ] ; then + [ -n "$test_id" ] && class=":TestForTool_$test_id" || class="" + extra_args="functional.test_toolbox$class -installed" +elif [ -n "$framework_test" ] ; then + [ -n "$test_id" ] && class=":TestForTool_$test_id" || class="" + extra_args="functional.test_toolbox$class -framework" +elif [ -n "$data_managers_test" ] ; then + [ -n "$test_id" ] && class=":TestForDataManagerTool_$test_id" || class="" + extra_args="functional.test_data_managers$class -data_managers" +elif [ -n "$workflow_test" ]; then + extra_args="functional.workflow:WorkflowTestCase $workflow_file" +elif [ -n "$toolshed_script" ]; then + extra_args="$toolshed_script" +elif [ -n "$section_id" ]; then + extra_args=`python tool_list.py $section_id` +elif [ -n "$test_id" ]; then + class=":TestForTool_$test_id" + extra_args="functional.test_toolbox$class" +elif [ -n "$1" ] ; then + extra_args="$1" +else + extra_args='--exclude="^get" functional' +fi + +python $test_script $coverage_arg -v --with-nosehtml --html-report-file $report_file $extra_args https://bitbucket.org/galaxy/galaxy-central/commits/08754cf82e62/ Changeset: 08754cf82e62 User: jmchilton Date: 2014-04-28 18:48:51 Summary: Add unit tests to run_tests.sh. Affected #: 1 file diff -r 9dd03dd46d04b83aeb270ebe538d6075facb5379 -r 08754cf82e625cb6c6fa88aeb6e4195b222da64b run_tests.sh --- a/run_tests.sh +++ b/run_tests.sh @@ -16,6 +16,8 @@ '${0##*/} -framework' for running through example tool tests testing framework features in test/functional/tools" '${0##*/} -framework -id toolid' for testing one framework tool (in test/functional/tools/) with id 'toolid' '${0##*/} -data_managers -id data_manager_id' for testing one Data Manager with id 'data_manager_id' +'${0##*/} -unit' for running all unit tests (doctests in lib and tests in test/unit) +'${0##*/} -unit testscriptath' running particular tests scripts EOF } @@ -104,6 +106,24 @@ exit 1 fi ;; + -c|--coverage) + # Must have coverage installed (try `which coverage`) - only valid with --unit + # for now. Would be great to get this to work with functional tests though. + coverage_arg="--with-coverage" + NOSE_WITH_COVERAGE=true + shift + ;; + -u|-unit|--unit) + report_file="run_unit_tests.html" + test_script="./scripts/nosetests.py" + if [ $# -gt 1 ]; then + unit_extra=$2 + shift 2 + else + unit_extra='--exclude=functional --exclude="^get" --exclude=controllers --exclude=runners lib test/unit' + shift 1 + fi + ;; --) shift break @@ -140,6 +160,8 @@ elif [ -n "$test_id" ]; then class=":TestForTool_$test_id" extra_args="functional.test_toolbox$class" +elif [ -n "$unit_extra" ]; then + extra_args="--with-doctest $unit_extra" elif [ -n "$1" ] ; then extra_args="$1" else https://bitbucket.org/galaxy/galaxy-central/commits/e8047b8f1e24/ Changeset: e8047b8f1e24 User: jmchilton Date: 2014-04-28 18:48:51 Summary: Add qunit test options to new run_tests.sh script. Can target specific tests or all tests and can optionally watch tests using Carl's work on 'grunt watch'. Affected #: 1 file diff -r 08754cf82e625cb6c6fa88aeb6e4195b222da64b -r e8047b8f1e24a4815138e5862f24096dfc4d9c79 run_tests.sh --- a/run_tests.sh +++ b/run_tests.sh @@ -18,6 +18,8 @@ '${0##*/} -data_managers -id data_manager_id' for testing one Data Manager with id 'data_manager_id' '${0##*/} -unit' for running all unit tests (doctests in lib and tests in test/unit) '${0##*/} -unit testscriptath' running particular tests scripts +'${0##*/} -qunit' for running qunit JavaScript tests +'${0##*/} -qunit testname' for running single JavaScript test with given name EOF } @@ -31,6 +33,8 @@ test_script="./scripts/functional_tests.py" report_file="run_functional_tests.html" +driver="python" + while : do case "$1" in @@ -124,6 +128,24 @@ shift 1 fi ;; + -q|-qunit|--qunit) + # Requires grunt installed and dependencies configured see + # test/qunit/README.txt for more information. + driver="grunt" + gruntfile="./test/qunit/Gruntfile.js" + if [ $# -gt 1 ]; then + qunit_name=$2 + shift 2 + else + shift 1 + fi + ;; + -watch|--watch) + # Have grunt watch test or directory for changes, only + # valid for javascript testing. + watch=1 + shift + ;; --) shift break @@ -168,4 +190,21 @@ extra_args='--exclude="^get" functional' fi -python $test_script $coverage_arg -v --with-nosehtml --html-report-file $report_file $extra_args +if [ "$driver" = "python" ]; then + python $test_script $coverage_arg -v --with-nosehtml --html-report-file $report_file $extra_args +else + if [ -n "$watch" ]; then + grunt_task="watch" + else + grunt_task="" + fi + if [ -n "$qunit_name" ]; then + grunt_args="--test=$qunit_name" + else + grunt_args="" + fi + # TODO: Exapnd javascript helpers to include setting up + # grunt deps in npm, "watch"ing directory, and running casper + # functional tests. + grunt --gruntfile=$gruntfile $grunt_task $grunt_args +fi https://bitbucket.org/galaxy/galaxy-central/commits/28b307e0c087/ Changeset: 28b307e0c087 User: jmchilton Date: 2014-04-28 18:48:51 Summary: Rework test framework testing and tools. -framework will still cause just a small set of non-default, non-user facing tools to be tested and exerise various aspects of the tool and test frameworks, but now an argument can be passed in -with_framework_test_tools to allow these tools to be used as part of the normal test framework (i.e. when not using -framework). This serves two purposes - to ensure the feature coverage of existing tests remains high as the tests are migrated out of the distribution and to allow a set of tests to be available for functional (e.g. API tests) that users don't need to see. Affected #: 15 files diff -r e8047b8f1e24a4815138e5862f24096dfc4d9c79 -r 28b307e0c08791b3073dd27c0189fdd28083045d run_tests.sh --- a/run_tests.sh +++ b/run_tests.sh @@ -32,6 +32,7 @@ test_script="./scripts/functional_tests.py" report_file="run_functional_tests.html" +with_framework_test_tools_arg="" driver="python" @@ -75,6 +76,10 @@ shift 1 fi ;; + -with_framework_test_tools|--with_framework_test_tools) + with_framework_test_tools_arg="-with_framework_test_tools" + shift + ;; -w|-workflow|--workflow) if [ $# -gt 1 ]; then workflow_file=$2 @@ -191,7 +196,7 @@ fi if [ "$driver" = "python" ]; then - python $test_script $coverage_arg -v --with-nosehtml --html-report-file $report_file $extra_args + python $test_script $coverage_arg -v --with-nosehtml --html-report-file $report_file $with_framework_test_tools_arg $extra_args else if [ -n "$watch" ]; then grunt_task="watch" diff -r e8047b8f1e24a4815138e5862f24096dfc4d9c79 -r 28b307e0c08791b3073dd27c0189fdd28083045d scripts/functional_tests.py --- a/scripts/functional_tests.py +++ b/scripts/functional_tests.py @@ -217,16 +217,19 @@ # Exclude all files except test_toolbox.py. ignore_files = ( re.compile( r'^test_[adghlmsu]*' ), re.compile( r'^test_ta*' ) ) else: + framework_tool_dir = os.path.join('test', 'functional', 'tools') framework_test = __check_arg( '-framework' ) # Run through suite of tests testing framework. if framework_test: - framework_tool_dir = os.path.join('test', 'functional', 'tools') tool_conf = os.path.join( framework_tool_dir, 'samples_tool_conf.xml' ) datatypes_conf_override = os.path.join( framework_tool_dir, 'sample_datatypes_conf.xml' ) - test_dir = os.path.join( framework_tool_dir, 'test-data') else: # Use tool_conf.xml toolbox. tool_conf = 'tool_conf.xml' - test_dir = default_galaxy_test_file_dir + if __check_arg( '-with_framework_test_tools' ): + # Some of these tools will not work without swapping + # default interactor to point to test. + tool_conf = "%s,%s" % ( tool_conf, os.path.join( framework_tool_dir, 'samples_tool_conf.xml' ) ) + test_dir = default_galaxy_test_file_dir tool_config_file = os.environ.get( 'GALAXY_TEST_TOOL_CONF', tool_conf ) galaxy_test_file_dir = os.environ.get( 'GALAXY_TEST_FILE_DIR', test_dir ) if not os.path.isabs( galaxy_test_file_dir ): diff -r e8047b8f1e24a4815138e5862f24096dfc4d9c79 -r 28b307e0c08791b3073dd27c0189fdd28083045d test-data/simple_line.txt --- /dev/null +++ b/test-data/simple_line.txt @@ -0,0 +1,1 @@ +This is a line of text. diff -r e8047b8f1e24a4815138e5862f24096dfc4d9c79 -r 28b307e0c08791b3073dd27c0189fdd28083045d test-data/simple_line_alternative.txt --- /dev/null +++ b/test-data/simple_line_alternative.txt @@ -0,0 +1,1 @@ +This is a different line of text. \ No newline at end of file diff -r e8047b8f1e24a4815138e5862f24096dfc4d9c79 -r 28b307e0c08791b3073dd27c0189fdd28083045d test-data/simple_line_x2.txt --- /dev/null +++ b/test-data/simple_line_x2.txt @@ -0,0 +1,2 @@ +This is a line of text. +This is a line of text. diff -r e8047b8f1e24a4815138e5862f24096dfc4d9c79 -r 28b307e0c08791b3073dd27c0189fdd28083045d test-data/simple_line_x3.txt --- /dev/null +++ b/test-data/simple_line_x3.txt @@ -0,0 +1,3 @@ +This is a line of text. +This is a line of text. +This is a line of text. diff -r e8047b8f1e24a4815138e5862f24096dfc4d9c79 -r 28b307e0c08791b3073dd27c0189fdd28083045d test-data/simple_line_x5.txt --- /dev/null +++ b/test-data/simple_line_x5.txt @@ -0,0 +1,5 @@ +This is a line of text. +This is a line of text. +This is a line of text. +This is a line of text. +This is a line of text. diff -r e8047b8f1e24a4815138e5862f24096dfc4d9c79 -r 28b307e0c08791b3073dd27c0189fdd28083045d test-data/simple_lines_interleaved.txt --- /dev/null +++ b/test-data/simple_lines_interleaved.txt @@ -0,0 +1,4 @@ +This is a line of text. +This is a different line of text. +This is a line of text. +This is a different line of text. \ No newline at end of file diff -r e8047b8f1e24a4815138e5862f24096dfc4d9c79 -r 28b307e0c08791b3073dd27c0189fdd28083045d test/functional/tools/test-data/simple_line.txt --- a/test/functional/tools/test-data/simple_line.txt +++ /dev/null @@ -1,1 +0,0 @@ -This is a line of text. diff -r e8047b8f1e24a4815138e5862f24096dfc4d9c79 -r 28b307e0c08791b3073dd27c0189fdd28083045d test/functional/tools/test-data/simple_line_alternative.txt --- a/test/functional/tools/test-data/simple_line_alternative.txt +++ /dev/null @@ -1,1 +0,0 @@ -This is a different line of text. \ No newline at end of file diff -r e8047b8f1e24a4815138e5862f24096dfc4d9c79 -r 28b307e0c08791b3073dd27c0189fdd28083045d test/functional/tools/test-data/simple_line_x2.txt --- a/test/functional/tools/test-data/simple_line_x2.txt +++ /dev/null @@ -1,2 +0,0 @@ -This is a line of text. -This is a line of text. diff -r e8047b8f1e24a4815138e5862f24096dfc4d9c79 -r 28b307e0c08791b3073dd27c0189fdd28083045d test/functional/tools/test-data/simple_line_x3.txt --- a/test/functional/tools/test-data/simple_line_x3.txt +++ /dev/null @@ -1,3 +0,0 @@ -This is a line of text. -This is a line of text. -This is a line of text. diff -r e8047b8f1e24a4815138e5862f24096dfc4d9c79 -r 28b307e0c08791b3073dd27c0189fdd28083045d test/functional/tools/test-data/simple_line_x5.txt --- a/test/functional/tools/test-data/simple_line_x5.txt +++ /dev/null @@ -1,5 +0,0 @@ -This is a line of text. -This is a line of text. -This is a line of text. -This is a line of text. -This is a line of text. diff -r e8047b8f1e24a4815138e5862f24096dfc4d9c79 -r 28b307e0c08791b3073dd27c0189fdd28083045d test/functional/tools/test-data/simple_lines_interleaved.txt --- a/test/functional/tools/test-data/simple_lines_interleaved.txt +++ /dev/null @@ -1,4 +0,0 @@ -This is a line of text. -This is a different line of text. -This is a line of text. -This is a different line of text. \ No newline at end of file diff -r e8047b8f1e24a4815138e5862f24096dfc4d9c79 -r 28b307e0c08791b3073dd27c0189fdd28083045d test/functional/tools/test-data/velveth_test1 --- a/test/functional/tools/test-data/velveth_test1 +++ /dev/null @@ -1,1 +0,0 @@ -../../../../test-data/velveth_test1 \ No newline at end of file Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: davebgx: Twill test framework cleanup. Update install and test methods to reflect changes in base twilltestcase.py.
by commits-noreply＠bitbucket.org 28 Apr '14

28 Apr '14

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/4a6cd1da30d0/ Changeset: 4a6cd1da30d0 User: davebgx Date: 2014-04-28 17:22:17 Summary: Twill test framework cleanup. Update install and test methods to reflect changes in base twilltestcase.py. Affected #: 2 files diff -r fe0df2e688999aa7e0087cbb95c0a7257e9ae6d2 -r 4a6cd1da30d0c4e3f8e0ed7b1c1e6342bed20e44 test/base/twilltestcase.py --- a/test/base/twilltestcase.py +++ b/test/base/twilltestcase.py @@ -1769,10 +1769,11 @@ tc.submit( "edit_external_service_button" ) for check_str in strings_displayed_after_submit: self.check_page_for_string( check_str ) + # Sample tracking stuff def check_request_grid( self, cntrller, state, deleted=False, strings_displayed=[] ): - self.visit_url( '%s/%s/browse_requests?sort=create_time&f-state=%s&f-deleted=%s' % \ - ( self.url, cntrller, state.replace( ' ', '+' ), str( deleted ) ) ) + params = { 'f-state': state, 'f-deleted': deleted, 'sort': 'create_time' } + self.visit_url( '/%s/browse_requests' % cntrller ) for check_str in strings_displayed: self.check_page_for_string( check_str ) def create_request_type( self, name, desc, request_form_id, sample_form_id, states, strings_displayed=[], strings_displayed_after_submit=[] ): @@ -1967,8 +1968,8 @@ for check_str in strings_displayed: self.check_page_for_string( check_str ) for sample_id in sample_ids: - tc.fv( "1", "select_sample_%i" % sample_id, True ) - tc.fv( "1", "sample_operation", 'Select data library and folder' ) + tc.fv( "edit_samples", "select_sample_%i" % sample_id, True ) + tc.fv( "edit_samples", "sample_operation", 'Select data library and folder' ) # refresh on change to show the data libraries selectfield self.refresh_form( "sample_operation", 'Select data library and folder' ) self.check_page_for_string( "Select data library:" ) diff -r fe0df2e688999aa7e0087cbb95c0a7257e9ae6d2 -r 4a6cd1da30d0c4e3f8e0ed7b1c1e6342bed20e44 test/install_and_test_tool_shed_repositories/base/twilltestcase.py --- a/test/install_and_test_tool_shed_repositories/base/twilltestcase.py +++ b/test/install_and_test_tool_shed_repositories/base/twilltestcase.py @@ -3,7 +3,6 @@ import re import test_db_util import time -import urllib import galaxy.model as model import galaxy.model.tool_shed_install as install_model @@ -72,22 +71,22 @@ encoded_repository_id = repository_info_dict[ 'repository_id' ] tool_shed_url = repository_info_dict[ 'tool_shed_url' ] # Pass galaxy_url to the tool shed in order to set cookies and redirects correctly. - install_params = urllib.urlencode( dict( repository_ids=encoded_repository_id, - changeset_revisions=changeset_revision, - galaxy_url=self.url ) ) + install_params = dict( repository_ids=encoded_repository_id, + changeset_revisions=changeset_revision, + galaxy_url=self.url ) # If the tool shed does not have the same hostname as the Galaxy server being used for these tests, # twill will not carry over previously set cookies for the Galaxy server when following the # install_repositories_by_revision redirect, so we have to include 403 in the allowed HTTP # status codes and log in again. - url = '%s/repository/install_repositories_by_revision?%s' % ( tool_shed_url, install_params ) - self.visit_url( url, allowed_codes=[ 200, 403 ] ) + url = '%s/repository/install_repositories_by_revision' % tool_shed_url + self.visit_url( url, params=install_params, allowed_codes=[ 200, 403 ] ) self.logout() self.login( email='test(a)bx.psu.edu', username='test' ) - install_params = urllib.urlencode( dict( repository_ids=encoded_repository_id, - changeset_revisions=changeset_revision, - tool_shed_url=tool_shed_url ) ) - url = '/admin_toolshed/prepare_for_install?%s' % install_params - self.visit_url( url ) + install_params = dict( repository_ids=encoded_repository_id, + changeset_revisions=changeset_revision, + tool_shed_url=tool_shed_url ) + url = '/admin_toolshed/prepare_for_install' + self.visit_url( url, params=install_params ) # This section is tricky, due to the way twill handles form submission. The tool dependency checkbox needs to # be hacked in through tc.browser, putting the form field in kwd doesn't work. form = tc.browser.get_form( 'select_tool_panel_section' ) @@ -124,13 +123,6 @@ del( kwd[ field_name ] ) return kwd - def visit_url( self, url, allowed_codes=[ 200 ] ): - new_url = tc.go( url ) - return_code = tc.browser.get_code() - assert return_code in allowed_codes, 'Invalid HTTP return code %s, allowed codes: %s' % \ - ( return_code, ', '.join( str( code ) for code in allowed_codes ) ) - return new_url - def wait_for_repository_installation( self, repository_ids ): final_states = [ install_model.ToolShedRepository.installation_status.ERROR, install_model.ToolShedRepository.installation_status.INSTALLED ] Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: jmchilton: Merged in jmchilton/galaxy-central-fork-1 (pull request #378)
by commits-noreply＠bitbucket.org 28 Apr '14

28 Apr '14

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/4d554f1875eb/ Changeset: 4d554f1875eb User: jmchilton Date: 2014-04-28 16:53:07 Summary: Merged in jmchilton/galaxy-central-fork-1 (pull request #378) Allow specification of environment variables for job destinations Affected #: 9 files diff -r b0067c6e06274e80157496e034c9c7d0a083ffb6 -r 4d554f1875eb9e1dfc5fc156dbae86c79c2efbee job_conf.xml.sample_advanced --- a/job_conf.xml.sample_advanced +++ b/job_conf.xml.sample_advanced @@ -74,6 +74,23 @@ <param id="Resource_List">walltime=72:00:00</param></destination><destination id="remote_cluster" runner="drmaa" tags="longjobs"/> + <destination id="java_cluster" runner="drmaa"> +  + <env id="_JAVA_OPTIONS">-Xmx=6GB</env> + <env id="ANOTHER_OPTION" raw="true">'5'</env> + <env file="/mnt/java_cluster/environment_setup.sh" /> + <env exec="module load javastuff/2.10" /> +  + </destination><destination id="real_user_cluster" runner="drmaa"><param id="galaxy_external_runjob_script">scripts/drmaa_external_runner.py</param> diff -r b0067c6e06274e80157496e034c9c7d0a083ffb6 -r 4d554f1875eb9e1dfc5fc156dbae86c79c2efbee lib/galaxy/jobs/__init__.py --- a/lib/galaxy/jobs/__init__.py +++ b/lib/galaxy/jobs/__init__.py @@ -179,6 +179,7 @@ job_metrics.set_destination_conf_element( id, metrics_elements[ 0 ] ) job_destination = JobDestination(**dict(destination.items())) job_destination['params'] = self.__get_params(destination) + job_destination['env'] = self.__get_envs(destination) self.destinations[id] = (job_destination,) if job_destination.tags is not None: for tag in job_destination.tags: @@ -350,6 +351,25 @@ rval[param.get('id')] = param.text return rval + def __get_envs(self, parent): + """Parses any child <env> tags in to a dictionary suitable for persistence. + + :param parent: Parent element in which to find child <param> tags. + :type parent: ``xml.etree.ElementTree.Element`` + + :returns: dict + """ + rval = [] + for param in parent.findall('env'): + rval.append( dict( + name=param.get('id'), + file=param.get('file'), + execute=param.get('exec'), + value=param.text, + raw=util.asbool(param.get('raw', 'false')) + ) ) + return rval + @property def default_job_tool_configuration(self): """The default JobToolConfiguration, used if a tool does not have an explicit defintion in the configuration. It consists of a reference to the default handler and default destination. diff -r b0067c6e06274e80157496e034c9c7d0a083ffb6 -r 4d554f1875eb9e1dfc5fc156dbae86c79c2efbee lib/galaxy/jobs/runners/__init__.py --- a/lib/galaxy/jobs/runners/__init__.py +++ b/lib/galaxy/jobs/runners/__init__.py @@ -17,6 +17,7 @@ from galaxy.util import DATABASE_MAX_STRING_SIZE, shrink_stream_by_size from galaxy.util import in_directory from galaxy.jobs.runners.util.job_script import job_script +from galaxy.jobs.runners.util.env import env_to_statement log = logging.getLogger( __name__ ) @@ -253,10 +254,17 @@ def get_job_file(self, job_wrapper, **kwds): job_metrics = job_wrapper.app.job_metrics job_instrumenter = job_metrics.job_instrumenters[ job_wrapper.job_destination.id ] + + env_setup_commands = kwds.get( 'env_setup_commands', [] ) + env_setup_commands.append( job_wrapper.get_env_setup_clause() or '' ) + destination = job_wrapper.job_destination or {} + envs = destination.get( "env", [] ) + for env in envs: + env_setup_commands.append( env_to_statement( env ) ) options = dict( job_instrumenter=job_instrumenter, galaxy_lib=job_wrapper.galaxy_lib_dir, - env_setup_commands=job_wrapper.get_env_setup_clause(), + env_setup_commands=env_setup_commands, working_directory=os.path.abspath( job_wrapper.working_directory ), command=job_wrapper.runner_command_line, ) diff -r b0067c6e06274e80157496e034c9c7d0a083ffb6 -r 4d554f1875eb9e1dfc5fc156dbae86c79c2efbee lib/galaxy/jobs/runners/lwr.py --- a/lib/galaxy/jobs/runners/lwr.py +++ b/lib/galaxy/jobs/runners/lwr.py @@ -134,6 +134,7 @@ tool=job_wrapper.tool, config_files=job_wrapper.extra_filenames, requirements=requirements, + env=client.env, rewrite_paths=rewrite_paths, arbitrary_files=unstructured_path_rewrites, ) @@ -222,14 +223,15 @@ for key, value in params.iteritems(): if value: params[key] = model.User.expand_user_properties( job_wrapper.get_job().user, value ) - return self.get_client( params, job_id ) + env = getattr( job_wrapper.job_destination, "env", [] ) + return self.get_client( params, job_id, env ) def get_client_from_state(self, job_state): job_destination_params = job_state.job_destination.params job_id = job_state.job_id return self.get_client( job_destination_params, job_id ) - def get_client( self, job_destination_params, job_id ): + def get_client( self, job_destination_params, job_id, env=[] ): # Cannot use url_for outside of web thread. #files_endpoint = url_for( controller="job_files", job_id=encoded_job_id ) @@ -243,6 +245,7 @@ get_client_kwds = dict( job_id=str( job_id ), files_endpoint=files_endpoint, + env=env ) return self.client_manager.get_client( job_destination_params, **get_client_kwds ) diff -r b0067c6e06274e80157496e034c9c7d0a083ffb6 -r 4d554f1875eb9e1dfc5fc156dbae86c79c2efbee lib/galaxy/jobs/runners/lwr_client/client.py --- a/lib/galaxy/jobs/runners/lwr_client/client.py +++ b/lib/galaxy/jobs/runners/lwr_client/client.py @@ -41,6 +41,7 @@ job_directory = None self.env = destination_params.get( "env", [] ) self.files_endpoint = destination_params.get("files_endpoint", None) + self.env = destination_params.get("env", []) self.job_directory = job_directory self.default_file_action = self.destination_params.get("default_file_action", "transfer") diff -r b0067c6e06274e80157496e034c9c7d0a083ffb6 -r 4d554f1875eb9e1dfc5fc156dbae86c79c2efbee lib/galaxy/jobs/runners/pbs.py --- a/lib/galaxy/jobs/runners/pbs.py +++ b/lib/galaxy/jobs/runners/pbs.py @@ -296,7 +296,7 @@ else: stage_commands = '' - env_setup_commands = '%s\n%s' % (stage_commands, job_wrapper.get_env_setup_clause()) + env_setup_commands = [ stage_commands ] script = self.get_job_file(job_wrapper, exit_code_path=ecfile, env_setup_commands=env_setup_commands) job_file = "%s/%s.sh" % (self.app.config.cluster_files_directory, job_wrapper.job_id) fh = file(job_file, "w") diff -r b0067c6e06274e80157496e034c9c7d0a083ffb6 -r 4d554f1875eb9e1dfc5fc156dbae86c79c2efbee lib/galaxy/jobs/runners/util/env.py --- /dev/null +++ b/lib/galaxy/jobs/runners/util/env.py @@ -0,0 +1,40 @@ + +RAW_VALUE_BY_DEFAULT = False + + +def env_to_statement(env): + ''' Return the abstraction description of an environment variable definition + into a statement for shell script. + + >>> env_to_statement(dict(name='X', value='Y')) + 'X="Y"; export X' + >>> env_to_statement(dict(name='X', value='Y', raw=True)) + 'X=Y; export X' + >>> env_to_statement(dict(name='X', value='"A","B","C"')) + 'X="\\\\"A\\\\",\\\\"B\\\\",\\\\"C\\\\""; export X' + >>> env_to_statement(dict(file="Y")) + '. "Y"' + >>> env_to_statement(dict(file="'RAW $FILE'", raw=True)) + ". 'RAW $FILE'" + >>> # Source file takes precedence + >>> env_to_statement(dict(name='X', value='"A","B","C"', file="S")) + '. "S"' + >>> env_to_statement(dict(execute="module load java/1.5.1")) + 'module load java/1.5.1' + ''' + source_file = env.get('file', None) + if source_file: + return '. %s' % __escape(source_file, env) + execute = env.get('execute', None) + if execute: + return execute + name = env['name'] + value = __escape(env['value'], env) + return '%s=%s; export %s' % (name, value, name) + + +def __escape(value, env): + raw = env.get('raw', RAW_VALUE_BY_DEFAULT) + if not raw: + value = '"' + value.replace('"', '\\"') + '"' + return value diff -r b0067c6e06274e80157496e034c9c7d0a083ffb6 -r 4d554f1875eb9e1dfc5fc156dbae86c79c2efbee lib/galaxy/jobs/runners/util/job_script/__init__.py --- a/lib/galaxy/jobs/runners/util/job_script/__init__.py +++ b/lib/galaxy/jobs/runners/util/job_script/__init__.py @@ -16,7 +16,7 @@ OPTIONAL_TEMPLATE_PARAMS = { 'galaxy_lib': None, 'headers': '', - 'env_setup_commands': '', + 'env_setup_commands': [], 'slots_statement': SLOTS_STATEMENT_CLUSTER_DEFAULT, 'instrument_pre_commands': '', 'instrument_post_commands': '', @@ -51,13 +51,15 @@ raise Exception("Failed to create job_script, a required parameter is missing.") job_instrumenter = kwds.get("job_instrumenter", None) if job_instrumenter: - del kwds[ "job_instrumenter" ] + del kwds["job_instrumenter"] working_directory = kwds["working_directory"] kwds["instrument_pre_commands"] = job_instrumenter.pre_execute_commands(working_directory) or '' kwds["instrument_post_commands"] = job_instrumenter.post_execute_commands(working_directory) or '' template_params = OPTIONAL_TEMPLATE_PARAMS.copy() template_params.update(**kwds) + env_setup_commands_str = "\n".join(template_params["env_setup_commands"]) + template_params["env_setup_commands"] = env_setup_commands_str if not isinstance(template, Template): template = Template(template) return template.safe_substitute(template_params) diff -r b0067c6e06274e80157496e034c9c7d0a083ffb6 -r 4d554f1875eb9e1dfc5fc156dbae86c79c2efbee test/unit/jobs/test_job_configuration.py --- a/test/unit/jobs/test_job_configuration.py +++ b/test/unit/jobs/test_job_configuration.py @@ -111,6 +111,20 @@ assert limits.concurrent_jobs[ "longjobs" ] == 1 assert limits.walltime_delta == datetime.timedelta( 0, 0, 0, 0, 0, 24 ) + def test_env_parsing( self ): + self.__with_advanced_config() + env_dest = self.job_config.destinations[ "java_cluster" ][ 0 ] + assert len( env_dest.env ) == 4, len( env_dest.env ) + assert env_dest.env[ 0 ][ "name" ] == "_JAVA_OPTIONS" + assert env_dest.env[ 0 ][ "value" ] == '-Xmx=6GB' + + assert env_dest.env[ 1 ][ "name" ] == "ANOTHER_OPTION" + assert env_dest.env[ 1 ][ "raw" ] is True + + assert env_dest.env[ 2 ][ "file" ] == "/mnt/java_cluster/environment_setup.sh" + + assert env_dest.env[ 3 ][ "execute" ] == "module load javastuff/2.10" + # TODO: Add job metrics parsing test. @property Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: 4 new changesets
by commits-noreply＠bitbucket.org 28 Apr '14

28 Apr '14

4 new commits in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/d48a5868cd1d/ Changeset: d48a5868cd1d User: jmchilton Date: 2014-04-24 07:36:54 Summary: Jobs - Convert job_script's env_setup_commands to list. Affected #: 3 files diff -r 641f7063ce3c5dcf375aafdd9a65a2e3f7ccfa4c -r d48a5868cd1dc986b1f9ea2b0b8931407e124733 lib/galaxy/jobs/runners/__init__.py --- a/lib/galaxy/jobs/runners/__init__.py +++ b/lib/galaxy/jobs/runners/__init__.py @@ -253,10 +253,13 @@ def get_job_file(self, job_wrapper, **kwds): job_metrics = job_wrapper.app.job_metrics job_instrumenter = job_metrics.job_instrumenters[ job_wrapper.job_destination.id ] + + env_setup_commands = kwds.get( 'env_setup_commands', [] ) + env_setup_commands.append( job_wrapper.get_env_setup_clause() or '' ) options = dict( job_instrumenter=job_instrumenter, galaxy_lib=job_wrapper.galaxy_lib_dir, - env_setup_commands=job_wrapper.get_env_setup_clause(), + env_setup_commands=[job_wrapper.get_env_setup_clause()], working_directory=os.path.abspath( job_wrapper.working_directory ), command=job_wrapper.runner_command_line, ) diff -r 641f7063ce3c5dcf375aafdd9a65a2e3f7ccfa4c -r d48a5868cd1dc986b1f9ea2b0b8931407e124733 lib/galaxy/jobs/runners/pbs.py --- a/lib/galaxy/jobs/runners/pbs.py +++ b/lib/galaxy/jobs/runners/pbs.py @@ -296,7 +296,7 @@ else: stage_commands = '' - env_setup_commands = '%s\n%s' % (stage_commands, job_wrapper.get_env_setup_clause()) + env_setup_commands = [ stage_commands ] script = self.get_job_file(job_wrapper, exit_code_path=ecfile, env_setup_commands=env_setup_commands) job_file = "%s/%s.sh" % (self.app.config.cluster_files_directory, job_wrapper.job_id) fh = file(job_file, "w") diff -r 641f7063ce3c5dcf375aafdd9a65a2e3f7ccfa4c -r d48a5868cd1dc986b1f9ea2b0b8931407e124733 lib/galaxy/jobs/runners/util/job_script/__init__.py --- a/lib/galaxy/jobs/runners/util/job_script/__init__.py +++ b/lib/galaxy/jobs/runners/util/job_script/__init__.py @@ -16,7 +16,7 @@ OPTIONAL_TEMPLATE_PARAMS = { 'galaxy_lib': None, 'headers': '', - 'env_setup_commands': '', + 'env_setup_commands': [], 'slots_statement': SLOTS_STATEMENT_CLUSTER_DEFAULT, 'instrument_pre_commands': '', 'instrument_post_commands': '', @@ -58,6 +58,8 @@ template_params = OPTIONAL_TEMPLATE_PARAMS.copy() template_params.update(**kwds) + env_setup_commands_str = "\n".join(template_params["env_setup_commands"]) + template_params["env_setup_commands"] = env_setup_commands_str if not isinstance(template, Template): template = Template(template) return template.safe_substitute(template_params) https://bitbucket.org/galaxy/galaxy-central/commits/63d6d9a9af79/ Changeset: 63d6d9a9af79 User: jmchilton Date: 2014-04-24 07:36:54 Summary: Jobs - Allow destinations to declare arbitrary envirnoment variables. See job_conf.xml.sample_advanced for examples. Affected #: 8 files diff -r d48a5868cd1dc986b1f9ea2b0b8931407e124733 -r 63d6d9a9af796d66f44e5105d306e3c46ff09549 job_conf.xml.sample_advanced --- a/job_conf.xml.sample_advanced +++ b/job_conf.xml.sample_advanced @@ -74,6 +74,13 @@ <param id="Resource_List">walltime=72:00:00</param></destination><destination id="remote_cluster" runner="drmaa" tags="longjobs"/> + <destination id="java_cluster" runner="drmaa"> +  + <env id="_JAVA_OPTIONS">-Xmx=6GB</env> + <env id="ANOTHER_OPTION" raw="true">'5'</env> + </destination><destination id="real_user_cluster" runner="drmaa"><param id="galaxy_external_runjob_script">scripts/drmaa_external_runner.py</param> diff -r d48a5868cd1dc986b1f9ea2b0b8931407e124733 -r 63d6d9a9af796d66f44e5105d306e3c46ff09549 lib/galaxy/jobs/__init__.py --- a/lib/galaxy/jobs/__init__.py +++ b/lib/galaxy/jobs/__init__.py @@ -179,6 +179,7 @@ job_metrics.set_destination_conf_element( id, metrics_elements[ 0 ] ) job_destination = JobDestination(**dict(destination.items())) job_destination['params'] = self.__get_params(destination) + job_destination['env'] = self.__get_envs(destination) self.destinations[id] = (job_destination,) if job_destination.tags is not None: for tag in job_destination.tags: @@ -350,6 +351,23 @@ rval[param.get('id')] = param.text return rval + def __get_envs(self, parent): + """Parses any child <env> tags in to a dictionary suitable for persistence. + + :param parent: Parent element in which to find child <param> tags. + :type parent: ``xml.etree.ElementTree.Element`` + + :returns: dict + """ + rval = [] + for param in parent.findall('env'): + rval.append( dict( + name=param.get('id'), + value=param.text, + raw=util.asbool(param.get('raw', 'false')) + ) ) + return rval + @property def default_job_tool_configuration(self): """The default JobToolConfiguration, used if a tool does not have an explicit defintion in the configuration. It consists of a reference to the default handler and default destination. diff -r d48a5868cd1dc986b1f9ea2b0b8931407e124733 -r 63d6d9a9af796d66f44e5105d306e3c46ff09549 lib/galaxy/jobs/runners/__init__.py --- a/lib/galaxy/jobs/runners/__init__.py +++ b/lib/galaxy/jobs/runners/__init__.py @@ -17,6 +17,7 @@ from galaxy.util import DATABASE_MAX_STRING_SIZE, shrink_stream_by_size from galaxy.util import in_directory from galaxy.jobs.runners.util.job_script import job_script +from galaxy.jobs.runners.util.env import env_to_statement log = logging.getLogger( __name__ ) @@ -256,10 +257,14 @@ env_setup_commands = kwds.get( 'env_setup_commands', [] ) env_setup_commands.append( job_wrapper.get_env_setup_clause() or '' ) + destination = job_wrapper.job_destination or {} + envs = destination.get( "env", [] ) + for env in envs: + env_setup_commands.append( env_to_statement( env ) ) options = dict( job_instrumenter=job_instrumenter, galaxy_lib=job_wrapper.galaxy_lib_dir, - env_setup_commands=[job_wrapper.get_env_setup_clause()], + env_setup_commands=env_setup_commands, working_directory=os.path.abspath( job_wrapper.working_directory ), command=job_wrapper.runner_command_line, ) diff -r d48a5868cd1dc986b1f9ea2b0b8931407e124733 -r 63d6d9a9af796d66f44e5105d306e3c46ff09549 lib/galaxy/jobs/runners/lwr.py --- a/lib/galaxy/jobs/runners/lwr.py +++ b/lib/galaxy/jobs/runners/lwr.py @@ -134,6 +134,7 @@ tool=job_wrapper.tool, config_files=job_wrapper.extra_filenames, requirements=requirements, + env=client.env, rewrite_paths=rewrite_paths, arbitrary_files=unstructured_path_rewrites, ) @@ -222,14 +223,15 @@ for key, value in params.iteritems(): if value: params[key] = model.User.expand_user_properties( job_wrapper.get_job().user, value ) - return self.get_client( params, job_id ) + env = getattr( job_wrapper.job_destination, "env", [] ) + return self.get_client( params, job_id, env ) def get_client_from_state(self, job_state): job_destination_params = job_state.job_destination.params job_id = job_state.job_id return self.get_client( job_destination_params, job_id ) - def get_client( self, job_destination_params, job_id ): + def get_client( self, job_destination_params, job_id, env=[] ): # Cannot use url_for outside of web thread. #files_endpoint = url_for( controller="job_files", job_id=encoded_job_id ) @@ -243,6 +245,7 @@ get_client_kwds = dict( job_id=str( job_id ), files_endpoint=files_endpoint, + env=env ) return self.client_manager.get_client( job_destination_params, **get_client_kwds ) diff -r d48a5868cd1dc986b1f9ea2b0b8931407e124733 -r 63d6d9a9af796d66f44e5105d306e3c46ff09549 lib/galaxy/jobs/runners/lwr_client/client.py --- a/lib/galaxy/jobs/runners/lwr_client/client.py +++ b/lib/galaxy/jobs/runners/lwr_client/client.py @@ -41,6 +41,7 @@ job_directory = None self.env = destination_params.get( "env", [] ) self.files_endpoint = destination_params.get("files_endpoint", None) + self.env = destination_params.get("env", []) self.job_directory = job_directory self.default_file_action = self.destination_params.get("default_file_action", "transfer") diff -r d48a5868cd1dc986b1f9ea2b0b8931407e124733 -r 63d6d9a9af796d66f44e5105d306e3c46ff09549 lib/galaxy/jobs/runners/util/env.py --- /dev/null +++ b/lib/galaxy/jobs/runners/util/env.py @@ -0,0 +1,21 @@ + +RAW_VALUE_BY_DEFAULT = False + + +def env_to_statement(env): + ''' Return the abstraction description of an environment variable definition + into a statement for shell script. + + >>> env_to_statement(dict(name='X', value='Y')) + 'X="Y"; export X' + >>> env_to_statement(dict(name='X', value='Y', raw=True)) + 'X=Y; export X' + >>> env_to_statement(dict(name='X', value='"A","B","C"')) + 'X="\\\\"A\\\\",\\\\"B\\\\",\\\\"C\\\\""; export X' + ''' + name = env['name'] + value = env['value'] + raw = env.get('raw', RAW_VALUE_BY_DEFAULT) + if not raw: + value = '"' + value.replace('"', '\\"') + '"' + return '%s=%s; export %s' % (name, value, name) diff -r d48a5868cd1dc986b1f9ea2b0b8931407e124733 -r 63d6d9a9af796d66f44e5105d306e3c46ff09549 lib/galaxy/jobs/runners/util/job_script/__init__.py --- a/lib/galaxy/jobs/runners/util/job_script/__init__.py +++ b/lib/galaxy/jobs/runners/util/job_script/__init__.py @@ -51,7 +51,7 @@ raise Exception("Failed to create job_script, a required parameter is missing.") job_instrumenter = kwds.get("job_instrumenter", None) if job_instrumenter: - del kwds[ "job_instrumenter" ] + del kwds["job_instrumenter"] working_directory = kwds["working_directory"] kwds["instrument_pre_commands"] = job_instrumenter.pre_execute_commands(working_directory) or '' kwds["instrument_post_commands"] = job_instrumenter.post_execute_commands(working_directory) or '' diff -r d48a5868cd1dc986b1f9ea2b0b8931407e124733 -r 63d6d9a9af796d66f44e5105d306e3c46ff09549 test/unit/jobs/test_job_configuration.py --- a/test/unit/jobs/test_job_configuration.py +++ b/test/unit/jobs/test_job_configuration.py @@ -111,6 +111,16 @@ assert limits.concurrent_jobs[ "longjobs" ] == 1 assert limits.walltime_delta == datetime.timedelta( 0, 0, 0, 0, 0, 24 ) + def test_env_parsing( self ): + self.__with_advanced_config() + env_dest = self.job_config.destinations[ "java_cluster" ][ 0 ] + assert len( env_dest.env ) == 2, len( env_dest.env ) + assert env_dest.env[ 0 ][ "name" ] == "_JAVA_OPTIONS" + assert env_dest.env[ 0 ][ "value" ] == '-Xmx=6GB' + + assert env_dest.env[ 1 ][ "name" ] == "ANOTHER_OPTION" + assert env_dest.env[ 1 ][ "raw" ] is True + # TODO: Add job metrics parsing test. @property https://bitbucket.org/galaxy/galaxy-central/commits/d72967b2b7b2/ Changeset: d72967b2b7b2 User: jmchilton Date: 2014-04-25 16:36:40 Summary: Extend destination specific env options. Add support sourcing files and executing arbitrary shell commands (see documented examples). Affected #: 4 files diff -r 63d6d9a9af796d66f44e5105d306e3c46ff09549 -r d72967b2b7b25ba02584880526c7b28cab6970e7 job_conf.xml.sample_advanced --- a/job_conf.xml.sample_advanced +++ b/job_conf.xml.sample_advanced @@ -77,9 +77,19 @@ <destination id="java_cluster" runner="drmaa"> + DRMAA/SLURM, PBS, Condor, and LWR. General dependencies for tools + should be configured via tool_depednency_dir and package options + and these options should be reserved for defining cluster specific + options. + --><env id="_JAVA_OPTIONS">-Xmx=6GB</env><env id="ANOTHER_OPTION" raw="true">'5'</env> + <env file="/mnt/java_cluster/environment_setup.sh" /> + <env exec="module load javastuff/2.10" /> + </destination><destination id="real_user_cluster" runner="drmaa"> diff -r 63d6d9a9af796d66f44e5105d306e3c46ff09549 -r d72967b2b7b25ba02584880526c7b28cab6970e7 lib/galaxy/jobs/__init__.py --- a/lib/galaxy/jobs/__init__.py +++ b/lib/galaxy/jobs/__init__.py @@ -363,6 +363,8 @@ for param in parent.findall('env'): rval.append( dict( name=param.get('id'), + file=param.get('file'), + execute=param.get('exec'), value=param.text, raw=util.asbool(param.get('raw', 'false')) ) ) diff -r 63d6d9a9af796d66f44e5105d306e3c46ff09549 -r d72967b2b7b25ba02584880526c7b28cab6970e7 lib/galaxy/jobs/runners/util/env.py --- a/lib/galaxy/jobs/runners/util/env.py +++ b/lib/galaxy/jobs/runners/util/env.py @@ -12,10 +12,29 @@ 'X=Y; export X' >>> env_to_statement(dict(name='X', value='"A","B","C"')) 'X="\\\\"A\\\\",\\\\"B\\\\",\\\\"C\\\\""; export X' + >>> env_to_statement(dict(file="Y")) + '. "Y"' + >>> env_to_statement(dict(file="'RAW $FILE'", raw=True)) + ". 'RAW $FILE'" + >>> # Source file takes precedence + >>> env_to_statement(dict(name='X', value='"A","B","C"', file="S")) + '. "S"' + >>> env_to_statement(dict(execute="module load java/1.5.1")) + 'module load java/1.5.1' ''' + source_file = env.get('file', None) + if source_file: + return '. %s' % __escape(source_file, env) + execute = env.get('execute', None) + if execute: + return execute name = env['name'] - value = env['value'] + value = __escape(env['value'], env) + return '%s=%s; export %s' % (name, value, name) + + +def __escape(value, env): raw = env.get('raw', RAW_VALUE_BY_DEFAULT) if not raw: value = '"' + value.replace('"', '\\"') + '"' - return '%s=%s; export %s' % (name, value, name) + return value diff -r 63d6d9a9af796d66f44e5105d306e3c46ff09549 -r d72967b2b7b25ba02584880526c7b28cab6970e7 test/unit/jobs/test_job_configuration.py --- a/test/unit/jobs/test_job_configuration.py +++ b/test/unit/jobs/test_job_configuration.py @@ -114,13 +114,17 @@ def test_env_parsing( self ): self.__with_advanced_config() env_dest = self.job_config.destinations[ "java_cluster" ][ 0 ] - assert len( env_dest.env ) == 2, len( env_dest.env ) + assert len( env_dest.env ) == 4, len( env_dest.env ) assert env_dest.env[ 0 ][ "name" ] == "_JAVA_OPTIONS" assert env_dest.env[ 0 ][ "value" ] == '-Xmx=6GB' assert env_dest.env[ 1 ][ "name" ] == "ANOTHER_OPTION" assert env_dest.env[ 1 ][ "raw" ] is True + assert env_dest.env[ 2 ][ "file" ] == "/mnt/java_cluster/environment_setup.sh" + + assert env_dest.env[ 3 ][ "execute" ] == "module load javastuff/2.10" + # TODO: Add job metrics parsing test. @property https://bitbucket.org/galaxy/galaxy-central/commits/4d554f1875eb/ Changeset: 4d554f1875eb User: jmchilton Date: 2014-04-28 16:53:07 Summary: Merged in jmchilton/galaxy-central-fork-1 (pull request #378) Allow specification of environment variables for job destinations Affected #: 9 files diff -r b0067c6e06274e80157496e034c9c7d0a083ffb6 -r 4d554f1875eb9e1dfc5fc156dbae86c79c2efbee job_conf.xml.sample_advanced --- a/job_conf.xml.sample_advanced +++ b/job_conf.xml.sample_advanced @@ -74,6 +74,23 @@ <param id="Resource_List">walltime=72:00:00</param></destination><destination id="remote_cluster" runner="drmaa" tags="longjobs"/> + <destination id="java_cluster" runner="drmaa"> +  + <env id="_JAVA_OPTIONS">-Xmx=6GB</env> + <env id="ANOTHER_OPTION" raw="true">'5'</env> + <env file="/mnt/java_cluster/environment_setup.sh" /> + <env exec="module load javastuff/2.10" /> +  + </destination><destination id="real_user_cluster" runner="drmaa"><param id="galaxy_external_runjob_script">scripts/drmaa_external_runner.py</param> diff -r b0067c6e06274e80157496e034c9c7d0a083ffb6 -r 4d554f1875eb9e1dfc5fc156dbae86c79c2efbee lib/galaxy/jobs/__init__.py --- a/lib/galaxy/jobs/__init__.py +++ b/lib/galaxy/jobs/__init__.py @@ -179,6 +179,7 @@ job_metrics.set_destination_conf_element( id, metrics_elements[ 0 ] ) job_destination = JobDestination(**dict(destination.items())) job_destination['params'] = self.__get_params(destination) + job_destination['env'] = self.__get_envs(destination) self.destinations[id] = (job_destination,) if job_destination.tags is not None: for tag in job_destination.tags: @@ -350,6 +351,25 @@ rval[param.get('id')] = param.text return rval + def __get_envs(self, parent): + """Parses any child <env> tags in to a dictionary suitable for persistence. + + :param parent: Parent element in which to find child <param> tags. + :type parent: ``xml.etree.ElementTree.Element`` + + :returns: dict + """ + rval = [] + for param in parent.findall('env'): + rval.append( dict( + name=param.get('id'), + file=param.get('file'), + execute=param.get('exec'), + value=param.text, + raw=util.asbool(param.get('raw', 'false')) + ) ) + return rval + @property def default_job_tool_configuration(self): """The default JobToolConfiguration, used if a tool does not have an explicit defintion in the configuration. It consists of a reference to the default handler and default destination. diff -r b0067c6e06274e80157496e034c9c7d0a083ffb6 -r 4d554f1875eb9e1dfc5fc156dbae86c79c2efbee lib/galaxy/jobs/runners/__init__.py --- a/lib/galaxy/jobs/runners/__init__.py +++ b/lib/galaxy/jobs/runners/__init__.py @@ -17,6 +17,7 @@ from galaxy.util import DATABASE_MAX_STRING_SIZE, shrink_stream_by_size from galaxy.util import in_directory from galaxy.jobs.runners.util.job_script import job_script +from galaxy.jobs.runners.util.env import env_to_statement log = logging.getLogger( __name__ ) @@ -253,10 +254,17 @@ def get_job_file(self, job_wrapper, **kwds): job_metrics = job_wrapper.app.job_metrics job_instrumenter = job_metrics.job_instrumenters[ job_wrapper.job_destination.id ] + + env_setup_commands = kwds.get( 'env_setup_commands', [] ) + env_setup_commands.append( job_wrapper.get_env_setup_clause() or '' ) + destination = job_wrapper.job_destination or {} + envs = destination.get( "env", [] ) + for env in envs: + env_setup_commands.append( env_to_statement( env ) ) options = dict( job_instrumenter=job_instrumenter, galaxy_lib=job_wrapper.galaxy_lib_dir, - env_setup_commands=job_wrapper.get_env_setup_clause(), + env_setup_commands=env_setup_commands, working_directory=os.path.abspath( job_wrapper.working_directory ), command=job_wrapper.runner_command_line, ) diff -r b0067c6e06274e80157496e034c9c7d0a083ffb6 -r 4d554f1875eb9e1dfc5fc156dbae86c79c2efbee lib/galaxy/jobs/runners/lwr.py --- a/lib/galaxy/jobs/runners/lwr.py +++ b/lib/galaxy/jobs/runners/lwr.py @@ -134,6 +134,7 @@ tool=job_wrapper.tool, config_files=job_wrapper.extra_filenames, requirements=requirements, + env=client.env, rewrite_paths=rewrite_paths, arbitrary_files=unstructured_path_rewrites, ) @@ -222,14 +223,15 @@ for key, value in params.iteritems(): if value: params[key] = model.User.expand_user_properties( job_wrapper.get_job().user, value ) - return self.get_client( params, job_id ) + env = getattr( job_wrapper.job_destination, "env", [] ) + return self.get_client( params, job_id, env ) def get_client_from_state(self, job_state): job_destination_params = job_state.job_destination.params job_id = job_state.job_id return self.get_client( job_destination_params, job_id ) - def get_client( self, job_destination_params, job_id ): + def get_client( self, job_destination_params, job_id, env=[] ): # Cannot use url_for outside of web thread. #files_endpoint = url_for( controller="job_files", job_id=encoded_job_id ) @@ -243,6 +245,7 @@ get_client_kwds = dict( job_id=str( job_id ), files_endpoint=files_endpoint, + env=env ) return self.client_manager.get_client( job_destination_params, **get_client_kwds ) diff -r b0067c6e06274e80157496e034c9c7d0a083ffb6 -r 4d554f1875eb9e1dfc5fc156dbae86c79c2efbee lib/galaxy/jobs/runners/lwr_client/client.py --- a/lib/galaxy/jobs/runners/lwr_client/client.py +++ b/lib/galaxy/jobs/runners/lwr_client/client.py @@ -41,6 +41,7 @@ job_directory = None self.env = destination_params.get( "env", [] ) self.files_endpoint = destination_params.get("files_endpoint", None) + self.env = destination_params.get("env", []) self.job_directory = job_directory self.default_file_action = self.destination_params.get("default_file_action", "transfer") diff -r b0067c6e06274e80157496e034c9c7d0a083ffb6 -r 4d554f1875eb9e1dfc5fc156dbae86c79c2efbee lib/galaxy/jobs/runners/pbs.py --- a/lib/galaxy/jobs/runners/pbs.py +++ b/lib/galaxy/jobs/runners/pbs.py @@ -296,7 +296,7 @@ else: stage_commands = '' - env_setup_commands = '%s\n%s' % (stage_commands, job_wrapper.get_env_setup_clause()) + env_setup_commands = [ stage_commands ] script = self.get_job_file(job_wrapper, exit_code_path=ecfile, env_setup_commands=env_setup_commands) job_file = "%s/%s.sh" % (self.app.config.cluster_files_directory, job_wrapper.job_id) fh = file(job_file, "w") diff -r b0067c6e06274e80157496e034c9c7d0a083ffb6 -r 4d554f1875eb9e1dfc5fc156dbae86c79c2efbee lib/galaxy/jobs/runners/util/env.py --- /dev/null +++ b/lib/galaxy/jobs/runners/util/env.py @@ -0,0 +1,40 @@ + +RAW_VALUE_BY_DEFAULT = False + + +def env_to_statement(env): + ''' Return the abstraction description of an environment variable definition + into a statement for shell script. + + >>> env_to_statement(dict(name='X', value='Y')) + 'X="Y"; export X' + >>> env_to_statement(dict(name='X', value='Y', raw=True)) + 'X=Y; export X' + >>> env_to_statement(dict(name='X', value='"A","B","C"')) + 'X="\\\\"A\\\\",\\\\"B\\\\",\\\\"C\\\\""; export X' + >>> env_to_statement(dict(file="Y")) + '. "Y"' + >>> env_to_statement(dict(file="'RAW $FILE'", raw=True)) + ". 'RAW $FILE'" + >>> # Source file takes precedence + >>> env_to_statement(dict(name='X', value='"A","B","C"', file="S")) + '. "S"' + >>> env_to_statement(dict(execute="module load java/1.5.1")) + 'module load java/1.5.1' + ''' + source_file = env.get('file', None) + if source_file: + return '. %s' % __escape(source_file, env) + execute = env.get('execute', None) + if execute: + return execute + name = env['name'] + value = __escape(env['value'], env) + return '%s=%s; export %s' % (name, value, name) + + +def __escape(value, env): + raw = env.get('raw', RAW_VALUE_BY_DEFAULT) + if not raw: + value = '"' + value.replace('"', '\\"') + '"' + return value diff -r b0067c6e06274e80157496e034c9c7d0a083ffb6 -r 4d554f1875eb9e1dfc5fc156dbae86c79c2efbee lib/galaxy/jobs/runners/util/job_script/__init__.py --- a/lib/galaxy/jobs/runners/util/job_script/__init__.py +++ b/lib/galaxy/jobs/runners/util/job_script/__init__.py @@ -16,7 +16,7 @@ OPTIONAL_TEMPLATE_PARAMS = { 'galaxy_lib': None, 'headers': '', - 'env_setup_commands': '', + 'env_setup_commands': [], 'slots_statement': SLOTS_STATEMENT_CLUSTER_DEFAULT, 'instrument_pre_commands': '', 'instrument_post_commands': '', @@ -51,13 +51,15 @@ raise Exception("Failed to create job_script, a required parameter is missing.") job_instrumenter = kwds.get("job_instrumenter", None) if job_instrumenter: - del kwds[ "job_instrumenter" ] + del kwds["job_instrumenter"] working_directory = kwds["working_directory"] kwds["instrument_pre_commands"] = job_instrumenter.pre_execute_commands(working_directory) or '' kwds["instrument_post_commands"] = job_instrumenter.post_execute_commands(working_directory) or '' template_params = OPTIONAL_TEMPLATE_PARAMS.copy() template_params.update(**kwds) + env_setup_commands_str = "\n".join(template_params["env_setup_commands"]) + template_params["env_setup_commands"] = env_setup_commands_str if not isinstance(template, Template): template = Template(template) return template.safe_substitute(template_params) diff -r b0067c6e06274e80157496e034c9c7d0a083ffb6 -r 4d554f1875eb9e1dfc5fc156dbae86c79c2efbee test/unit/jobs/test_job_configuration.py --- a/test/unit/jobs/test_job_configuration.py +++ b/test/unit/jobs/test_job_configuration.py @@ -111,6 +111,20 @@ assert limits.concurrent_jobs[ "longjobs" ] == 1 assert limits.walltime_delta == datetime.timedelta( 0, 0, 0, 0, 0, 24 ) + def test_env_parsing( self ): + self.__with_advanced_config() + env_dest = self.job_config.destinations[ "java_cluster" ][ 0 ] + assert len( env_dest.env ) == 4, len( env_dest.env ) + assert env_dest.env[ 0 ][ "name" ] == "_JAVA_OPTIONS" + assert env_dest.env[ 0 ][ "value" ] == '-Xmx=6GB' + + assert env_dest.env[ 1 ][ "name" ] == "ANOTHER_OPTION" + assert env_dest.env[ 1 ][ "raw" ] is True + + assert env_dest.env[ 2 ][ "file" ] == "/mnt/java_cluster/environment_setup.sh" + + assert env_dest.env[ 3 ][ "execute" ] == "module load javastuff/2.10" + # TODO: Add job metrics parsing test. @property Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: greg: Reorder eggs.require for some fabric files.
by commits-noreply＠bitbucket.org 28 Apr '14

28 Apr '14

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/b0067c6e0627/ Changeset: b0067c6e0627 User: greg Date: 2014-04-28 16:37:44 Summary: Reorder eggs.require for some fabric files. Affected #: 3 files diff -r 9faa9a2ab1e544b8ba705b3e320d2166d7a6ee88 -r b0067c6e06274e80157496e034c9c7d0a083ffb6 lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py --- a/lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py +++ b/lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py @@ -3,9 +3,9 @@ from galaxy import eggs +eggs.require( 'paramiko' ) +eggs.require( 'ssh' ) eggs.require( 'Fabric' ) -eggs.require( 'ssh' ) -eggs.require( 'paramiko' ) from fabric.api import env from fabric.api import lcd diff -r 9faa9a2ab1e544b8ba705b3e320d2166d7a6ee88 -r b0067c6e06274e80157496e034c9c7d0a083ffb6 lib/tool_shed/galaxy_install/tool_dependencies/recipe/recipe_manager.py --- a/lib/tool_shed/galaxy_install/tool_dependencies/recipe/recipe_manager.py +++ b/lib/tool_shed/galaxy_install/tool_dependencies/recipe/recipe_manager.py @@ -12,9 +12,10 @@ # TODO: eliminate the use of fabric here. from galaxy import eggs + +eggs.require( 'paramiko' ) +eggs.require( 'ssh' ) eggs.require( 'Fabric' ) -eggs.require( 'ssh' ) -eggs.require( 'paramiko' ) from fabric.operations import _AttributeString from fabric import state diff -r 9faa9a2ab1e544b8ba705b3e320d2166d7a6ee88 -r b0067c6e06274e80157496e034c9c7d0a083ffb6 lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py --- a/lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py +++ b/lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py @@ -13,9 +13,10 @@ # TODO: eliminate the use of fabric here. from galaxy import eggs + +eggs.require( 'paramiko' ) +eggs.require( 'ssh' ) eggs.require( 'Fabric' ) -eggs.require( 'ssh' ) -eggs.require( 'paramiko' ) from fabric.api import settings from fabric.api import lcd Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: greg: Fix for the install_repository_revision function in the Galaxy API: a more correct and informative message is now displayed if any of the request parameters is invalid.
by commits-noreply＠bitbucket.org 28 Apr '14

28 Apr '14

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/9faa9a2ab1e5/ Changeset: 9faa9a2ab1e5 User: greg Date: 2014-04-28 15:45:01 Summary: Fix for the install_repository_revision function in the Galaxy API: a more correct and informative message is now displayed if any of the request parameters is invalid. Affected #: 1 file diff -r dc9fb7fba975f6169d9ee8eb257bf477d7d4274c -r 9faa9a2ab1e544b8ba705b3e320d2166d7a6ee88 lib/galaxy/webapps/galaxy/api/tool_shed_repositories.py --- a/lib/galaxy/webapps/galaxy/api/tool_shed_repositories.py +++ b/lib/galaxy/webapps/galaxy/api/tool_shed_repositories.py @@ -230,8 +230,18 @@ ( str( tool_shed_url ), str( changeset_revision ), str( name ), str( owner ), str( e ) ) log.debug( message ) return dict( status='error', error=message ) + # Make sure the tool shed returned everything we need for installing the repository. + if not repository_revision_dict or not repo_info_dict: + key = kwd.get( 'key', None ) + invalid_parameter_message = "No information is available for the requested repository revision.\n" + invalid_parameter_message += "One or more of the following parameter values is likely invalid:\n" + invalid_parameter_message += "key: %s\n" % str( key ) + invalid_parameter_message += "tool_shed_url: %s\n" % str( tool_shed_url ) + invalid_parameter_message += "name: %s\n" % str( name ) + invalid_parameter_message += "owner: %s\n" % str( owner ) + invalid_parameter_message += "changeset_revision: %s\n" % str( changeset_revision ) + raise HTTPBadRequest( detail=invalid_parameter_message ) repo_info_dicts = [ repo_info_dict ] - # Make sure the tool shed returned everything we need for installing the repository. try: has_repository_dependencies = repository_revision_dict[ 'has_repository_dependencies' ] except: Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: greg: A bit of rework for recently introduced assert action types for tool dependency package install recipes: 1) assert_file_exists - true for both files and symlinks, but false if path is a directory 2) assert_file_executable - true for both files and symlinks but false if path is a directory 3) assert_directory_exists - true for both directories and symlinks, but false if path is a file 4) assert_directory_executable - true for both directories and symlinks, but fa
by commits-noreply＠bitbucket.org 28 Apr '14

28 Apr '14

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/dc9fb7fba975/ Changeset: dc9fb7fba975 User: greg Date: 2014-04-28 13:03:37 Summary: A bit of rework for recently introduced assert action types for tool dependency package install recipes: 1) assert_file_exists - true for both files and symlinks, but false if path is a directory 2) assert_file_executable - true for both files and symlinks but false if path is a directory 3) assert_directory_exists - true for both directories and symlinks, but false if path is a file 4) assert_directory_executable - true for both directories and symlinks, but false if path is a file. Affected #: 3 files diff -r 56dd83576ce67763863b61ad85104f8d14d978e0 -r dc9fb7fba975f6169d9ee8eb257bf477d7d4274c lib/tool_shed/galaxy_install/tool_dependencies/recipe/recipe_manager.py --- a/lib/tool_shed/galaxy_install/tool_dependencies/recipe/recipe_manager.py +++ b/lib/tool_shed/galaxy_install/tool_dependencies/recipe/recipe_manager.py @@ -414,7 +414,8 @@ return tool_dependency, filtered_actions, dir def load_step_handlers( self ): - step_handlers_by_type = dict( assert_directory_exists=step_handler.AssertDirectoryExists(), + step_handlers_by_type = dict( assert_directory_executable=step_handler.AssertDirectoryExecutable(), + assert_directory_exists=step_handler.AssertDirectoryExists(), assert_file_executable=step_handler.AssertFileExecutable(), assert_file_exists=step_handler.AssertFileExists(), autoconf=step_handler.Autoconf(), diff -r 56dd83576ce67763863b61ad85104f8d14d978e0 -r dc9fb7fba975f6169d9ee8eb257bf477d7d4274c lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py --- a/lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py +++ b/lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py @@ -36,6 +36,39 @@ raise "Unimplemented Method" +class AssertDirectoryExecutable( RecipeStep ): + + def __init__( self ): + self.type = 'assert_directory_executable' + + def execute_step( self, app, tool_dependency, package_name, actions, action_dict, filtered_actions, env_file_builder, + install_environment, work_dir, install_dir, current_dir=None, initial_download=False ): + """ + Make sure a symbolic link or directory on disk exists and is executable, but is not a file. + Since this class is not used in the initial download stage, no recipe step filtering is + performed here, and None values are always returned for filtered_actions and dir. + """ + if os.path.isabs( action_dict[ 'full_path' ] ): + full_path = action_dict[ 'full_path' ] + else: + full_path = os.path.join( current_dir, action_dict[ 'full_path' ] ) + if not td_common_util.assert_directory_executable( full_path=full_path ): + status = app.install_model.ToolDependency.installation_status.ERROR + error_message = 'The path %s is not a directory or is not executable by the owner.' % str( full_path ) + tool_dependency = tool_dependency_util.set_tool_dependency_attributes( app, + tool_dependency, + status=status, + error_message=error_message, + remove_from_disk=False ) + return tool_dependency, None, None + + def prepare_step( self, app, tool_dependency, action_elem, action_dict, install_dir, is_binary_download ): + # <action type="assert_executable">$INSTALL_DIR/mira/my_file</action> + if action_elem.text: + action_dict[ 'full_path' ] = td_common_util.evaluate_template( action_elem.text, install_dir ) + return action_dict + + class AssertDirectoryExists( RecipeStep ): def __init__( self ): @@ -44,9 +77,9 @@ def execute_step( self, app, tool_dependency, package_name, actions, action_dict, filtered_actions, env_file_builder, install_environment, work_dir, install_dir, current_dir=None, initial_download=False ): """ - Make sure a directory on disk exists. Since this class is not used in the initial download stage, - no recipe step filtering is performed here, and None values are always returned for filtered_actions - and dir. + Make sure a a symbolic link or directory on disk exists, but is not a file. Since this + class is not used in the initial download stage, no recipe step filtering is performed + here, and None values are always returned for filtered_actions and dir. """ if os.path.isabs( action_dict[ 'full_path' ] ): full_path = action_dict[ 'full_path' ] @@ -54,7 +87,7 @@ full_path = os.path.join( current_dir, action_dict[ 'full_path' ] ) if not td_common_util.assert_directory_exists( full_path=full_path ): status = app.install_model.ToolDependency.installation_status.ERROR - error_message = 'The required directory %s does not exist.' % str( full_path ) + error_message = 'The path %s is not a directory or does not exist.' % str( full_path ) tool_dependency = tool_dependency_util.set_tool_dependency_attributes( app, tool_dependency, status=status, @@ -77,9 +110,9 @@ def execute_step( self, app, tool_dependency, package_name, actions, action_dict, filtered_actions, env_file_builder, install_environment, work_dir, install_dir, current_dir=None, initial_download=False ): """ - Make sure a file on disk exists and is executable. Since this class is not used in the initial - download stage, no recipe step filtering is performed here, and None values are always returned - for filtered_actions and dir. + Make sure a symbolic link or file on disk exists and is executable, but is not a directory. + Since this class is not used in the initial download stage, no recipe step filtering is + performed here, and None values are always returned for filtered_actions and dir. """ if os.path.isabs( action_dict[ 'full_path' ] ): full_path = action_dict[ 'full_path' ] @@ -87,7 +120,7 @@ full_path = os.path.join( current_dir, action_dict[ 'full_path' ] ) if not td_common_util.assert_file_executable( full_path=full_path ): status = app.install_model.ToolDependency.installation_status.ERROR - error_message = 'The file %s is not executable by the owner.' % str( full_path ) + error_message = 'The path %s is not a file or is not executable by the owner.' % str( full_path ) tool_dependency = tool_dependency_util.set_tool_dependency_attributes( app, tool_dependency, status=status, @@ -110,9 +143,9 @@ def execute_step( self, app, tool_dependency, package_name, actions, action_dict, filtered_actions, env_file_builder, install_environment, work_dir, install_dir, current_dir=None, initial_download=False ): """ - Make sure a file on disk exists. Since this class is not used in the initial download stage, - no recipe step filtering is performed here, and None values are always returned for - filtered_actions and dir. + Make sure a symbolic link or file on disk exists, but is not a directory. Since this + class is not used in the initial download stage, no recipe step filtering is performed + here, and None values are always returned for filtered_actions and dir. """ if os.path.isabs( action_dict[ 'full_path' ] ): full_path = action_dict[ 'full_path' ] @@ -120,7 +153,7 @@ full_path = os.path.join( current_dir, action_dict[ 'full_path' ] ) if not td_common_util.assert_file_exists( full_path=full_path ): status = app.install_model.ToolDependency.installation_status.ERROR - error_message = 'The required file %s does not exist.' % str( full_path ) + error_message = 'The path %s is not a file or does not exist.' % str( full_path ) tool_dependency = tool_dependency_util.set_tool_dependency_attributes( app, tool_dependency, status=status, diff -r 56dd83576ce67763863b61ad85104f8d14d978e0 -r dc9fb7fba975f6169d9ee8eb257bf477d7d4274c lib/tool_shed/galaxy_install/tool_dependencies/td_common_util.py --- a/lib/tool_shed/galaxy_install/tool_dependencies/td_common_util.py +++ b/lib/tool_shed/galaxy_install/tool_dependencies/td_common_util.py @@ -111,31 +111,63 @@ def open_zip( self, filepath, mode ): return zipfile.ZipFile( filepath, mode ) -def assert_directory_exists( full_path ): - """Return True if a directory exists and is not a symbolic link.""" - if os.path.islink( full_path ): +def assert_directory_executable( full_path ): + """ + Return True if a symbolic link or directory exists and is executable, but if + full_path is a file, return False. + """ + if full_path is None: return False - if os.path.is_dir( full_path ): - return True + if os.path.isfile( full_path ): + return False + if os.path.isdir( full_path ): + # Make sure the owner has execute permission on the directory. + # See http://docs.python.org/2/library/stat.html + if stat.S_IXUSR & os.stat( full_path )[ stat.ST_MODE ] == 64: + return True return False -def assert_file_exists( full_path ): - """Return True if a file exists. This will work for both symbolic linke and files.""" - if os.path.exists( full_path ): +def assert_directory_exists( full_path ): + """ + Return True if a symbolic link or directory exists, but if full_path is a file, + return False. """ + if full_path is None: + return False + if os.path.isfile( full_path ): + return False + if os.path.isdir( full_path ): return True return False def assert_file_executable( full_path ): - """Return True if a file exists and is executable.""" - if os.path.islink( full_path ): + """ + Return True if a symbolic link or file exists and is executable, but if full_path + is a directory, return False. + """ + if full_path is None: return False - if os.path.is_file( full_path ): + if os.path.isdir( full_path ): + return False + if os.path.exists( full_path ): # Make sure the owner has execute permission on the file. # See http://docs.python.org/2/library/stat.html if stat.S_IXUSR & os.stat( full_path )[ stat.ST_MODE ] == 64: return True return False +def assert_file_exists( full_path ): + """ + Return True if a symbolic link or file exists, but if full_path is a directory, + return False. + """ + if full_path is None: + return False + if os.path.isdir( full_path ): + return False + if os.path.exists( full_path ): + return True + return False + def create_env_var_dict( elem, tool_dependency_install_dir=None, tool_shed_repository_install_dir=None ): env_var_name = elem.get( 'name', 'PATH' ) env_var_action = elem.get( 'action', 'prepend_to' ) Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

← Newer
1
2
3
4
5
6
7
...
27
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

galaxy-commits April 2014