- galaxy-commits - lists.galaxyproject.org

[galaxyproject/usegalaxy-playbook] 7b729c: Pull Main from galaxy-central so we can get stable...
by GitHub 18 Sep '14

18 Sep '14

Branch: refs/heads/master Home: https://github.com/galaxyproject/usegalaxy-playbook Commit: 7b729c1a8fac476da384a916eb304155bb6c7aca https://github.com/galaxyproject/usegalaxy-playbook/commit/7b729c1a8fac476d… Author: Nate Coraor <nate(a)bx.psu.edu> Date: 2014-09-17 (Wed, 17 Sep 2014) Changed paths: M production/group_vars/all.yml Log Message: ----------- Pull Main from galaxy-central so we can get stable commits before they're on galaxy-dist.

1 0

commit/galaxy-central: natefoo: With the exception of shed_*.xml configs, which are still instantiated from
by commits-noreply＠bitbucket.org 18 Sep '14

18 Sep '14

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/9300f9bd7ba1/ Changeset: 9300f9bd7ba1 User: natefoo Date: 2014-09-17 19:23:51+00:00 Summary: With the exception of shed_*.xml configs, which are still instantiated from their samples by common_startup.sh, reverse the stream polarity and prefer the config/ version of config files, since their existence suggests the administrator took an intentional action to create them. Note that none of this magical detection applies unless you leave the options in galaxy.ini that set the paths to these configs commented/unset. Credit to John for the suggestion. Affected #: 2 files diff -r 69bb888b05cd2cbb0f553fe7f2b7c24d05e6ac97 -r 9300f9bd7ba1b791b97e91fd1db75fc88280f013 lib/galaxy/config.py --- a/lib/galaxy/config.py +++ b/lib/galaxy/config.py @@ -378,24 +378,34 @@ Backwards compatibility for config files moved to the config/ dir. """ defaults = dict( - data_manager_config_file = [ 'data_manager_conf.xml', 'config/data_manager_conf.xml', 'config/data_manager_conf.xml.sample' ], - datatypes_config_file = [ 'datatypes_conf.xml', 'config/datatypes_conf.xml', 'config/datatypes_conf.xml.sample' ], - external_service_type_config_file = [ 'external_service_types_conf.xml', 'config/external_service_types_conf.xml.sample', 'config/external_service_types_conf.xml.sample' ], - job_config_file = [ 'job_conf.xml', 'config/job_conf.xml' ], - job_metrics_config_file = [ 'job_metrics_conf.xml', 'config/job_metrics_conf.xml' ], - dependency_resolvers_config_file = [ 'dependency_resolvers_conf.xml', 'config/dependency_resolvers_conf.xml' ], - job_resource_params_file = [ 'job_resource_params_conf.xml', 'config/job_resource_params_conf.xml' ], + data_manager_config_file = [ 'config/data_manager_conf.xml', 'data_manager_conf.xml', 'config/data_manager_conf.xml.sample' ], + datatypes_config_file = [ 'config/datatypes_conf.xml', 'datatypes_conf.xml', 'config/datatypes_conf.xml.sample' ], + external_service_type_config_file = [ 'config/external_service_types_conf.xml', 'external_service_types_conf.xml', 'config/external_service_types_conf.xml.sample' ], + job_config_file = [ 'config/job_conf.xml', 'job_conf.xml' ], + job_metrics_config_file = [ 'config/job_metrics_conf.xml', 'job_metrics_conf.xml' ], + dependency_resolvers_config_file = [ 'config/dependency_resolvers_conf.xml', 'dependency_resolvers_conf.xml' ], + job_resource_params_file = [ 'config/job_resource_params_conf.xml', 'job_resource_params_conf.xml' ], migrated_tools_config = [ 'migrated_tools_conf.xml', 'config/migrated_tools_conf.xml' ], - object_store_config_file = [ 'object_store_conf.xml', 'config/object_store_conf.xml' ], - openid_config_file = [ 'openid_conf.xml', 'config/openid_conf.xml' 'config/openid_conf.xml.sample' ], + object_store_config_file = [ 'config/object_store_conf.xml', 'object_store_conf.xml' ], + openid_config_file = [ 'config/openid_conf.xml', 'openid_conf.xml' 'config/openid_conf.xml.sample' ], shed_data_manager_config_file = [ 'shed_data_manager_conf.xml', 'config/shed_data_manager_conf.xml' ], shed_tool_data_table_config = [ 'shed_tool_data_table_conf.xml', 'config/shed_tool_data_table_conf.xml' ], - tool_sheds_config_file = [ 'tool_sheds_conf.xml', 'config/tool_sheds_conf.xml', 'config/tool_sheds_conf.xml.sample' ], + tool_sheds_config_file = [ 'config/tool_sheds_conf.xml', 'tool_sheds_conf.xml', 'config/tool_sheds_conf.xml.sample' ], ) listify_defaults = dict( - tool_data_table_config_path = [ 'tool_data_table_conf.xml', 'config/tool_data_table_conf.xml', 'config/tool_data_table_conf.xml.sample' ], - tool_config_file = [ 'tool_conf.xml,shed_tool_conf.xml', 'config/tool_conf.xml,config/shed_tool_conf.xml', 'config/tool_conf.xml.sample,config/shed_tool_conf.xml' ] + tool_data_table_config_path = [ 'config/tool_data_table_conf.xml', 'config/tool_data_table_conf.xml', 'config/tool_data_table_conf.xml.sample' ], + # rationale: + # [0]: user has explicitly created config/tool_conf.xml but did not + # move their existing shed_tool_conf.xml, don't use + # config/shed_tool_conf.xml, which is probably the empty + # version copied from the sample, or else their shed tools + # will disappear + # [1]: user has created config/tool_conf.xml and, having passed + # [0], probably moved their shed_tool_conf.xml as well + # [2]: user has done nothing, use the old files + # [3]: fresh install + tool_config_file = [ 'config/tool_conf.xml,shed_tool_conf.xml', 'config/tool_conf.xml,config/shed_tool_conf.xml', 'tool_conf.xml,shed_tool_conf.xml', 'config/tool_conf.xml.sample,config/shed_tool_conf.xml' ] ) for var, defaults in defaults.items(): diff -r 69bb888b05cd2cbb0f553fe7f2b7c24d05e6ac97 -r 9300f9bd7ba1b791b97e91fd1db75fc88280f013 lib/galaxy/webapps/tool_shed/config.py --- a/lib/galaxy/webapps/tool_shed/config.py +++ b/lib/galaxy/webapps/tool_shed/config.py @@ -139,12 +139,12 @@ def __parse_config_file_options( self, kwargs ): defaults = dict( - datatypes_config_file = [ 'datatypes_conf.xml', 'config/datatypes_conf.xml', 'config/datatypes_conf.xml.sample' ], + datatypes_config_file = [ 'config/datatypes_conf.xml', 'datatypes_conf.xml', 'config/datatypes_conf.xml.sample' ], shed_tool_data_table_config = [ 'shed_tool_data_table_conf.xml', 'config/shed_tool_data_table_conf.xml' ], ) listify_defaults = dict( - tool_data_table_config_path = [ 'tool_data_table_conf.xml', 'config/tool_data_table_conf.xml', 'config/tool_data_table_conf.xml.sample' ], + tool_data_table_config_path = [ 'config/tool_data_table_conf.xml', 'tool_data_table_conf.xml', 'config/tool_data_table_conf.xml.sample' ], ) for var, defaults in defaults.items(): Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: jmchilton: Merge latest stable.
by commits-noreply＠bitbucket.org 18 Sep '14

18 Sep '14

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/69bb888b05cd/ Changeset: 69bb888b05cd User: jmchilton Date: 2014-09-17 19:21:34+00:00 Summary: Merge latest stable. Affected #: 3 files diff -r 818c24105d055f65047595a38ae505d5e45f7520 -r 69bb888b05cd2cbb0f553fe7f2b7c24d05e6ac97 .hgtags --- a/.hgtags +++ b/.hgtags @@ -18,4 +18,4 @@ 81fbe25bd02edcd53065e8e4476dd1dfb5a72cf2 latest_2013.11.04 2a756ca2cb1826db7796018e77d12e2dd7b67603 latest_2014.02.10 ca45b78adb4152fc6e7395514d46eba6b7d0b838 release_2014.08.11 -20f4fdf1735aeff23a6e7ab00389781fe3f3142c latest_2014.08.11 +21f2a42a06d7d2b8c408aed22c691f5e8b209311 latest_2014.08.11 diff -r 818c24105d055f65047595a38ae505d5e45f7520 -r 69bb888b05cd2cbb0f553fe7f2b7c24d05e6ac97 lib/galaxy/tools/__init__.py --- a/lib/galaxy/tools/__init__.py +++ b/lib/galaxy/tools/__init__.py @@ -2538,6 +2538,14 @@ try: current_case = input.get_current_case( value, trans ) except ValueError, e: + if input.is_job_resource_conditional: + # Unless explicitly given job resource parameters + # (e.g. from the run tool form) don't populate the + # state. Along with other hacks prevents workflow + # saving from populating resource defaults - which + # are meant to be much more transient than the rest + # of tool state. + continue #load default initial value if not test_param_error: test_param_error = str( e ) Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

[galaxyproject/usegalaxy-playbook] 73d6de: Dynamic job punting: send jobs to the local cluste...
by GitHub 17 Sep '14

17 Sep '14

Branch: refs/heads/master Home: https://github.com/galaxyproject/usegalaxy-playbook Commit: 73d6dee53f5101add50127a7cb8d38ce3721bee8 https://github.com/galaxyproject/usegalaxy-playbook/commit/73d6dee53f5101ad… Author: Nate Coraor <nate(a)bx.psu.edu> Date: 2014-09-17 (Wed, 17 Sep 2014) Changed paths: M files/galaxy/test.galaxyproject.org/dynamic_rules/roundup_multi_walltime.py M files/galaxy/usegalaxy.org/dynamic_rules/roundup_stampede_select.py Log Message: ----------- Dynamic job punting: send jobs to the local cluster when the user has selected a reference genome from the history, so we don't waste Stampede SUs on single-threaded indexing steps.

1 0

[galaxyproject/usegalaxy-playbook] 7bae64: Update Main
by GitHub 17 Sep '14

17 Sep '14

Branch: refs/heads/master Home: https://github.com/galaxyproject/usegalaxy-playbook Commit: 7bae6424631d25b4999d78dffbc4e91baff95771 https://github.com/galaxyproject/usegalaxy-playbook/commit/7bae6424631d25b4… Author: Nate Coraor <nate(a)bx.psu.edu> Date: 2014-09-17 (Wed, 17 Sep 2014) Changed paths: M production/group_vars/all.yml Log Message: ----------- Update Main

1 0

[galaxyproject/usegalaxy-playbook] 911dd5: We're not using the walltime resubmission on Main ...
by GitHub 17 Sep '14

17 Sep '14

Branch: refs/heads/master Home: https://github.com/galaxyproject/usegalaxy-playbook Commit: 911dd500b2995e955d8d13e2d8074aaf23fe0094 https://github.com/galaxyproject/usegalaxy-playbook/commit/911dd500b2995e95… Author: Nate Coraor <nate(a)bx.psu.edu> Date: 2014-09-17 (Wed, 17 Sep 2014) Changed paths: R files/galaxy/usegalaxy.org/dynamic_rules/roundup_multi_walltime.py M production/group_vars/galaxyservers.yml Log Message: ----------- We're not using the walltime resubmission on Main yet, this dynamic rule will be added back later.

1 0

commit/galaxy-central: jmchilton: Fix saving tool state modifications in workflow editor when resource parameters are enabled.
by commits-noreply＠bitbucket.org 17 Sep '14

17 Sep '14

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/9f495158c5ba/ Changeset: 9f495158c5ba Branch: stable User: jmchilton Date: 2014-09-17 15:56:03+00:00 Summary: Fix saving tool state modifications in workflow editor when resource parameters are enabled. Resource parameter selection is only available at runtime in the tool form - not in the workflow editor. It is fairly important that this state not be associated with the workflow since that would cause odd-ness when exporting one workflows to other deployments or as these unversioned resource parameters evolve over time. This fix addresses a consequence of that form not appearing and goes a step further in preventing this state from being associated with worklfow steps. Thanks to Dan for the bug report. Affected #: 1 file diff -r 2855d1a712c45fe45da7585f1ea05ecbb2a6c2fd -r 9f495158c5ba45fcf61fc37d82aa8545eaba0bd5 lib/galaxy/tools/__init__.py --- a/lib/galaxy/tools/__init__.py +++ b/lib/galaxy/tools/__init__.py @@ -2423,6 +2423,14 @@ try: current_case = input.get_current_case( value, trans ) except ValueError, e: + if input.is_job_resource_conditional: + # Unless explicitly given job resource parameters + # (e.g. from the run tool form) don't populate the + # state. Along with other hacks prevents workflow + # saving from populating resource defaults - which + # are meant to be much more transient than the rest + # of tool state. + continue #load default initial value if not test_param_error: test_param_error = str( e ) Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: carlfeberhard: Client build system, pack_scripts: the build system will now call pack_scripts on all files in static/scripts when called via 'grunt' or 'grunt pack', and pack_scripts automatically only on those that have changed when using 'grunt watch'; add README.txt
by commits-noreply＠bitbucket.org 17 Sep '14

17 Sep '14

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/818c24105d05/ Changeset: 818c24105d05 User: carlfeberhard Date: 2014-09-17 15:28:16+00:00 Summary: Client build system, pack_scripts: the build system will now call pack_scripts on all files in static/scripts when called via 'grunt' or 'grunt pack', and pack_scripts automatically only on those that have changed when using 'grunt watch'; add README.txt Affected #: 3 files diff -r 96c96957d0c6d301371ce1cf7dc40521138df51a -r 818c24105d055f65047595a38ae505d5e45f7520 client/GruntFile.js --- a/client/GruntFile.js +++ b/client/GruntFile.js @@ -19,11 +19,31 @@ } }, + // use 'grunt pack' to call pack_scripts.py to pack all or selected files in static/scripts + exec: { + packScripts: { + cwd: '../static/scripts', + target: [], + cmd: function(){ + var targets = grunt.config( 'exec.packScripts.target' ); + // if nothing was passed in pack all scripts + if( !targets.length ){ + return './pack_scripts.py'; + } + + grunt.log.write( 'packing: ' + targets + '\n' ); + return targets.map( function( target ){ + return './pack_scripts.py ' + target; + }).join( '; ' ); + } + } + }, + // use 'grunt watch' (from a new tab in your terminal) to have grunt re-copy changed files automatically watch: { // watch for changes in the src dir files: [ 'galaxy/scripts/**' ], - tasks: [ 'copy' ], + tasks: [ 'copy', 'pack' ], options: { spawn: false } @@ -32,11 +52,20 @@ grunt.loadNpmTasks( 'grunt-contrib-watch' ); grunt.loadNpmTasks( 'grunt-contrib-copy'); + grunt.loadNpmTasks('grunt-exec'); + grunt.registerTask( 'pack', [ 'exec' ] ); + grunt.registerTask( 'default', [ 'copy', 'pack' ] ); + + // -------------------------------------------------------------------------- copy,pack only those changed // adapted from grunt-contrib-watch jslint example //TODO: a bit hacky and there's prob. a better way //NOTE: copy will fail silently if a file isn't found + + // outer scope variable for the event handler and onChange fn - begin with empty hash var changedFiles = Object.create(null); + + // when files are changed, set the copy src and packScripts target to the filenames of the updated files var onChange = grunt.util._.debounce(function() { grunt.config( 'copy.main.files', [{ expand: true, @@ -44,8 +73,10 @@ src: Object.keys( changedFiles ), dest: '../static/scripts/' }]); + grunt.config( 'exec.packScripts.target', Object.keys( changedFiles ) ); changedFiles = Object.create(null); }, 200); + grunt.event.on('watch', function(action, filepath) { // store each filepath in a Files obj, the debounced fn above will use it as an aggregate list for copying // we need to take galaxy/scripts out of the filepath or it will be copied to the wrong loc @@ -54,5 +85,4 @@ onChange(); }); - grunt.registerTask( 'default', [ 'copy' ] ); }; diff -r 96c96957d0c6d301371ce1cf7dc40521138df51a -r 818c24105d055f65047595a38ae505d5e45f7520 client/README.txt --- /dev/null +++ b/client/README.txt @@ -0,0 +1,42 @@ +Client Build System +=================== + +Builds and moves the client-side scripts necessary for running the Galaxy webapps. There's no need to use this system +unless you are modifying or developing client-side scripts. + +You'll need Node and the Node Package Manager (npm): nodejs.org. + +Once npm is installed, install the grunt task manager and it's command line into your global scope: + + npm install -g grunt grunt-cli + +Next, from within this directory, install the local build dependencies: + + cd client + npm install + +You're now ready to re-build the client scripts after modifying them. + + +Rebuilding +========== + +There are two methods for rebuilding: a complete rebuild and automatic, partial rebuilds while you develop. + +A complete rebuild can be done with the following (from the `client` directory): + + grunt + +This will copy any files in `client/galaxy/scripts` to `static/scripts` and run `static/scripts/pack_scripts.py` on all. + +Grunt can also do an automatic, partial rebuild of any files you change *as you develop* by: + + 1. opening a new terminal session + 2. `cd client` + 3. `grunt watch` + +This starts a new grunt watch process that will monitor the files in `client/galaxy/scripts` for changes and copy and +pack them when they change. + +You can stop the `grunt watch` task by pressing `Ctrl+C`. Note: you should also be able to background that task if you +prefer. diff -r 96c96957d0c6d301371ce1cf7dc40521138df51a -r 818c24105d055f65047595a38ae505d5e45f7520 client/package.json --- a/client/package.json +++ b/client/package.json @@ -8,6 +8,7 @@ "devDependencies": { "grunt": "^0.4.5", "grunt-contrib-copy": "^0.5.0", - "grunt-contrib-watch": "^0.6.1" + "grunt-contrib-watch": "^0.6.1", + "grunt-exec": "^0.4.6" } } Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: dan: Turn off sanitation of __ARCHIVE_SOURCE__ and __DEST_DIR__ in History Import Tool. Secure command-line using base64 encoding. Fixes issue where you couldn't import from a URL that has common characters such as e.g. '~' in it.
by commits-noreply＠bitbucket.org 17 Sep '14

17 Sep '14

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/96c96957d0c6/ Changeset: 96c96957d0c6 User: dan Date: 2014-09-17 15:14:50+00:00 Summary: Turn off sanitation of __ARCHIVE_SOURCE__ and __DEST_DIR__ in History Import Tool. Secure command-line using base64 encoding. Fixes issue where you couldn't import from a URL that has common characters such as e.g. '~' in it. Affected #: 2 files diff -r 3203caa787c1af0ec130995cebca7b7a86a06839 -r 96c96957d0c6d301371ce1cf7dc40521138df51a lib/galaxy/tools/imp_exp/imp_history_from_archive.xml --- a/lib/galaxy/tools/imp_exp/imp_history_from_archive.xml +++ b/lib/galaxy/tools/imp_exp/imp_history_from_archive.xml @@ -1,10 +1,14 @@ <tool id="__IMPORT_HISTORY__" name="Import History" version="0.1" tool_type="import_history"><type class="ImportHistoryTool" module="galaxy.tools"/><action module="galaxy.tools.actions.history_imp_exp" class="ImportHistoryToolAction"/> - <command interpreter="python">unpack_tar_gz_archive.py $__ARCHIVE_SOURCE__ $__DEST_DIR__ --$__ARCHIVE_TYPE__</command> + <command interpreter="python">unpack_tar_gz_archive.py "${ str( $__ARCHIVE_SOURCE__ ).encode( 'base64' ) }" "${ str( $__DEST_DIR__ ).encode( 'base64' ) }" --$__ARCHIVE_TYPE__ --encoded</command><inputs> - <param name="__ARCHIVE_SOURCE__" type="text"/> + <param name="__ARCHIVE_SOURCE__" type="text"> + <sanitizer sanitize="False"/> + </param><param name="__ARCHIVE_TYPE__" type="text"/> - <param name="__DEST_DIR__" type="text"/> + <param name="__DEST_DIR__" type="text"> + <sanitizer sanitize="False"/> + </param></inputs></tool> \ No newline at end of file diff -r 3203caa787c1af0ec130995cebca7b7a86a06839 -r 96c96957d0c6d301371ce1cf7dc40521138df51a lib/galaxy/tools/imp_exp/unpack_tar_gz_archive.py --- a/lib/galaxy/tools/imp_exp/unpack_tar_gz_archive.py +++ b/lib/galaxy/tools/imp_exp/unpack_tar_gz_archive.py @@ -12,6 +12,7 @@ import tempfile import urllib2 import math +from base64 import b64decode # Set max size of archive/file that will be handled to be 100 GB. This is # arbitrary and should be adjusted as needed. @@ -55,11 +56,16 @@ parser = optparse.OptionParser() parser.add_option( '-U', '--url', dest='is_url', action="store_true", help='Source is a URL.' ) parser.add_option( '-F', '--file', dest='is_file', action="store_true", help='Source is a URL.' ) + parser.add_option( '-e', '--encoded', dest='is_b64encoded', action="store_true", default=False, help='Source and destination dir values are base64 encoded.' ) (options, args) = parser.parse_args() is_url = bool( options.is_url ) is_file = bool( options.is_file ) archive_source, dest_dir = args + if options.is_b64encoded: + archive_source = b64decode( archive_source ) + dest_dir = b64decode( dest_dir ) + try: # Get archive from URL. if is_url: Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: natefoo: Provide feedback on job failures due to tools removed from the config now that new state jobs are not recovered when tracking in the databsae (3d92d6f).
by commits-noreply＠bitbucket.org 17 Sep '14

17 Sep '14

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/3203caa787c1/ Changeset: 3203caa787c1 User: natefoo Date: 2014-09-17 14:26:48+00:00 Summary: Provide feedback on job failures due to tools removed from the config now that new state jobs are not recovered when tracking in the databsae (3d92d6f). Affected #: 1 file diff -r 0111073204bca68f004c2830eb1a249f17d5741e -r 3203caa787c1af0ec130995cebca7b7a86a06839 lib/galaxy/jobs/handler.py --- a/lib/galaxy/jobs/handler.py +++ b/lib/galaxy/jobs/handler.py @@ -355,8 +355,13 @@ """ job_destination = None try: + assert job_wrapper.tool is not None, 'This tool was disabled before the job completed. Please contact your Galaxy administrator.' # Cause the job_destination to be set and cached by the mapper job_destination = job_wrapper.job_destination + except AssertionError as e: + log.warning( "(%s) Tool '%s' removed from tool config, unable to run job" % ( job.id, job.tool_id ) ) + job_wrapper.fail( e ) + return JOB_ERROR, job_destination except JobNotReadyException as e: job_state = e.job_state or JOB_WAIT return job_state, None Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0