- galaxy-dev - lists.galaxyproject.org

condor compute cluster
by Shrum, Donald C 06 Aug '14

06 Aug '14

Our galaxy server has an apache front end with ldap authentication. I've added a condor destination... <plugin id="condor" type="runner" load="galaxy.jobs.runners.condor:CondorJobRunner"/> <destination id="condor" runner="condor"/> There are some tools I'd like to point to condor but it appears all the jobs will be sent as the user condor. Is there an existing fix that will submit jobs to our condor cluster as the logged in user? If not could someone give me a small pointer for a hack fix? Something along the lines of: galaxy-dist/lib/galaxy/jobs/runners/util/condor/__init__.py line 71: submit = Popen(('condor_submit', submit_file), stdout=PIPE, stderr=STDOUT) Could be changed to sudo -u xxx -c condor_submit... Thanks all, Don FSU Research Computing Center

2 1

August 2014 Galaxy CloudMan release
by Enis Afgan 06 Aug '14

06 Aug '14

*We just released an update to Galaxy CloudMan.* CloudMan offers an easy way to get a personal and completely functional instance of Galaxy in the cloud in just a few minutes, without any manual configuration. This is mostly an incremental bug fix release with the following summary of changes: - On AWS, updated *galaxyFS* snapshot (snap-e6e1c04a), which includes the June 2, 2014 Galaxy release with the July 30th security fix. All the tools installed via the Tool Shed have been updated and a number of new tools added, most notably: Tophat2, Bowtie2, FastQC, several FASTQ manipulation tools, several QC tools. - For AWS, added support for VPC - For OpenStack clouds, added the ability to automatically recover worker instances on cluster reboot - Added support for creating a file system based on a downloadable archive - Do not run Galaxy with multiple processes by default. This is because Tool Shed installs do not work properly in the multi-process mode. This feature can be enabled by setting user data option configure_multiple_galaxy_processes to True when launching an instance. - Set SGE slots in each queue to be equal to the number of cores on the instance - Set instance IP in the Galaxy's FTP data upload tool message - Added support for Nginx v1.4 and allow it (with the PAM module) to used as the authentication mechanism when accessing Galaxy Reports app - Fixed cluster deletion when performed via the API - No longer automatically start Hadoop and HTCondor services - On manually-invoked instance reboots, do not increment the instance reboot count that otherwise eventually leads to instance termination - Limit the size of the log message buffer used in the UI to 1000 lines. Long-running instances had issues with this log growing large and that led to poor UI performance. The complete log is still available from the Admin page (or the command line). - Automatically delete the bucket/container for Test type (*ie*, 'SGE only') clusters on cluster termination For complete details on implemented changes, please see the source code commits <https://bitbucket.org/galaxy/cloudman/commits/all?search=835%3A903> . Enjoy and please let us know what you think, Enis & Dannon & The Galaxy Team https://wiki.galaxyproject.org/Cloud

1 0

Packaging tools with language specific dependencies
by Renato Alves 06 Aug '14

06 Aug '14

Hi everyone, Is there any standard or commonly used way to package tools that have language specific dependencies. I know that with Python libraries one can use setup_virtualenv and with Java jars the JAVA_JAR_LIB strategy is used. Is there anything equivalent for R, Perl and Ruby libraries? Thanks Renato

2 6

Issues with dependencies for toolshed packages
by Iry Witham 06 Aug '14

06 Aug '14

Hi Team, I have been trying to install package_snpEff_3_5 from the Galaxy main tool shed and package_snpEff_3_6 from the Galaxy test tool shed and have the same issue. Once the installer completes it shows that the dependency snpEff.x.x was not installed. When I try to install the dependency it reports that the directory /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/database/tmp/tmp-toolshed-mtdi48tQV/snpEff does not exist. I have checked to confirm that this is true and discovered that rather then a directory I find a null file named "snpEff and the snpEff_v3.x_core.zip file. I checked my shedtool dependencies folder and the directory for these tools are empty. I have since uninstalled the package. I just tried again to install the snpEff tool and am getting the followig error message: Internal Server Error Galaxy was unable to successfully complete your request URL: http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://too… Module galaxy.web.framework.middleware.error:149 in __call__ >> <http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://too…> app_iter = self.application(environ, sr_checker) Module paste.recursive:84 in __call__ >> <http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://too…> return self.application(environ, start_response) Module paste.httpexceptions:633 in __call__ >> <http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://too…> return self.application(environ, start_response) Module galaxy.web.framework.base:132 in __call__ >> <http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://too…> return self.handle_request( environ, start_response ) Module galaxy.web.framework.base:190 in handle_request >> <http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://too…> body = method( trans, **kwargs ) Module galaxy.web.framework:377 in decorator >> <http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://too…> return func( self, trans, *args, **kwargs ) Module galaxy.webapps.galaxy.controllers.admin_toolshed:1035 in prepare_for_install >> <http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://too…> raw_text = common_util.tool_shed_get( trans.app, tool_shed_url, url ) Module tool_shed.util.common_util:310 in tool_shed_get >> <http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://too…> response = urlopener.open( uri ) Module urllib2:395 in open >> <http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://too…> response = meth(req, response) Module urllib2:508 in http_response >> <http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://too…> 'http', request, response, code, msg, hdrs) Module urllib2:427 in error >> <http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://too…> result = self._call_chain(*args) Module urllib2:367 in _call_chain >> <http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://too…> result = func(*args) Module urllib2:603 in http_error_302 >> <http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://too…> return self.parent.open(new) Module urllib2:395 in open >> <http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://too…> response = meth(req, response) Module urllib2:508 in http_response >> <http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://too…> 'http', request, response, code, msg, hdrs) Module urllib2:433 in error >> <http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://too…> return self._call_chain(*args) Module urllib2:367 in _call_chain >> <http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://too…> result = func(*args) Module urllib2:516 in http_error_default >> <http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://too…> raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) HTTPError: HTTP Error 500: Internal Server Error extra data<http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://too…> full traceback<http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://too…> text version<http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://too…> This may be an intermittent problem due to load or other unpredictable factors, reloading the page may address the problem. I appreciate any possible assistance, Regards, __________________________________ Iry T. Witham Scientific Applications Administrator Scientific Computing Group Computational Sciences Dept. The Jackson Laboratory 600 Main Street Bar Harbor, ME 04609 Phone: 207-288-6744 email: iry.witham(a)jax.org [cid:8C546FFA-BCF0-42A9-8AB5-4F9CE2ADED3C] The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.

3 4

Re: [galaxy-dev] Main Galaxy Tool Shed is running the next-stable branch
by Eric Kuyt 06 Aug '14

06 Aug '14

Hi Greg, Could there be some problems with https://toolshed.g2.bx.psu.edu/ I have some troubles installing toolshed tools in our galaxy. When I try installing for instance 'suite_samtools_0_1_19', I am presented with a 500 error. I tried then pulling a new stable galaxy from bitbucket, which seems to have the same problem. Maybe I'm doing something wrong both times or toolshed is having problems. I opt for the latter of course. -- part of galaxy traceback -- File '/usr/lib/python2.7/urllib2.py', line 527 in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) HTTPError: HTTP Error 500: Internal Server Error Thanks, Eric On 29 July 2014 23:35, Greg Von Kuster <greg(a)bx.psu.edu> wrote: > Hello all, > > The main Galaxy Tool Shed is now running the next-stable branch ( > e6813f244a2a (next-stable) tip ) in preparation for next upcoming Galaxy > release. Please let us know if you encounter anything strange or broken > and we'll get things fixed asap. The main Tool Shed will be updated to the > stable branch when it is created for the upcoming Galaxy release. > > Thanks very much, > > Greg Von Kuster > > > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ >

2 1

error using get( $__user_id__ ) in xml file in new Galaxy
by Neil.Burdett＠csiro.au 06 Aug '14

06 Aug '14

Hi, I have recently upgraded Galaxy from a version I had last year. In my tool xml file I could state: ${ __app__.model.User.get( $__user_id__ ).api_keys[0].key } in the <command> section to get the user's API. However, after updating to the current version of Galaxy I get the following error when using this entry in the xml file: Traceback (most recent call last): File "/home/bmladmin/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 153, in prepare_job job_wrapper.prepare() File "/home/bmladmin/galaxy-dist/lib/galaxy/jobs/__init__.py", line 712, in prepare self.command_line, self.extra_filenames = tool_evaluator.build() File "/home/bmladmin/galaxy-dist/lib/galaxy/tools/evaluation.py", line 348, in build self.__build_command_line( ) File "/home/bmladmin/galaxy-dist/lib/galaxy/tools/evaluation.py", line 364, in __build_command_line command_line = fill_template( command, context=param_dict ) File "/home/bmladmin/galaxy-dist/lib/galaxy/util/template.py", line 9, in fill_template return str( Template( source=template_text, searchList=[context] ) ) File "/home/bmladmin/galaxy-dist/eggs/Cheetah-2.2.2-py2.7-linux-x86_64-ucs4.egg/Cheetah/Template.py", line 1004, in __str__ return getattr(self, mainMethName)() File "cheetah_DynamicallyCompiledCheetahTemplate_1407207170_7_53169.py", line 92, in respond NotFound: cannot find 'get' while searching for 'get' Any ideas? Thanks Neil

3 3

Serious start-up delay in next-stable
by Björn Grüning 06 Aug '14

06 Aug '14

Hi all, does anyone encountered a long start-up delay with the recent next-stable and starting a instance from scratch? For me its hanging at: tool_shed.galaxy_install.migrate.check DEBUG 2014-08-05 18:39:45,207 psycopg2 egg successfully loaded for postgres dialect for 3-5 minutes. Cheers, Bjoern

5 4

samtools installation error
by Beginner TI 05 Aug '14

05 Aug '14

1 0

Re: [galaxy-dev] Galaxy Security Vulnerability
by Dooley, Damion 05 Aug '14

05 Aug '14

Does this apply to all past galaxy installs? I have an older galaxy site I've been wanting to phase out rather than upgrade. For now I'd like to use a patch but site version (parent: 7148:17d57db9a7c0 ) predates any of the tags. I presume I'd have to just implement the patch by hand? Regards, Damion Message: 7 Date: Thu, 31 Jul 2014 14:55:57 -0400 From: Nate Coraor <nate(a)bx.psu.edu> To: Galaxy Development <galaxy-dev(a)lists.bx.psu.edu>, galaxy-announce(a)lists.bx.psu.edu Subject: [galaxy-dev] Galaxy Security Vulnerability Message-ID: <D482333D-384E-49C8-8DD8-C752E4B0AF76(a)bx.psu.edu> Content-Type: text/plain; charset="us-ascii" A security vulnerability was recently discovered by Inge Alexander Raknes that would allow a malicious person to execute arbitrary code on a Galaxy server. The vulnerability was in a method that uses Python "pickle" functionality to decode state information from tool forms. Because pickles can be used to instantiate arbitrary Python objects, tool states could be constructed to exploit this vulnerability. ... - pickle-2013.01.13.patch - This patch should apply cleanly (with offset/fuzz) to releases from 2013.01.13 up to 2013.08.12, and possibly older versions of Galaxy as well. Available at: https://depot.galaxyproject.org/patch/pickle-2013.01.13.patch

2 1

job not starting on cluster
by Langhorst, Brad 05 Aug '14

05 Aug '14

I’m not aware of any recent changes to the configuration of galaxy… It seems that jobs are not being distributed at the moment. I have 2 job handlers... It seems that when I restart them only a single job is queued, it has an error and then sticks on "Cleaning up external metadata files” forever. any ideas? I have 700+ jobs in the queue and would like to get them running asap... one hanlder just stays at Starting server in PID 4868. serving on http://127.0.0.1:8090 here’s the log from the other. Starting server in PID 4873. serving on http://127.0.0.1:8091 galaxy.jobs DEBUG 2014-08-05 17:34:08,388 (125206) Working directory for job is: /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206 galaxy.jobs.handler DEBUG 2014-08-05 17:34:08,410 (125206) Dispatching to gridengine runner galaxy.jobs DEBUG 2014-08-05 17:34:08,479 (125206) Persisting job destination (destination id: gridengine) galaxy.jobs.handler INFO 2014-08-05 17:34:08,498 (125206) Job dispatched galaxy.tools.deps DEBUG 2014-08-05 17:34:08,862 Building dependency shell command for dependency 'bedtools2' galaxy.tools.deps WARNING 2014-08-05 17:34:08,865 Failed to resolve dependency on 'bedtools2', ignoring galaxy.jobs.runners DEBUG 2014-08-05 17:34:08,966 (125206) command is: bedtools --version > /mnt/galaxy/data/galaxy/tmp/GALAXY_VERSION_STRING_125206 2>&1; bedtools intersect -v -a /mnt/galaxy/data/galaxy/user-data/000/170/dataset_170995.dat -b /mnt/galaxy/data/galaxy/user-data/000/169/dataset_169046.dat > "/mnt/galaxy/data/galaxy/user-data/000/170/dataset_170999.dat"; return_code=$?; cd /mnt/galaxy/data/galaxy/galaxy-dist; /mnt/galaxy/data/galaxy/galaxy-dist/set_metadata.sh /mnt/galaxy/data/galaxy/user-data /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206 . /mnt/galaxy/data/galaxy/galaxy-dist/universe_wsgi.ini /mnt/galaxy/data/galaxy/tmp/tmpLCBPvF /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/galaxy.json /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_in_HistoryDatasetAssociation_194826_qIy33T,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_kwds_HistoryDatasetAssociation_194826_SJhToq,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_out_HistoryDatasetAssociation_194826_4Lf956,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_results_HistoryDatasetAssociation_194826_YzEZK8,,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_override_HistoryDatasetAssociation_194826_19nZAm; sh -c "exit $return_code" galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:08,994 (125206) submitting file /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/galaxy_125206.sh galaxy.jobs.runners.drmaa INFO 2014-08-05 17:34:09,023 (125206) queued as 1539118 galaxy.jobs DEBUG 2014-08-05 17:34:09,048 (125206) Persisting job destination (destination id: gridengine) galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:09,451 (125206/1539118) state change: job is queued and active galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:11,510 (125206/1539118) state change: job is running galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:31,208 (125206/1539118) state change: job finished normally galaxy.jobs DEBUG 2014-08-05 17:34:31,440 setting dataset state to ERROR galaxy.jobs INFO 2014-08-05 17:34:31,652 Collecting job metrics for <galaxy.model.Job object at 0x7f42e030f850> galaxy.jobs DEBUG 2014-08-05 17:34:31,664 job 125206 ended galaxy.datatypes.metadata DEBUG 2014-08-05 17:34:31,665 Cleaning up external metadata files -- Bradley W. Langhorst, Ph.D. Applications and Product Development Scientist

2 1