- galaxy-dev - lists.galaxyproject.org

uWSGI threading issue
by Durdevic, Marija 10 Jul '18

10 Jul '18

Dear developers, Since last Galaxy upgrade some issue with uwsgi and threading arise. I used Galaxy documentation<https://docs.galaxyproject.org/en/master/admin/scaling.html> as a guideline to configure galaxy.yml: uwsgi: socket: 127.0.0.1:4001 http: 0.0.0.0:8080 buffer-size: 16384 processes: 4 threads: 8 offload-threads: 2 static-map: /static/style=static/style/blue static-map: /static=static master: true virtualenv: .venv pythonpath: lib module: galaxy.webapps.galaxy.buildapp:uwsgi_app() thunder-lock: true die-on-term: true hook-master-start: unix_signal:2 gracefully_kill_them_all hook-master-start: unix_signal:15 gracefully_kill_them_all py-call-osafterfork: true enable-threads: true mule: lib/galaxy/main.py mule: lib/galaxy/main.py mule: lib/galaxy/main.py farm: job-handlers:1,2,3 When I run Galaxy, I got error message - error: can't start new thread Also if I run some job which require a lot of threads, I always get error message - RuntimeError: can't start new thread I tried to increase number of processes in /etc/security/limits.conf for galaxy user but it didn’t help. What can be solution? Do you have any idea what could be wrong? Thanks a lot for your support. Best, Marija Marija Đurđević PhD, Diagnostic & Research Center for Molecular BioMedicine Institute of Pathology Bioinformatician, Core Facility Computational Bioanalytics Center for Medical Research Medical University of Graz Stiftingtalstraße 24, A-8010 Graz Austria Phone: +43 316/385-73024 Fax:+43 316/385-73009 Email: marija.durdevic(a)medunigraz.at<mailto:marija.durdevic@medunigraz.at> Email: marija.djurdjevic(a)klinikum-graz.at<mailto:marija.djurdjevic@klinikum-graz.at> Web: https://zmf.medunigraz.at<https://zmf.medunigraz.at/>

1 0

From Paste to uWSGI
by Jochen Bick 10 Jul '18

10 Jul '18

Hi all, we would like to update to 18.05 and thereby also update our Paste Server. https://docs.galaxyproject.org/en/latest/admin/scaling.html Is there any tutorial on how to update from Paste to uWSGI? Cheers Jochen -- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 / LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland Phone +41 52 354 92 06 jochen.bick(a)usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch

3 3

Re: [galaxy-dev] Conda problems: Failed to activate conda environment
by Matthias Bernt 06 Jul '18

06 Jul '18

Dear Gerhard, I don't know about the python error.. could you tell us the output of `locale` when logged in as the user running the Galaxy server? For the second problem, I remember that I also have a similar problem. For some reasons some symlinks in the conda environment are not created. you can check if in the respective environment's bin directory symlinks to the conda, activate and deactivate binaries/scripts exist. If not you can fix the missing links (I run the following after each tool installation) ``` for env in "{{ galaxy_server_dir }}"/database/dependencies/_conda/envs/*; do for link in conda activate deactivate; do \[ -h $env/bin/$link \] || ln -s "{{ galaxy_server_dir }}"/database/dependencies/_conda/bin/$link $env/bin/$link; done done ``` Hope this helps. Cheers, Matthias On 05.07.2018 18:00, galaxy-dev-request(a)lists.galaxyproject.org wrote: > Message: 1 > Date: Thu, 5 Jul 2018 08:14:35 +0000 > From: "Van Wageningen, GERHARD [gerhardv(a)sun.ac.za]" > <gerhardv(a)sun.ac.za> > To:"galaxy-dev(a)lists.galaxyproject.org" > <galaxy-dev(a)lists.galaxyproject.org> > Subject: [galaxy-dev] Conda problems: Failed to activate conda > environment > Message-ID: > <AM0PR07MB4354EEAD1AFB5809409F9F3692400(a)AM0PR07MB4354.eurprd07.prod.outlook.com> > > Content-Type: text/plain; charset="iso-8859-1" > > Hi All > > I have recently started learning the Galaxy setup starting with an upgrade. > Made some headway but I was wondering if someone could comment on a problem that has emerged. > The system is a Galaxy 18.05 installation (upgraded from 17.01) on an Altair PBS scheduler with drmaa setup (working) > > Conda problem: Failed to activate conda environment. > > Conda works during tool installation or reinstallation. > > Galaxy tools have been updated/reinstalled so they all look "green" in manage tools. > > In Manage Dependencies there are a number of "Dependency resolved but version not found" warnings, otherwise the deps look clean. > > When running tools in Galaxy errors occur, after the jobs have correctly deployed on the cluster. > > > The galaxy application raised errors like the two below. > > "Failed to activate conda environment! Error was: > Fatal Python error: Py_Initialize: Unable to get the locale encoding > File "/home/.galaxy/tooldeps/python/2.7/iuc/package_python_2_7/8b09fe018cac/lib/python2.7/encodings/__init__.py", line 123 > raise C" > > > " > > Failed to activate conda environment! Error was: CondaEnvironmentError: Environment error: Cannot activate environment bash. User does not have write access for conda symlinks. > > > " > > > Galaxy started and stopped using supervisord with config below: -- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt(a)ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann -------------------------------------------

2 3

Strange error in galaxy
by Evan Clark 27 Jun '18

27 Jun '18

Has anyone seen this error in galaxy before? I'm not sure exactly what it indicates. Traceback (most recent call last): File "/mnt/ceph/home/galaxy/galaxy/tools/data_source/upload.py", line 425, in<module> __main__() File "/mnt/ceph/home/galaxy/galaxy/tools/data_source/upload.py", line 413, in __main__ add_file( dataset, registry, json_file, output_path ) File "/mnt/ceph/home/galaxy/galaxy/tools/data_source/upload.py", line 325, in add_file shutil.move( dataset.path, output_path ) File "/usr/lib64/python2.7/shutil.py", line 301, in move copy2(src, real_dst) File "/usr/lib64/python2.7/shutil.py", line 130, in copy2 copyfile(src, dst) File "/usr/lib64/python2.7/shutil.py", line 84, in copyfile copyfileobj(fsrc, fdst) File "/usr/lib64/python2.7/shutil.py", line 52, in copyfileobj fdst.write(buf) IOError: [Errno 4] Interrupted system call

1 0

REST API api/tools
by Dmitry Repchevsky 27 Jun '18

27 Jun '18

Hello, It looks like there where changes in the data returned by the API. I used to get "citations" and "requirements" properties from GET /api/tools JSON... How can I get bioconda packeges for the Galaxy tool? Thank you, Dmitry

1 0

jobs not running with galaxy.yml
by Matthias Bernt 27 Jun '18

27 Jun '18

Dear list, I'm trying to switch from galaxy.ini to yml on my dev instance (using a drmaa runner starting jobs as real user). Somehow my jobs are not starting anymore (it seems that they are not given to the job handler). The only output I get is: galaxy.tools.actions INFO 2018-06-27 13:11:37,445 [p:19863,w:1,m:0] [uWSGIWorker1Core2] Handled output named output for tool idba_ud (25965.752 ms) galaxy.tools.actions INFO 2018-06-27 13:11:37,459 [p:19863,w:1,m:0] [uWSGIWorker1Core2] Added output datasets to history (13.657 ms) galaxy.tools.actions INFO 2018-06-27 13:11:37,484 [p:19863,w:1,m:0] [uWSGIWorker1Core2] Verified access to datasets for Job[unflushed,tool_id=idba_ud] (5.423 ms) galaxy.tools.actions INFO 2018-06-27 13:11:37,484 [p:19863,w:1,m:0] [uWSGIWorker1Core2] Setup for job Job[unflushed,tool_id=idba_ud] complete, ready to flush (25.058 ms) galaxy.tools.actions INFO 2018-06-27 13:11:37,522 [p:19863,w:1,m:0] [uWSGIWorker1Core2] Flushed transaction for job Job[id=4366,tool_id=idba_ud] (37.777 ms) galaxy.tools.execute DEBUG 2018-06-27 13:11:37,522 [p:19863,w:1,m:0] [uWSGIWorker1Core2] Tool [idba_ud] created job [4366] (26050.612 ms) galaxy.tools.execute DEBUG 2018-06-27 13:11:37,538 [p:19863,w:1,m:0] [uWSGIWorker1Core2] Executed 1 job(s) for tool idba_ud request: (26075.593 ms) In addition to manual conversion I tried the auto conversion using `make config-convert` with the same result. I also tried to validate the yml file: ``` make config-validate ``` but get only: ``` pykwalify.errors.RuleError: <RuleError: error code 4: Types do not match: 'None' --> '<type 'str'>': Path: '/mapping/galaxy/mapping/error_email_to'> ``` Any ideas where I could start digging? Cheers, Matthias -- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt(a)ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann -------------------------------------------

1 0

MySQL errors and warnings with a fresh install of Galaxy 18-05
by SAPET, Frederic 26 Jun '18

26 Jun '18

Hi, I try to install galaxy 18-05. My database backend is MysqL (5.7.11-4 Percona Server) I start from scratch (drop database, new directory with galaxy files fetched with git) So the database is first created when run.sh is used : : Creating database for URI [mysql://.... I see warnings about duplicated indexes and OperationalError from SQLAlchemy. Some tables are then missing in my database (workflow_invocation_output_dataset_collection_association and workflow_invocation_output_dataset_collection_association). And fields are still present (ie in request table : library_id and folder_id) Could you please help me ? Thank you. Fred >grep "^OperationalError:" FirstRun.log | cut -f 2- -d "," "Cannot drop index 'ix_request_library_id': needed in a foreign key constraint") [SQL: u'\nALTER TABLE request DROP COLUMN library_id'] (Background on this error at: http://sqlalche.me/e/e3q8) "Cannot drop index 'ix_request_folder_id': needed in a foreign key constraint") [SQL: u'\nALTER TABLE request DROP COLUMN folder_id'] (Background on this error at: http://sqlalche.me/e/e3q8) "BLOB/TEXT column 'annotation' used in key specification without a key length") [SQL: u'CREATE INDEX ix_page_annotation_association_annotation ON page_annotation_association (annotation)'] (Background on this error at: http://sqlalche.me/e/e3q8) "Cannot drop index 'sequencer_id': needed in a foreign key constraint") [SQL: u'\nALTER TABLE request_type DROP COLUMN sequencer_id'] (Background on this error at: http://sqlalche.me/e/e3q8) "BLOB/TEXT column 'guid' used in key specification without a key length") [SQL: u'CREATE UNIQUE INDEX ix_tool_id_guid_map_guid ON tool_id_guid_map (guid)'] (Background on this error at: http://sqlalche.me/e/e3q8) "BLOB/TEXT column 'layout' used in key specification without a key length") [SQL: u'CREATE INDEX ix_form_definition_layout ON form_definition (layout)'] (Background on this error at: http://sqlalche.me/e/e3q8) "BLOB/TEXT column 'notification' used in key specification without a key length") [SQL: u'CREATE INDEX ix_request_notification ON request (notification)'] (Background on this error at: http://sqlalche.me/e/e3q8) "BLOB/TEXT column 'data_manager_id' used in key specification without a key length") [SQL: u'CREATE INDEX ix_data_manager_job_association_data_manager_id ON data_manager_job_association (data_manager_id)'] (Background on this error at: http://sqlalche.me/e/e3q8) "Identifier name 'workflow_invocation_output_dataset_collection_association_ibfk_1' is too long") [SQL: u'\nCREATE TABLE workflow_invocation_output_dataset_collection_association (\n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tworkflow_invocation_id INTEGER, \n\tworkflow_step_id INTEGER, \n\tdataset_collection_id INTEGER, \n\tworkflow_output_id INTEGER, \n\tPRIMARY KEY (id), \n\tFOREIGN KEY(workflow_invocation_id) REFERENCES workflow_invocation (id), \n\tFOREIGN KEY(workflow_step_id) REFERENCES workflow_step (id), \n\tFOREIGN KEY(dataset_collection_id) REFERENCES history_dataset_collection_association (id), \n\tFOREIGN KEY(workflow_output_id) REFERENCES workflow_output (id)\n)\n\n'] (Background on this error at: http://sqlalche.me/e/e3q8) "Identifier name 'workflow_invocation_step_output_dataset_collection_association_ibfk_1' is too long") [SQL: u'\nCREATE TABLE workflow_invocation_step_output_dataset_collection_association (\n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tworkflow_invocation_step_id INTEGER, \n\tworkflow_step_id INTEGER, \n\tdataset_collection_id INTEGER, \n\toutput_name VARCHAR(255), \n\tPRIMARY KEY (id), \n\tFOREIGN KEY(workflow_invocation_step_id) REFERENCES workflow_invocation_step (id), \n\tFOREIGN KEY(workflow_step_id) REFERENCES workflow_step (id), \n\tFOREIGN KEY(dataset_collection_id) REFERENCES history_dataset_collection_association (id)\n)\n\n'] (Background on this error at: http://sqlalche.me/e/e3q8) >grep "Warning" FirstRun.log | cut -f 3,4 -d ":" Warning: Duplicate index 'ix_galaxy_user_deleted' defined on the table 'galaxydbBGAdev.galaxy_user'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_galaxy_user_purged' defined on the table 'galaxydbBGAdev.galaxy_user'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_hdadaa_history_dataset_association_id' defined on the table 'galaxydbBGAdev.history_dataset_association_display_at_authorization'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_lddaia_ldda_id' defined on the table 'galaxydbBGAdev.library_dataset_dataset_info_association'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_hda_ta_history_dataset_association_id' defined on the table 'galaxydbBGAdev.history_dataset_association_tag_association'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_galaxy_user_username' defined on the table 'galaxydbBGAdev.galaxy_user'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_galaxy_user_form_values_id' defined on the table 'galaxydbBGAdev.galaxy_user'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_lia_deleted' defined on the table 'galaxydbBGAdev.library_info_association'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_lfia_deleted' defined on the table 'galaxydbBGAdev.library_folder_info_association'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_lddia_deleted' defined on the table 'galaxydbBGAdev.library_dataset_dataset_info_association'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_sample_folder_id' defined on the table 'galaxydbBGAdev.sample'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_lia_inheritable' defined on the table 'galaxydbBGAdev.library_info_association'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_lfia_inheritable' defined on the table 'galaxydbBGAdev.library_folder_info_association'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_sd_external_service_id' defined on the table 'galaxydbBGAdev.sample_dataset'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_icda_ldda_parent_id' defined on the table 'galaxydbBGAdev.implicitly_converted_dataset_association'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_icda_ldda_id' defined on the table 'galaxydbBGAdev.implicitly_converted_dataset_association'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_tsr_includes_datatypes' defined on the table 'galaxydbBGAdev.tool_shed_repository'. This is deprecated and will be disallowed in a future release. >grep ERROR FirstRun.log | grep failed 0037_samples_library ERROR 2018-06-13 12:01:45,817 [p:17852,w:0,m:0] [MainThread] Deleting column 'library_id' to request table failed. 0037_samples_library ERROR 2018-06-13 12:01:45,832 [p:17852,w:0,m:0] [MainThread] Deleting column 'folder_id' to request table failed. galaxy.model.migrate.check INFO 2018-06-13 12:01:47,129 [p:17852,w:0,m:0] [MainThread] 0037_samples_library ERROR 2018-06-13 12:01:45,817 Deleting column 'library_id' to request table failed. galaxy.model.migrate.check INFO 2018-06-13 12:01:47,133 [p:17852,w:0,m:0] [MainThread] 0037_samples_library ERROR 2018-06-13 12:01:45,817 Deleting column 'library_id' to request table failed. galaxy.model.migrate.check INFO 2018-06-13 12:01:47,137 [p:17852,w:0,m:0] [MainThread] 0037_samples_library ERROR 2018-06-13 12:01:45,832 Deleting column 'folder_id' to request table failed. galaxy.model.migrate.check INFO 2018-06-13 12:01:47,142 [p:17852,w:0,m:0] [MainThread] 0037_samples_library ERROR 2018-06-13 12:01:45,832 Deleting column 'folder_id' to request table failed. 0040_page_annotations ERROR 2018-06-13 12:01:49,598 [p:17852,w:0,m:0] [MainThread] Creating page_annotation_association table failed. 0068_rename_sequencer_to_external_services ERROR 2018-06-13 12:02:26,658 [p:17852,w:0,m:0] [MainThread] Deleting column 'sequencer_id' from the 'request_type' table failed. 0087_tool_id_guid_map_table ERROR 2018-06-13 12:02:48,300 [p:17852,w:0,m:0] [MainThread] Creating tool_id_guid_map table failed. galaxy.model.migrate.check INFO 2018-06-13 12:02:48,309 [p:17852,w:0,m:0] [MainThread] 0087_tool_id_guid_map_table ERROR 2018-06-13 12:02:48,300 Creating tool_id_guid_map table failed. galaxy.model.migrate.check INFO 2018-06-13 12:02:48,313 [p:17852,w:0,m:0] [MainThread] 0087_tool_id_guid_map_table ERROR 2018-06-13 12:02:48,300 Creating tool_id_guid_map table failed. 0112_add_data_manager_history_association_and_data_manager_job_association_tables ERROR 2018-06-13 12:03:22,252 [p:17852,w:0,m:0] [MainThread] Creating data_manager_job_association table failed. 0136_collection_and_workflow_state ERROR 2018-06-13 12:03:57,766 [p:17852,w:0,m:0] [MainThread] Creating workflow_invocation_output_dataset_collection_association table failed. 0136_collection_and_workflow_state ERROR 2018-06-13 12:03:58,326 [p:17852,w:0,m:0] [MainThread] Creating workflow_invocation_step_output_dataset_collection_association table failed.

3 4

instance management inspiration
by Matthias Bernt 22 Jun '18

22 Jun '18

Dear list, I'm looking for some inspiration for managing my galaxy instance (with ansible). In particular I'm looking for ansible roles to - setup quotas - manage the tool panel (I need to get order in the chaos) I'm wondering how the usegalaxy instances manage this? Is there some github that I could browse? Cheers, Matthias -- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt(a)ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann -------------------------------------------

2 3

pbs/torque jobs
by Briand, Sheldon (NRC/CNRC) 14 Jun '18

14 Jun '18

Hi, I have upgraded to Galaxy 18.05 (from 17.05). I am running a torque job scheduler (version 6.02). When I submit a job through galaxy and I look in the admin/manage jobs section I see that the job has been submitted successfully. It shows that the job is queued and waiting to run. On the cluster I see that the job runs and goes to completion. However, the status in galaxy continue to show that the job is waiting to run and the status never gets updated. I'm using postgres as my database and I am running through a nginx proxy. I see no errors in my galaxy.log file. I switched from galaxy.ini to galaxy.yml and from paste to uwsgi. Is this a configuration problem on my end? Where should I be looking? Thanks, -Sheldon Sheldon Briand Computer Systems and Applications Analyst National Research Council/Government of Canada Sheldon.briand(a)canada.ca/<mailto:Sheldon.briand@canada.ca/> Tel: (902) 426-1677

2 10

Conflicting conda dependencies for Galaxy tool
by Peter Briggs 14 Jun '18

14 Jun '18

Dear devs I'm currently trying to update a tool developed to wrap an in-house scripted pipeline written by a local researcher, so that it gets its dependencies from conda. The pipeline uses a variety of third-party software packages (many of which are now at "legacy" versions), and I've hit a problem where two of these packages turn out to have conflicting dependencies. Specifically the issue is with qiime=1.8.0, which it turns out is not compatible with any version of R which is available in the channels used by Galaxy. So when Galaxy builds the conda environment for the tool at execution time, it fails with an error from conda: UnsatisfiableError: The following specifications were found to be in conflict: - cairo==1.14.10=0 -> freetype==2.8.1 - cairo==1.14.10=0 -> pixman=0.34 - py2cairo==1.10.0=py27_1 -> cairo==1.12.2 -> freetype=2.4 - py2cairo==1.10.0=py27_1 -> cairo==1.12.2 -> pixman==0.26.2 (The py2cairo requirement comes from an older version of matplotlib that qiime 1.8.0 pulls in.) It's a long shot, but: does Galaxy offer any options for working around conflicting conda dependencies like these? Or is the only option to try and resolve them at the conda packaging level (which might not be practical, or even possible)? Thanks in advance for any advice, Best wishes Peter -- Peter Briggs peter.briggs(a)manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482

1 0