June 2018 - galaxy-dev - lists.galaxyproject.org

Monitoring Dashboard for Galaxy
by evan clark 03 Sep '18

03 Sep '18

I remember there previously being an additional tool that ran on a different port from galaxy that allowed for monitoring of performance and jobs running. Is this tool still packaged with galaxy and if so how can it be activated.

3 3

Unable to install tools from toolshed/mercurial issue in release_18.05 instance
by Peter Briggs 16 Aug '18

16 Aug '18

Dear devs I've encountered what appears to be a subtle bug with release 18.05, which breaks the installation of tools from the toolshed, and appears to be a result of not having mercurial (hg) available in /usr/bin on the system that Galaxy is installed on (in this case Scientific Linux 6.5). When attempting to install a tool (e.g. devteam/fastqc) from the main toolshed via the admin interface, after clicking "install" the tool installation status goes immediately to "Error". The tool repository isn't cloned to "shed_tools" and no dependencies are installed. I've been unable to find any error messages in the logs. However, attempting to install via the API does return the message: Error cloning repository: [Errno 2] No such file or directory which comes from the "clone_repository" function in lib/tool_shed/util/hg_util.py (when something goes wrong with the "hg clone ..." command). Installing Mercurial 1.3 via yum on the server and attempting tool installation again gives a slightly different error via the API: Error cloning repository: Command '['hg', 'clone', '-r', u'17', u'https://toolshed.g2.bx.psu.edu/repos/devteam/fastqc', u'/XXXXXXXXXXXXXX/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/fastqc/c15237684a01/fastqc']' returned non-zero exit status 255 Output was: abort: No such file or directory: /XXXXXXXXXXXXXXXXXX/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/fastqc/c15237684a01/fastqc Uninstalling the system Mercurial and instead installing version 3.7.3 and making a link from /usr/bin/hg seems to fix the problem, and tools can be installed without problems. I couldn't find any evidence of this being reported before, and I don't know if I've missed some configuration detail which means that Galaxy isn't picking up hg from its virtualenv instead of /usr/bin. Has anyone else encountered this problem? and is there a fix/workaround for it (other than horrible links to /usr/bin/hg)? Any advice gratefully received! Best wishes Peter -- Peter Briggs peter.briggs(a)manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482

3 3

Strange error in galaxy
by Evan Clark 27 Jun '18

27 Jun '18

Has anyone seen this error in galaxy before? I'm not sure exactly what it indicates. Traceback (most recent call last): File "/mnt/ceph/home/galaxy/galaxy/tools/data_source/upload.py", line 425, in<module> __main__() File "/mnt/ceph/home/galaxy/galaxy/tools/data_source/upload.py", line 413, in __main__ add_file( dataset, registry, json_file, output_path ) File "/mnt/ceph/home/galaxy/galaxy/tools/data_source/upload.py", line 325, in add_file shutil.move( dataset.path, output_path ) File "/usr/lib64/python2.7/shutil.py", line 301, in move copy2(src, real_dst) File "/usr/lib64/python2.7/shutil.py", line 130, in copy2 copyfile(src, dst) File "/usr/lib64/python2.7/shutil.py", line 84, in copyfile copyfileobj(fsrc, fdst) File "/usr/lib64/python2.7/shutil.py", line 52, in copyfileobj fdst.write(buf) IOError: [Errno 4] Interrupted system call

1 0

REST API api/tools
by Dmitry Repchevsky 27 Jun '18

27 Jun '18

Hello, It looks like there where changes in the data returned by the API. I used to get "citations" and "requirements" properties from GET /api/tools JSON... How can I get bioconda packeges for the Galaxy tool? Thank you, Dmitry

1 0

jobs not running with galaxy.yml
by Matthias Bernt 27 Jun '18

27 Jun '18

Dear list, I'm trying to switch from galaxy.ini to yml on my dev instance (using a drmaa runner starting jobs as real user). Somehow my jobs are not starting anymore (it seems that they are not given to the job handler). The only output I get is: galaxy.tools.actions INFO 2018-06-27 13:11:37,445 [p:19863,w:1,m:0] [uWSGIWorker1Core2] Handled output named output for tool idba_ud (25965.752 ms) galaxy.tools.actions INFO 2018-06-27 13:11:37,459 [p:19863,w:1,m:0] [uWSGIWorker1Core2] Added output datasets to history (13.657 ms) galaxy.tools.actions INFO 2018-06-27 13:11:37,484 [p:19863,w:1,m:0] [uWSGIWorker1Core2] Verified access to datasets for Job[unflushed,tool_id=idba_ud] (5.423 ms) galaxy.tools.actions INFO 2018-06-27 13:11:37,484 [p:19863,w:1,m:0] [uWSGIWorker1Core2] Setup for job Job[unflushed,tool_id=idba_ud] complete, ready to flush (25.058 ms) galaxy.tools.actions INFO 2018-06-27 13:11:37,522 [p:19863,w:1,m:0] [uWSGIWorker1Core2] Flushed transaction for job Job[id=4366,tool_id=idba_ud] (37.777 ms) galaxy.tools.execute DEBUG 2018-06-27 13:11:37,522 [p:19863,w:1,m:0] [uWSGIWorker1Core2] Tool [idba_ud] created job [4366] (26050.612 ms) galaxy.tools.execute DEBUG 2018-06-27 13:11:37,538 [p:19863,w:1,m:0] [uWSGIWorker1Core2] Executed 1 job(s) for tool idba_ud request: (26075.593 ms) In addition to manual conversion I tried the auto conversion using `make config-convert` with the same result. I also tried to validate the yml file: ``` make config-validate ``` but get only: ``` pykwalify.errors.RuleError: <RuleError: error code 4: Types do not match: 'None' --> '<type 'str'>': Path: '/mapping/galaxy/mapping/error_email_to'> ``` Any ideas where I could start digging? Cheers, Matthias -- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt(a)ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann -------------------------------------------

1 0

MySQL errors and warnings with a fresh install of Galaxy 18-05
by SAPET, Frederic 26 Jun '18

26 Jun '18

Hi, I try to install galaxy 18-05. My database backend is MysqL (5.7.11-4 Percona Server) I start from scratch (drop database, new directory with galaxy files fetched with git) So the database is first created when run.sh is used : : Creating database for URI [mysql://.... I see warnings about duplicated indexes and OperationalError from SQLAlchemy. Some tables are then missing in my database (workflow_invocation_output_dataset_collection_association and workflow_invocation_output_dataset_collection_association). And fields are still present (ie in request table : library_id and folder_id) Could you please help me ? Thank you. Fred >grep "^OperationalError:" FirstRun.log | cut -f 2- -d "," "Cannot drop index 'ix_request_library_id': needed in a foreign key constraint") [SQL: u'\nALTER TABLE request DROP COLUMN library_id'] (Background on this error at: http://sqlalche.me/e/e3q8) "Cannot drop index 'ix_request_folder_id': needed in a foreign key constraint") [SQL: u'\nALTER TABLE request DROP COLUMN folder_id'] (Background on this error at: http://sqlalche.me/e/e3q8) "BLOB/TEXT column 'annotation' used in key specification without a key length") [SQL: u'CREATE INDEX ix_page_annotation_association_annotation ON page_annotation_association (annotation)'] (Background on this error at: http://sqlalche.me/e/e3q8) "Cannot drop index 'sequencer_id': needed in a foreign key constraint") [SQL: u'\nALTER TABLE request_type DROP COLUMN sequencer_id'] (Background on this error at: http://sqlalche.me/e/e3q8) "BLOB/TEXT column 'guid' used in key specification without a key length") [SQL: u'CREATE UNIQUE INDEX ix_tool_id_guid_map_guid ON tool_id_guid_map (guid)'] (Background on this error at: http://sqlalche.me/e/e3q8) "BLOB/TEXT column 'layout' used in key specification without a key length") [SQL: u'CREATE INDEX ix_form_definition_layout ON form_definition (layout)'] (Background on this error at: http://sqlalche.me/e/e3q8) "BLOB/TEXT column 'notification' used in key specification without a key length") [SQL: u'CREATE INDEX ix_request_notification ON request (notification)'] (Background on this error at: http://sqlalche.me/e/e3q8) "BLOB/TEXT column 'data_manager_id' used in key specification without a key length") [SQL: u'CREATE INDEX ix_data_manager_job_association_data_manager_id ON data_manager_job_association (data_manager_id)'] (Background on this error at: http://sqlalche.me/e/e3q8) "Identifier name 'workflow_invocation_output_dataset_collection_association_ibfk_1' is too long") [SQL: u'\nCREATE TABLE workflow_invocation_output_dataset_collection_association (\n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tworkflow_invocation_id INTEGER, \n\tworkflow_step_id INTEGER, \n\tdataset_collection_id INTEGER, \n\tworkflow_output_id INTEGER, \n\tPRIMARY KEY (id), \n\tFOREIGN KEY(workflow_invocation_id) REFERENCES workflow_invocation (id), \n\tFOREIGN KEY(workflow_step_id) REFERENCES workflow_step (id), \n\tFOREIGN KEY(dataset_collection_id) REFERENCES history_dataset_collection_association (id), \n\tFOREIGN KEY(workflow_output_id) REFERENCES workflow_output (id)\n)\n\n'] (Background on this error at: http://sqlalche.me/e/e3q8) "Identifier name 'workflow_invocation_step_output_dataset_collection_association_ibfk_1' is too long") [SQL: u'\nCREATE TABLE workflow_invocation_step_output_dataset_collection_association (\n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tworkflow_invocation_step_id INTEGER, \n\tworkflow_step_id INTEGER, \n\tdataset_collection_id INTEGER, \n\toutput_name VARCHAR(255), \n\tPRIMARY KEY (id), \n\tFOREIGN KEY(workflow_invocation_step_id) REFERENCES workflow_invocation_step (id), \n\tFOREIGN KEY(workflow_step_id) REFERENCES workflow_step (id), \n\tFOREIGN KEY(dataset_collection_id) REFERENCES history_dataset_collection_association (id)\n)\n\n'] (Background on this error at: http://sqlalche.me/e/e3q8) >grep "Warning" FirstRun.log | cut -f 3,4 -d ":" Warning: Duplicate index 'ix_galaxy_user_deleted' defined on the table 'galaxydbBGAdev.galaxy_user'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_galaxy_user_purged' defined on the table 'galaxydbBGAdev.galaxy_user'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_hdadaa_history_dataset_association_id' defined on the table 'galaxydbBGAdev.history_dataset_association_display_at_authorization'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_lddaia_ldda_id' defined on the table 'galaxydbBGAdev.library_dataset_dataset_info_association'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_hda_ta_history_dataset_association_id' defined on the table 'galaxydbBGAdev.history_dataset_association_tag_association'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_galaxy_user_username' defined on the table 'galaxydbBGAdev.galaxy_user'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_galaxy_user_form_values_id' defined on the table 'galaxydbBGAdev.galaxy_user'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_lia_deleted' defined on the table 'galaxydbBGAdev.library_info_association'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_lfia_deleted' defined on the table 'galaxydbBGAdev.library_folder_info_association'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_lddia_deleted' defined on the table 'galaxydbBGAdev.library_dataset_dataset_info_association'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_sample_folder_id' defined on the table 'galaxydbBGAdev.sample'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_lia_inheritable' defined on the table 'galaxydbBGAdev.library_info_association'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_lfia_inheritable' defined on the table 'galaxydbBGAdev.library_folder_info_association'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_sd_external_service_id' defined on the table 'galaxydbBGAdev.sample_dataset'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_icda_ldda_parent_id' defined on the table 'galaxydbBGAdev.implicitly_converted_dataset_association'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_icda_ldda_id' defined on the table 'galaxydbBGAdev.implicitly_converted_dataset_association'. This is deprecated and will be disallowed in a future release. Warning: Duplicate index 'ix_tsr_includes_datatypes' defined on the table 'galaxydbBGAdev.tool_shed_repository'. This is deprecated and will be disallowed in a future release. >grep ERROR FirstRun.log | grep failed 0037_samples_library ERROR 2018-06-13 12:01:45,817 [p:17852,w:0,m:0] [MainThread] Deleting column 'library_id' to request table failed. 0037_samples_library ERROR 2018-06-13 12:01:45,832 [p:17852,w:0,m:0] [MainThread] Deleting column 'folder_id' to request table failed. galaxy.model.migrate.check INFO 2018-06-13 12:01:47,129 [p:17852,w:0,m:0] [MainThread] 0037_samples_library ERROR 2018-06-13 12:01:45,817 Deleting column 'library_id' to request table failed. galaxy.model.migrate.check INFO 2018-06-13 12:01:47,133 [p:17852,w:0,m:0] [MainThread] 0037_samples_library ERROR 2018-06-13 12:01:45,817 Deleting column 'library_id' to request table failed. galaxy.model.migrate.check INFO 2018-06-13 12:01:47,137 [p:17852,w:0,m:0] [MainThread] 0037_samples_library ERROR 2018-06-13 12:01:45,832 Deleting column 'folder_id' to request table failed. galaxy.model.migrate.check INFO 2018-06-13 12:01:47,142 [p:17852,w:0,m:0] [MainThread] 0037_samples_library ERROR 2018-06-13 12:01:45,832 Deleting column 'folder_id' to request table failed. 0040_page_annotations ERROR 2018-06-13 12:01:49,598 [p:17852,w:0,m:0] [MainThread] Creating page_annotation_association table failed. 0068_rename_sequencer_to_external_services ERROR 2018-06-13 12:02:26,658 [p:17852,w:0,m:0] [MainThread] Deleting column 'sequencer_id' from the 'request_type' table failed. 0087_tool_id_guid_map_table ERROR 2018-06-13 12:02:48,300 [p:17852,w:0,m:0] [MainThread] Creating tool_id_guid_map table failed. galaxy.model.migrate.check INFO 2018-06-13 12:02:48,309 [p:17852,w:0,m:0] [MainThread] 0087_tool_id_guid_map_table ERROR 2018-06-13 12:02:48,300 Creating tool_id_guid_map table failed. galaxy.model.migrate.check INFO 2018-06-13 12:02:48,313 [p:17852,w:0,m:0] [MainThread] 0087_tool_id_guid_map_table ERROR 2018-06-13 12:02:48,300 Creating tool_id_guid_map table failed. 0112_add_data_manager_history_association_and_data_manager_job_association_tables ERROR 2018-06-13 12:03:22,252 [p:17852,w:0,m:0] [MainThread] Creating data_manager_job_association table failed. 0136_collection_and_workflow_state ERROR 2018-06-13 12:03:57,766 [p:17852,w:0,m:0] [MainThread] Creating workflow_invocation_output_dataset_collection_association table failed. 0136_collection_and_workflow_state ERROR 2018-06-13 12:03:58,326 [p:17852,w:0,m:0] [MainThread] Creating workflow_invocation_step_output_dataset_collection_association table failed.

3 4

instance management inspiration
by Matthias Bernt 22 Jun '18

22 Jun '18

Dear list, I'm looking for some inspiration for managing my galaxy instance (with ansible). In particular I'm looking for ansible roles to - setup quotas - manage the tool panel (I need to get order in the chaos) I'm wondering how the usegalaxy instances manage this? Is there some github that I could browse? Cheers, Matthias -- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt(a)ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann -------------------------------------------

2 3

pbs/torque jobs
by Briand, Sheldon (NRC/CNRC) 14 Jun '18

14 Jun '18

Hi, I have upgraded to Galaxy 18.05 (from 17.05). I am running a torque job scheduler (version 6.02). When I submit a job through galaxy and I look in the admin/manage jobs section I see that the job has been submitted successfully. It shows that the job is queued and waiting to run. On the cluster I see that the job runs and goes to completion. However, the status in galaxy continue to show that the job is waiting to run and the status never gets updated. I'm using postgres as my database and I am running through a nginx proxy. I see no errors in my galaxy.log file. I switched from galaxy.ini to galaxy.yml and from paste to uwsgi. Is this a configuration problem on my end? Where should I be looking? Thanks, -Sheldon Sheldon Briand Computer Systems and Applications Analyst National Research Council/Government of Canada Sheldon.briand(a)canada.ca/<mailto:Sheldon.briand@canada.ca/> Tel: (902) 426-1677

2 10

Conflicting conda dependencies for Galaxy tool
by Peter Briggs 14 Jun '18

14 Jun '18

Dear devs I'm currently trying to update a tool developed to wrap an in-house scripted pipeline written by a local researcher, so that it gets its dependencies from conda. The pipeline uses a variety of third-party software packages (many of which are now at "legacy" versions), and I've hit a problem where two of these packages turn out to have conflicting dependencies. Specifically the issue is with qiime=1.8.0, which it turns out is not compatible with any version of R which is available in the channels used by Galaxy. So when Galaxy builds the conda environment for the tool at execution time, it fails with an error from conda: UnsatisfiableError: The following specifications were found to be in conflict: - cairo==1.14.10=0 -> freetype==2.8.1 - cairo==1.14.10=0 -> pixman=0.34 - py2cairo==1.10.0=py27_1 -> cairo==1.12.2 -> freetype=2.4 - py2cairo==1.10.0=py27_1 -> cairo==1.12.2 -> pixman==0.26.2 (The py2cairo requirement comes from an older version of matplotlib that qiime 1.8.0 pulls in.) It's a long shot, but: does Galaxy offer any options for working around conflicting conda dependencies like these? Or is the only option to try and resolve them at the conda packaging level (which might not be practical, or even possible)? Thanks in advance for any advice, Best wishes Peter -- Peter Briggs peter.briggs(a)manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482

1 0

python: error while loading shared libraries
by Floreline TOUCHARD 13 Jun '18

13 Jun '18

Dear Galaxy community, I'm currently testing cross-platform compatibility with our Galaxy instance (15.10) : - [on PROD environment] Galaxy service runs on a RedHat 6.5 server (using python2.6 installed in a virtualenv) and submits jobs on cluster nodes also in RedHat 6.5/python2.6 (default version installed on the system). - [on TEST environment] I changed the configuration of our Galaxy TEST instance (still RedHat 6.5 / python 2.6) to submit jobs on cluster nodes in CentOS7/python2.7 (default version installed on the system). - Galaxy installations (test and prod) and virtualenv folders are located in a volume mounted on Galaxy servers and all cluster nodes. On TEST environment python tools (like the internal Galaxy tool "upload1", or the NCBI BLAST toolkit) if submitted on the cluster failed with the following message : "python: error while loading shared libraries: libpython2.6.so.1.0: cannot open shared object file: No such file or directory" I guess the problem lies in the shared library libpython2.6.so.1.0 located in /usr/lib64/libpython2.6.so.1.0 on the Galaxy server and absent from the cluster nodes (it should be something like /usr/lib64/python2.7/libpython2.7.so.1.0). Is it possible to integrate this shared library in the Python virtualenv and use a specific configuration to indicate Galaxy to use it when submitting jobs on the cluster ? Thanks in advance, Floreline.

1 0