August 2019 - galaxy-dev - lists.galaxyproject.org

pbs-python issues
by Sandra Maksimovic 11 Oct '19

11 Oct '19

Hi all, As a galaxy newbie, I'm struggling to get the pbs-python module working on our galaxy instance. PBS is installed to a custom location on a shared filesystem mounted via NFS to our galaxy server, however, installing pbs-python using the git clone / python venv method in the documentation fails because it can't find PBS and there does not appear to be any way to define it for pbs-python (I could be wrong?). ... gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DTORQUE_4 -I/usr/include/torque -Isrc/C++ -I/usr/include/python2.7 -c src/C++/pbs_wrap.cxx -o build/temp.linux-x86_64-2.7/src/C++/pbs_wrap.o In file included from src/C++/pbs_wrap.cxx:2978:0: src/C++/pbs_ifl.h:90:32: fatal error: u_hash_map_structs.h: No such file or directory #include "u_hash_map_structs.h" ^ compilation terminated. error: command 'gcc' failed with exit status 1 So I went ahead and installed pbs_python from source (which does allow you define a PBS_PYTHON_INCLUDEDIR environment variable), however, galaxy does not seem to like this as evidenced by errors during startup. I suspect this has to do with pbs_python not being installed into the galaxy virtual environment. galaxy[97486]: Traceback (most recent call last): galaxy[97486]: File "<string>", line 1, in <module> galaxy[97486]: File "/hpc/software/installed/galaxy/19.05/lib/galaxy/dependencies/__init__.py", line 179, in optional galaxy[97486]: conditional = ConditionalDependencies(config_file) galaxy[97486]: File "/hpc/software/installed/galaxy/19.05/lib/galaxy/dependencies/__init__.py", line 32, in __init__ galaxy[97486]: self.parse_configs() galaxy[97486]: File "/hpc/software/installed/galaxy/19.05/lib/galaxy/dependencies/__init__.py", line 41, in parse_configs galaxy[97486]: for plugin in ElementTree.parse(job_conf_xml).find('plugins').findall('plugin'): galaxy[97486]: File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1182, in parse galaxy[97486]: tree.parse(source, parser) galaxy[97486]: File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 656, in parse galaxy[97486]: parser.feed(data) galaxy[97486]: File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1642, in feed galaxy[97486]: self._raiseerror(v) galaxy[97486]: File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1506, in _raiseerror galaxy[97486]: raise err galaxy[97486]: xml.etree.ElementTree.ParseError: junk after document element: line 4, column 0 I've attempted reinstalls of the git clone / python venv method with the PBS environment variable to no avail. I was wondering if someone might have any ideas about working around this roadblock, or may have encountered a similar module installation issue in the past? Also, I was hoping to get some ideas/examples of generic job_conf.xml definitions for PBS clusters? Things like best practices, caveats, etc. My understanding is that, unless configured otherwise, galaxy will submit jobs as the galaxy user and that configuring the server to run jobs as end users themselves is difficult/risky. Just wondering if you guys might have opinions/thoughts/recommendations about this? And finally, what would be a good way to test that galaxy is submitting jobs to the queue properly? Is there some generic test data/procedure to verify that the galaxy instance is working as expected? Thanks, Sandra Maksimovic Systems Administrator Information Technology Murdoch Children's Research Institute The Royal Children's Hospital, 50 Flemington Road Parkville, Victoria 3052 Australia T +61 3 8341 6498 E sandra.maksimovic(a)mcri.edu.au<mailto:sandra.maksimovic@mcri.edu.au> W mcri.edu.au<https://www.mcri.edu.au/> Disclaimer This e-mail and any attachments to it (the "Communication") are, unless otherwise stated, confidential, may contain copyright material and is for the use only of the intended recipient. If you receive the Communication in error, please notify the sender immediately by return e-mail, delete the Communication and the return e-mail, and do not read, copy, retransmit or otherwise deal with it. Any views expressed in the Communication are those of the individual sender only, unless expressly stated to be those of Murdoch Children’s Research Institute (MCRI) ABN 21 006 566 972 or any of its related entities. MCRI does not accept liability in connection with the integrity of or errors in the Communication, computer virus, data corruption, interference or delay arising from or in respect of the Communication.

3 8

September Galaxy News: INCOB, Galaxy Africa, ACM BCB, GTN, QFAB, ...
by Dave Clements 30 Aug '19

30 Aug '19

Hello all, The September 2019 Galactic news has been published: There are a bunch of upcoming events, including Galaxy @ INCOB, Galaxy Africa, two Bioconda / ToolShed events, a GTN CoFest, QFAB training, a Galaxy Netherlands gathering, (plus more than 20 more). We are also highlighting the new Climate Science Workbench Galaxy server, three recent blog posts, six doc & training updates, and 6 (or 11, depending on how you count) open positions. Find information on all this and more on the Galaxy Community Hub: https://galaxyproject.org/news/2019-09-galaxy-update/ And it's all in a spiffy new format. Hope to see you in Jakarta, Niagara Falls, Ghana, Brisbane, Utrecht, Sri Lanka, Ireland, online, ... Dave C and the Galaxy Community -- https://galaxyproject.org/

1 0

Job going in Melbourne for Galaxy Australia
by Simon Gladman 28 Aug '19

28 Aug '19

Hi all, Just a heads up that we are looking fill a position for an Application Support/Software Developer at Melbourne Bioinformatics (part of the University of Melbourne) ostensibly to work on Galaxy Australia. This position represents an opportunity join an engaged team of software developers, systems administrators and bioinformaticians to contribute to the operation, support and functionality developments of Galaxy Australia, an award-winning publicly available platform for performing bioinformatics analyses and workflows for Australian researchers. Galaxy Australia is part of the UseGalaxy.* initiative. http://jobs.unimelb.edu.au/caw/en/job/900700/application-support-software-d… Applications close on the 19th of September at 23:55 Australian Eastern Standard Time. (UMT+10) Cheers, Simon. -- Simon Gladman | Bioinformatician Melbourne Bioinformatics 187 Grattan Street, The University of Melbourne | Victoria 3010 Australia T +61 3 9035 5822 | M 0418 421 077 Mail: simon.gladman(a)unimelb.edu.au Web: http://www.melbournebioinformatics.org.au

1 0

set up an asynchronous data source in galaxy
by SAPET, Frederic 26 Aug '19

26 Aug '19

Hi, I'm trying to set up an asynchronous data source in galaxy. I follow the examples that Helena provide here : https://github.com/erasche/galaxy-data_source-examples/tree/master/flask It's a success for synchronous modE. But when I test the asynchronous mode, it fails. from what I understand, the file in history (that would contains the data) is not well created. The example creates a link like this one : https://galaxydev.biogemma.fr:4430/export/?gx_url=https%3A%2F%2Fgalaxydev.b… When I click on the link, no file is created in my history. The error in the log is : File "lib/galaxy/webapps/galaxy/controllers/async.py", line 150, in index open(data.file_name, 'wb').close() # create the file IOError: [Errno 2] No such file or directory: '' The filename is missing, so I have no new file in my history waiting for the asynchronous process to fill it. I've described the error here (with complete log messages) : https://github.com/erasche/galaxy-data_source-examples/issues/6 Can anybody reproduce this? Is this an issue? or a misconfiguration from my side? thank you. Fred

1 2

upload issues with gff3.gz files.
by SAPET, Frederic 26 Aug '19

26 Aug '19

Hi, I'm using bioblend to populate a library in galaxy. it's work well (bam and vcf). But I have issues with some gff3.gz files. For few of them (over 40 files) I see errors in the log. If I upload that file (that fails with bioblend) it works using the upload tool on the interface. It's auto-detected as "tabular". I can then change datatype to gff3.gz. It works (with a little issue, the file is described as: 1 line, 41,085 comments. That's not true, there is not comments in my file). If I upload that file using the upload tool on the interface, it will fail if I set the datatype to gff3.gz during the upload process. I see then the same type of error (as I see with my bioblend script). I need some ideas, do you have tips ? Maybe some clues to follow... thank you. error type is : Traceback (most recent call last): File "lib/galaxy/jobs/runners/__init__.py", line 510, in _finish_or_resubmit_job job_wrapper.finish(tool_stdout, tool_stderr, exit_code, check_output_detected_state=check_output_detected_state, job_stdout=job_stdout, job_stderr=job_stderr) File "lib/galaxy/jobs/__init__.py", line 1428, in finish self.sa_session.flush() File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/orm/scoping.py", line 162, in do return getattr(self.registry(), name)(*args, **kwargs) File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2446, in flush self._flush(objects) File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2584, in _flush transaction.rollback(_capture_exception=True) File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 67, in __exit__ compat.reraise(exc_type, exc_value, exc_tb) File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2544, in _flush flush_context.execute() File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 416, in execute rec.execute(self) File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 583, in execute uow, File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj insert, File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 1116, in _emit_insert_statements statement, params File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 980, in execute return meth(self, multiparams, params) File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement distilled_params, File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1174, in _execute_context e, util.text_type(statement), parameters, None, None File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1458, in _handle_dbapi_exception util.raise_from_cause(sqlalchemy_exception, exc_info) File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1171, in _execute_context context = constructor(dialect, self, conn, *args) File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 719, in _init_compiled for key in compiled_params File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 719, in <genexpr> for key in compiled_params File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/sql/type_api.py", line 1189, in process return impl_processor(process_param(value, dialect)) File "lib/galaxy/model/custom_types.py", line 319, in process_bind_param value = json_encoder.encode(value).encode() File "/softs/add-ons/Python-2.7.12/lib/python2.7/json/encoder.py", line 209, in encode chunks = list(chunks) File "/softs/add-ons/Python-2.7.12/lib/python2.7/json/encoder.py", line 434, in _iterencode for chunk in _iterencode_dict(o, _current_indent_level): File "/softs/add-ons/Python-2.7.12/lib/python2.7/json/encoder.py", line 408, in _iterencode_dict for chunk in chunks: File "/softs/add-ons/Python-2.7.12/lib/python2.7/json/encoder.py", line 387, in _iterencode_dict yield _encoder(key) StatementError: (exceptions.UnicodeDecodeError) 'utf8' codec can't decode byte 0xa5 in position 9: invalid start byte [SQL: u'INSERT INTO history_dataset_association_history (history_dataset_association_id, update_time, version, name, extension, metadata, extended_metadata_id) VALUES (%(history_dataset_association_id)s, %(update_time)s, %(version)s, %(name)s, %(extension)s, %(_metadata)s, %(extended_metadata_id)s) RETURNING history_dataset_association_history.id'] [parameters: [{'update_time': datetime.datetime(2019, 8, 26, 7, 53, 29, 457330), 'history_dataset_association_id': 3, 'extension': u'gff3.gz', 'extended_metadata_id': None, '_metadata': {'attribute_types': {'\xc4\xa5v\x10\x04>\xd5\x81U\xa5\\\xc7\xa2\xb3t_d\x07': 'str'}, 'comment_lines': 2528, u'dbkey': u'?', 'delimiter': '\t', 'columns': 9, 'column_types': ['str', 'str', 'str', 'int', 'int', 'float', 'str', 'str', 'str'], 'attributes': 1, 'data_lines': 38558}, 'version': 1, 'name': u'Final_ZmCML247v1a.collinearity.gff.gz'}]] Fred

1 0

Gateways 2019 (San Diego): Register by Sept. 9 -- Keynote by James Taylor of Galaxy!
by Katherine Lawrence 22 Aug '19

22 Aug '19

* Gateways 2019 Monday through Wednesday, September 23-25, 2019Hosted at the Catamaran Resort in San Diego, CAhttps://sciencegateways.org/gateways2019 Science gateways allow science & engineering communities to access shared data, software, computing services, instruments, educational materials, and other resources specific to their disciplines. They are typically a web portal or a suite of desktop applications. Join gateway creators and enthusiasts to learn, share, connect, and shape the future of gateways. This year’s conference is co-located with eScience. Regular registration closes Monday, Sept. 9. (Book your hotel by Friday, August 23 for the best rates!) The conference program <https://sciencegateways.org/web/gateways2019/program>is already brimming with exciting keynotes, tutorials, and sessions, including James Taylor of Galaxy and author and science communication expert Randy Olson. The Poster Session is open to all.The deadline to submit a poster abstract has been extended to Tuesday, Sept. 3. <https://sciencegateways.org/web/gateways2019/program/cfp#poster> This conference is organized by the Science Gateways Community Institute (https://sciencegateways.org) Questions?Email help(a)sciencegateways.org. *

1 0

King Skrupellos hack
by Pat Gillevet 21 Aug '19

21 Aug '19

Has anyone been hacked through the galaxy port 8080? We have an aberrant link on our install and I cannot find where it is hidden. The message that comes up is attached. Pat ******************************************************************************** Patrick M. Gillevet, Ph.D Director, Microbiome Analysis Center Professor, Department of Biology George Mason University, Science & Technology Campus 10900 University Boulevard, MSN 4D4 Manassas, Virginia 20110 Office 703-993-1057 Room Colgan Hall-426 FAX 703-993-8430 http://mbac.gmu.edu **********************************************************************************

3 3

hudge size of _cleared_contents dir in job_working_directory
by Franck Giacomoni 21 Aug '19

21 Aug '19

Dear all, On a 19.1 galaxy version, my job_working_directory (database/ jobs_directory) contains a dir named _cleared_contents which continuously growing in terms of size… What is the exact process run by Galaxy to move a job from database/ jobs_directory/000/… to database/ jobs_directory/_cleared_contents Is there any configuration (in galaxy.yml) to manage this process? Can I delete this directory regularly or do I need to be careful of unfinished job already present in this specific dir (_cleared_contents)? Thank you very much for your reply Kindly regards Franck Franck Giacomoni Biological computing & Metabolomics franck.giacomoni(a)inra.fr<mailto:franck.giacomoni@inra.fr> UMR1019 - Human Nutrition Unit - Metabolism Exploration Platform Tél. : +33 (0)4 73 62 47 72 Fax : +33 (0)4 73 62 46 88 **Télétravail les mercredis 8h15-12h15 / 13h00-16h40** Centre de recherche de Clermont-Ferrand / Theix 63122 Saint Genès Champanelle France INRA PFEM portal<http://www6.clermont.inra.fr/plateforme_exploration_metabolisme> INRA Ingenum portal<https://www.ingenum.inra.fr/> MetaboHUB consortium website<http://www.metabohub.fr/metabohub.html> MetaboHUB Open Metabolic e-resources (OMeR)<https://informatique-mia.inra.fr/omer/> Workflow4Metabolomics.org portal<http://workflow4metabolomics.org/> PhytoHub Database website<http://phytohub.eu/> [/Users/fgiacomoni/Library/Containers/com.microsoft.Outlook/Data/Library/Caches/Signatures/signature_2031791895]<https://workflow4metabolomics.org/w4e2020>

1 0

August 2019 Galactic News: GCC2019 is done; BCC2020 is coming
by Dave Clements 01 Aug '19

01 Aug '19

Hello all, The August 2019 Galactic News is here! https://galaxyproject.org/news/2019-08-galaxy-update/ * Events: * GCC2019 is done (and yow, it was a success) * BCC2020 is coming to Toronto! * Galaxy ♥ BioHackathon Europe! * Plus deadlines this month and 19 other events in the next 90 days. * 178 new publications, including 5 highlighted ones. * Two new Galactic Blog posts: * Getting the most out of 1) Galaxy on Twitter, and 2) Pan-Galactic Search * Galaxy Platform News from UseGalaxy.* and the Human Cell Atlas. * Seven Open positions in four countries on two continents. * Updates to training materials and documentation. * New releases. This month's newsletter was a community effort unlike any before it. Thanks to everyone who contributed (and keep those contributions coming!). See you in San Diego, Milwaukee, Honolulu, Toronto, ... Dave C -- https://galaxyproject.org/

1 0