pbs-python issues
by Sandra Maksimovic
Hi all,
As a galaxy newbie, I'm struggling to get the pbs-python module working on our galaxy instance. PBS is installed to a custom location on a shared filesystem mounted via NFS to our galaxy server, however, installing pbs-python using the git clone / python venv method in the documentation fails because it can't find PBS and there does not appear to be any way to define it for pbs-python (I could be wrong?).
...
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DTORQUE_4 -I/usr/include/torque -Isrc/C++ -I/usr/include/python2.7 -c src/C++/pbs_wrap.cxx -o build/temp.linux-x86_64-2.7/src/C++/pbs_wrap.o
In file included from src/C++/pbs_wrap.cxx:2978:0:
src/C++/pbs_ifl.h:90:32: fatal error: u_hash_map_structs.h: No such file or directory
#include "u_hash_map_structs.h"
^
compilation terminated.
error: command 'gcc' failed with exit status 1
So I went ahead and installed pbs_python from source (which does allow you define a PBS_PYTHON_INCLUDEDIR environment variable), however, galaxy does not seem to like this as evidenced by errors during startup. I suspect this has to do with pbs_python not being installed into the galaxy virtual environment.
galaxy[97486]: Traceback (most recent call last):
galaxy[97486]: File "<string>", line 1, in <module>
galaxy[97486]: File "/hpc/software/installed/galaxy/19.05/lib/galaxy/dependencies/__init__.py", line 179, in optional
galaxy[97486]: conditional = ConditionalDependencies(config_file)
galaxy[97486]: File "/hpc/software/installed/galaxy/19.05/lib/galaxy/dependencies/__init__.py", line 32, in __init__
galaxy[97486]: self.parse_configs()
galaxy[97486]: File "/hpc/software/installed/galaxy/19.05/lib/galaxy/dependencies/__init__.py", line 41, in parse_configs
galaxy[97486]: for plugin in ElementTree.parse(job_conf_xml).find('plugins').findall('plugin'):
galaxy[97486]: File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1182, in parse
galaxy[97486]: tree.parse(source, parser)
galaxy[97486]: File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 656, in parse
galaxy[97486]: parser.feed(data)
galaxy[97486]: File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1642, in feed
galaxy[97486]: self._raiseerror(v)
galaxy[97486]: File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1506, in _raiseerror
galaxy[97486]: raise err
galaxy[97486]: xml.etree.ElementTree.ParseError: junk after document element: line 4, column 0
I've attempted reinstalls of the git clone / python venv method with the PBS environment variable to no avail. I was wondering if someone might have any ideas about working around this roadblock, or may have encountered a similar module installation issue in the past?
Also, I was hoping to get some ideas/examples of generic job_conf.xml definitions for PBS clusters? Things like best practices, caveats, etc. My understanding is that, unless configured otherwise, galaxy will submit jobs as the galaxy user and that configuring the server to run jobs as end users themselves is difficult/risky. Just wondering if you guys might have opinions/thoughts/recommendations about this?
And finally, what would be a good way to test that galaxy is submitting jobs to the queue properly? Is there some generic test data/procedure to verify that the galaxy instance is working as expected?
Thanks,
Sandra Maksimovic
Systems Administrator
Information Technology
Murdoch Children's Research Institute
The Royal Children's Hospital, 50 Flemington Road
Parkville, Victoria 3052 Australia
T +61 3 8341 6498
E sandra.maksimovic(a)mcri.edu.au<mailto:sandra.maksimovic@mcri.edu.au>
W mcri.edu.au<https://www.mcri.edu.au/>
Disclaimer
This e-mail and any attachments to it (the "Communication") are, unless otherwise stated, confidential, may contain copyright material and is for the use only of the intended recipient. If you receive the Communication in error, please notify the sender immediately by return e-mail, delete the Communication and the return e-mail, and do not read, copy, retransmit or otherwise deal with it. Any views expressed in the Communication are those of the individual sender only, unless expressly stated to be those of Murdoch Children’s Research Institute (MCRI) ABN 21 006 566 972 or any of its related entities. MCRI does not accept liability in connection with the integrity of or errors in the Communication, computer virus, data corruption, interference or delay arising from or in respect of the Communication.
1 month, 4 weeks
September Galaxy News: INCOB, Galaxy Africa, ACM BCB, GTN, QFAB, ...
by Dave Clements
Hello all,
The September 2019 Galactic news has been published:
There are a bunch of upcoming events, including Galaxy @ INCOB, Galaxy
Africa, two Bioconda / ToolShed events, a GTN CoFest, QFAB training, a
Galaxy Netherlands gathering, (plus more than 20 more).
We are also highlighting the new Climate Science Workbench Galaxy
server, three recent blog posts, six doc & training updates, and 6 (or
11, depending on how you count) open positions.
Find information on all this and more on the Galaxy Community Hub:
https://galaxyproject.org/news/2019-09-galaxy-update/
And it's all in a spiffy new format.
Hope to see you in Jakarta, Niagara Falls, Ghana, Brisbane, Utrecht,
Sri Lanka, Ireland, online, ...
Dave C and the Galaxy Community
--
https://galaxyproject.org/
3 months, 1 week
Job going in Melbourne for Galaxy Australia
by Simon Gladman
Hi all,
Just a heads up that we are looking fill a position for an Application
Support/Software Developer at Melbourne Bioinformatics (part of the
University of Melbourne) ostensibly to work on Galaxy Australia. This
position represents an opportunity join an engaged team of software
developers, systems administrators and bioinformaticians to contribute to
the operation, support and functionality developments of Galaxy Australia,
an award-winning publicly available platform for performing bioinformatics
analyses and workflows for Australian researchers. Galaxy Australia is part
of the UseGalaxy.* initiative.
http://jobs.unimelb.edu.au/caw/en/job/900700/application-support-software...
Applications close on the 19th of September at 23:55 Australian Eastern
Standard Time. (UMT+10)
Cheers,
Simon.
--
Simon Gladman | Bioinformatician
Melbourne Bioinformatics
187 Grattan Street, The University of Melbourne | Victoria 3010 Australia
T +61 3 9035 5822 | M 0418 421 077
Mail: simon.gladman(a)unimelb.edu.au
Web: http://www.melbournebioinformatics.org.au
3 months, 1 week
upload issues with gff3.gz files.
by SAPET, Frederic
Hi,
I'm using bioblend to populate a library in galaxy. it's work well (bam and vcf). But I have issues with some gff3.gz files.
For few of them (over 40 files) I see errors in the log.
If I upload that file (that fails with bioblend) it works using the upload tool on the interface. It's auto-detected as "tabular". I can then change datatype to gff3.gz. It works (with a little issue, the file is described as: 1 line, 41,085 comments. That's not true, there is not comments in my file).
If I upload that file using the upload tool on the interface, it will fail if I set the datatype to gff3.gz during the upload process. I see then the same type of error (as I see with my bioblend script).
I need some ideas, do you have tips ? Maybe some clues to follow...
thank you.
error type is :
Traceback (most recent call last):
File "lib/galaxy/jobs/runners/__init__.py", line 510, in _finish_or_resubmit_job
job_wrapper.finish(tool_stdout, tool_stderr, exit_code, check_output_detected_state=check_output_detected_state, job_stdout=job_stdout, job_stderr=job_stderr)
File "lib/galaxy/jobs/__init__.py", line 1428, in finish
self.sa_session.flush()
File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/orm/scoping.py", line 162, in do
return getattr(self.registry(), name)(*args, **kwargs)
File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2446, in flush
self._flush(objects)
File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2584, in _flush
transaction.rollback(_capture_exception=True)
File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 67, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2544, in _flush
flush_context.execute()
File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 416, in execute
rec.execute(self)
File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 583, in execute
uow,
File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
insert,
File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 1116, in _emit_insert_statements
statement, params
File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 980, in execute
return meth(self, multiparams, params)
File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement
distilled_params,
File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1174, in _execute_context
e, util.text_type(statement), parameters, None, None
File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1458, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1171, in _execute_context
context = constructor(dialect, self, conn, *args)
File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 719, in _init_compiled
for key in compiled_params
File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 719, in <genexpr>
for key in compiled_params
File "/softs/bioinfo/galaxy-dev/.venv/lib/python2.7/site-packages/sqlalchemy/sql/type_api.py", line 1189, in process
return impl_processor(process_param(value, dialect))
File "lib/galaxy/model/custom_types.py", line 319, in process_bind_param
value = json_encoder.encode(value).encode()
File "/softs/add-ons/Python-2.7.12/lib/python2.7/json/encoder.py", line 209, in encode
chunks = list(chunks)
File "/softs/add-ons/Python-2.7.12/lib/python2.7/json/encoder.py", line 434, in _iterencode
for chunk in _iterencode_dict(o, _current_indent_level):
File "/softs/add-ons/Python-2.7.12/lib/python2.7/json/encoder.py", line 408, in _iterencode_dict
for chunk in chunks:
File "/softs/add-ons/Python-2.7.12/lib/python2.7/json/encoder.py", line 387, in _iterencode_dict
yield _encoder(key)
StatementError: (exceptions.UnicodeDecodeError) 'utf8' codec can't decode byte 0xa5 in position 9: invalid start byte [SQL: u'INSERT INTO history_dataset_association_history (history_dataset_association_id, update_time, version, name, extension, metadata, extended_metadata_id) VALUES (%(history_dataset_association_id)s, %(update_time)s, %(version)s, %(name)s, %(extension)s, %(_metadata)s, %(extended_metadata_id)s) RETURNING history_dataset_association_history.id'] [parameters: [{'update_time': datetime.datetime(2019, 8, 26, 7, 53, 29, 457330), 'history_dataset_association_id': 3, 'extension': u'gff3.gz', 'extended_metadata_id': None, '_metadata': {'attribute_types': {'\xc4\xa5v\x10\x04>\xd5\x81U\xa5\\\xc7\xa2\xb3t_d\x07': 'str'}, 'comment_lines': 2528, u'dbkey': u'?', 'delimiter': '\t', 'columns': 9, 'column_types': ['str', 'str', 'str', 'int', 'int', 'float', 'str', 'str', 'str'], 'attributes': 1, 'data_lines': 38558}, 'version': 1, 'name': u'Final_ZmCML247v1a.collinearity.gff.gz'}]]
Fred
3 months, 2 weeks
Gateways 2019 (San Diego): Register by Sept. 9 -- Keynote by James Taylor of Galaxy!
by Katherine Lawrence
*
Gateways 2019
Monday through Wednesday, September 23-25, 2019Hosted at the Catamaran
Resort in San Diego, CAhttps://sciencegateways.org/gateways2019
Science gateways allow science & engineering communities to access
shared data, software, computing services, instruments, educational
materials, and other resources specific to their disciplines. They are
typically a web portal or a suite of desktop applications. Join gateway
creators and enthusiasts to learn, share, connect, and shape the future
of gateways. This year’s conference is co-located with eScience.
Regular registration closes Monday, Sept. 9. (Book your hotel by Friday,
August 23 for the best rates!)
The conference program
<https://sciencegateways.org/web/gateways2019/program>is already
brimming with exciting keynotes, tutorials, and sessions, including
James Taylor of Galaxy and author and science communication expert Randy
Olson.
The Poster Session is open to all.The deadline to submit a poster
abstract has been extended to Tuesday, Sept. 3.
<https://sciencegateways.org/web/gateways2019/program/cfp#poster>
This conference is organized by the Science Gateways Community Institute
(https://sciencegateways.org). Questions?Email help(a)sciencegateways.org.
*
3 months, 2 weeks
King Skrupellos hack
by Pat Gillevet
Has anyone been hacked through the galaxy port 8080? We have an aberrant link on our install and I cannot find where it is hidden.
The message that comes up is attached.
Pat
********************************************************************************
Patrick M. Gillevet, Ph.D
Director, Microbiome Analysis Center
Professor, Department of Biology
George Mason University, Science & Technology Campus
10900 University Boulevard, MSN 4D4
Manassas, Virginia 20110
Office 703-993-1057 Room Colgan Hall-426 FAX 703-993-8430
http://mbac.gmu.edu
**********************************************************************************
3 months, 2 weeks
August 2019 Galactic News: GCC2019 is done; BCC2020 is coming
by Dave Clements
Hello all,
The August 2019 Galactic News is here!
https://galaxyproject.org/news/2019-08-galaxy-update/
* Events:
* GCC2019 is done (and yow, it was a success)
* BCC2020 is coming to Toronto!
* Galaxy ♥ BioHackathon Europe!
* Plus deadlines this month and 19 other events in the next 90 days.
* 178 new publications, including 5 highlighted ones.
* Two new Galactic Blog posts:
* Getting the most out of 1) Galaxy on Twitter, and 2) Pan-Galactic Search
* Galaxy Platform News from UseGalaxy.* and the Human Cell Atlas.
* Seven Open positions in four countries on two continents.
* Updates to training materials and documentation.
* New releases.
This month's newsletter was a community effort unlike any before it.
Thanks to everyone who contributed (and keep those contributions
coming!).
See you in San Diego, Milwaukee, Honolulu, Toronto, ...
Dave C
--
https://galaxyproject.org/
4 months, 1 week
