September 2015 - galaxy-dev - lists.galaxyproject.org

Galaxy Interactive Environment devs
by Eric Rasche 15 Oct '15

15 Oct '15

Howdy y'all, I'm building some changes to the IE infrastructure that will fix some initial hacks we did, however they will be backwards incompatible. Specifically these changes will fix some of the enterprise deployment pains (running with an external upstream proxy like apache) and allow for safely running IEs on a separate host. I'm emailing the dev list to see if anyone else has started building IEs, or if it just Björn and myself. Feel free to make yourselves known to us so we know who to contact when we're planning new features :) If you're just building on top of our existing containers, this won't be an issue for you, you'll just need to rebuild your containers once we release 15.10 versions. Cheers, Eric -- Eric Rasche Programmer II Center for Phage Technology Rm 312A, BioBio Texas A&M University College Station, TX 77843 404-692-2048 esr(a)tamu.edu

5 8

Re: [galaxy-dev] pbs_python can't be installed via scramble
by Makis Ladoukakis 14 Oct '15

14 Oct '15

Hello Will, The second command returns the container.hpp inside the directory of the torque source which i used to install it: find / -name container.hpp /home/user/torque-5.1.1.2-1_18e4a5f1/src/include/container.hpp I noticed there is also a log.h file there. the first command doesn't return anything (I've search manually too, nothing inside /usr/local/include/). How do I fix this? Shall I change the export command of PBS_PYTHON_INCLUDEDIR? Thank you, Makis Date: Thu, 24 Sep 2015 08:31:07 -0700 Subject: Re: [galaxy-dev] pbs_python can't be installed via scramble From: wholtz(a)lygos.com To: makis4ever(a)hotmail.com CC: galaxy-dev(a)lists.galaxyproject.org And does find /usr/local/include -name container.hppor find / -name container.hppreturn torque/src/include/container.hpp ? My guess is you installed binaries for Torque, but the scrambling requires the Torque source (or at least the header files form Torque). -Will On Thu, Sep 24, 2015 at 1:16 AM, Makis Ladoukakis <makis4ever(a)hotmail.com> wrote: Hello Will, Thanks again for your reply. Yes I had already tried these but I got the same container.hpp error. The only thing different in my case is that the names of the directories of the directories are a bit different: export PBS_PYTHON_INCLUDEDIR=/usr/local/include/ export PBSCONFIG=/usr/local/bin/pbs-config export LIBTORQUE_DIR=/usr/local/lib/libtorque.so python scripts/scramble.py -e pbs_python Any ideas? Kind regards, Makis Date: Wed, 23 Sep 2015 07:06:32 -0700 Subject: Re: [galaxy-dev] pbs_python can't be installed via scramble From: wholtz(a)lygos.com To: makis4ever(a)hotmail.com CC: galaxy-dev(a)lists.galaxyproject.org Hi Makis, The thread I posted does have the exact same container.hpp error and proposes a solution. Did you try setting your PBS_PYTHON_INCLUDEDIR to point to your torque/include directory, as mentioned in that thread? from the thread:export PBS_PYTHON_INCLUDEDIR=/usr/local/torque/include/ export PBSCONFIG=/usr/local/torque/bin/pbs-config export LIBTORQUE_DIR=/usr/local/torque/lib/libtorque.so -Will On Wed, Sep 23, 2015 at 5:49 AM, Makis Ladoukakis <makis4ever(a)hotmail.com> wrote: Hello Will, Thank you for your reply. I had already checked this thread before as I browsed through the mailing list for "pbs_python" issues. It doesn't have the same errors (can't locate log.h and when i provide it it wants the container.hpp ) and up to one point i've already followed the "rescrambling" of the pbs_python. Anyone else had any experience with that? Thank you, Makis Date: Tue, 22 Sep 2015 09:35:11 -0700 Subject: Re: [galaxy-dev] pbs_python can't be installed via scramble From: wholtz(a)lygos.com To: makis4ever(a)hotmail.com CC: galaxy-dev(a)lists.galaxyproject.org Here is an old thread that looks rather similar to your problems:https://www.mail-archive.com/galaxy-dev@lists.galaxyproject.org/ms… -Will On Tue, Sep 22, 2015 at 2:43 AM, Makis Ladoukakis <makis4ever(a)hotmail.com> wrote: Hello everyone, I'm trying to set up a Galaxy instance on a multi-core server in my university so according to the instructions here: https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#PBS I installed Torque, changed the eggs.ini file by adding the 4.4.0. version of pbs_python and tried to setup pbs_python via scramble: LIBTORQUE_DIR=/usr/local/lib/ python scripts/scramble.py -e pbs_python When I did that I got the error: Failed to find log.h in inlcude dir /usr/include/torque. (Set incude dir via PBS_PYTHON_INCLUDEDIR variable) Traceback (most recent call last): File "scripts/scramble.py", line 50, in <module> egg.scramble() File "/home/galaxy/galaxy/lib/galaxy/eggs/scramble.py", line 57, in scramble self.run_scramble_script() File "/home/galaxy/galaxy/lib/galaxy/eggs/scramble.py", line 210, in run_scramble_script raise ScrambleFailure( self, "%s(): Egg build failed for %s %s" % ( sys._getframe().f_code.co_name, self.name, self.version ) ) galaxy.eggs.scramble.ScrambleFailure: run_scramble_script(): Egg build failed for pbs_python 4.4.0 so I did some digging around and found that the file I need is in /scripts/scramble/build/py2.7-linux-x86_64-ucs4/pbs_python/src/C++/ (please correct me if I am wrong) So I tried again using: PBS_PYTHON_INCLUDEDIR=/home/galaxy/galaxy/scripts/scramble/build/py2.7-linux-x86_64-ucs4/pbs_python/src/C++/ LIBTORQUE_DIR=/usr/local/lib/ python scripts/scramble.py -e pbs_python but then I got the error: In file included from src/C++/pbs_ifl.h:90:0, from src/C++/pbs_wrap.cxx:2978: /usr/local/include/u_hash_map_structs.h:82:25: fatal error: container.hpp: No such file or directory #include "container.hpp" Can someone help me please? Kind regards, Makis ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- The information contained in this e-mail message or any attachment(s) may be confidential and/or privileged and is intended for use only by the individual(s) to whom this message is addressed. If you are not the intended recipient, any dissemination, distribution, copying, or use is strictly prohibited. If you receive this e-mail message in error, please e-mail the sender at wholtz(a)lygos.com and destroy this message and remove the transmission from all computer directories (including e-mail servers). Please consider the environment before printing this email. -- The information contained in this e-mail message or any attachment(s) may be confidential and/or privileged and is intended for use only by the individual(s) to whom this message is addressed. If you are not the intended recipient, any dissemination, distribution, copying, or use is strictly prohibited. If you receive this e-mail message in error, please e-mail the sender at wholtz(a)lygos.com and destroy this message and remove the transmission from all computer directories (including e-mail servers). Please consider the environment before printing this email. -- The information contained in this e-mail message or any attachment(s) may be confidential and/or privileged and is intended for use only by the individual(s) to whom this message is addressed. If you are not the intended recipient, any dissemination, distribution, copying, or use is strictly prohibited. If you receive this e-mail message in error, please e-mail the sender at wholtz(a)lygos.com and destroy this message and remove the transmission from all computer directories (including e-mail servers). Please consider the environment before printing this email.

1 3

External User Authenticaion
by Ryan G 13 Oct '15

13 Oct '15

Hi all - I'm trying to use external user authentication with Galaxy. The external authentication passes to Galaxy the username with the mail domain at HTTP_USER. In galaxy.ini, I enable: use_remote_user = True When I try to access Galaxy, I get the message: Galaxy is configured to authenticate users via an external method (such as HTTP authentication in Apache), but a username was not provided by the upstream (proxy) server. This is generally due to a misconfiguration in the upstream server. But nothing in paster.log indicating what the error is. How do I track this down?

3 13

Installing Cairo into Gakaxy
by Christian Brenninkmeijer 13 Oct '15

13 Oct '15

Hi All, Has anyone got a working example tool_dependencies.xml to install the Cairo package into R. (I actually need a package that depends on R Cairo but the current issue is Cairo) I am having the cairo.h not found issue I have cloned https://toolshed.g2.bx.psu.edu/view/devteam/package_cairo_1_12_14/b39299b4b… (and its dependencies) As well as https://toolshed.g2.bx.psu.edu/view/iuc/package_r_3_1_2/9f2fddb9d6e2 I am trying to use <action type="setup_r_environment"> But can not get the package_cairo to be seen by the R I have tried <action type="set_environment_for_install"> both inside and outside of the setup_r_environment but it does not appear to be working, Full attempted tool_dependencies.xml attached, Thanks in advance Christian University of Manchester

4 24

GCC2016 Training Day Topic Nominations are now open
by Dave Clements 12 Oct '15

12 Oct '15

Hello all, The 2016 Galaxy Community Conference (GCC2016) <https://wiki.galaxyproject.org/Events/GCC2016> features two full days of training on June 26-27. Each day will have multiple tracks, with each track featuring several sessions throughout the day. GCC2016 will be held at Indiana University <http://indiana.edu/> in Bloomington, Indiana, United States. *GCC2016 Training topics <https://wiki.galaxyproject.org/Events/GCC2016/Training> are nominated by you <http://bit.ly/gcc2016tdnom>, the Galaxy Community.* Please give some thought to what topics you would like to see offered, or that you would like to offer, and then submit them as topic nominations <http://bit.ly/gcc2016tdnom>. Any topic of interest to the Galaxy Community can be nominated and you are encouraged to nominate more than one topic. If you are looking for ideas, see - Topics nominated for GCC2015 <http://bit.ly/gcc2015vote> - Topics nominated for GCC2014 <http://bit.ly/1s6NtMN> - Topics nominated for GCC2013 <http://bit.ly/1i2j1gN> - The Galaxy Events page <https://wiki.galaxyproject.org/Events>, which lists what training the community is offering outside of GCCs. - Topics that have already been nominated for GCC2016 <https://wiki.galaxyproject.org/Events/GCC2016/Training#Nominated_Topics> *Please nominate a topic now! <http://bit.ly/gcc2016tdnom> * Nominated topics will be published as they come in <https://wiki.galaxyproject.org/Events/GCC2016/Training#Nominated_Topics>. *Topic nomination closes Friday, October, 16.*. Topics will be compiled into a uniform list by the GCC2016 Organizing Committee, and topics will be voted on by the Galaxy Community from October 26 through November 6. Topics will then be selected and scheduled based on topic interest, and the organisers' ability to confirm instructors for each session. Some very popular sessions may be scheduled more than once. The final schedule will be posted before registration opens. Looking forward to a stellar set of nominations, and hoping to see you in Bloomington, GCC2016 Exec Committee <https://wiki.galaxyproject.org/Events/GCC2016/Organizers> -- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ https://wiki.galaxyproject.org/

1 1

Get the working directory within the ncbi_blastp_wrapper.xml file
by Ryan Johnson 12 Oct '15

12 Oct '15

Hello all, We are setting up a galaxy instance that will submit jobs to our cluster. We could not get pbs_python to work so we have added all PBS logic to the cheetah template wrapper ncbi_blastp_wrapper.xml. Currently, all jobs are running out of the home directory. We want to add logic to the wrapper which will get the working directory from the JobWrapper class and then change to that directory and run from there. Can anyone help us to access the JobWrapper.working_underscore value? Or suggest a better way to do this? Thank you very much, Ryan -- Ryan S. Johnson, PhD Applications Scientist Center for Advanced Research Computing P: (650) 430-6194 E: rjohns03(a)carc.unm.edu

2 1

Question about using dynamic options and refresh_on_change for rendering 2 associated select lists
by Greg Von Kuster 09 Oct '15

09 Oct '15

Hello all, I’m working on adding support to the Galaxy framework for datatypes in the constructive solid geometry (CSG) space and I have several working Galaxy tools and a visualization plugin that deal with 3d shapes in this environment. All of this work will soon be contributed to open source. For one of the tools, I need to provide 2 select lists on the tool form that are associated in such a way that when an option in select list 1 is chosen, the options in select list 2 are altered and re-rendered. This behavior would probably use a combination of dynamic options and refresh_on_change between the 2 associated select lists. I’ve not seen a tool that does this precisely, so I’m hoping that if it is currently possible to do this within Galaxy tools, someone will point me to an example. If it is not currently possible, I will contribute a PR for supporting it, but would like some input as to how it should be done. Dynamic options are currently rendered using 4 approaches; from_file, from_dataset, from_parameter and from_data_table. One or more of several filters can then be applied to the options to alter them if desired. Since these CSG tools involve new datatypes, I have flexibility into how the metadata elements are set up that are needed to render these 2 select lists. A dictionary may work where the metadata element contents look like this: {a: [‘1’, ‘2’], b: [‘3’, ‘4’] } The options in select list 1 would be ‘a’ and ‘b’. When ‘a’ is selected, select list 2 would have options ‘1’ and ‘2’, and when ‘b’ is selected, select list 2 would have options ‘3’ and ‘4’. To support this approach, perhaps dynamic options could be enhanced to include a from_dict option. There would be some complxity involved in associating the lists using refresh_on_change so that select list 2 is refreshed with the correct options when select list one is changed. I haven’t yet worked out these details. Also, using a dictionary will not keep order for the options, so perhaps a list of tuples would be better? [ [ ‘a’, [‘1’, ‘2’]], [‘b’, [‘3’, ‘4’]]] Can someone let me know if this is currently possible, and if so, an example tool that does it? If it’s not possible, does the above approach seem reasonable, or is there a better way? Thanks very much, Greg Von Kuster ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ <https://lists.galaxyproject.org/> To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ <http://galaxyproject.org/search/mailinglists/>

4 6

Some issues with job submission to pulsar
by Raj Ayyampalayam 07 Oct '15

07 Oct '15

Hello, I am setting up a new instance of Galaxy (based on the latest dev branch on github repo) configured to submit the jobs to a remote cluster via pulsar. I used the usegalaxy playbook to get the configuration parameters for the galaxy job_conf.xml and pulsar app.yml files. I was partially successful in running the jobs via pulsar, the regular tools (analysis tools, etc..) are running OK on the remote cluster. I am having trouble getting the upload tool to run on the the remote cluster. Here is the error I see in the galaxy error report for the job: It looks like the upload tool running on the remote machine is trying to access a file in the database/tmp area of the galaxy server machine. Traceback (most recent call last): File "/escratch4/apps/galaxy_scratch/pulsar/files/staging/46/tool_files/upload.py", line 430, in <module> __main__() File "/escratch4/apps/galaxy_scratch/pulsar/files/staging/46/tool_files/upload.py", line 405, in __main__ registry.load_datatypes( root_dir=sys.argv[1], config=sys.argv[2] ) File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy/lib/galaxy/datatypes/registry.py", line 94, in load_datatypes tree = galaxy.util.parse_xml( config ) File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy/lib/galaxy/util/__init__.py", line 178, in parse_xml root = tree.parse( fname, parser=ElementTree.XMLParser( target=DoctypeSafeCallbackTarget() ) ) File "/usr/local/python/2.7.8/lib/python2.7/xml/etree/ElementTree.py", line 647, in parse source = open(source, "rb") IOError: [Errno 2] No such file or directory: '/home/galaxy/wkdir/galaxy/database/tmp/tmpp6j83l' I've created a Gist with the contents of job_conf.xml, app.yml, file action file, error content at https://gist.github.com/raj76/7be86f6a56714deef050 . Any suggestions on how to debug this issue? Thanks, -Raj

2 2

Documentation of Galaxy API metadata objects
by Clare Sloggett 01 Oct '15

01 Oct '15

Hi all, I'm wondering if there is any documentation of the JSON structures returned by particular Galaxy REST API calls. Some aspects of the Galaxy API are effectively documented via bioblend's structure, and others are documented at http://galaxy.readthedocs.org/en/master/api_doc.html . However I don't know where to look to find out the actual JSON structure that should be returned by an API call. Does this exist? For instance, if I call (using bioblend) gi.histories.show_history(..., contents=True, details="all"), I will get back a list of dicts where each dict contains dataset metadata, e.g.: { u'api_type': u'file', u'create_time': u'2015-09-28T06:00:19.850421', u'data_type': u'galaxy.datatypes.sequence.FastqSanger', u'file_name': u'/mnt/galaxy/files/000/dataset_2.dat', .... I'd like to know how to find out what dictionary keys to expect for a particular API call in a particular Galaxy version. In this example, IIRC at some point the key 'file_name' changed to 'file_path', but I don't know how to determine which version of Galaxy uses which convention. I don't think this change would necessarily be reflected in bioblend, because bioblend just returns the entire JSON structure as a Python object without caring much about its contents. I also don't know how to derive this information from the Galaxy source code itself (even though I know where the API code is under webapp), so alternatively any guidance on that would be helpful! Thanks, Clare

3 3

Picard sortSam python error inside its Java app?
by Dooley, Damion 30 Sep '15

30 Sep '15

If any Picard / Samtools experts can help ... Ran into a picard sortSam libc / python error that is only showing up when launched within Galaxy - AND ONLY WITH SORT_ORDER="queryname" option. I run the same command line as the tool does, but as galaxy user on command line, it runs ok?! Is there some resource limit I need to raise or lower in Galaxy context? The server has tonnes of memory and cores. After the crash, I get this at bottom of STDERR along with error message. But I presume this may only be a side-effect: [bam_index_core] the alignment is not sorted (HWI-ST521:273:C4EH9ACXX:4:1101:10000:53458/1): 2632 > 1357 in 75-th chr [bam_index_build2] fail to index the BAM file. since no such message occurs via command line invocation. Advice greatly appreciated, Damion ********************************************* Details... Galaxy tool run: after a few minutes finally yields this on stderr: Picked up _JAVA_OPTIONS: -Xmx2048m -Xms256m *** glibc detected *** python: double free or corruption (top): 0x0000000005fb5200 *** ======= Backtrace: ========= /lib64/libc.so.6[0x3f4d675e66] ... /usr/lib64/libpython2.6.so.1.0(Py_Main+0xb62)[0x3f4f2ff8a2] /lib64/libc.so.6(__libc_start_main+0xfd)[0x3f4d61ed5d] python[0x400649] ... /galaxy_data/production1/job_working_directory/003/3583/galaxy_3583.sh: line 21: 46594 Aborted (core dumped) python ... Command line version:(Taken directly from Galaxy tool execution info; extract below shows with QUIET=false though I get same success with QUIET=true, and I loaded the right picard.jar path into JAVA_JAR_PATH) -salk tmp]$ _JAVA_OPTIONS=${_JAVA_OPTIONS:-'-Xmx2048m -Xms256m'} && export _JAVA_OPTIONS && java -jar $JAVA_JAR_PATH/picard.jar SortSam INPUT="/galaxy_data/production1/files/004/dataset_4987.dat" OUTPUT="/galaxy_data/production1/files/005/dataset_5408.dat" SORT_ORDER="queryname" VERBOSITY=ERROR VALIDATION_STRINGENCY=LENIENT Picked up _JAVA_OPTIONS: -Xmx2048m -Xms256m [Wed Sep 30 12:29:39 PDT 2015] picard.sam.SortSam INPUT=/galaxy_data/production1/files/004/dataset_4987.dat OUTPUT=/galaxy_data/production1/files/005/dataset_5408.dat SORT_ORDER=queryname VERBOSITY=ERROR VALIDATION_STRINGENCY=LENIENT QUIET=false COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json [Wed Sep 30 12:29:39 PDT 2015] Executing as galaxy@grl-salk on Linux 2.6.32-220.17.1.el6.x86_64 amd64; OpenJDK 64-Bit Server VM 1.6.0_30-b30; Picard version: 1.136(f187319bf8bbde56892d5b5a1ce3fc0529b71a49_1436805856) IntelDeflater [Wed Sep 30 12:42:51 PDT 2015] picard.sam.SortSam done. Elapsed time: 13.21 minutes. Runtime.totalMemory()=1677066240 Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for Disease Control 655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada

1 0