- galaxy-dev - lists.galaxyproject.org

Galaxy and PBS/Torque Cluster
by Ryan Johnson 27 Aug '15

27 Aug '15

Hello All, We are using a Bio-Linux 8 Galaxy instance and are currently trying to integrate in into a PBS/Torque cluster. Currently we are able to run jobs locally but not via PBS. Can anyone provide us with a step by step guide to configure Galaxy and the PBS. I have included some of the errors we are receiving below: galaxy.datatypes.metadata DEBUG 2015-08-19 14:56:42,920 Failed to cleanup MetadataTempFile temp files from /usr/lib/galaxy-server/database/job_working_directory/000/96 /metadata_out_HistoryDatasetAssociation_100_Kk1ajw: No JSON object could be decoded galaxy.jobs.runners.pbs ERROR 2015-08-19 14:56:42,960 Connection to PBS server for submit failed: 0: no error We can not figure out what is stopping our jobs from entering the PBS queue. Below is our job_conf.xml. <?xml version="1.0"?> <job_conf> <plugins workers="2">   <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="2"/> <plugin id="pbs" type="runner" load="galaxy.jobs.runners.pbs:PBSJobRunner"/> </plugins> <handlers> <handler id="main"/>  </handlers> <destinations default="defaultq">  <destination id="defaultq" runner="pbs" tags="cluster"> <param id="-q">defaultq</param> <param id="-l">walltime=24:00:00</param> </destination> </destinations> </job_conf> Thank you very much for all the assistance! Regards, Ryan -- Ryan S. Johnson, PhD Applications Scientist Center for Advanced Research Computing P: (650) 430-6194 E:rjohns03@carc.unm.edu

2 1

Change genome build for multiple files
by Frederik Coppens 27 Aug '15

27 Aug '15

Hi is there a convenient way to change the genome build for a lot of files at once (e.g. a whole data library)? I want to ‘update’ to a newer genome build and the only way I found is to go through ‘Edit information’ for each file separately. I assume it can be changed directly in the db, but I’m not familiar with the scheme. thanks Frederik ================================================================== Frederik Coppens, Ph.D. Project Leader Applied Bioinformatics & Biostatistics IT manager Tel: 32 (0)9 3313812 Fax:32 (0)9 3313809 VIB DEPARTMENT OF PLANT SYSTEMS BIOLOGY, UGent Technologiepark 927, 9052 Gent, BELGIUM frcop(a)psb.ugent.be http://www.psb.ugent.be/abb ==================================================================

2 2

API for job and workflow input metadata
by Marco Ocana 24 Aug '15

24 Aug '15

Hi, I am working on writing some code that would allow for remotely executing jobs and workflows in Galaxy. Looking through the docs and examples that I have encountered, we are able to run specific tools if we know what are the inputs needed. Does the API (plain REST, bioblend, blend4j) allow for discovering what are the inputs needed to execute a specific tool? I'd be interested in things like argument name, type, label, description. And in the case where the input parameter is restricted to a number of options (e.g., pulldown selections), can I obtain those that would be available from the web pages? File types? etc. Big plus if we can find out similar information on the available output result. Thanks Marco

2 2

Change in toolshed disallowing install of previous versions of tools?
by Aaron Petkau 21 Aug '15

21 Aug '15

Hey everyone, So I rely a lot on specific versions of tools in the toolshed, and am often testing out the install process to make sure everything's working. Today I noticed that my install process was failing, and it looks like it's due to not being able to install previous versions of tools through the API. It seems to default to installing the latest version of a tool. For example, for the tool samtools_mpileup <https://toolshed.g2.bx.psu.edu/repository?repository_id=01d08a1b766b864e&ch…>, if I run the code: gi.toolShed.install_repository_revision("http://toolshed.g2.bx.psu.edu/ ",'samtools_mpileup','devteam','973fea5b4bdf',install_tool_dependencies=True,install_repository_dependencies=True) in bioblend (to install revision 973fea5b4bdf) I instead get the latest (aa0ef6f0ee89) revision installed. I can see the previous revision still exists https://toolshed.g2.bx.psu.edu/repository?repository_id=01d08a1b766b864e&ch… . I'm wondering if something has changed here? I remember being able to install old versions of tools through the API before, but maybe I'm getting mixed up? Thanks, Aaron

2 4

Automated training environments - was Re: 20 August GalaxyAdmins Meetup: Genomic Data
by mark fernandes (IFR) 21 Aug '15

21 Aug '15

Thanks Bjorn, Have already pulled a few of your Docker containers (And have been learning an awful lot from your Github code & the BioBlend source) Will try and get further with this later (My Docker machine is being restored from backup and on my desktop Kitematic is hanging at 99% - obviously Friday is not the day to be doing these sorts of things!). Enjoyed listening in on the Admin meetup - v. interesting to see where things are going. Will have to add Ansible to my (growing) reading list! :-) Tried the Flavour generator previously and generated some Dockerfiles (I had been shown an earlier version by Martin Page). It looks really good. Good to see tooltip info on returned tools now. Can I be bold and suggest that it might help to have Docker short description as a tool-tip for the starting point Docker image so it's clear what the specific mage provides? Keep up the good work! Best, Mark Fernandes T: +44 (0)1603 255000 Institute of Food Research, Norwich Research Park, Norwich, NR4 7UA, UK www.ifr.ac.uk Facebook | Twitter | LinkedIn | YouTube The Institute of Food Research receives strategic funding from BBSRC

2 1

Re: [galaxy-dev] 20 August GalaxyAdmins Meetup: Genomic Data Management; Tool Installation Automation
by mark fernandes (IFR) 20 Aug '15

20 Aug '15

Sounds like this will be an interesting meeting and quite germane to a requirement of mine. I want to achieve an environment as follows: 1) Galaxy Docker container (Thanks Bjorn :-) )with minimal tools deliverable to course developers machine e.g. via Kitematic 2) Course developer adds minimal set of Galaxy tools required to deliver course 3) Developer then adds shared data, workflows, notes & materials and tests everything. 4) Runs tools to extract tools etc to an export set 5) Course can be then deployed to a (local) Training server's Docker container using complementary set of tools to import from the export set (or to a user's desktop e.g.via Kitematic as part of an e-learning environment) Currently working my way through Bioblend API docs & notes from GCC 2015 workshop to see how much of this is achievable (also have Bjorn's notes on cloning tools from Galaxy Main which I hope to adapt). Any advice/insights welcomed along with any advances in automation from Development team. Regards, Mark Fernandes Bioinformatics/Biomathematics Training Training information blog: http://scicomp-training.ifr.ac.uk/ 'Open door' BI/BM training surgery (i.e. just turn up) every Thursday 2.30pm-3.30pm T: +44 (0)1603 255000 Institute of Food Research, Norwich Research Park, Norwich, NR4 7UA, UK www.ifr.ac.uk Facebook | Twitter | LinkedIn | YouTube The Institute of Food Research receives strategic funding from BBSRC

2 1

Re: [galaxy-dev] new visualization tool for composite datatype?
by Carl Eberhard 19 Aug '15

19 Aug '15

Yes, you can still use the framework to create interactive python-only visualizations. The simplest way to do this is form-based interaction. 1. The visualization mako page renders a form that contains the options the user can choose from 2. The user chooses and submits the form 3. Galaxy will re-render the visualization mako and now, since those options are passed as parameters, the chosen visualization can be rendered So you can render the form with your options: %if not query.get( 'param2', None ): ## initial render of form <form> <input name="param2" /> <input name="param3" /> <input name="param4" /> <button type="submit">Draw</button> </form> %else: ## second render drawing with param2 ... Note: this will default to a GET based form whose 'action' url will be the visualization's url. Any html5 form based controls will work: radio buttons, ranges, etc. When the user submits the form, each of the params, will now be part of the query string when the mako renders the second time: /visualization/show/myvis?param2=something&param3=... You can then use these params to call your python code with the user chosen options: arg2 = query.get( 'param2', default ) arg3 = query.get( 'param3', ... from draw import main returned = main( arg2, arg3, ... ) <svg> ${ returned } </svg> (or something similar) Other things that may help?: - you can use action="post" if you want to post data instead of get - handy for larger parameters - try to not bootstrap/load all your data into the mako file as this will be re-fetched everytime the form is re-submitted - you can add the arg2, arg3, etc. params to the config file and the registry will parse those for you and put them in the template's 'config' variable, e.g.: int_param2 = config.param2 - when using these query/form parameters you can now save/generate urls that will auto-start your visualizations since all the configuration options are sent in the url Let me know if I can give more info or that doesn't end up working. On Wed, Aug 19, 2015 at 9:30 AM, Anne Claire Fouilloux < a.c.fouilloux(a)geo.uio.no> wrote: > Hi Carl, > > > Yes I have a python code to manipulate and draw the data (it creates svg > files and I wish to add this visualization tool to my galaxy instance: > > Let's say my composite datatype contains 3 files: > > -file1.dat > > -file2.dat > > -file3.dat > > My python code takes 4 arguments (for instance draw.py arg1 arg2 arg3 > arg4). > > arg1: The first argument is a path giving the location of these 3 files. > > arg2, arg3 and arg4 are list of options. These lists are dynamic as they > depend on the content of these 3 files. > > Once the user has selected some options, I have a "draw" button and would > like the python script draw.py to be executed and generate a plot. > > Your answer was very helpful; at least it clarifies a number of things. > Now I understood I can access to these files using hda.extra_files_path in > my mako file: > > <% > > composite_file_path=hda.extra_files_path + '/' + date + '/' > > %> > > Which is helpful to generate my select buttons (for arg2, arg3 and arg4) > as I fist need to read the content of these files for the generation of > these buttons. > > > I still need to call my python script with the option chosen by the user. > Is it something I can easily do with galaxy? > > > Thanks, > > > Anne. > > > > ------------------------------ > *From:* Carl Eberhard <carlfeberhard(a)gmail.com> > *Sent:* 18 August 2015 18:59 > *To:* Anne Claire Fouilloux > *Cc:* galaxy-dev(a)lists.galaxyproject.org > *Subject:* Re: [galaxy-dev] new visualization tool for composite datatype? > > Hi, Anne > > Can you be more specific about how you'd like to visualize the files? It > sounds like you've already got a tool that will build an html file which is > the classic way to make visualizations with galaxy. > > Now you'd like to make it more interactive? Do you already have javascript > or python code that will be used to manipulate or draw the data? The > visualization registry will help you link a user's data to your > visualization code but will not create the code that does the rendering or > interaction. > > Aside from that, if you know the structure and filenames of your > composite's extra files directory, currently you can access those files > using: > api/histories/{history_id}/contents/{content_id}/display?filename=<some > file> > The meme type will be guessed based on the extension generally and more > complex paths can also be specified using the filename parameter. > > So, for example, if I have a FastQC Html composite file with an extra > files path of: > database/files/000/dataset_83_files > > And within that directory, fastqc makes an additional subdirectory: > sample1_fastqc/Icons > sample1_fastqc/Images > sample1_fastqc/summary.txt > > I can fetch the summary.txt file (with the 'text/plain' memetype) using: > > api/histories/{history_id}/contents/{content_id}/display?filename=sample1_fastqc/summary.txt > > Let me know if that wasn't the info you were after or I misunderstood the > question. > Carl > > > On Fri, Aug 14, 2015 at 7:50 AM, Anne Claire Fouilloux < > a.c.fouilloux(a)geo.uio.no> wrote: > >> Hi, >> >> >> I created a new composite datatype (class derived from Html class) >> myDatatype and I sucessfully use it in a new created tool. >> >> >> When I click on "View data" it shows my Html file with a list of binary >> files and everything is Ok. >> >> >> Now I would like to Visualize my data. I created a new directory in >> >> galaxy-dist/config/plugins/visualizations/ following instructions given >> at >> >> https://wiki.galaxyproject.org/VisualizationsRegistry and read >> >> https://wiki.galaxyproject.org/VisualizationsRegistry/Configuration and >> >> https://wiki.galaxyproject.org/DataProviders >> >> >> However, I still don't understand what I should do to be able to access >> files from my composite datatype to be able to visualize myDatatype (I mean >> I need to visualize the files containing in the directory listed by the >> html file). >> >> >> Is it possible? Where can I find some information on how to do it? >> >> >> Thanks, >> >> >> Anne. >> >> >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> https://lists.galaxyproject.org/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ >> > >

1 0

Dataset Collections status
by Keith Suderman 19 Aug '15

19 Aug '15

Greetings, I started pulling Galaxy code from the dev branch a few months ago to take advantage of the (then just emerging) dataset collections feature. However, it is not clear to me from the latest release notes if the data collections are now fully merged into master, or if I should continue to use the code in the dev branch to take advantage of bleeding edge code. I would like to move back to the master branch as soon as feasible. When running workflows over dataset collections I will frequently see errors like: /bin/sh: 1: /home/galaxy/galaxy_old/database/job_working_directory/001/1216/galaxy_1216.sh: Text file busy Which, from what I can tell, occurs when one process is trying to modify/delete a file open in another process. While the error seems to be repeatable, it also seems random as the errors do not occur in the same places if I run the workflow multiple times. Given that I am working from the dev branch I don't want to open/raise issues on features still in development. But if this is unexpected then I can do some more investigating and file a proper bug report. Cheers, Keith ------------------------------ Research Associate Department of Computer Science Vassar College Poughkeepsie, NY

3 2

20 August GalaxyAdmins Meetup: Genomic Data Management; Tool Installation Automation
by Dave Clements 19 Aug '15

19 Aug '15

Hello all, The next GalaxyAdmins online meetup is next week, on Thursday, August 20 <https://wiki.galaxyproject.org/Community/GalaxyAdmins/Meetups/2015_08_20>. The featured topics are: - *Genomic data management at Canada's National Microbiology Laboratory <https://www.nml-lnm.gc.ca/index-eng.htm> with IRIDA <http://www.irida.ca/> and Galaxy*, presented by Aaron Petkau <https://github.com/apetkau> - *Adding transparency and automation into the Galaxy tool installation process <https://github.com/afgane/ansible-tools>, *presented by Enis Afgan <https://wiki.galaxyproject.org/EnisAfgan> The meetup will use the same Adobe Connect technology <https://wiki.galaxyproject.org/Community/GalaxyAdmins/Meetups/2015_08_20#Ca…> used on previous calls. It takes a couple of minutes to connect, so you are encouraged to start connecting a few minutes early. See your local time <http://www.timeanddate.com/worldclock/fixedtime.html?msg=GalaxyAdmins+Augus…> to know when to connect. Looking forward to our first post-GCC gathering, Dave C and Hans-Rudolf -- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ https://wiki.galaxyproject.org/

1 1

Galaxy in Amazon
by Ryan G 19 Aug '15

19 Aug '15

Hi all - We are running a local instance of Galaxy on our internal infrastructure. It seems to be going well. We've gotten to the point where we are ready to migrate our NGS data to Amazon for storage in S3. We are also looking at how Galaxy can be used in Amazon. Specifically, we are interested in understanding: 1) Should we run an instance of Galaxy in Amazon, or continue to run it locally (to minimize costs) but have it run analyses in Amazon? 2) Regardless of how we run it, data will be stored in S3. How will Galaxy interact with S3 for its Data Libraries? 3) Is it even possible to separate the Galaxy web interface from the HPC cluster? 3) We understand Galaxy in Amazon uses CloudMan. Can we run this in our VPC with our own AMI? If anyone can provide insights into how they are using Galaxy in Amazon, I am very interested to hear your thoughts. Ryan

2 1