- galaxy-dev - lists.galaxyproject.org

Fwd: Dynamically Generated Tools
by Ross 19 Jul '13

19 Jul '13

AFAIK it's not yet possible, but a tool shed API to expose subversion hooks for repository creation and update is a logical development which is already on the radar AFAIK. For the bigger picture of complexity for the user, I think some key clarifications required include: which 'user' is experiencing complexity; how big a barrier it is to generating tools; how readily can we make it go away; how much will it help? Tool builders (even 'users' of the TF) might reasonably be expected to have a capacity for substantial complexity that the end user of the generated tools should not need to possess - after all, once a generated tool is installed in Galaxy, the Galaxy user can't tell the difference and sees no more complexity from the generated tool than a hand written one. On absent local tool sheds: it's trivial to run a toolshed on a dev laptop and not hard to run in production - but tool makers can always use the public test and main TS if they want to distribute new tools so I'm not sure how big a barrier a missing local toolshed really is. On Thu, Jul 18, 2013 at 12:08 PM, Michael E. Cotterell < mepcotterell(a)gmail.com> wrote: > I didn't mean to imply it was unclear. It definitely made sense. > > I completely agree with you. However, is there for a way (maybe via the > toolshed api) for my tool to push these tools to a TS repo? Also, doesn't > that add some more complexity for the user? Not everyone runs a local tool > shed. > > Sincerely, > Michael E. Cotterell > > Ph.D. Student in Computer Science, University of Georgia > Instructor of Record, Graduate RA & TA, University of Georgia > Faculty Liaison, CS Graduate Student Association, University of Georgia > mepcotterell(a)gmail.com (mailto:mepcotterell@gmail.com) > mepcott(a)uga.edu (mailto:mepcott@uga.edu) > mec(a)cs.uga.edu (mailto:mec@cs.uga.edu) > http://michaelcotterell.com/ > > > On Wednesday, July 17, 2013 at 10:05 PM, Ross wrote: > > > Sorry if my post was rambling and unclear. > > > > Here's the executive summary for what it's worth: > > IMHO, a Galaxy tool that generates tools should emit them as tool shed > compatible artefacts, ready for uploading to a TS repository for automated > installation to any target instance. > > > > > > > > > > > > On Thu, Jul 18, 2013 at 11:25 AM, Michael E. Cotterell < > mepcotterell(a)gmail.com (mailto:mepcotterell@gmail.com)> wrote: > > > Ross, > > > > > > Thanks for the comments. > > > > > > I've already made my main tool "tool_shed" compatible. What I'm > wondering about is what do I do with the tools that my tool creates? > > > > > > Sincerely, > > > Michael E. Cotterell > > > > > > Ph.D. Student in Computer Science, University of Georgia > > > Instructor of Record, Graduate RA & TA, University of Georgia > > > Faculty Liaison, CS Graduate Student Association, University of Georgia > > > mepcotterell(a)gmail.com (mailto:mepcotterell@gmail.com) (mailto: > mepcotterell(a)gmail.com) > > > mepcott(a)uga.edu (mailto:mepcott@uga.edu) (mailto:mepcott@uga.edu) > > > mec(a)cs.uga.edu (mailto:mec@cs.uga.edu) (mailto:mec@cs.uga.edu) > > > http://michaelcotterell.com/ > > > > > > > > > On Wednesday, July 17, 2013 at 7:44 PM, Ross wrote: > > > > > > > Hi, Michael, > > > > > > > > As others have said, in the long term, tool wrappers and their > dependencies will be distributed and installed through tool sheds rather > than being distributed with Galaxy source, so you might want to plan to > generate and upload tool shed compatible archives from the get-go. The test > and main tool sheds contain plenty of examples and Björn's tools exercise > pretty much all of the functionality - you can browse the repo structure > which is identical to the gz archive structure you need to upload to a new > repo, or download the repo as a gz, unpack it and get exactly the kind of > directory structure and contents you need to emulate for your tools. Once > you have a working tool packaging the archive up is straightforward. > > > > > > > > The format for tool shed repo uploads and syntax for the tags used > to define dependencies is very well documented in the tool shed section of > the wiki. As Björn points out, the tool factory python wrapper might be a > useful source of ideas and perhaps code. Your tool generator will need to > do something similar to write the content and generate complete tool shed > archives. When generating a new tool, the TF uses an ugly XML wrapper > generator (contributed improvements would be very welcome!) and (probably > more usefully) the few lines of code you need to package up the functional > test data and the XML and wrapper if you need one in toolshed archive > format. > > > > > > > > > > > > On Thu, Jul 18, 2013 at 5:30 AM, Björn Grüning < > bjoern.gruening(a)pharmazie.uni-freiburg.de (mailto: > bjoern.gruening(a)pharmazie.uni-freiburg.de) (mailto: > bjoern.gruening(a)pharmazie.uni-freiburg.de)> wrote: > > > > > Hi Michael, > > > > > > > > > > I think you will enter new ground with your tool. The closest tool > that > > > > > will do something similar is Ross toolfactory, I think: > > > > > > > > > > http://toolshed.g2.bx.psu.edu/view/fubar/toolfactory > > > > > > > > > > For me one question is, do you really want to offer that service to > > > > > every user and is it save? > > > > > > > > > > > TL;DR I have a tool that generates other tools. Where should > those tools go, and how should I add them (without modifying Galaxy code)? > > > > > > > > > > > > I apologize, in advance, for the wall of text. I would like some > advice on what the best way to go about setting up a tool that generates > other tools... > > > > > > > > > > > > As some of you (especially those who saw my boss's talk at GCC) > already know, I'm working on a tool that lets galaxy users add web service > operations as tools to Galaxy. > > > > > > We have a general purpose client for invoking web services, and > we have a tool that let's a user enter a WSDL/WADL URL, select the > operations they want to add, and generates tool config XML files for Galaxy. > > > > > > > > > > > > Right now, I'm in the process of making our tool "tool shed" > compatible, and that's going along great > > > > > > (I especially like the tool dependency installation stuff), but > I'm trying to determine the best course of action on the following: > > > > > > > > > > > > 1. Where is the best place to put the tool config (XML) files > that are generated by my tool? My predecessors just placed them in a > subdirectory under our tool's directory. > > > > > > I'm ok with this, but does that mean that all the generated > files will get removed if they upgrade our tool via the tool shed (haven't > taken the time to see if > > > > > > Galaxy just does an "hg pull" or if it's more complicated than > that)? > > > > > > > > > > > > > > > > > > > > It is mainly a hg pull. But if you deinstall that repository and > > > > > reinstall it, all your files are lost. > > > > > > > > > > > Would it be better to have my tool create a local tool shed > repository > > > > > > (just for these generated tools), and if so, can the tool shed > API let my tool install this repository? Having not looked at > install_tool_shed_repositories.py yet, > > > > > > it's unclear from the wiki documentation whether or not POST > /api/tool_shed_repositories/install_repository_revision requires that the > tool shed already registered in tool_sheds_conf.xml)? > > > > > > > > > > > > > > > > > > > > I think that is a better way to do it. The API can be used to > install > > > > > your tools. But it need to be configured beforehand from the user, > > > > > administrator. > > > > > > > > > > > 2. What's the best way to add the generated tools to Galaxy? My > predecessors modified the tool_config.xml directly and required users to > restart Galaxy. > > > > > > Now, If these tools are in a tool shed repo, I know that I can > use the add_to_tool_panel() method in lib/tool_shed/util/tool_util.py if > these tools are in a tool shed repo, however, > > > > > > if that's not the recommended course of action, then I'd like > some advice on what to do? > > > > > > > > > > > > > > > > > > > > There is a feature request, that you can specify in your > > > > > toolshed-repository the preferred location in the tool panel. For > now I > > > > > would ignore it and just install the tools somewhere and hope that > > > > > feature request is quiet high in Gregs ToDo list :) > > > > > > > > > > Cheers, > > > > > Bjoern > > > > > > > > > > -- Ross Lazarus MBBS MPH; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444 http://scholar.google.com/citations?hl=en&user=UCUuEM4AAAAJ

2 2

permissions
by Lee Katz 19 Jul '13

19 Jul '13

Hi, I buried this into another email but I would like to get some help on it and therefore have to make it visible. Newer users to my Galaxy instance are not showing up under permissions when I try to give them access to MiSeq data. Is there any kind of troubleshooting steps I can take to understand why? The new users and their new roles do not show up under "roles associated" when I try to edit permissions. However, they do show up when I try to edit folder permissions. Thank you! -- Lee Katz, Ph.D.

1 0

tool inheritance
by Lee Katz 19 Jul '13

19 Jul '13

One more question for the Galaxy gurus. I have a series of tools that 1) interleave raw reads, 2) assemble them, and 3) predict genes from the assembly. How can I display inheritance on the data produced from these tools? It would be really nice if a set of genes showed that they came from an assembly, which came from a set of raw reads. I can't find this on the site, and so I am sure I am using the wrong keywords or something... Thank you! -- Lee Katz, Ph.D.

2 2

Re: [galaxy-dev] FastQC files are not being recognized by
by Ambrose Andongabo (RRes-Roth) 19 Jul '13

19 Jul '13

Try the fastqsanger datatype if it works fine if not try the others such as fastqillumina etc Thanks Ambrose -----Original Message----- From: galaxy-dev-bounces(a)lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Ricardo Perez Sent: 18 July 2013 20:41 To: galaxy-dev(a)lists.bx.psu.edu Subject: [galaxy-dev] FastQC files are not being recognized by Dear all, We are doing some processes in our server that use fastq files. We have obtained the data in .sra format and then convert them into fastq format using the sra tool kit. However, when we upload this files into galaxy, some tools does not recognize the file as fastq. When we apply FasQC:Read QC, under summary statistics it says that encoding is Sanger / Illumina 1.9. Is this an error of our part while installing galaxy in our server? Thank you for your time, --Ricardo Perez ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- This message has been scanned for viruses and dangerous content by MailScanner, and we believe but do not warrant that this e-mail and any attachments thereto do not contain any viruses. However, you are fully responsible for performing any virus scanning. -- This message has been scanned for viruses and dangerous content by MailScanner, and we believe but do not warrant that this e-mail and any attachments thereto do not contain any viruses. However, you are fully responsible for performing any virus scanning.

1 0

Strange behaviour with galaxy history
by Groot, Philip de 19 Jul '13

19 Jul '13

Hi, First of all, I apologize for interrupting into this discussion, but I have exactly the same problem and did not find a solution yet. And no: prefix and cookies are set properly in our particular setting (as "/")! This is not the cause... I think that the problem occurred with the May Galaxy update, but it was unnoticed for a while. Problem is that the histories of all users are not listed when being asked for it within Galaxy. Only the current history is present for every user. In addition, when I login using another computer, a different history is being depicted (which suggest a cookie issue but I just cannot find it). Copying the current history works but the copy is not listed either. In the postgres database, the histories are properly listed. If I type in the proper history name in Galaxy, I even get the stored history back! So this all works, but users cannot see their histories. The java console does not reveal any problem. prefix and cookie are set properly (I think). So I don't know. Does anyone have any clue on how to tackle this issue? I am happy to provide more information if needed! Thanks! Dr. Philip de Groot Ph.D. Bioinformatics Researcher Wageningen University / TIFN Nutrigenomics Consortium Nutrition, Metabolism & Genomics Group Division of Human Nutrition PO Box 8129, 6700 EV Wageningen Visiting Address: Erfelijkheidsleer: De Valk, Building 304 Dreijenweg 2, 6703 HA Wageningen Room: 0052a T: +31-317-485786 F: +31-317-483342 E-mail: Philip.deGroot(a)wur.nl<mailto:Philip.deGroot@wur.nl> Internet: http://www.nutrigenomicsconsortium.nl<http://www.nutrigenomicsconsortium.nl/> http://humannutrition.wur.nl/ https://madmax.bioinformatics.nl/

3 5

Support for Galaxy tools on Python
by Hyungro Lee 19 Jul '13

19 Jul '13

Hi, I was wondering if Galaxy provides python library for Galaxy tools. For example, if it did happen, filter and sort tools can be expressed like this: galaxy.filter(param1, param2, ...) and galaxy.sort(param1, param2, ...) instead of using command line tools like filtering.py $param1, $param2, ... I thought it might be useful for python developers with bioblend if they want to extend developments on python after launching galaxy and importing workflows through bioblend. Any plans and thoughts? Best, Hyungro

1 0

possible bug
by Berner, Thomas 18 Jul '13

18 Jul '13

Hey guys, we found a curios bug, but it is a little bit tricky to explain. Here is an example: - Click on "Concatenate datasets tail-to-head" - add a new data set - click on the first data field and select your data set of choice - press "tab" to get to the next data set an select the second data set by typing the number of the data set via the numpad (might only be possible if you have a long history with many files) => the first number is written as an "character", e.g., 1=a, 2=b, and so on. This bug only shows up if you use the numpad to type in the numbers. We tested it with different tools in our local Galaxy instance and in the Galaxy Main instance. Can anybody confirm this behavior? Best regards, Thomas

1 0

Re: [galaxy-dev] Cann't restart server after updating to the latest version
by Joachim Jacob | VIB | 18 Jul '13

18 Jul '13

Hi, Please keep questions on the galaxy-dev list. Can you paste the part of your universe_wsgi.ini that is under # --- HTTP server -------------------- ? Can you also provide the output (as user galaxy) of $ lsof -Pni | grep galaxy I think this error can happen if you try to start Galaxy when another instance is already running; or when you have misconfigured your universe_wsgi.ini. Cheers, Joachim Joachim Jacob Contact details: http://www.bits.vib.be/index.php/about/80-team On 07/18/2013 10:46 AM, Karen Chait wrote: > > Hello Jacob, > > Thanks for your help solving restart problem > http://dev.list.galaxyproject.org/Cann-t-restart-server-after-updating-to-t… > > I am facing the same problem again but this time when running ./run.sh > I get the following error: > > socket.error: [Errno 98] Address already in use > > galaxy.jobs.handler INFO 2013-07-18 03:28:46,328 sending stop signal > to worker thread > > galaxy.jobs.handler INFO 2013-07-18 03:28:46,328 job handler queue stopped > > galaxy.jobs.runners INFO 2013-07-18 03:28:46,328 LWRRunner: Sending > stop signal to monitor thread > > galaxy.jobs.runners INFO 2013-07-18 03:28:46,328 LWRRunner: Sending > stop signal to 3 worker threads > > galaxy.jobs.runners INFO 2013-07-18 03:28:46,328 LocalRunner: Sending > stop signal to 5 worker threads > > galaxy.jobs.handler INFO 2013-07-18 03:28:46,329 sending stop signal > to worker thread > > galaxy.jobs.handler INFO 2013-07-18 03:28:46,329 job handler stop > queue stopped > > Exception in thread JobHandlerQueue.monitor_thread (most likely raised > during interpreter shutdown): > > Do you know how to solve the problem? > > Thanks > > Karen >

1 0

Dynamically Generated Tools
by Michael E. Cotterell 18 Jul '13

18 Jul '13

TL;DR I have a tool that generates other tools. Where should those tools go, and how should I add them (without modifying Galaxy code)? I apologize, in advance, for the wall of text. I would like some advice on what the best way to go about setting up a tool that generates other tools... As some of you (especially those who saw my boss's talk at GCC) already know, I'm working on a tool that lets galaxy users add web service operations as tools to Galaxy. We have a general purpose client for invoking web services, and we have a tool that let's a user enter a WSDL/WADL URL, select the operations they want to add, and generates tool config XML files for Galaxy. Right now, I'm in the process of making our tool "tool shed" compatible, and that's going along great (I especially like the tool dependency installation stuff), but I'm trying to determine the best course of action on the following: 1. Where is the best place to put the tool config (XML) files that are generated by my tool? My predecessors just placed them in a subdirectory under our tool's directory. I'm ok with this, but does that mean that all the generated files will get removed if they upgrade our tool via the tool shed (haven't taken the time to see if Galaxy just does an "hg pull" or if it's more complicated than that)? Would it be better to have my tool create a local tool shed repository (just for these generated tools), and if so, can the tool shed API let my tool install this repository? Having not looked at install_tool_shed_repositories.py yet, it's unclear from the wiki documentation whether or not POST /api/tool_shed_repositories/install_repository_revision requires that the tool shed already registered in tool_sheds_conf.xml)? 2. What's the best way to add the generated tools to Galaxy? My predecessors modified the tool_config.xml directly and required users to restart Galaxy. Now, If these tools are in a tool shed repo, I know that I can use the add_to_tool_panel() method in lib/tool_shed/util/tool_util.py if these tools are in a tool shed repo, however, if that's not the recommended course of action, then I'd like some advice on what to do? Thanks! Sincerely, Michael E. Cotterell Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Faculty Liaison, CS Graduate Student Association, University of Georgia mepcotterell(a)gmail.com (mailto:mepcotterell@gmail.com) mepcott(a)uga.edu (mailto:mepcott@uga.edu) mec(a)cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/

4 6

Java JAXB for Tool Configuration Files
by Michael E. Cotterell 18 Jul '13

18 Jul '13

Hey guys, I don't know if there will be much interest in this, but in the course of revising code that my predecessors wrote, I discovered that they were hand crafting tool XML configuration files. Since we all know that makes reading and writing such files hard, I started work on creating a Java model for the tool config XML using JAXB. In short, I can create and populate some Java objects and then marshal them directly to XML, knowing that they'll be in the correct format. Also, it makes reading/unmarshalling tool config files easy as well. If there's any interest in this, I can probably separate out the model code into a separate project. At the moment, it's a subset of the tool config XML model that I need for my Web service tools, but I could foresee completing it in hopes of it being more useful to others. Imagine: define your tool using Java and have it print out the pretty XML! Also, there should be an XSD for the tool config files. I can generate one once my Java model is complete, but it's kind of important. In the wiki, it says something to the effect that all is needed is valid XML… That's not entirely correct. It should valid XML that conforms to an XML schema definition (now I'm ranting :P). If one already exists, then all the better! Anyway, let me know if there's any interest. Sincerely, Michael E. Cotterell Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Faculty Liaison, CS Graduate Student Association, University of Georgia mepcotterell(a)gmail.com (mailto:mepcotterell@gmail.com) mepcott(a)uga.edu (mailto:mepcott@uga.edu) mec(a)cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/

1 0