October 2012 - galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] Using $NSLOTS in tools to control thread number
by Peter Cock 19 Oct '12

19 Oct '12

On Fri, Jun 15, 2012 at 4:38 PM, James Taylor <james(a)jamestaylor.org> wrote: > This is exactly what I think we should do (and have for a long time), but I > think the variable should be something like: > > GALAXY_CPUS > > (threads is not accurate, a multithread or multiprocess job might want to use > this info, something even more abstract than CPUS might make sense, but > SLOTS has never made sense to me). I agree that a Galaxy specific name makes a lot of sense, and that the SGE term "slots" is a bit odd. Using CPUS however is potentially ambiguous with CPUs vs cores - my desktop has two quad core CPUs, i.e. 2 CPUs but 8 cores. Where do you think this number should come from? A new entry in the runner URL is simple albeit potentially redundant with cluster-specific entries in the runner URL. As to the alternative (doing it automatically), for PBS and SGE determining the number of cores from the cluster configuration and/or parsing the cluster runner URL sounds doable - what about the other backends? Peter

2 2

Confusing error message when tool not installed
by Peter Cock 19 Oct '12

19 Oct '12

Hi all, I have the assorted tools & wrappers under hg on a branch of my galaxy-central repository (the 'tools' branch), but they are not on the main branch (i.e. the 'default' branch). I switched from the 'tools' branch (which has the BLAST+ tools) to the 'default' branch, and then attempted to rerun a BLAST job in my history. This gave a red error but with a misleading error message: This dataset was created by an obsolete tool (ncbi_tblastx_wrapper). Can't re-run. I think it should have said something like: Tool 'ncbi_tblastn_wrapper' is not installed. This is not specific to the BLAST+ wrappers, it happens with all the locally installed tools. Peter P.S. If I clicked on the tool in the left hand menu (still there as while I had restarted the Galaxy server I had not reloaded the page in my browser), it said: Tool 'ncbi_tblastn_wrapper' does not exist, kwd={} (This was a plain white message, it lacked the nice red background). Again, the wording isn't ideal but this is a corner case.

1 0

windows nodes on cloudman/ec2
by Jorrit Boekel 19 Oct '12

19 Oct '12

Dear list, Our analysis pipeline is normally fed with proprietary raw file types from mass spectrometry instruments. These can currently only be read on windows systems where vendor DLLs are installed. Since Cloudman/Galaxy does not interface with Windows nodes out of the box, I implemented a conversion tool wrapper that instantiates windows nodes on EC2 and lets them convert the raw files. It currently only does this, but there would of course be room for a number of other tools on the same windows image. I know there are limitations such as Amazon's 20 node limit and the nodes not being controlled in Cloudman, but I wonder if my implementation would cause any serious problems with running this. Problems like security, etc (after all, the tool wrapper curls the user data to get the AWS access/secret keys. If anyone has comments on it, I'd love to hear them. Current code for the windows image can be found on: https://bitbucket.org/glormph/galaxy-ec2-windows And the tool that creates the windows node is on: https://bitbucket.org/glormph/adapt/changeset/ca13e548b85132049a918af84cfb0… cheers, jorrit -- Scientific programmer Mass spec analysis support @ BILS Janne Lehtiö / Lukas Käll labs SciLifeLab Stockholm

2 1

Cuff merge and cuffcompare sequence reference
by Jonas Grauholm 19 Oct '12

19 Oct '12

Greetings, I'm trying to analyse a number of mouse RNA-seq samples on a local galaxy install. When trying to run cuffmerge and cuffcompare I get the following error. No sequence data found for dbkey M_musculus_Ensembl_NCBIM37, so sequence data cannot be used. The tool works fine with a human reference genome so im fairly confident I just fumbled a config file somewhere. The problem is I been having difficulty locating exactly what file defines what sequence file is being used? I looked through all my config files but so far I been unable to locate the error, could anyone possibly point me in the right direction? Best Regards Jonas Grauholm AROS Applied Biotechnology A/S SciencePark Skejby Brendstrupgaardsvej 102 DK+8200 Aarhus N Phone: +45 70271170

2 1

Fwd: 'UsesFormDefinitions' is not defined
by Alexander Graf 19 Oct '12

19 Oct '12

Hi, yesterday I updated Galaxy to the latest revision and now it is not possible to access any Data Libraries or to create a new one. Galaxy shows me a read box with following error message: *Error attempting to display contents of library (100326_HWUSI-EAS632R_0002): name 'UsesFormDefinitions' is not defined.* The log file message is: *"GET /galaxy/library_common/browse_library?sort=name&webapp=galaxy&f-description=All&f-name=All&page=1&cntrller=library&show_item_checkboxes=false&async=false&operation=browse&id=9d04ee723dc2cac7 HTTP/1.1" 302 - " http://x.x.x.x/galaxy/library/browse_libraries?status=error&use_panels=Fals…." "Mozilla/5.0 (X11; Linux x86_64) * Any ideas how to solve this problem or were to start troubleshooting? Thank you in advance Alex

4 5

Re: [galaxy-dev] pass more information on a dataset merge
by Alex.Khassapov＠csiro.au 19 Oct '12

19 Oct '12

Hi John, what I don't get - I specify the output format "m:grd", my tool generates multiple output files in dataset_id_files folder, but the dataset_id.dat file is empty. I need to call this regenerate_primary_file() to add the HTML with the file list to the dat file. But I'm not sure where? -Alex -----Original Message----- From: jmchilton(a)gmail.com [mailto:jmchilton@gmail.com] On Behalf Of John Chilton Sent: Friday, 19 October 2012 6:16 AM To: Khassapov, Alex (CSIRO IM&T, Clayton) Subject: Re: [galaxy-dev] pass more information on a dataset merge On Tue, Oct 16, 2012 at 11:11 PM, <Alex.Khassapov(a)csiro.au> wrote: > Hi John, > > I am definitely interested in this idea, not only me - we are currently working on moving a few scientific tools (not related to genome) into cloud using Galaxy. Great. My interests in Galaxy are mostly outside of genomics as well, it is good to have more people utilizing Galaxy in this way because it will force the platform to become more generic and address more broader use cases. > > We will try it further and see if we need any changes. For now one improvement would be nice, make dataset_id.dat contain list of paths to the location of the uploaded files, so by displaying html page the user could just click on the link and download the file. > Code that attempted to do this was in there, but didn't work obviously. I have now fixed it up. Thanks for beta testing. -John > We are pretty new to Galaxy, so our understanding of Galaxy is pretty limited. > > Thanks again, > > Alex > > > -----Original Message----- > From: jmchilton(a)gmail.com [mailto:jmchilton@gmail.com] On Behalf Of > John Chilton > Sent: Wednesday, 17 October 2012 3:21 AM > To: Khassapov, Alex (CSIRO IM&T, Clayton) > Subject: Re: [galaxy-dev] pass more information on a dataset merge > > Wow, thanks for the rapid feedback! I have made the changes you have suggested. It seems you must be interested in this idea/implementation. Let me know if you have specific use cases/requirements in mind and/or if you would be interested in write access to the repository. > > -John > > On Mon, Oct 15, 2012 at 11:51 PM, <Alex.Khassapov(a)csiro.au> wrote: >> Hi John, >> >> I tried your galaxy-central-homogeneous-composite-datatypes implementation, works great thank you (and Jorrit). >> >> A couple of fixes: >> >> 1. Add multi_upload.xml to too_conf.xml 2. >> lib/galaxy/tools/parameters/grouping.py line 322 (in get_filenames( context )) - >> "if ftp_files is not None:" >> Remove "is not None" as ftp_files is empty [], but not None, then line 331 "user_ftp_dir = os.path.join( trans.app.config.ftp_upload_dir, trans.user.email )" throws an exeption if ftp_upload_dir isn't set. >> >> Alex >> >> -----Original Message----- >> From: galaxy-dev-bounces(a)lists.bx.psu.edu >> [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of John >> Chilton >> Sent: Tuesday, 16 October 2012 1:07 AM >> To: Jorrit Boekel >> Cc: galaxy-dev(a)lists.bx.psu.edu >> Subject: Re: [galaxy-dev] pass more information on a dataset merge >> >> Here is an implementation of the implicit multi-file composite datatypes piece of that idea. I think the implicit parallelism may be harder. >> >> https://bitbucket.org/galaxyp/galaxy-central-homogeneous-composite-da >> t >> atypes/compare >> >> Jorrit do you have any objection to me trying to get this included in galaxy-central (this is 95% code I stole from you)? I made the changes against a clean galaxy-central fork and included nothing proteomics specific in anticipation of trying to do that. I have talked with Jim Johnson about the idea and he believes it would be useful his mothur metagenomics tools, so the idea is valuable outside of proteomics. >> >> Galaxy team, would you be okay with including this and if so is there anything you would like to see either at a high level or at the level of the actual implementation. >> >> -John >> >> ------------------------------------------------ >> John Chilton >> Senior Software Developer >> University of Minnesota Supercomputing Institute >> Office: 612-625-0917 >> Cell: 612-226-9223 >> Bitbucket: https://bitbucket.org/jmchilton >> Github: https://github.com/jmchilton >> Web: http://jmchilton.net >> >> On Mon, Oct 8, 2012 at 9:24 AM, John Chilton <chilton(a)msi.umn.edu> wrote: >>> Jim Johnson and I have been discussing that approach to handling >>> fractionated proteomics samples as well (composite datatypes, not >>> the specifics of the interface for parallelizing). >>> >>> My perspective has been that Galaxy should be augmented with better >>> native mechanisms for grouping objects in histories, operating over >>> those groups, building workflows that involve arbitrary numbers of >>> inputs, etc... Composite data types are kindof a kludge, I think >>> they are more useful for grouping HTML files together when you don't >>> care about operating on the constituent parts you just want to view >>> pages a as a report or something. With this proteomic data we are >>> working with, the individual pieces are really interesting right? >>> You want to operate on the individual pieces with the full array of >>> tools (not just these special tools that have the logic for dealing >>> with the composite datatypes), you want to visualize the files, >>> etc... Putting these component pieces in the composite data type >>> extra_files path really limits what you can do with the pieces in Galaxy. >>> >>> I have a vague idea of something that I think could bridge some of >>> the gaps between the approaches (though I have no clue on the >>> feasibility). Looking through your implementation on bitbucket it >>> looks like you are defining your core datatypes (MS2, CruxSequest) >>> as subclasses of this composite data type (CompositeMultifile). My >>> recommendation would be to try to define plain datatypes for these >>> core datatype (MS2, CruxSequest) and then have the separate >>> composite datatype sort of delegate to the plain datatypes. >>> >>> You could then continue to explicitly declare subclasses of the >>> composite datatype (maybe MS2Set, CruxSequestSet), but also maybe >>> augement the tool xml so you can do implicit data type instances the >>> way you can with tabular data for instance (instead of defining >>> columns you would define the datatype to delegate to). >>> >>> The next step would be to make the parallelism implicit (i.e pull it >>> out of the tool wrapper). Your tool wrappers wouldn't reference the >>> composite datatypes, they would reference the simple datatypes, but >>> you could add a little icon next to any input that let you replace a >>> single input with a composite input for that type. It would be kind >>> of like the run workflow page where you can replace an input with a >>> multiple inputs. If a composite input (or inputs) are selected the >>> tool would then produce composite outputs. >>> >>> For the steps that actually combine multiple inputs, I think in your >>> case this is perculator maybe (a tool like interprophet or Scaffold >>> that merges peptide probabilities across runs and groups proteins), >>> then you could have the same sort of implicit replacement but >>> instead of for single inputs it could do that for multi-inputs >>> (assuming the Galaxy powers that be accept my fixes for multi-input tool parameters: >>> https://bitbucket.org/galaxy/galaxy-central/pull-request/76/multi-input-dat…) >>> >>> The upshot of all of that would be that then even if these >>> composites datatypes aren't used widely, other people could still >>> use your proteomics tools (my users are definitely interested in >>> Crux for >>> instance) and you could then use other developers' proteomic tools >>> with your composite datatypes even though they weren't designed with >>> that use case in mind (I have msconvert, myrimatch, idpicker, >>> proteinpilot, Ira Cooke has X! Tandem, OMSSA, TPP, and NBIC has an >>> entire suite of label free quant tools). A third benefit would be >>> that people working in other -omicses could make use of the >>> homogenous composite datatype implementation without needing to >>> rewrite their wrappers and datatypes. >>> >>> There is probably something that I am missing that makes this very >>> difficult, let me know if you think this is a good idea and what its >>> feasibility might be. I forked your repo and set off to try to >>> implement some of this stuff last week and I ended up with my galaxy >>> pull requests to improve batching workflows and multi-input tool >>> parameters instead, but I hope to eventually get around to it. >>> >>> -John >>> >>> ------------------------------------------------ >>> John Chilton >>> Senior Software Developer >>> University of Minnesota Supercomputing Institute >>> Office: 612-625-0917 >>> Cell: 612-226-9223 >>> Bitbucket: https://bitbucket.org/jmchilton >>> Github: https://github.com/jmchilton >>> Web: http://jmchilton.net >>> >>> On Mon, Oct 1, 2012 at 8:24 AM, Jorrit Boekel >>> <jorrit.boekel(a)scilifelab.se> wrote: >>>> Dear list, >>>> >>>> I thought I was working with fairly large datasets, but they have >>>> recently started to include ~2Gb files in sets of >50. I have ran >>>> these sort of things before as merged data by using tar to roll >>>> them up in one set, but when dealing with >100Gb tarfiles, Galaxy >>>> on EC2 seems to get very slow, although that's probably because of >>>> my implementation of dataset type detection (untar and read through files). >>>> >>>> Since tarring/untarring isn't very clean, I want to switch from >>>> tarring to creating composite files on merge by putting a tool's >>>> results into the dataset.extra_files_path. This doesn't seem to be >>>> supported yet, because we currently pass in do_merge the output >>>> dataset.filename to the respective datatype's merge method. >>>> >>>> I would like to pass more data to the merge method (let's say the >>>> whole dataset object) to be able to get the composite files directory and 'merge' >>>> the files in there. Good idea, bad idea? If anyone has views on >>>> this, I'd love to hear them. >>>> >>>> cheers, >>>> jorrit >>>> >>>> ___________________________________________________________ >>>> Please keep all replies on the list by using "reply all" >>>> in your mail client. To manage your subscriptions to this and >>>> other Galaxy lists, please use the interface at: >>>> >>>> http://lists.bx.psu.edu/ >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: >> >> http://lists.bx.psu.edu/

1 0

Retreiving a library data type in <command>
by Simon Gladman 19 Oct '12

19 Oct '12

Hi all, Have searched mailing list to no avail. Therefore: How can I get a data library's data_type in the <command> tagset? Example: <command> dosomething.pl

2 2

Jobs crash only in workflow context
by Todd Oakley 18 Oct '12

18 Oct '12

Hello, We just did a few tweaks to improve Galaxy performance, and a new issue popped up that I would like advice on troubleshooting. When we run workflows, we see that tools later in the workflow run and crash before the results they depend on have completed running. We can re-run the crashed jobs later and they work fine, suggesting that they are only failing in the context of running workflows. I'd appreciate any advice on how to start troubleshooting this problem. Thanks much! Todd -- *************************************** Todd Oakley, Professor Ecology Evolution and Marine Biology University of California, Santa Barbara Santa Barbara, CA 93106 USA ***************************************

2 2

Error while running freshly installed velvet
by Philipe Moncuquet 18 Oct '12

18 Oct '12

Hi galaxy-dev community, I recently installed velevetoptimiser via the toolshed. I copied binaries in the tool directory on our local Galaxy. I wanted to give it a try to confirm installation was correct and get the following error message (see below). I know that the output files are created via Galaxy with some name looking like dataset_###.dat, one of this should be corresponding to 'contig.fa'. Could it be that python script is not looking for the right file name ? As anybody saw the same error before? Thanks Traceback (most recent call last): File "/data/galaxyTools/shed_tools_dev/toolshed.g2.bx.psu.edu/repos/konradpaszkiewicz/velvetoptimiser/50ae1360fbbe/velvetoptimiser/velvet_optimiser.py", line 102, in <module> if __name__ == "__main__": __main__() File "/data/galaxyTools/shed_tools_dev/toolshed.g2.bx.psu.edu/repos/konradpaszkiewicz/velvetoptimiser/50ae1360fbbe/velvetoptimiser/velvet_optimiser.py", line 67, in __main__ for line in open(contigs_path ): IOError: [Errno 2] No such file or directory: 'contigs.fa'

1 0

problem viewing galaxy menu items using IE, version number
by Sukha Malladi 17 Oct '12

17 Oct '12

Hello, 1. When we access galaxy (local installation) using IE (internet explorer) 8 and 9, we cannot expand/collapse uploaded data files visible from history window. Neither are the info and save icons visible. How do I fix this ? Also items under the main menu are not clickable (e.g. when I click on User menu item, it expands to show User and Register, but I cannot select User or Register items). I can view/click these items using Firefox browser. 2. How do I determine the version number of my installation ? I do not have the original tar file that has the version name. Appreciate your help. Sukha

1 0