- galaxy-dev - lists.galaxyproject.org

Upload files from filesystem paths - backwards checkboxes
by Peter Cock 11 Mar '11

11 Mar '11

Hi all, Via the Galaxy admin interface, I'm currently importing some BAM files into a library from the local file system. Since they are big, I don't want Galaxy to copy them. So I looked at the "Copy data into Galaxy?" option and made sure it wasn't ticked. Galaxy did copy the file :( I looked at the interface again, and see you have: > Copy data into Galaxy? > [check box] No > Normally data uploaded with this tool is copied into Galaxy's "files" > directory so any later changes to the data will not affect Galaxy. > However, this may not be desired (especially for large NGS > datasets), so use of this option will force Galaxy to always > read the data from its original path. You've got a tick box labelled "No" so it does the opposite to the question above it. That is a HORRIBLE interface. I can't be the first person to be tricked by it, can I? I would urge you to remove the work "No" and flip the meaning (and default) to match. But that could confuse existing admin users used to clicking from previous habit. How about making it a select parameter instead, with options "Copy files (default)" and "Link to files" which is very explicit? Likewise for the "Preserve directory structure?" setting, I'd remove the "No" caption and flip the setting, or just replace the backwards tick box with a select parameter offering "Preserve directory structure" and "Place in selected folder" (or something like that). Pretty please? How about if I write and test the patch? Peter

3 5

inputs sanitization...
by Kostas Karasavvas 11 Mar '11

11 Mar '11

Hi all! I found the following in the mailing list archives. Does anyone know where I can find a complete list of the sanitization taking place? '>' __gt__ '@' __at__ etc. Thank you, Kostas ************************ It's worth noting that command line parameters are sanitized for security purposes, so passing them unsanitized should only be done in controlled environments where security is not a concern. Otherwise, parameters should be restored to their original values by the tool or a wrapper around the tool. --nate On Oct 5, 2010, at 10:10, Jelle Scholtalbers <j.scholtalbers at gmail.com> wrote: > Hi, > > take a look at <options sanitize="False" /> as used also in > tools/fastq/fastq_manipulation.xml > > Cheers, > Jelle > > > On Tue, Oct 5, 2010 at 12:39 AM, Zhe Chen <zhe at lanl.gov> wrote: >> Hi, >> >> I was implementing a galaxy tool. A problem occurs when my input contains >> ">", linebreak, galaxy seems traslate them to __gt__, __cr____cn__. Is >> there a way to stop this translation? >> >> Or suggestion to fix this problem? >> >> Thanks

2 1

NoneType dereference on the jobs view
by Ry4an Brase 11 Mar '11

11 Mar '11

Intermittently, and always during periods of high load we'll get a 500 Server error from the Admin 'Manage Jobs' list. In the logs the stacktrace looks like: http://paste.pocoo.org/show/351374/ Attached is the patch JJ provided to work around jobs without histories, but I thought I'd bring it up here too in case either others are seeing it or someone knows a root cause. Thanks! -- Ry4an Brase 612-626-6575 Software Developer Application Development University of Minnesota Supercomputing Institute http://www.msi.umn.edu

2 1

Re: [galaxy-dev] Server error from history view
by Isabelle Phan 11 Mar '11

11 Mar '11

Hello Sarah, (my galaxy-dev account is broken, read your post on nabble) You need to disable filter-with=gzip in order to set debug=False But in any case, I have not seen a difference in the log, so my take is that galaxy only ever runs in debug mode. Isabelle

2 1

Using Cytoscape web start as External Data Visualization
by charlie 11 Mar '11

11 Mar '11

Hi all, Have anyone has experience to use Cytoscape web start as External Data Visualization tool? Thanks! Charlie

1 0

Creating Roles and groups
by Peter Cock 10 Mar '11

10 Mar '11

Hi all, I've had a little play with our local Galaxy to try and get a feel for libraries, groups and roles. I see when you create a new role (e.g. "Project X"), you have a tick box to create a group of the same name. However, when you create a new group, these is no such tick box to create a role of the same name. Could there be? Thanks, Peter P.S. The following were helpful, https://bitbucket.org/galaxy/galaxy-central/wiki/SecurityFeatures https://bitbucket.org/galaxy/galaxy-central/wiki/DataLibraries/LibrarySecur…

2 1

Enabling iterative test-driven development of tools
by John Chilton 10 Mar '11

10 Mar '11

Hello, The current tool testing options provided by Galaxy are excellent for verifying that Galaxy is functioning properly and that tools provide reproducible results. However, I believe there are two related shortcomings to the current Galaxy approach, and I would like to propose a solution for these shortcomings. The first is that the workflow for the tool developer is very clunky, especially if the developer is building up a tool incrementally with a test-driven development (TDD) approach. For each parameter addition a whole new output file must be created externally, manually verified by the developer, possibly converted to a regular expression, and placed in the test data directory. Put another way, it is easy to iteratively build up a tool, but not to iteratively build a test case. The second point is a bit pedantic. The current approach only verifies that the output is the same as the supplied output, not that the output is actually "correct". The typical workflow I described above relies on manual inspection of the expected output file to verify it is "correct". I believe it is better to programmatically state assertions about what makes an output correct than to rely on manual verification, this serves both to reduce human error and act as documentation about what makes an output correct. To address these two points, I propose an extensible addition to the Galaxy tool syntax (and provide an implementation) to declaratively make assertions about output files. The syntax change is to the output child element of test elements. With this change, the output element may have a child element named assert_contents, which in turn may have any number of child elements each of which describes an assertion about the contents of the referenced output file. The file attribute on the output element is still required if an assert_contents element is not found, but it is optional if an assert_contents element is found. The whole file check described by the file attribute will be executed in addition to the listed assertions if both are present. As an example, the following fragment assserts that an output file contains the text 'chr7', does not contain the text 'chr8', and has a line matching the regular expression '.*\s+127489808\s+127494553'. <test> <param name="input" value="maf_stats_interval_in.dat" /> <output name="out_file1"> <assert_contents> <has_text text="chr7" /> <not_has_text text="chr8" /> <has_line_matching expression=".*\s+127489808\s+127494553" /> </assert_contents> </output> </test> Each potential child element of assert_contents corresponds to a simple python function. These functions are broken out into modules which are dynamically (mostly) loaded at test time. The extensibility of this approach comes from how trivial it is to add new assertion functions to a module and whole new modules of such functions. I have started work on three modules of assertion functions, these are for text, tabular, and XML output files respectively. has_text, not_has_text, and has_line_matching above are examples of three such assertion functions from the text module. To see how it works, here is a function from the file test/base/asserts/text.py defining the has_line_matching element: def assert_has_line_matching(output, expression): """ Asserts the specified output contains a line matching the regular expression specified by the argument expression.""" match = re.search("^%s$" % expression, output, flags = re.MULTILINE) assert match != None, "No line matching expression '%s' was found in output file." % expression As demonstrated, the function name corresponding to the element element_name is just assert_element_name. The code that calls these assertion functions, automatically matches XML attributes with function arguments by names, and matches an argument named output with a string containing the contents of the output file resulting from the test run. Matching function arguments this way gracefully allows for multiple arguments and optional arguments. There is additional information about the implementation at the end of this e-mail. This approach should really aide iterative development of tools. Each new parameter you add to a tool is going to change the output in some way, hopefully you will be able to describe how it affects the output as an assertion. As you add new parameters, the previous parameters will hopefully affect the output in the same way and the old assertion will not need to change, you will just need to add new ones. Obviously this won't always be the case, but hopefully changes to previous assertions will be minimal over time. I believe this process will be faster over time than repeatedly producing output files or interactive GUI based testing, and the final product will be a richer test case. I have attached two patches. The first patch (implementation.patch) is the patch that I propose merging into galaxy-central. It modifies the tool parser to parse these new elements, modifies twilltestcase.py to perform the assertions, and includes the three modules of assertions described above. The second patch (examples.patch) adds a data files to the test-data directory and modifies tools/filters/headWrapper.xml to demonstrate each of the initial assertions elements I have defined. This patch merely proves it works and provides a sandbox to quickly play around with these assertions from working examples, this is not meant to be merged into galaxy-central. The first patch can be imported by executing the following command from an up-to-date galaxy-central pull: hg patch /path/to/implementation.patch To try it out, apply the second patch (examples.patch) and run the functional test Show_beginning1 hg patch /path/to/examples.patch ./run_functional_tests.sh -id Show_beginning1 To view the examples ran with the test see tools/filters/headWrapper.xml after applying the second patch. Let me know if you have any questions, concerns, or if there are any changes I can make to get this work included in galaxy-central. Thanks for your time and consideration, -John ------------------------------------------------ John Chilton Software Developer University of Minnesota Supercomputing Institute Office: 612-625-0917 Cell: 612-226-9223 E-Mail: chilton(a)msi.umn.edu Advanced Usage: In addition to the argument output defined above, there are two other argument names that when used have a special meaning - children and verify_assertions_function. children is a parsed dictionary-based python description of the child element of the assertion element. verify_assertions_function is a function that takes in a string and the same parsed dictionary-based python description of assertion XML elements and checks them. Used in conjunction these can be used to by assertion function authors to allow for the expression recursively defining assertion over some subset of the output. Here is an example: <output name="out_file1"> <assert_contents> <element_text path="BlastOutput_iterations/Iteration/Iteration_hits/Hit/Hit_def"> <not_has_text text="EDK72998.1" /> <has_text_matching expression="ABK[\d\.]+" /> </element_text> </assert_contents> </output> With corresponding Python function definition: def assert_element_text(output, path, verify_assertions_function, children): """ Recursively checks the specified assertions against the text of the first element matching the specified path.""" text = xml_find_text(output, path) verify_assertions_function(text, children) The children argument could also be used to define other assertion specific syntaxes not dependent on verify_assertions_function. A note on this example: This example is admittedly convoluted, but it is working and included in the patch described above. My real desire for this functionality is for some tools I am developing that produce zip files. Stock Galaxy doesn't really play well with zip file base datatypes so this code is not included in the implementation, but you can imagine why this would be useful. I hope to be able to define tests like: <zip_has_file name="subfile"> <has_text text="subfile text" /> </zip_has_file>

1 0

Re: [galaxy-dev] [galaxy-user] irc channel for galaxy
by Dave Clements 10 Mar '11

10 Mar '11

Hi George, Thanks to your suggestion and work by Nate Coraor, Galaxy now has an IRC channel: Server: irc.freenode.net Channel: #galaxyproject This is an informal online gathering place for the Galaxy community to post questions and help each other out. Note that while IRC is conducive to quick discussion, it doesn't work so well as an official support channel (and therefore IRC is not an official Galaxy support channel). If you have a question, bug, or feature suggestion that you want to make sure the Galaxy team knows about, please continue to send these to the mailing lists. Please reply to this thread if you have any questions, or you can post them on the IRC. Thanks, Dave C. On Mon, Mar 7, 2011 at 1:57 PM, George Marselis < George.MARSELIS(a)kaust.edu.sa> wrote: > Hey guys, > > Is there an irc channel for galaxy? I googled for "galaxy irc" but all I > see in the first fifty results are entries for the samsung galaxy (which > is an excellent phone ;) > > Best Regards, > ---- > George Marselis, systems administrator > Building #2, Level 4, room 4327 > Computational Bioscience Research Center, KAUST > Land: +966-2-808-2944, Mobile: +966-56-321-7714, Skype: project2501a > > > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ > -- http://galaxy.psu.edu/gcc2011/ http://getgalaxy.org http://usegalaxy.org/

1 0

setup.sh missing
by Curt Palm 09 Mar '11

09 Mar '11

I just cloned the current release version of galaxy: hg clone http://bitbucket.org/galaxy/galaxy-dist and can not run setup.sh because it is not present in the /galaxy-dist/ directory. setup.sh is also not listed on the https://bitbucket.org/galaxy/galaxy-dist/src page, is this an error or has the installation process changed? thanks ******************************************************* Curtis J. Palm cpalm(a)stanford.edu Stanford Genome Technology Center MC: 8307 office: 650-812-1994 cell: 408 858-7849 *******************************************************

2 1

Re: [galaxy-dev] divide fq into 2
by Musa A. Hassan 09 Mar '11

09 Mar '11

Yes I can't get the file into galaxy at all. Am uploading from a file path. the file is 35mb. Musa ________________________________________ From: Ry4an Brase [ry4an+galaxy(a)msi.umn.edu] Sent: Tuesday, March 08, 2011 11:06 PM To: Musa A. Hassan Subject: Re: [galaxy-dev] divide fq into 2 On Tue, Mar 08, 2011 at 10:44:51PM -0500, Musa A. Hassan wrote: > Hi Ry4an, > > I'd like to do this in galaxy, but the problem is it wont load into > galaxy. As for using split, the file generated from this returns a > length mismatch in say Tophat, maybe in the process of splitting the > file some changes happen to the format. So you can't get the file into galaxy at all? Are you trying to upload it through your browser (suitable only for non-huge files) or are you using 'upload from file path'? How big (bytes) is the file. Also, you should try to keep your replies on the mailing list so that other searching in the future find the same help. -- Ry4an Brase 612-626-6575 Software Developer Application Development University of Minnesota Supercomputing Institute http://www.msi.umn.edu

2 1