August 2016 - galaxy-dev - lists.galaxyproject.org

uploading multi-file archives and creation of potentially large collections
by Stephan Oepen 08 Aug '16

08 Aug '16

colleagues, in our adaptation of galaxy for large-scale natural language processing, a fairly common use pattern is to invoke a workflow on a potentially large number of text files. hence, i am wondering about facilities for uploading an archive (in ‘.zip’ or ‘.tgz’ format, say) containing several files, where i would like the upload tool to extract the files from the archive, import each individually into my history, and (maybe optionally) create a list collection for the set of files. in my current galaxy instance (running version 2015.03), when i upload a multi-file ‘.zip’ file, part of the above actually happens: however, the upload tool only imports the first file extracted from the archive (and helpfully shows a warning message on the corresponding history entry). have there been relevant changes in this neighborhood in more recent galaxy releases? related to the above, we have started to experiment with potentially large collections and are beginning to worry about the scalability of the collection mechanism. in principle, we would like to operate on collections comprised of tens or hundreds of thousands of individual datasets. what are common collection sizes (in the number of components, not so much in the aggregate file size) used in other galaxy instances to date? what kind of gut reaction do galaxy developers have to the idea of a collection containing, say, a hundred thousand entries? with thanks in advance,

4 3

provide a field to upload a file as part of tool input
by Ryan G 06 Aug '16

06 Aug '16

Hi all - I have a custom tool that can optionally take a file as input. Right now, users have to upload the file into their history, then select the file when running the tool. Is there a way to let the user select a file local on their computer as input to the tool, then when run, the file is uploaded? And (second question), how can I make this field optional, such that the user does not need to provide a file at all when running the tool? Right now, my input field is: <param name="factorSheet" type="data" label="Factor Sheet" help="Select a factor sheet (if any)." />

2 2

Parameter Storing in Database
by Katherine Beaulieu 05 Aug '16

05 Aug '16

Would anyone be able to tell me in what file the storing of tool parameters into the database occurs? Thanks! Katherine

2 1

Re: [galaxy-dev] Command line building
by Katherine Beaulieu 05 Aug '16

05 Aug '16

On Fri, Aug 5, 2016 at 11:20 AM, Katherine Beaulieu < katherine.beaulieu014(a)gmail.com> wrote: > Thats perfect thanks so much!:) > > On Fri, Aug 5, 2016 at 11:18 AM, Dannon Baker <dannon.baker(a)gmail.com> > wrote: > >> Got it, misunderstood your interest. You want this, see the >> fill_template usage in this method: >> https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/tools/evaluatio… >> >> On Fri, Aug 5, 2016 at 11:10 AM Katherine Beaulieu < >> katherine.beaulieu014(a)gmail.com> wrote: >> >>> Hi Dannon, >>> This is sort of what I'm looking for. I'm looking for the place where >>> cheetah variables get replaced with input parameter values. I think this >>> may be in another file? >>> >>> On Fri, Aug 5, 2016 at 10:57 AM, Dannon Baker <dannon.baker(a)gmail.com> >>> wrote: >>> >>>> Check out >>>> https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/jobs/command_fa… >>>> >>>> Let me know if this wasn't what you were looking for. >>>> >>>> -Dannon >>>> >>>> On Fri, Aug 5, 2016 at 10:49 AM Katherine Beaulieu < >>>> katherine.beaulieu014(a)gmail.com> wrote: >>>> >>>>> Within the galaxy app itself, does anyone know where the command line >>>>> is being built? >>>>> Thanks! >>>>> Katherine >>>>> ___________________________________________________________ >>>>> Please keep all replies on the list by using "reply all" >>>>> in your mail client. To manage your subscriptions to this >>>>> and other Galaxy lists, please use the interface at: >>>>> https://lists.galaxyproject.org/ >>>>> >>>>> To search Galaxy mailing lists use the unified search at: >>>>> http://galaxyproject.org/search/mailinglists/ >>>> >>>> >>> >

1 0

Command line building
by Katherine Beaulieu 05 Aug '16

05 Aug '16

Within the galaxy app itself, does anyone know where the command line is being built? Thanks! Katherine

2 1

conversion issue
by Rathert, Philipp, Dr. 05 Aug '16

05 Aug '16

Dear All, I get the following errors when I try to Convert BAM to BigWig Traceback (most recent call last): File "/shed_tools/toolshed.g2.bx.psu.edu/repos/brad-chapman/bam_to_bigwig/52bcd04ee0d6/bam_to_bigwig/bam_to_bigwig/bam_to_bigwig.py", line 103, in <module> main(*args, **kwargs) File "/shed_tools/toolshed.g2.bx.psu.edu/repos/brad-chapman/bam_to_bigwig/52bcd04ee0d6/bam_to_bigwig/bam_to_bigwig/bam_to_bigwig.py", line 48, in main convert_to_graph(bam_file, split, config, temp_handle) File "/shed_tools/toolshed.g2.bx.psu.edu/repos/brad-chapman/bam_to_bigwig/52bcd04ee0d6/bam_to_bigwig/bam_to_bigwig/bam_to_bigwig.py", line 76, in convert_to_graph subprocess.check_call(cl, stdout=out_handle) File "/usr/lib/python2.7/subprocess.py", line 535, in check_call retcode = call(*popenargs, **kwargs) File "/usr/lib/python2.7/subprocess.py", line 522, in call return Popen(*popenargs, **kwargs).wait() File "/usr/lib/python2.7/subprocess.py", line 710, in __init__ errread, errwrite) File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory The tool produced the following additional output: Have 93 references Calculating coverage... or Fatal error: Exit code 1 () bedtools: /galaxy-central/tool_deps/ucsc_tools/312/iuc/package_ucsc_tools_312/2d6bafd63401/lib/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by bedtools) needLargeMem: trying to allocate 0 bytes (limit: 100000000000) Traceback (most recent call last): File "/shed_tools/toolshed.g2.bx.psu.edu/repos/brad-chapman/bam_to_bigwig/9163e1db4c16/bam_to_bigwig/bam_to_bigwig.py", line 122, in <module> main(*args, **kwargs) File "/shed_tools/toolshed.g2.bx.psu.edu/repos/brad-chapman/bam_to_bigwig/9163e1db4c16/bam_to_bigwig/bam_to_bigwig.py", line 57, in main convert_to_bigwig(temp_file, sizes, config, outfile) File "/shed_tools/toolshed.g2.bx.psu.edu/repos/brad-chapman/bam_to_bigwig/9163e1db4c16/bam_to_bigwig/bam_to_bigwig.py", line 104, in convert_to_bigwig subprocess.check_call(cl) File "/galaxy-central/tool_deps/python/2.7.10/iuc/package_python_2_7_10/0339c4a9b87b/lib/python2.7/subprocess.py", line 540, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['bedGraphToBigWig', '/tmp/tmp2F67r6', '/export/galaxy-central/database/files/003/dataset_3116-sizes.txt', '/export/galaxy-central/database/files/003/dataset_3116.dat']' returned non-zero exit status 255 The tool produced the following additional output: Have 93 references Calculating coverage... Converting 0 MB graph file to bigwig.. depending what version of the tool i am using. Any idea what this can be? I am using the latest Docker-galaxy-stable version (16.04) but I think this is not related to docker. All tools are updated. Thank you very much for your help, Philipp

2 4

Retrieving bam index file for visualization outside of Galaxy
by Ouellette, Scott 05 Aug '16

05 Aug '16

Hi all, I have a use case where I need to programmatically download both the .bam and it's index file to run IGV outside of Galaxy, but have not found a straightforward way to do so. It seems that Galaxy always generates this .bai file under the hood, and I can download it manually by clicking the "save" icon on the dataset in my history. Although, the application I am coding for relies on the bioblend Python library. --- In bioblend I see a field returned upon a `DatasetClient(<galaxy_instance>).show_dataset(<dataset_id>)` but that is as far as I've got: "meta_files": [ { "file_type": "bam_index" } ], Any help is appreciated! Thanks, Scott O.

2 1

Removing job in error state
by Ryan G 04 Aug '16

04 Aug '16

Hi - In the Manage Jobs admin job, I have a job from a user that never ran successfully. It shows: Job ID User Last Update Tool State Inputs Command Line Job Runner PID/Cluster ID 12772 user(a)company.com 654 hours ago cshl_fastx_collapser new 13653 error None None None How do I remove this job? I can't find a way to delete it.

2 2

Galaxy URL and api key tool config
by Katherine Beaulieu 03 Aug '16

03 Aug '16

Hi everyone, I was wondering if anyone knew how to pass the url of your galaxy instance to a tool as well as a way to authenticate so I could use the bioblend api within my tool? Thanks, Katherine

2 1

Re: [galaxy-dev] Deprecated code file tag
by Katherine Beaulieu 03 Aug '16

03 Aug '16

On Wed, Aug 3, 2016 at 11:36 AM, Katherine Beaulieu < katherine.beaulieu014(a)gmail.com> wrote: > Ok thanks for the information! > > On Wed, Aug 3, 2016 at 11:21 AM, Eric Rasche <esr(a)tamu.edu> wrote: > >> Hi Katherine, >> >> There is unfortunately no direct replacement to code files (but I hear it >> might still work...if you have internal-only tools, that might be an >> option). I have the exact same use case as you but do not have a clean >> Galaxy solution to offer. I've opened an issue on github >> <https://github.com/galaxyproject/galaxy/issues/2712> for this. >> >> Cheers, >> Eric >> >> >> On 03. aug. 2016 12:35, Katherine Beaulieu wrote: >> >> Hi everyone, >> I can't remember where I came across this but I read somewhere that the >> <code file tag is deprecated and I was just wondering if anyone knew what >> it was being replaced with? I use it to get a list of items from a remote >> service to display in a select list for two of my tools. >> Thanks! >> Katherine >> >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> https://lists.galaxyproject.org/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ >> >> >> -- >> Eric Rasche >> Programmer II >> >> Center for Phage Technology >> Rm 312A, Biochemistry & Biophysics >> Texas A&M University >> College Station, TX 77843 >> esr(a)tamu.edu >> > >

1 0