John, that's seems great.
I will read this stuff and see if  I can use it (The bed format isn't that essential, bowtie can bam instead).

If it wont work I will try the other solution which doesn't need to change the galaxy own code (Creating hundreds of workflow run, linking to their outputs and running last workflow with the merging tool - this solution also distribute in a better way).

Because galaxy is used a lot on sequencers output, I think someday it should support this kind of jobs internally.
When I will have a running solution, I will publish what solution I have used.

Its really great to know I'm not the first one to attack this problem.
Thanks for the advices.
Hagai




On Tue, Feb 12, 2013 at 5:42 PM, Joachim Jacob |VIB| <joachim.jacob@vib.be> wrote:
You cannot directly couple different workflows.

But you could indeed copy all outputs of the different workflows into one history, and create a separate workflow with your tool to work on all those input files.

Cheers,

Joachim

Joachim Jacob

Rijvisschestraat 120, 9052 Zwijnaarde
Tel: +32 9 244.66.34
Bioinformatics Training and Services (BITS)
http://www.bits.vib.be
@bitsatvib

On 02/12/2013 04:31 PM, Hagai Cohen wrote:

Thanks for your answer.
I figured that there is an option to run a workflow on multiple files, but I can't merge the outputs afterwardsl. I would like the workflow to return one final output.

But you gave me another idea.
Can I somehow tell one workflow to run on other workflow output?
If this can be done, I can run 100 different workflows with bowtie & statistics, each working on one fastq file, than run another workflow which gets 100 xls inputs and merge them to one.




On Tue, Feb 12, 2013 at 5:20 PM, Joachim Jacob |VIB| <joachim.jacob@vib.be <mailto:joachim.jacob@vib.be>> wrote:

    Hi Hagai,

    Actually, using a workflow, you are able to select multiple input
    files, and let the workflow run separately on all input files.

    I would proceed by creating a data library for all your fastq
    files, which you can upload via FTP, or via a system directory.
    You can use a sample of your fastq files to create the steps in a
    history you want to perform, and extract a workflow out of it.
    Next, copy all fastq files from a data library in a new history,
    and run your workflow on the all input files.

    I hope this helps you further,
    Joachim


    Joachim Jacob

    Rijvisschestraat 120, 9052 Zwijnaarde
    Tel: +32 9 244.66.34 <tel:%2B32%209%20244.66.34>

    Bioinformatics Training and Services (BITS)
    http://www.bits.vib.be
    @bitsatvib


    On 02/12/2013 04:02 PM, Hagai Cohen wrote:

        Hi,
        I'm looking for a preferred way of running Bowtie (or any
        other tool) on multiple input files and run statistics on the
        Bowtie output afterwards.

        The input is a directory of files fastq1..fastq100
        The bowtie output should be bed1...bed100
        The statistics tool should run on bed1...bed100 and return
        xls1..xls100
        Then I will write a tool which will get xls1..xls100 and merge
        them to one final output.

        I searched for a smiliar cases, and I couldn't figure anyone
        which had this problem before.
        Can't use the parallelism tag, because what will be the input
        for each tool? it should be a fastq file not a directory of
        fastq files.
        Neither I would like to run each fastq file in a different
        workflow - creating a mess.

        I thought only on two solutions:
        1. Implement new datatypes: bed_dir & fastq_dir and implements
        new tool wrappers which will get a folder instead of a file.
        2. merge the input files before sending to bowtie, and use
        parallelism tag to make them be splitted & merged again on
        each tool.

        Does anyone has any better suggestion?

        Thanks,
        Hagai











        ___________________________________________________________
        Please keep all replies on the list by using "reply all"
        in your mail client.  To manage your subscriptions to this
        and other Galaxy lists, please use the interface at:

        http://lists.bx.psu.edu/