OK OK, cloud computing is expensive.<div><br></div><div>But I also know from my own experience that you can cut I/O by a factor of 10-20 and CPU by a factor of ten as well:</div><div>- use bowtie for mapping (but index is quite big): saves a lot of CPU</div>

<div>- compress input fastq files (reduces size to 1/5) and read only compressed files</div><div>- extreme solution: strip all quality values from fastq (reduces size to 1/4)</div><div>- remove all file-concatenation steps</div>

<div>- pipe into samtools to convert to bam immediately after mapping, always save in bam format</div><div>- strip all unmapped reads directly with samtools -F4</div><div><br></div><div>but I wonder how much that would save in the end...?</div>

<div><br></div><div>cheers</div><div>Max<br clear="all">--<br>Maximilian Haussler<br>Tel: +447574246789<br><a href="http://www.manchester.ac.uk/research/maximilian.haussler/" target="_blank">http://www.manchester.ac.uk/research/maximilian.haussler/</a><br>


<br><br><div class="gmail_quote">On Tue, Nov 23, 2010 at 9:02 PM, Enis Afgan <span dir="ltr">&lt;<a href="mailto:eafgan@emory.edu">eafgan@emory.edu</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">

It&#39;s just that computing, and cloud computing with that, is expensive. Depending on the usage, either the EBS volumes or the CPU time (i.e., instances) is what will represent majority of the cost. Most likely, it will be the instances, unless you use very few instances for a short period and a lot of storage.<div>



<br></div><div>There are a couple of papers I can recall analyzing the cost of science in the cloud, if you want to take a look:</div><div>- Deelman E, Singh G, Livny M, Berriman B, Good J: The cost of doing science on the cloud: the Montage example</div>



<div>- Wilkening J, Wilke A, Desai N, Meyer F: Using Clouds for Metagenomics: A Case Study<br><font color="#888888">
































<br><div class="gmail_quote"><br></div><div class="gmail_quote">Enis</div></font><div><div></div><div class="h5"><div class="gmail_quote"><br></div><div class="gmail_quote">On Tue, Nov 23, 2010 at 3:43 PM, Maximilian Haussler <span dir="ltr">&lt;<a href="mailto:maximilianh@gmail.com" target="_blank">maximilianh@gmail.com</a>&gt;</span> wrote:<br>



<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I&#39;d be interested in why AWS is so expensive for these datasets:<div><br></div><div>Is it mostly</div><div>a) the data transfer between nodes?</div>



<div>b) the data storage on EBS?</div><div>c) the CPU time ?</div><div>

why next-gen analysis is expensive on the cloud? </div><div><br></div><div>Can anyone who is actively using AWS look up the distribution of the total cost on the individual types?</div><div><br></div><div>I guess that there is a lot of room for improvement for the different costs, depending on the type of algorithm that you&#39;re using.</div>





<div><br></div><div>thanks in advance</div><div>Max<br clear="all"><br><br><div class="gmail_quote"><div><div></div><div>On Tue, Nov 23, 2010 at 8:17 PM, David Martin <span dir="ltr">&lt;<a href="mailto:dmarti@lsuhsc.edu" target="_blank">dmarti@lsuhsc.edu</a>&gt;</span> wrote:<br>





</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div></div><div>



<div>
<font size="4"><font face="Calibri, Verdana, Helvetica, Arial"><span style="font-size:11pt">Hello,<br>
<br>
We are about to get about 200 GB of illumina reads(43 bp) from 20 samples, two groups of 10 animals.  We are hoping to use Galaxy on the Cloud to compare gene expression between the two groups.  First of all, do you think this is possible with the current state of Galaxy Cloud development?  Secondly, we are currently practicing with small drosophila datasets (4 sets of 2 GB each), and over the course of a few days of doing relatively little besides grooming and filtering the data, we had already been charged $60 by Amazon, which we thought was a bit inefficient.  What is the best way to proceed working from one day to the next?  Should one terminate the cluster at Cloud Console and then stop(pause) the cluster at the AWS console, and then restart the instance the next day?  Does one have to reattach all of the EBS volumes before restarting the cluster?  We were just terminating the instance and then bringing it back up and all the data was still there, ie it worked fine, but when we looked after a couple days there were 45 EBS volumes there -  much of it was surely redundant as our data wasn’t very large.   Perhaps we need to take a snapshot and reboot the instance from this? Thank you for any hints regarding this matter, this is all very new to me.  Let me know if you need clarification or more information.<br>





<font color="#888888">
<br>
David Martin<br>
<a href="mailto:dmarti@lsuhsc.edu" target="_blank">dmarti@lsuhsc.edu</a></font></span></font></font>
</div>


<br></div></div>_______________________________________________<br>
galaxy-user mailing list<br>
<a href="mailto:galaxy-user@lists.bx.psu.edu" target="_blank">galaxy-user@lists.bx.psu.edu</a><br>
<a href="http://lists.bx.psu.edu/listinfo/galaxy-user" target="_blank">http://lists.bx.psu.edu/listinfo/galaxy-user</a><br>
<br></blockquote></div><br></div>
<br>_______________________________________________<br>
galaxy-user mailing list<br>
<a href="mailto:galaxy-user@lists.bx.psu.edu" target="_blank">galaxy-user@lists.bx.psu.edu</a><br>
<a href="http://lists.bx.psu.edu/listinfo/galaxy-user" target="_blank">http://lists.bx.psu.edu/listinfo/galaxy-user</a><br>
<br></blockquote></div><br></div></div></div>
</blockquote></div><br></div>