<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">Thanks for sharing these results with us! It looks encouraging at first glance.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br>By the way, is this a system that I'll be able to try out at some point?</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">Best,<br>George</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>George K. Thiruvathukal, PhD<br></div><div style="font-size:12.7272720336914px"><div style="font-size:12.7272720336914px"><i style="font-size:12.7272720336914px">Professor of Computer Science</i><span style="font-size:12.7272720336914px">, Loyola University Chicago</span><br></div><div style="font-size:12.7272720336914px"><span style="font-size:12.7272720336914px"><i>Director</i>, Center for Textual Studies and Digital Humanities</span></div><div style="font-size:12.7272720336914px"><span style="font-size:12.7272720336914px"><i>Guest Faculty</i>, Argonne National Laboratory, Math and Computer Science Division</span></div><div style="font-size:12.7272720336914px"><div style="font-size:12.7272720336914px">Editor in Chief, <a href="http://www.computer.org/portal/web/computingnow/cise" target="_blank">Computing in Science and Engineering</a> (IEEE CS/AIP)<br></div><div><span style="font-size:12.7272720336914px">(w) <a href="http://thiruvathukal.com" target="_blank">thiruvathukal.com</a> </span><span style="font-size:12.7272720336914px">(v) 773.829.4872</span><br></div><div><span style="font-size:12.7272720336914px"><br></span></div></div></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Mon, Nov 24, 2014 at 11:43 AM, Boisvert, Sebastien <span dir="ltr"><<a href="mailto:boisvert@anl.gov" target="_blank">boisvert@anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello everyone,<br>
<br>
As expected, Edison (Cray XC30) is twice faster than Beagle (Cray XE6).<br>
This was expected because NERSC is using the factor 2.0 for Edison and the factor<br>
1.0 for Hopper.<br>
<br>
And Beagle is around ~ 10 X faster than BGQ for the same number of nodes.<br>
<br>
<br>
<br>
Some unordered timers<br>
==================<br>
<br>
boisvert@edison12:/project/projectdirs/m1523/Jobs> grep TIMER spate-iowa-continuous-corn-soil-2.*<br>
spate-iowa-continuous-corn-soil-2.00253.txt:TIMER [Build assembly graph / Distribute vertices] 2 minutes, 0.706993 secondscore_manager/1021181 dies<br>
spate-iowa-continuous-corn-soil-2.00253.txt:TIMER [Build assembly graph / Distribute arcs] 4 minutes, 30.089874 seconds<br>
spate-iowa-continuous-corn-soil-2.00253.txt:TIMER [Build assembly graph] 6 minutes, 30.796875 seconds<br>
spate-iowa-continuous-corn-soil-2.00255.txt:TIMER [Load input / Count input data] 21.429867 seconds<br>
spate-iowa-continuous-corn-soil-2.00255.txt:TIMER [Load input / Distribute input data] 25.910631 seconds<br>
spate-iowa-continuous-corn-soil-2.00255.txt:TIMER [Load input] 47.340496 seconds<br>
Fichier binaire spate-iowa-continuous-corn-soil-2.spate concordant<br>
<br>
The file system is much faster than that of Beagle ( ~ 8X faster)..<br>
Graph is generated in just 6min30s. This is very fast.<br>
<br>
The load during "Distribute vertices" looks like this:<br>
thorium_worker_pool: node/253 EPOCH LOAD 150 s 15.71/22 (0.71) 0.72 0.71 0.70 0.73 0.71 0.64 0.74 0.71 0.73 0.70 0.71 0.71 0.71 0.71 0.66 0.73 0.75 0.72 0.73 0.72 0.73 0.73<br>
<br>
The load during "Distribute arcs" looks like this:<br>
thorium_worker_pool: node/253 EPOCH LOAD 410 s 13.28/22 (0.60) 0.58 0.62 0.61 0.61 0.58 0.61 0.61 0.61 0.59 0.60 0.59 0.60 0.61 0.59 0.65 0.61 0.60 0.60 0.60 0.61 0.58 0.61<br>
<br>
<br>
Memory usage<br>
============<br>
<br>
<br>
thorium_node: node/250 METRICS AliveActorCount: 2245 ByteCount: 18765324288 / 67657900032<br>
<br>
Heap usage per node is very low too: 18 GiB / 64 GiB. This is because Linux uses a sparse memory model, copy-on-write zero pages, 4K pages,<br>
and a sane model for memory pressure. CNK seems to have none of these features in comparison.<br>
<br>
<br>
Messaging system health check<br>
=========================<br>
<br>
The messaging system looks very healthy too:<br>
<br>
1280 s<br>
thorium_node: node/250 MESSAGES Tick: 1567752844 ReceivedMessageCount: 279612987 SentMessageCount: 277693878 BufferedInboundMessageCount: 0 BufferedOutboundMessageCount: 790 ActiveRequestCo<br>
<br>
1290 s<br>
thorium_node: node/250 MESSAGES Tick: 1567909882 ReceivedMessageCount: 282757208 SentMessageCount: 280834514 BufferedInboundMessageCount: 0 BufferedOutboundMessageCount: 14 ActiveRequestCou<br>
nt: 22<br>
<br>
That's 314422.1 messages / s for each node, or around 80 M messages / s for the whole job (256 nodes).<br>
The multiplexer is clearly working hard !<br>
<br>
<br>
<br>
Graph traversal velocity (small messages, the peer-to-peer multiplexer is utilized a lot !)<br>
==================================================================<br>
<br>
<br>
In the graph traversal, the load is at 9% (it is at 12% on BGQ at this step):<br>
thorium_worker_pool: node/250 EPOCH LOAD 1290 s 1.90/22 (0.09) 0.09 0.09 0.09 0.09 0.09 0.09 0.09 0.09 0.09 0.09 0.09 0.09 0.09 0.09 0.09 0.09 0.09 0.08 0.08 0.09 0.09 0.09<br>
<br>
<br>
Timelines are basically empty since everything is waiting for data.<br>
<br>
thorium_worker_pool: node/250 EPOCH FUTURE_TIMELINE 1290 s 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0<br>
thorium_worker_pool: node/250 EPOCH WAKE_UP_COUNT 1290 s 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0<br>
<br>
<br>
<br>
I am getting a sustained velocity of 10 vertices / s per actor.<br>
<br>
biosal_unitig_visitor/1118714 visited 8500 vertices so far (velocity: 9.953161 vertices / s)<br>
biosal_unitig_visitor/1256698 visited 8500 vertices so far (velocity: 9.964830 vertices / s)<br>
biosal_unitig_visitor/1111034 visited 8500 vertices so far (velocity: 9.953161 vertices / s)<br>
biosal_unitig_visitor/1124090 visited 8500 vertices so far (velocity: 9.953161 vertices / s)<br>
<br>
spate-iowa-continuous-corn-soil-2.00252.txt:DEBUG the system has 563200 visitors<br>
<br>
Across the board, the throughput is 5603840.0 vertices / s.<br>
<br>
<br>
I can compute an expected running time:<br>
<br>
boisvert@edison12:/project/projectdirs/m1523/Jobs> grep GRAPH spate-iowa-continuous-corn-soil-2.*.txt<br>
spate-iowa-continuous-corn-soil-2.00253.txt:GRAPH -> 148375705714 vertices, 298256036296 vertex observations, and 146235667225 arcs.<br>
<br>
So the number of canonical DNA bits with at least a sequencing coverage depth of 2 is:<br>
<br>
$ irb<br>
irb(main):001:0> 148375705714 / 2 - 56665010890<br>
=> 17522841967<br>
<br>
<br>
boisvert@edison12:/project/projectdirs/m1523/Jobs> head spate-iowa-continuous-corn-soil-2/coverage_distribution.txt-canonical -n 4<br>
1 56665010890<br>
2 7970399985<br>
3 3453812029<br>
4 1787385290<br>
<br>
It should run in under 52 minutes for the graph traversal.<br>
<br>
irb(main):006:0> 17522841967 / 5603840 / 60<br>
=> 52<br>
<br>
<br>
<br>
Comparison with others<br>
==================<br>
<br>
With MegaHIT ( <a href="http://arxiv.org/pdf/1409.7208v1.pdf" target="_blank">http://arxiv.org/pdf/1409.7208v1.pdf</a> )<br>
<br>
MEGAHIT, GPU... 44.1 hours<br>
MEGAHIT, CPU only... 99.6 hours<br>
SPATE with 256 Cray XC30 nodes... probably < 2 hours (spate-iowa-continuous-corn-soil-2, see <a href="https://github.com/GeneAssembly/biosal/issues/822" target="_blank">https://github.com/GeneAssembly/biosal/issues/822</a> )<br>
<br>
<br>
Actors, actors, actors<br>
================<br>
<br>
The complex stuff in BioSAL is definitely Thorium, not the actor code. Actor scripts are easy to write and understand.<br>
And the scope of an actor is very small too.<br>
<br>
On the other hand, Thorium has more complex code paths -- it is a runtime for actors.<br>
_______________________________________________<br>
BIOSAL mailing list<br>
<a href="mailto:BIOSAL@lists.cels.anl.gov">BIOSAL@lists.cels.anl.gov</a><br>
<a href="https://lists.cels.anl.gov/mailman/listinfo/biosal" target="_blank">https://lists.cels.anl.gov/mailman/listinfo/biosal</a><br>
</blockquote></div><br></div>