<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">Hi there,</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">Yes, now that I wrote my first Thorium program, I can help with this. I may have to work on this over the weekend, but I will keep you posted.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br>Best,<br>George</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"> </div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>George K. Thiruvathukal, PhD<br></div><div style="font-size:12.7272720336914px"><div style="font-size:12.7272720336914px"><i style="font-size:12.7272720336914px">Professor of Computer Science</i><span style="font-size:12.7272720336914px">, Loyola University Chicago</span><br></div><div style="font-size:12.7272720336914px"><span style="font-size:12.7272720336914px"><i>Director</i>, Center for Textual Studies and Digital Humanities</span></div><div style="font-size:12.7272720336914px"><span style="font-size:12.7272720336914px"><i>Guest Faculty</i>, Argonne National Laboratory, Math and Computer Science Division</span></div><div style="font-size:12.7272720336914px"><div style="font-size:12.7272720336914px">Editor in Chief, <a href="http://www.computer.org/portal/web/computingnow/cise" target="_blank">Computing in Science and Engineering</a> (IEEE CS/AIP)<br></div><div><span style="font-size:12.7272720336914px">(w) <a href="http://gkt.tv/" target="_blank">gkt.tv</a> </span><span style="font-size:12.7272720336914px">(v) 773.829.4872</span><br></div><div><span style="font-size:12.7272720336914px"><br></span></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Thu, Nov 13, 2014 at 3:52 PM, Boisvert, Sebastien <span dir="ltr"><<a href="mailto:boisvert@anl.gov" target="_blank">boisvert@anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">> From: George K. Thiruvathukal [<a href="mailto:gkt@cs.luc.edu">gkt@cs.luc.edu</a>]<br>
> Sent: Thursday, November 13, 2014 3:37 PM<br>
> To: Boisvert, Sebastien<br>
</span>> Cc: <a href="mailto:biosal@lists.cels.anl.gov">biosal@lists.cels.anl.gov</a><br>
<span class="">> Subject: Re: Travis and failure<br>
><br>
><br>
><br>
><br>
</span><span class="">> It looks like the run-unit-tests.sh script is actually detecting errors but not returning an exit code anywhere in the code.<br>
<br>
</span>If you want to add the return codes:<br>
<br>
- unit-tests: you already found the script<br>
- example-tests: tests/run-examples.sh<br>
- application-tests: tests/run-application-tests.sh<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
><br>
><br>
> George<br>
><br>
><br>
><br>
> George K. Thiruvathukal, PhD<br>
><br>
> Professor of Computer Science, Loyola University Chicago<br>
><br>
> Director, Center for Textual Studies and Digital Humanities<br>
> Guest Faculty, Argonne National Laboratory, Math and Computer Science Division<br>
> Editor in Chief, Computing in<br>
> Science and Engineering (IEEE CS/AIP)<br>
><br>
> (w) <a href="http://gkt.tv" target="_blank">gkt.tv</a> (v)<br>
> 773.829.4872<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> On Thu, Nov 13, 2014 at 3:35 PM, George K. Thiruvathukal<br>
> <<a href="mailto:gkt@cs.luc.edu">gkt@cs.luc.edu</a>> wrote:<br>
><br>
><br>
> FYI, when Scala runs unit tests using "sbt test", it returns 1 if any test fails.<br>
><br>
><br>
><br>
> George<br>
><br>
><br>
><br>
> George K. Thiruvathukal, PhD<br>
><br>
> Professor of Computer Science, Loyola University Chicago<br>
><br>
> Director, Center for Textual Studies and Digital Humanities<br>
> Guest Faculty, Argonne National Laboratory, Math and Computer Science Division<br>
> Editor in Chief, Computing in<br>
> Science and Engineering (IEEE CS/AIP)<br>
><br>
> (w) <a href="http://gkt.tv" target="_blank">gkt.tv</a> (v)<br>
> 773.829.4872<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> On Thu, Nov 13, 2014 at 3:34 PM, George K. Thiruvathukal<br>
> <<a href="mailto:gkt@cs.luc.edu">gkt@cs.luc.edu</a>> wrote:<br>
><br>
><br>
> Séb,<br>
><br>
><br>
><br>
> I believe the project you showed me is an example of a broken project, so that's why Travis is not detecting failure. When running unit tests, failure is supposed to result not only in *junit.xml but<br>
> also propagate an error code back to the shell, not just rely on the *junit.xml (or standard) output. The convention for this (I think you already know) is to return non-zero.<br>
><br>
> Can you let me know what Thorium was intended to do when tests don't pass? I just did a test, and it seems like something might be broken. I modified one of the tests to fail (and it failed) but "make test" returns success to the OS:<br>
><br>
><br>
> gkt@cs-nuc-wtc:~/Work/biosal$ make test<br>
><br>
> UnitTestSuite test_assembly_arc PASSED: 5 failed: 0 TOTAL: 5<br>
> UnitTestSuite test_assembly_vertex PASSED: 22 failed: 0 TOTAL: 22<br>
> UnitTestSuite test_dna_codec PASSED: 6 failed: 0 TOTAL: 6<br>
> UnitTestSuite test_dna_kmer PASSED: 6 failed: 0 TOTAL: 6<br>
> UnitTestSuite test_dna_sequence PASSED: 3 failed: 0 TOTAL: 3<br>
> UnitTestSuite test_dynamic_hash_table PASSED: 1020411 failed: 0 TOTAL: 1020411<br>
> UnitTestSuite test_fast_queue_iterator PASSED: 152 failed: 0 TOTAL: 152<br>
> UnitTestSuite test_fast_ring PASSED: 892 failed: 0 TOTAL: 892<br>
> UnitTestSuite test_hash_table_group PASSED: 43 failed: 0 TOTAL: 43<br>
> UnitTestSuite test_hash_table PASSED: 37921 failed: 0 TOTAL: 37921<br>
> UnitTestSuite test_map_delete PASSED: 400000 failed: 0 TOTAL: 400000<br>
> UnitTestSuite test_map PASSED: 2535104 failed: 0 TOTAL: 2535104<br>
> UnitTestSuite test_memory_pool PASSED: 60013 failed: 0 TOTAL: 60013<br>
> UnitTestSuite test_node PASSED: 14 failed: 0 TOTAL: 14<br>
> UnitTestSuite test_packer PASSED: 1 failed: 0 TOTAL: 1<br>
> UnitTestSuite test_queue PASSED: 33075 failed: 0 TOTAL: 33075<br>
> UnitTestSuite test_red_black_tree PASSED: 900002 failed: 0 TOTAL: 900002<br>
> Error File: tests/test_ring.c, Function: main, Line: 18<br>
> UnitTestSuite test_ring PASSED: 256 FAILED: 1 TOTAL: 257<br>
> UnitTestSuite test_ring_queue PASSED: 62139 failed: 0 TOTAL: 62139<br>
> UnitTestSuite test_set PASSED: 409052 failed: 0 TOTAL: 409052<br>
> UnitTestSuite test_statistics PASSED: 1 failed: 0 TOTAL: 1<br>
> UnitTestSuite test_string PASSED: 5 failed: 0 TOTAL: 5<br>
> UnitTestSuite test_vector PASSED: 706709 failed: 0 TOTAL: 706709<br>
> UnitTestSuite PASSED 6165832 / 6165833 FAILED 1 / 6165833<br>
> see unit-tests.junit.xml<br>
> gkt@cs-nuc-wtc:~/Work/biosal$ echo $?<br>
> 0<br>
><br>
><br>
> Travis is not parsing the junit output (at least not for C projects). I can obviously modify my script on Travis to ensure it does fail. But I think we *should* return non-zero when any test fails.<br>
><br>
> Best,<br>
> George<br>
><br>
><br>
><br>
><br>
><br>
> George K. Thiruvathukal, PhD<br>
><br>
> Professor of Computer Science, Loyola University Chicago<br>
><br>
> Director, Center for Textual Studies and Digital Humanities<br>
> Guest Faculty, Argonne National Laboratory, Math and Computer Science Division<br>
> Editor in Chief, Computing in<br>
> Science and Engineering (IEEE CS/AIP)<br>
><br>
> (w) <a href="http://gkt.tv" target="_blank">gkt.tv</a> (v)<br>
> 773.829.4872<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> On Thu, Nov 13, 2014 at 3:10 PM, George K. Thiruvathukal<br>
> <<a href="mailto:gkt@cs.luc.edu">gkt@cs.luc.edu</a>> wrote:<br>
><br>
><br>
> Thanks for clarifying. I should have looked more closely at the project name. :-) Today has been busy since I arrived at the office.<br>
><br>
><br>
> I need to read up more on Travis to understand its approach to tests in general. I will also ask my colleague, who uses it for more traditional xUnit testing projects, too.<br>
><br>
><br>
> George<br>
><br>
><br>
><br>
> George K. Thiruvathukal, PhD<br>
><br>
> Professor of Computer Science, Loyola University Chicago<br>
><br>
> Director, Center for Textual Studies and Digital Humanities<br>
> Guest Faculty, Argonne National Laboratory, Math and Computer Science Division<br>
> Editor in Chief, Computing in<br>
> Science and Engineering (IEEE CS/AIP)<br>
><br>
> (w) <a href="http://gkt.tv" target="_blank">gkt.tv</a> (v)<br>
> 773.829.4872<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> On Thu, Nov 13, 2014 at 3:05 PM, Boisvert, Sebastien<br>
> <<a href="mailto:boisvert@anl.gov">boisvert@anl.gov</a>> wrote:<br>
><br>
> > From: George K. Thiruvathukal [<a href="mailto:gkt@cs.luc.edu">gkt@cs.luc.edu</a>]<br>
> > Sent: Thursday, November 13, 2014 2:56 PM<br>
> > To: Boisvert, Sebastien<br>
> > Subject: Re: Travis and failure<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > Hi there!<br>
> ><br>
> ><br>
> > I think the problem is that the failure is not being reflected as a Unix error code (non-zero). Is it possible that your wrapper for the JUnit results is not returning a non-zero code when tests fail?<br>
><br>
> The link I gave to you is for the project "Mean.js".<br>
><br>
> Also, I can confirm to you that "make tests" always returns 0.<br>
><br>
> The result is available in *junit.xml. Or you can simply grep for "FAILED" in stdout.<br>
><br>
> No FAILED string means it worked.<br>
><br>
><br>
> ><br>
> > George<br>
> ><br>
> ><br>
> ><br>
> > George K. Thiruvathukal, PhD<br>
> ><br>
> > Professor of Computer Science, Loyola University Chicago<br>
> ><br>
> > Director, Center for Textual Studies and Digital Humanities<br>
> > Guest Faculty, Argonne National Laboratory, Math and Computer Science Division<br>
> > Editor in Chief, Computing in<br>
> > Science and Engineering (IEEE CS/AIP)<br>
> ><br>
> > (w)<br>
> <a href="http://gkt.tv" target="_blank">gkt.tv</a> (v)<br>
> > 773.829.4872<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > On Thu, Nov 13, 2014 at 2:54 PM, Boisvert, Sebastien<br>
> > <<a href="mailto:boisvert@anl.gov">boisvert@anl.gov</a>> wrote:<br>
> ><br>
> > If you look at the following build:<br>
> ><br>
> <a href="https://travis-ci.org/meanjs/mean/jobs/40710659" target="_blank">https://travis-ci.org/meanjs/mean/jobs/40710659</a><br>
> ><br>
> > it is written "passed" in green.<br>
> ><br>
> > Yet, if you look closely, the log file says:<br>
> ><br>
> > 13 passing (3s)<br>
> > 1 failing<br>
> ><br>
> > 1) Article Model Unit Tests: Method Save should be able to save without problems:<br>
> > Error: timeout of 2000ms exceeded<br>
> > at null.<anonymous> (/home/travis/build/meanjs/mean/node_modules/mocha/lib/runnable.js:158:19)<br>
> > at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)<br>
> ><br>
> ><br>
> ><br>
> > I am not sure what is going on there.<br>
> ><br>
> ><br>
> ><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
</div></div></blockquote></div><br></div>