<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">It looks like the run-unit-tests.sh script is actually detecting errors but not returning an exit code anywhere in the code.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">George</div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>George K. Thiruvathukal, PhD<br></div><div style="font-size:12.7272720336914px"><div style="font-size:12.7272720336914px"><i style="font-size:12.7272720336914px">Professor of Computer Science</i><span style="font-size:12.7272720336914px">, Loyola University Chicago</span><br></div><div style="font-size:12.7272720336914px"><span style="font-size:12.7272720336914px"><i>Director</i>, Center for Textual Studies and Digital Humanities</span></div><div style="font-size:12.7272720336914px"><span style="font-size:12.7272720336914px"><i>Guest Faculty</i>, Argonne National Laboratory, Math and Computer Science Division</span></div><div style="font-size:12.7272720336914px"><div style="font-size:12.7272720336914px">Editor in Chief, <a href="http://www.computer.org/portal/web/computingnow/cise" target="_blank">Computing in Science and Engineering</a> (IEEE CS/AIP)<br></div><div><span style="font-size:12.7272720336914px">(w) <a href="http://gkt.tv/" target="_blank">gkt.tv</a> </span><span style="font-size:12.7272720336914px">(v) 773.829.4872</span><br></div><div><span style="font-size:12.7272720336914px"><br></span></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Thu, Nov 13, 2014 at 3:35 PM, George K. Thiruvathukal <span dir="ltr"><<a href="mailto:gkt@cs.luc.edu" target="_blank">gkt@cs.luc.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">FYI, when Scala runs unit tests using "sbt test", it returns 1 if any test fails.</div><span class="HOEnZb"><font color="#888888"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">George</div></font></span></div><div class="gmail_extra"><span class=""><br clear="all"><div><div><div dir="ltr"><div><div dir="ltr"><div>George K. Thiruvathukal, PhD<br></div><div style="font-size:12.7272720336914px"><div style="font-size:12.7272720336914px"><i style="font-size:12.7272720336914px">Professor of Computer Science</i><span style="font-size:12.7272720336914px">, Loyola University Chicago</span><br></div><div style="font-size:12.7272720336914px"><span style="font-size:12.7272720336914px"><i>Director</i>, Center for Textual Studies and Digital Humanities</span></div><div style="font-size:12.7272720336914px"><span style="font-size:12.7272720336914px"><i>Guest Faculty</i>, Argonne National Laboratory, Math and Computer Science Division</span></div><div style="font-size:12.7272720336914px"><div style="font-size:12.7272720336914px">Editor in Chief, <a href="http://www.computer.org/portal/web/computingnow/cise" target="_blank">Computing in Science and Engineering</a> (IEEE CS/AIP)<br></div><div><span style="font-size:12.7272720336914px">(w) <a href="http://gkt.tv/" target="_blank">gkt.tv</a> </span><span style="font-size:12.7272720336914px">(v) 773.829.4872</span><br></div><div><span style="font-size:12.7272720336914px"><br></span></div></div></div></div></div></div></div></div>
<br></span><div><div class="h5"><div class="gmail_quote">On Thu, Nov 13, 2014 at 3:34 PM, George K. Thiruvathukal <span dir="ltr"><<a href="mailto:gkt@cs.luc.edu" target="_blank">gkt@cs.luc.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">S<span style="font-size:13px;font-family:arial,sans-serif">éb,</span><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><span style="font-size:13px;font-family:arial,sans-serif"><br></span></div><div class="gmail_default"><font face="arial, sans-serif">I believe the project you showed me is an example of a broken project, so that's why Travis is not detecting failure. When running unit tests, failure is supposed to result not only in *junit.xml but also propagate an error code back to the shell, not just rely on the *junit.xml (or standard) output. The convention for this (I think you already know) is to return non-zero.</font></div><div class="gmail_default"><br>Can you let me know what Thorium was intended to do when tests don't pass? I just did a test, and it seems like something might be broken. I modified one of the tests to fail (and it failed) but "make test" returns success to the OS:</div><div class="gmail_default"><div class="gmail_default"><br></div><div class="gmail_default">gkt@cs-nuc-wtc:~/Work/biosal$ make test</div></div><div class="gmail_default"><div class="gmail_default">UnitTestSuite test_assembly_arc PASSED: 5 failed: 0 TOTAL: 5</div><div class="gmail_default">UnitTestSuite test_assembly_vertex PASSED: 22 failed: 0 TOTAL: 22</div><div class="gmail_default">UnitTestSuite test_dna_codec PASSED: 6 failed: 0 TOTAL: 6</div><div class="gmail_default">UnitTestSuite test_dna_kmer PASSED: 6 failed: 0 TOTAL: 6</div><div class="gmail_default">UnitTestSuite test_dna_sequence PASSED: 3 failed: 0 TOTAL: 3</div><div class="gmail_default">UnitTestSuite test_dynamic_hash_table PASSED: 1020411 failed: 0 TOTAL: 1020411</div><div class="gmail_default">UnitTestSuite test_fast_queue_iterator PASSED: 152 failed: 0 TOTAL: 152</div><div class="gmail_default">UnitTestSuite test_fast_ring PASSED: 892 failed: 0 TOTAL: 892</div><div class="gmail_default">UnitTestSuite test_hash_table_group PASSED: 43 failed: 0 TOTAL: 43</div><div class="gmail_default">UnitTestSuite test_hash_table PASSED: 37921 failed: 0 TOTAL: 37921</div><div class="gmail_default">UnitTestSuite test_map_delete PASSED: 400000 failed: 0 TOTAL: 400000</div><div class="gmail_default">UnitTestSuite test_map PASSED: 2535104 failed: 0 TOTAL: 2535104</div><div class="gmail_default">UnitTestSuite test_memory_pool PASSED: 60013 failed: 0 TOTAL: 60013</div><div class="gmail_default">UnitTestSuite test_node PASSED: 14 failed: 0 TOTAL: 14</div><div class="gmail_default">UnitTestSuite test_packer PASSED: 1 failed: 0 TOTAL: 1</div><div class="gmail_default">UnitTestSuite test_queue PASSED: 33075 failed: 0 TOTAL: 33075</div><div class="gmail_default">UnitTestSuite test_red_black_tree PASSED: 900002 failed: 0 TOTAL: 900002</div><div class="gmail_default">Error File: tests/test_ring.c, Function: main, Line: 18</div><div class="gmail_default">UnitTestSuite test_ring PASSED: 256 FAILED: 1 TOTAL: 257</div><div class="gmail_default">UnitTestSuite test_ring_queue PASSED: 62139 failed: 0 TOTAL: 62139</div><div class="gmail_default">UnitTestSuite test_set PASSED: 409052 failed: 0 TOTAL: 409052</div><div class="gmail_default">UnitTestSuite test_statistics PASSED: 1 failed: 0 TOTAL: 1</div><div class="gmail_default">UnitTestSuite test_string PASSED: 5 failed: 0 TOTAL: 5</div><div class="gmail_default">UnitTestSuite test_vector PASSED: 706709 failed: 0 TOTAL: 706709</div><div class="gmail_default">UnitTestSuite PASSED 6165832 / 6165833 FAILED 1 / 6165833</div><div class="gmail_default">see unit-tests.junit.xml</div><div><div>gkt@cs-nuc-wtc:~/Work/biosal$ echo $?</div><div>0</div></div><div><br></div><div>Travis is not parsing the junit output (at least not for C projects). I can obviously modify my script on Travis to ensure it does fail. But I think we *should* return non-zero when any test fails. </div><div><br></div><div>Best,</div><div>George</div><div><br></div></div></div><div class="gmail_extra"><span><br clear="all"><div><div><div dir="ltr"><div><div dir="ltr"><div>George K. Thiruvathukal, PhD<br></div><div style="font-size:12.7272720336914px"><div style="font-size:12.7272720336914px"><i style="font-size:12.7272720336914px">Professor of Computer Science</i><span style="font-size:12.7272720336914px">, Loyola University Chicago</span><br></div><div style="font-size:12.7272720336914px"><span style="font-size:12.7272720336914px"><i>Director</i>, Center for Textual Studies and Digital Humanities</span></div><div style="font-size:12.7272720336914px"><span style="font-size:12.7272720336914px"><i>Guest Faculty</i>, Argonne National Laboratory, Math and Computer Science Division</span></div><div style="font-size:12.7272720336914px"><div style="font-size:12.7272720336914px">Editor in Chief, <a href="http://www.computer.org/portal/web/computingnow/cise" target="_blank">Computing in Science and Engineering</a> (IEEE CS/AIP)<br></div><div><span style="font-size:12.7272720336914px">(w) <a href="http://gkt.tv/" target="_blank">gkt.tv</a> </span><span style="font-size:12.7272720336914px">(v) 773.829.4872</span><br></div><div><span style="font-size:12.7272720336914px"><br></span></div></div></div></div></div></div></div></div>
<br></span><div><div><div class="gmail_quote">On Thu, Nov 13, 2014 at 3:10 PM, George K. Thiruvathukal <span dir="ltr"><<a href="mailto:gkt@cs.luc.edu" target="_blank">gkt@cs.luc.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">Thanks for clarifying. I should have looked more closely at the project name. :-) Today has been busy since I arrived at the office.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">I need to read up more on Travis to understand its approach to tests in general. I will also ask my colleague, who uses it for more traditional xUnit testing projects, too.</div><span><font color="#888888"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br>George</div></font></span></div><div class="gmail_extra"><span><br clear="all"><div><div><div dir="ltr"><div><div dir="ltr"><div>George K. Thiruvathukal, PhD<br></div><div style="font-size:12.7272720336914px"><div style="font-size:12.7272720336914px"><i style="font-size:12.7272720336914px">Professor of Computer Science</i><span style="font-size:12.7272720336914px">, Loyola University Chicago</span><br></div><div style="font-size:12.7272720336914px"><span style="font-size:12.7272720336914px"><i>Director</i>, Center for Textual Studies and Digital Humanities</span></div><div style="font-size:12.7272720336914px"><span style="font-size:12.7272720336914px"><i>Guest Faculty</i>, Argonne National Laboratory, Math and Computer Science Division</span></div><div style="font-size:12.7272720336914px"><div style="font-size:12.7272720336914px">Editor in Chief, <a href="http://www.computer.org/portal/web/computingnow/cise" target="_blank">Computing in Science and Engineering</a> (IEEE CS/AIP)<br></div><div><span style="font-size:12.7272720336914px">(w) <a href="http://gkt.tv/" target="_blank">gkt.tv</a> </span><span style="font-size:12.7272720336914px">(v) 773.829.4872</span><br></div><div><span style="font-size:12.7272720336914px"><br></span></div></div></div></div></div></div></div></div>
<br></span><div><div><div class="gmail_quote">On Thu, Nov 13, 2014 at 3:05 PM, Boisvert, Sebastien <span dir="ltr"><<a href="mailto:boisvert@anl.gov" target="_blank">boisvert@anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">> From: George K. Thiruvathukal [<a href="mailto:gkt@cs.luc.edu" target="_blank">gkt@cs.luc.edu</a>]<br>
> Sent: Thursday, November 13, 2014 2:56 PM<br>
> To: Boisvert, Sebastien<br>
> Subject: Re: Travis and failure<br>
<span>><br>
><br>
><br>
><br>
> Hi there!<br>
><br>
><br>
> I think the problem is that the failure is not being reflected as a Unix error code (non-zero). Is it possible that your wrapper for the JUnit results is not returning a non-zero code when tests fail?<br>
<br>
</span>The link I gave to you is for the project "Mean.js".<br>
<br>
Also, I can confirm to you that "make tests" always returns 0.<br>
<br>
The result is available in *junit.xml. Or you can simply grep for "FAILED" in stdout.<br>
<br>
No FAILED string means it worked.<br>
<div><div><br>
><br>
> George<br>
><br>
><br>
><br>
> George K. Thiruvathukal, PhD<br>
><br>
> Professor of Computer Science, Loyola University Chicago<br>
><br>
> Director, Center for Textual Studies and Digital Humanities<br>
> Guest Faculty, Argonne National Laboratory, Math and Computer Science Division<br>
> Editor in Chief, Computing in<br>
> Science and Engineering (IEEE CS/AIP)<br>
><br>
> (w) <a href="http://gkt.tv" target="_blank">gkt.tv</a> (v)<br>
> 773.829.4872<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> On Thu, Nov 13, 2014 at 2:54 PM, Boisvert, Sebastien<br>
> <<a href="mailto:boisvert@anl.gov" target="_blank">boisvert@anl.gov</a>> wrote:<br>
><br>
> If you look at the following build:<br>
> <a href="https://travis-ci.org/meanjs/mean/jobs/40710659" target="_blank">https://travis-ci.org/meanjs/mean/jobs/40710659</a><br>
><br>
> it is written "passed" in green.<br>
><br>
> Yet, if you look closely, the log file says:<br>
><br>
> 13 passing (3s)<br>
> 1 failing<br>
><br>
> 1) Article Model Unit Tests: Method Save should be able to save without problems:<br>
> Error: timeout of 2000ms exceeded<br>
> at null.<anonymous> (/home/travis/build/meanjs/mean/node_modules/mocha/lib/runnable.js:158:19)<br>
> at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)<br>
><br>
><br>
><br>
> I am not sure what is going on there.<br>
><br>
><br>
><br>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div>