<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">Yes, of course! :) I was not speaking in terms of concrete syntax but just pointing out that you've already done the work to make it happen. (I forgot to put the word "with" after "exit".)</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">Best,<br>George</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>George K. Thiruvathukal, PhD<br></div><div style="font-size:12.7272720336914px"><div style="font-size:12.7272720336914px"><i style="font-size:12.7272720336914px">Professor of Computer Science</i><span style="font-size:12.7272720336914px">, Loyola University Chicago</span><br></div><div style="font-size:12.7272720336914px"><span style="font-size:12.7272720336914px"><i>Director</i>, Center for Textual Studies and Digital Humanities</span></div><div style="font-size:12.7272720336914px"><span style="font-size:12.7272720336914px"><i>Guest Faculty</i>, Argonne National Laboratory, Math and Computer Science Division</span></div><div style="font-size:12.7272720336914px"><div style="font-size:12.7272720336914px">Editor in Chief, <a href="http://www.computer.org/portal/web/computingnow/cise" target="_blank">Computing in Science and Engineering</a> (IEEE CS/AIP)<br></div><div><span style="font-size:12.7272720336914px">(w) <a href="http://gkt.tv/" target="_blank">gkt.tv</a> </span><span style="font-size:12.7272720336914px">(v) 773.829.4872</span><br></div><div><span style="font-size:12.7272720336914px"><br></span></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Thu, Nov 13, 2014 at 4:17 PM, Boisvert, Sebastien <span dir="ltr"><<a href="mailto:boisvert@anl.gov" target="_blank">boisvert@anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">> From: George K. Thiruvathukal [<a href="mailto:gkt@cs.luc.edu">gkt@cs.luc.edu</a>]<br>
</span>> Sent: Thursday, November 13, 2014 4:14 PM<br>
<span class="">> To: Boisvert, Sebastien<br>
> Cc: <a href="mailto:biosal@lists.cels.anl.gov">biosal@lists.cels.anl.gov</a><br>
> Subject: Re: Travis and failure<br>
><br>
><br>
><br>
><br>
</span><span class="">> Looks like you're right! I can probably just exit $total_failures and be done with it.<br>
<br>
</span>It is better if you use "return".<br>
<br>
<br>
This works:<br>
<br>
$ cat test.sh<br>
#!/bin/bash<br>
<br>
function main()<br>
{<br>
return 1<br>
}<br>
<br>
main<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
><br>
><br>
> I'll work on this at home.<br>
><br>
> George<br>
><br>
><br>
><br>
> George K. Thiruvathukal, PhD<br>
><br>
> Professor of Computer Science, Loyola University Chicago<br>
><br>
> Director, Center for Textual Studies and Digital Humanities<br>
> Guest Faculty, Argonne National Laboratory, Math and Computer Science Division<br>
> Editor in Chief, Computing in<br>
> Science and Engineering (IEEE CS/AIP)<br>
><br>
> (w) <a href="http://gkt.tv" target="_blank">gkt.tv</a> (v)<br>
> 773.829.4872<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> On Thu, Nov 13, 2014 at 4:04 PM, George K. Thiruvathukal<br>
> <<a href="mailto:gkt@cs.luc.edu">gkt@cs.luc.edu</a>> wrote:<br>
><br>
><br>
> Yes, it's very quick. I will try to work on it before the weekend.<br>
><br>
><br>
> Tomorrow I have to meet students to talk about their semester projects. Then it should be smooth sailing (I hope).<br>
><br>
><br>
> George<br>
><br>
><br>
><br>
> George K. Thiruvathukal, PhD<br>
><br>
> Professor of Computer Science, Loyola University Chicago<br>
><br>
> Director, Center for Textual Studies and Digital Humanities<br>
> Guest Faculty, Argonne National Laboratory, Math and Computer Science Division<br>
> Editor in Chief, Computing in<br>
> Science and Engineering (IEEE CS/AIP)<br>
><br>
> (w) <a href="http://gkt.tv" target="_blank">gkt.tv</a> (v)<br>
> 773.829.4872<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> On Thu, Nov 13, 2014 at 4:01 PM, Boisvert, Sebastien<br>
> <<a href="mailto:boisvert@anl.gov">boisvert@anl.gov</a>> wrote:<br>
><br>
> > From: George K. Thiruvathukal [<a href="mailto:gkt@cs.luc.edu">gkt@cs.luc.edu</a>]<br>
> > Sent: Thursday, November 13, 2014 3:55 PM<br>
> > To: Boisvert, Sebastien<br>
> > Cc:<br>
> <a href="mailto:biosal@lists.cels.anl.gov">biosal@lists.cels.anl.gov</a><br>
> > Subject: Re: Travis and failure<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > Hi there,<br>
> ><br>
> ><br>
> > Yes, now that I wrote my first Thorium program, I can help with this. I may have to work on this over the weekend, but I will keep you posted.<br>
><br>
> I created an issue:<br>
><br>
> <a href="https://github.com/GeneAssembly/biosal/issues/818" target="_blank">https://github.com/GeneAssembly/biosal/issues/818</a><br>
><br>
> This should be fairly easy to implement since all 3 scripts already have a variable containing the number of<br>
> failures.<br>
><br>
> Thanks.<br>
><br>
><br>
> ><br>
> > Best,<br>
> > George<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > George K. Thiruvathukal, PhD<br>
> ><br>
> > Professor of Computer Science, Loyola University Chicago<br>
> ><br>
> > Director, Center for Textual Studies and Digital Humanities<br>
> > Guest Faculty, Argonne National Laboratory, Math and Computer Science Division<br>
> > Editor in Chief, Computing in<br>
> > Science and Engineering (IEEE CS/AIP)<br>
> ><br>
> > (w)<br>
> <a href="http://gkt.tv" target="_blank">gkt.tv</a> (v)<br>
> > 773.829.4872<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > On Thu, Nov 13, 2014 at 3:52 PM, Boisvert, Sebastien<br>
> > <<a href="mailto:boisvert@anl.gov">boisvert@anl.gov</a>> wrote:<br>
> ><br>
> > > From: George K. Thiruvathukal [<a href="mailto:gkt@cs.luc.edu">gkt@cs.luc.edu</a>]<br>
> > > Sent: Thursday, November 13, 2014 3:37 PM<br>
> > > To: Boisvert, Sebastien<br>
> > > Cc:<br>
> ><br>
> <a href="mailto:biosal@lists.cels.anl.gov">biosal@lists.cels.anl.gov</a><br>
> > > Subject: Re: Travis and failure<br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > It looks like the run-unit-tests.sh script is actually detecting errors but not returning an exit code anywhere in the code.<br>
> ><br>
> > If you want to add the return codes:<br>
> ><br>
> > - unit-tests: you already found the script<br>
> > - example-tests: tests/run-examples.sh<br>
> > - application-tests: tests/run-application-tests.sh<br>
> ><br>
> ><br>
> ><br>
> > ><br>
> > ><br>
> > > George<br>
> > ><br>
> > ><br>
> > ><br>
> > > George K. Thiruvathukal, PhD<br>
> > ><br>
> > > Professor of Computer Science, Loyola University Chicago<br>
> > ><br>
> > > Director, Center for Textual Studies and Digital Humanities<br>
> > > Guest Faculty, Argonne National Laboratory, Math and Computer Science Division<br>
> > > Editor in Chief, Computing in<br>
> > > Science and Engineering (IEEE CS/AIP)<br>
> > ><br>
> > > (w)<br>
> ><br>
> <a href="http://gkt.tv" target="_blank">gkt.tv</a> (v)<br>
> > > 773.829.4872<br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > On Thu, Nov 13, 2014 at 3:35 PM, George K. Thiruvathukal<br>
> > > <<a href="mailto:gkt@cs.luc.edu">gkt@cs.luc.edu</a>> wrote:<br>
> > ><br>
> > ><br>
> > > FYI, when Scala runs unit tests using "sbt test", it returns 1 if any test fails.<br>
> > ><br>
> > ><br>
> > ><br>
> > > George<br>
> > ><br>
> > ><br>
> > ><br>
> > > George K. Thiruvathukal, PhD<br>
> > ><br>
> > > Professor of Computer Science, Loyola University Chicago<br>
> > ><br>
> > > Director, Center for Textual Studies and Digital Humanities<br>
> > > Guest Faculty, Argonne National Laboratory, Math and Computer Science Division<br>
> > > Editor in Chief, Computing in<br>
> > > Science and Engineering (IEEE CS/AIP)<br>
> > ><br>
> > > (w)<br>
> ><br>
> <a href="http://gkt.tv" target="_blank">gkt.tv</a> (v)<br>
> > > 773.829.4872<br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > On Thu, Nov 13, 2014 at 3:34 PM, George K. Thiruvathukal<br>
> > > <<a href="mailto:gkt@cs.luc.edu">gkt@cs.luc.edu</a>> wrote:<br>
> > ><br>
> > ><br>
> > > Séb,<br>
> > ><br>
> > ><br>
> > ><br>
> > > I believe the project you showed me is an example of a broken project, so that's why Travis is not detecting failure. When running unit tests, failure is supposed to result not only in *junit.xml but<br>
> > > also propagate an error code back to the shell, not just rely on the *junit.xml (or standard) output. The convention for this (I think you already know) is to return non-zero.<br>
> > ><br>
> > > Can you let me know what Thorium was intended to do when tests don't pass? I just did a test, and it seems like something might be broken. I modified one of the tests to fail (and it failed) but "make test" returns success to the OS:<br>
> > ><br>
> > ><br>
> > > gkt@cs-nuc-wtc:~/Work/biosal$ make test<br>
> > ><br>
> > > UnitTestSuite test_assembly_arc PASSED: 5 failed: 0 TOTAL: 5<br>
> > > UnitTestSuite test_assembly_vertex PASSED: 22 failed: 0 TOTAL: 22<br>
> > > UnitTestSuite test_dna_codec PASSED: 6 failed: 0 TOTAL: 6<br>
> > > UnitTestSuite test_dna_kmer PASSED: 6 failed: 0 TOTAL: 6<br>
> > > UnitTestSuite test_dna_sequence PASSED: 3 failed: 0 TOTAL: 3<br>
> > > UnitTestSuite test_dynamic_hash_table PASSED: 1020411 failed: 0 TOTAL: 1020411<br>
> > > UnitTestSuite test_fast_queue_iterator PASSED: 152 failed: 0 TOTAL: 152<br>
> > > UnitTestSuite test_fast_ring PASSED: 892 failed: 0 TOTAL: 892<br>
> > > UnitTestSuite test_hash_table_group PASSED: 43 failed: 0 TOTAL: 43<br>
> > > UnitTestSuite test_hash_table PASSED: 37921 failed: 0 TOTAL: 37921<br>
> > > UnitTestSuite test_map_delete PASSED: 400000 failed: 0 TOTAL: 400000<br>
> > > UnitTestSuite test_map PASSED: 2535104 failed: 0 TOTAL: 2535104<br>
> > > UnitTestSuite test_memory_pool PASSED: 60013 failed: 0 TOTAL: 60013<br>
> > > UnitTestSuite test_node PASSED: 14 failed: 0 TOTAL: 14<br>
> > > UnitTestSuite test_packer PASSED: 1 failed: 0 TOTAL: 1<br>
> > > UnitTestSuite test_queue PASSED: 33075 failed: 0 TOTAL: 33075<br>
> > > UnitTestSuite test_red_black_tree PASSED: 900002 failed: 0 TOTAL: 900002<br>
> > > Error File: tests/test_ring.c, Function: main, Line: 18<br>
> > > UnitTestSuite test_ring PASSED: 256 FAILED: 1 TOTAL: 257<br>
> > > UnitTestSuite test_ring_queue PASSED: 62139 failed: 0 TOTAL: 62139<br>
> > > UnitTestSuite test_set PASSED: 409052 failed: 0 TOTAL: 409052<br>
> > > UnitTestSuite test_statistics PASSED: 1 failed: 0 TOTAL: 1<br>
> > > UnitTestSuite test_string PASSED: 5 failed: 0 TOTAL: 5<br>
> > > UnitTestSuite test_vector PASSED: 706709 failed: 0 TOTAL: 706709<br>
> > > UnitTestSuite PASSED 6165832 / 6165833 FAILED 1 / 6165833<br>
> > > see unit-tests.junit.xml<br>
> > > gkt@cs-nuc-wtc:~/Work/biosal$ echo $?<br>
> > > 0<br>
> > ><br>
> > ><br>
> > > Travis is not parsing the junit output (at least not for C projects). I can obviously modify my script on Travis to ensure it does fail. But I think we *should* return non-zero when any test fails.<br>
> > ><br>
> > > Best,<br>
> > > George<br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > George K. Thiruvathukal, PhD<br>
> > ><br>
> > > Professor of Computer Science, Loyola University Chicago<br>
> > ><br>
> > > Director, Center for Textual Studies and Digital Humanities<br>
> > > Guest Faculty, Argonne National Laboratory, Math and Computer Science Division<br>
> > > Editor in Chief, Computing in<br>
> > > Science and Engineering (IEEE CS/AIP)<br>
> > ><br>
> > > (w)<br>
> ><br>
> <a href="http://gkt.tv" target="_blank">gkt.tv</a> (v)<br>
> > > 773.829.4872<br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > On Thu, Nov 13, 2014 at 3:10 PM, George K. Thiruvathukal<br>
> > > <<a href="mailto:gkt@cs.luc.edu">gkt@cs.luc.edu</a>> wrote:<br>
> > ><br>
> > ><br>
> > > Thanks for clarifying. I should have looked more closely at the project name. :-) Today has been busy since I arrived at the office.<br>
> > ><br>
> > ><br>
> > > I need to read up more on Travis to understand its approach to tests in general. I will also ask my colleague, who uses it for more traditional xUnit testing projects, too.<br>
> > ><br>
> > ><br>
> > > George<br>
> > ><br>
> > ><br>
> > ><br>
> > > George K. Thiruvathukal, PhD<br>
> > ><br>
> > > Professor of Computer Science, Loyola University Chicago<br>
> > ><br>
> > > Director, Center for Textual Studies and Digital Humanities<br>
> > > Guest Faculty, Argonne National Laboratory, Math and Computer Science Division<br>
> > > Editor in Chief, Computing in<br>
> > > Science and Engineering (IEEE CS/AIP)<br>
> > ><br>
> > > (w)<br>
> ><br>
> <a href="http://gkt.tv" target="_blank">gkt.tv</a> (v)<br>
> > > 773.829.4872<br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > On Thu, Nov 13, 2014 at 3:05 PM, Boisvert, Sebastien<br>
> > > <<a href="mailto:boisvert@anl.gov">boisvert@anl.gov</a>> wrote:<br>
> > ><br>
> > > > From: George K. Thiruvathukal [<a href="mailto:gkt@cs.luc.edu">gkt@cs.luc.edu</a>]<br>
> > > > Sent: Thursday, November 13, 2014 2:56 PM<br>
> > > > To: Boisvert, Sebastien<br>
> > > > Subject: Re: Travis and failure<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > Hi there!<br>
> > > ><br>
> > > ><br>
> > > > I think the problem is that the failure is not being reflected as a Unix error code (non-zero). Is it possible that your wrapper for the JUnit results is not returning a non-zero code when tests fail?<br>
> > ><br>
> > > The link I gave to you is for the project "Mean.js".<br>
> > ><br>
> > > Also, I can confirm to you that "make tests" always returns 0.<br>
> > ><br>
> > > The result is available in *junit.xml. Or you can simply grep for "FAILED" in stdout.<br>
> > ><br>
> > > No FAILED string means it worked.<br>
> > ><br>
> > ><br>
> > > ><br>
> > > > George<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > George K. Thiruvathukal, PhD<br>
> > > ><br>
> > > > Professor of Computer Science, Loyola University Chicago<br>
> > > ><br>
> > > > Director, Center for Textual Studies and Digital Humanities<br>
> > > > Guest Faculty, Argonne National Laboratory, Math and Computer Science Division<br>
> > > > Editor in Chief, Computing in<br>
> > > > Science and Engineering (IEEE CS/AIP)<br>
> > > ><br>
> > > > (w)<br>
> > ><br>
> ><br>
> <a href="http://gkt.tv" target="_blank">gkt.tv</a> (v)<br>
> > > > 773.829.4872<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > On Thu, Nov 13, 2014 at 2:54 PM, Boisvert, Sebastien<br>
> > > > <<a href="mailto:boisvert@anl.gov">boisvert@anl.gov</a>> wrote:<br>
> > > ><br>
> > > > If you look at the following build:<br>
> > > ><br>
> > ><br>
> ><br>
> <a href="https://travis-ci.org/meanjs/mean/jobs/40710659" target="_blank">https://travis-ci.org/meanjs/mean/jobs/40710659</a><br>
> > > ><br>
> > > > it is written "passed" in green.<br>
> > > ><br>
> > > > Yet, if you look closely, the log file says:<br>
> > > ><br>
> > > > 13 passing (3s)<br>
> > > > 1 failing<br>
> > > ><br>
> > > > 1) Article Model Unit Tests: Method Save should be able to save without problems:<br>
> > > > Error: timeout of 2000ms exceeded<br>
> > > > at null.<anonymous> (/home/travis/build/meanjs/mean/node_modules/mocha/lib/runnable.js:158:19)<br>
> > > > at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > I am not sure what is going on there.<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
</div></div></blockquote></div><br></div>