[BIOSAL] Travis and failure

Thu Nov 13 15:52:40 CST 2014

> From: George K. Thiruvathukal [gkt at cs.luc.edu]
> Sent: Thursday, November 13, 2014 3:37 PM
> To: Boisvert, Sebastien
> Cc: biosal at lists.cels.anl.gov
> Subject: Re: Travis and failure
> 
> 
> 
> 
> It looks like the run-unit-tests.sh script is actually detecting errors but not returning an exit code anywhere in the code.

If you want to add the return codes:

- unit-tests: you already found the script
- example-tests: tests/run-examples.sh
- application-tests: tests/run-application-tests.sh

> 
> 
> George
> 
> 
> 
> George K. Thiruvathukal, PhD
> 
> Professor of Computer Science, Loyola University Chicago
> 
> Director, Center for Textual Studies and Digital Humanities
> Guest Faculty, Argonne National Laboratory, Math and Computer Science Division
> Editor in Chief, Computing in
>  Science and Engineering (IEEE CS/AIP)
> 
> (w) gkt.tv (v)
>  773.829.4872
> 
> 
> 
> 
> 
> 
> 
> On Thu, Nov 13, 2014 at 3:35 PM, George K. Thiruvathukal
> <gkt at cs.luc.edu> wrote:
> 
> 
> FYI, when Scala runs unit tests using "sbt test", it returns 1 if any test fails.
> 
> 
> 
> George
> 
> 
> 
> George K. Thiruvathukal, PhD
> 
> Professor of Computer Science, Loyola University Chicago
> 
> Director, Center for Textual Studies and Digital Humanities
> Guest Faculty, Argonne National Laboratory, Math and Computer Science Division
> Editor in Chief, Computing in
>  Science and Engineering (IEEE CS/AIP)
> 
> (w) gkt.tv (v)
>  773.829.4872
> 
> 
> 
> 
> 
> 
> 
> 
> On Thu, Nov 13, 2014 at 3:34 PM, George K. Thiruvathukal
> <gkt at cs.luc.edu> wrote:
> 
> 
> Séb,
> 
> 
> 
> I believe the project you showed me is an example of a broken project, so that's why Travis is not detecting failure. When running unit tests, failure is supposed to result not only in *junit.xml but
>  also propagate an error code back to the shell, not just rely on the *junit.xml (or standard) output. The convention for this (I think you already know) is to return non-zero.
> 
> Can you let me know what Thorium was intended to do when tests don't pass? I just did a test, and it seems like something might be broken. I modified one of the tests to fail (and it failed) but "make test" returns success to the OS:
> 
> 
> gkt at cs-nuc-wtc:~/Work/biosal$ make test
> 
> UnitTestSuite        test_assembly_arc   PASSED:       5   failed:       0   TOTAL:       5
> UnitTestSuite     test_assembly_vertex   PASSED:      22   failed:       0   TOTAL:      22
> UnitTestSuite           test_dna_codec   PASSED:       6   failed:       0   TOTAL:       6
> UnitTestSuite            test_dna_kmer   PASSED:       6   failed:       0   TOTAL:       6
> UnitTestSuite        test_dna_sequence   PASSED:       3   failed:       0   TOTAL:       3
> UnitTestSuite  test_dynamic_hash_table   PASSED: 1020411   failed:       0   TOTAL: 1020411
> UnitTestSuite test_fast_queue_iterator   PASSED:     152   failed:       0   TOTAL:     152
> UnitTestSuite           test_fast_ring   PASSED:     892   failed:       0   TOTAL:     892
> UnitTestSuite    test_hash_table_group   PASSED:      43   failed:       0   TOTAL:      43
> UnitTestSuite          test_hash_table   PASSED:   37921   failed:       0   TOTAL:   37921
> UnitTestSuite          test_map_delete   PASSED:  400000   failed:       0   TOTAL:  400000
> UnitTestSuite                 test_map   PASSED: 2535104   failed:       0   TOTAL: 2535104
> UnitTestSuite         test_memory_pool   PASSED:   60013   failed:       0   TOTAL:   60013
> UnitTestSuite                test_node   PASSED:      14   failed:       0   TOTAL:      14
> UnitTestSuite              test_packer   PASSED:       1   failed:       0   TOTAL:       1
> UnitTestSuite               test_queue   PASSED:   33075   failed:       0   TOTAL:   33075
> UnitTestSuite      test_red_black_tree   PASSED:  900002   failed:       0   TOTAL:  900002
> Error File: tests/test_ring.c, Function: main, Line: 18
> UnitTestSuite                test_ring   PASSED:     256   FAILED:       1   TOTAL:     257
> UnitTestSuite          test_ring_queue   PASSED:   62139   failed:       0   TOTAL:   62139
> UnitTestSuite                 test_set   PASSED:  409052   failed:       0   TOTAL:  409052
> UnitTestSuite          test_statistics   PASSED:       1   failed:       0   TOTAL:       1
> UnitTestSuite              test_string   PASSED:       5   failed:       0   TOTAL:       5
> UnitTestSuite              test_vector   PASSED:  706709   failed:       0   TOTAL:  706709
> UnitTestSuite PASSED 6165832 / 6165833 FAILED 1 / 6165833
> see unit-tests.junit.xml
> gkt at cs-nuc-wtc:~/Work/biosal$ echo $?
> 0
> 
> 
> Travis is not parsing the junit output (at least not for C projects). I can obviously modify my script on Travis to ensure it does fail. But I think we *should* return non-zero when any test fails. 
> 
> Best,
> George
> 
> 
> 
> 
> 
> George K. Thiruvathukal, PhD
> 
> Professor of Computer Science, Loyola University Chicago
> 
> Director, Center for Textual Studies and Digital Humanities
> Guest Faculty, Argonne National Laboratory, Math and Computer Science Division
> Editor in Chief, Computing in
>  Science and Engineering (IEEE CS/AIP)
> 
> (w) gkt.tv (v)
>  773.829.4872
> 
> 
> 
> 
> 
> 
> 
> 
> On Thu, Nov 13, 2014 at 3:10 PM, George K. Thiruvathukal
> <gkt at cs.luc.edu> wrote:
> 
> 
> Thanks for clarifying. I should have looked more closely at the project name. :-) Today has been busy since I arrived at the office.
> 
> 
> I need to read up more on Travis to understand its approach to tests in general. I will also ask my colleague, who uses it for more traditional xUnit testing projects, too.
> 
> 
> George
> 
> 
> 
> George K. Thiruvathukal, PhD
> 
> Professor of Computer Science, Loyola University Chicago
> 
> Director, Center for Textual Studies and Digital Humanities
> Guest Faculty, Argonne National Laboratory, Math and Computer Science Division
> Editor in Chief, Computing in
>  Science and Engineering (IEEE CS/AIP)
> 
> (w) gkt.tv (v)
>  773.829.4872
> 
> 
> 
> 
> 
> 
> 
> 
> On Thu, Nov 13, 2014 at 3:05 PM, Boisvert, Sebastien 
> <boisvert at anl.gov> wrote:
> 
> > From: George K. Thiruvathukal [gkt at cs.luc.edu]
> > Sent: Thursday, November 13, 2014 2:56 PM
> > To: Boisvert, Sebastien
> > Subject: Re: Travis and failure
> >
> >
> >
> >
> > Hi there!
> >
> >
> > I think the problem is that the failure is not being reflected as a Unix error code (non-zero). Is it possible that your wrapper for the JUnit results is not returning a non-zero code when tests fail?
> 
> The link I gave to you is for the project "Mean.js".
> 
> Also, I can confirm to you that "make tests" always returns 0.
> 
> The result is available in *junit.xml. Or you can simply grep for "FAILED" in stdout.
> 
> No FAILED string means it worked.
> 
> 
> >
> > George
> >
> >
> >
> > George K. Thiruvathukal, PhD
> >
> > Professor of Computer Science, Loyola University Chicago
> >
> > Director, Center for Textual Studies and Digital Humanities
> > Guest Faculty, Argonne National Laboratory, Math and Computer Science Division
> > Editor in Chief, Computing in
> >  Science and Engineering (IEEE CS/AIP)
> >
> > (w) 
> gkt.tv (v)
> >  773.829.4872
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Nov 13, 2014 at 2:54 PM, Boisvert, Sebastien
> > <boisvert at anl.gov> wrote:
> >
> > If you look at the following build:
> > 
> https://travis-ci.org/meanjs/mean/jobs/40710659
> >
> > it is written "passed" in green.
> >
> > Yet, if you look closely, the log file says:
> >
> >  13 passing (3s)
> > 1 failing
> >
> > 1) Article Model Unit Tests: Method Save should be able to save without problems:
> > Error: timeout of 2000ms exceeded
> > at null.<anonymous> (/home/travis/build/meanjs/mean/node_modules/mocha/lib/runnable.js:158:19)
> > at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)
> >
> >
> >
> > I am not sure what is going on there.
> >
> >
> >
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 