[BIOSAL] Travis and failure

Thu Nov 13 15:37:02 CST 2014

It looks like the run-unit-tests.sh script is actually detecting errors but
not returning an exit code anywhere in the code.

George

George K. Thiruvathukal, PhD
*Professor of Computer Science*, Loyola University Chicago
*Director*, Center for Textual Studies and Digital Humanities
*Guest Faculty*, Argonne National Laboratory, Math and Computer Science
Division
Editor in Chief, Computing in Science and Engineering
<http://www.computer.org/portal/web/computingnow/cise> (IEEE CS/AIP)
(w) gkt.tv (v) 773.829.4872

On Thu, Nov 13, 2014 at 3:35 PM, George K. Thiruvathukal <gkt at cs.luc.edu>
wrote:

> FYI, when Scala runs unit tests using "sbt test", it returns 1 if any test
> fails.
>
> George
>
> George K. Thiruvathukal, PhD
> *Professor of Computer Science*, Loyola University Chicago
> *Director*, Center for Textual Studies and Digital Humanities
> *Guest Faculty*, Argonne National Laboratory, Math and Computer Science
> Division
> Editor in Chief, Computing in Science and Engineering
> <http://www.computer.org/portal/web/computingnow/cise> (IEEE CS/AIP)
> (w) gkt.tv (v) 773.829.4872
>
>
> On Thu, Nov 13, 2014 at 3:34 PM, George K. Thiruvathukal <gkt at cs.luc.edu>
> wrote:
>
>> Séb,
>>
>> I believe the project you showed me is an example of a broken project, so
>> that's why Travis is not detecting failure. When running unit tests,
>> failure is supposed to result not only in *junit.xml but also propagate an
>> error code back to the shell, not just rely on the *junit.xml (or standard)
>> output. The convention for this (I think you already know) is to return
>> non-zero.
>>
>> Can you let me know what Thorium was intended to do when tests don't
>> pass? I just did a test, and it seems like something might be broken. I
>> modified one of the tests to fail (and it failed) but "make test" returns
>> success to the OS:
>>
>> gkt at cs-nuc-wtc:~/Work/biosal$ make test
>> UnitTestSuite        test_assembly_arc   PASSED:       5   failed:
>> 0   TOTAL:       5
>> UnitTestSuite     test_assembly_vertex   PASSED:      22   failed:
>> 0   TOTAL:      22
>> UnitTestSuite           test_dna_codec   PASSED:       6   failed:
>> 0   TOTAL:       6
>> UnitTestSuite            test_dna_kmer   PASSED:       6   failed:
>> 0   TOTAL:       6
>> UnitTestSuite        test_dna_sequence   PASSED:       3   failed:
>> 0   TOTAL:       3
>> UnitTestSuite  test_dynamic_hash_table   PASSED: 1020411   failed:
>> 0   TOTAL: 1020411
>> UnitTestSuite test_fast_queue_iterator   PASSED:     152   failed:
>> 0   TOTAL:     152
>> UnitTestSuite           test_fast_ring   PASSED:     892   failed:
>> 0   TOTAL:     892
>> UnitTestSuite    test_hash_table_group   PASSED:      43   failed:
>> 0   TOTAL:      43
>> UnitTestSuite          test_hash_table   PASSED:   37921   failed:
>> 0   TOTAL:   37921
>> UnitTestSuite          test_map_delete   PASSED:  400000   failed:
>> 0   TOTAL:  400000
>> UnitTestSuite                 test_map   PASSED: 2535104   failed:
>> 0   TOTAL: 2535104
>> UnitTestSuite         test_memory_pool   PASSED:   60013   failed:
>> 0   TOTAL:   60013
>> UnitTestSuite                test_node   PASSED:      14   failed:
>> 0   TOTAL:      14
>> UnitTestSuite              test_packer   PASSED:       1   failed:
>> 0   TOTAL:       1
>> UnitTestSuite               test_queue   PASSED:   33075   failed:
>> 0   TOTAL:   33075
>> UnitTestSuite      test_red_black_tree   PASSED:  900002   failed:
>> 0   TOTAL:  900002
>> Error File: tests/test_ring.c, Function: main, Line: 18
>> UnitTestSuite                test_ring   PASSED:     256   FAILED:
>> 1   TOTAL:     257
>> UnitTestSuite          test_ring_queue   PASSED:   62139   failed:
>> 0   TOTAL:   62139
>> UnitTestSuite                 test_set   PASSED:  409052   failed:
>> 0   TOTAL:  409052
>> UnitTestSuite          test_statistics   PASSED:       1   failed:
>> 0   TOTAL:       1
>> UnitTestSuite              test_string   PASSED:       5   failed:
>> 0   TOTAL:       5
>> UnitTestSuite              test_vector   PASSED:  706709   failed:
>> 0   TOTAL:  706709
>> UnitTestSuite PASSED 6165832 / 6165833 FAILED 1 / 6165833
>> see unit-tests.junit.xml
>> gkt at cs-nuc-wtc:~/Work/biosal$ echo $?
>> 0
>>
>> Travis is not parsing the junit output (at least not for C projects). I
>> can obviously modify my script on Travis to ensure it does fail. But I
>> think we *should* return non-zero when any test fails.
>>
>> Best,
>> George
>>
>>
>> George K. Thiruvathukal, PhD
>> *Professor of Computer Science*, Loyola University Chicago
>> *Director*, Center for Textual Studies and Digital Humanities
>> *Guest Faculty*, Argonne National Laboratory, Math and Computer Science
>> Division
>> Editor in Chief, Computing in Science and Engineering
>> <http://www.computer.org/portal/web/computingnow/cise> (IEEE CS/AIP)
>> (w) gkt.tv (v) 773.829.4872
>>
>>
>> On Thu, Nov 13, 2014 at 3:10 PM, George K. Thiruvathukal <gkt at cs.luc.edu>
>> wrote:
>>
>>> Thanks for clarifying. I should have looked more closely at the project
>>> name. :-) Today has been busy since I arrived at the office.
>>>
>>> I need to read up more on Travis to understand its approach to tests in
>>> general. I will also ask my colleague, who uses it for more traditional
>>> xUnit testing projects, too.
>>>
>>> George
>>>
>>> George K. Thiruvathukal, PhD
>>> *Professor of Computer Science*, Loyola University Chicago
>>> *Director*, Center for Textual Studies and Digital Humanities
>>> *Guest Faculty*, Argonne National Laboratory, Math and Computer Science
>>> Division
>>> Editor in Chief, Computing in Science and Engineering
>>> <http://www.computer.org/portal/web/computingnow/cise> (IEEE CS/AIP)
>>> (w) gkt.tv (v) 773.829.4872
>>>
>>>
>>> On Thu, Nov 13, 2014 at 3:05 PM, Boisvert, Sebastien <boisvert at anl.gov>
>>> wrote:
>>>
>>>> > From: George K. Thiruvathukal [gkt at cs.luc.edu]
>>>> > Sent: Thursday, November 13, 2014 2:56 PM
>>>> > To: Boisvert, Sebastien
>>>> > Subject: Re: Travis and failure
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > Hi there!
>>>> >
>>>> >
>>>> > I think the problem is that the failure is not being reflected as a
>>>> Unix error code (non-zero). Is it possible that your wrapper for the JUnit
>>>> results is not returning a non-zero code when tests fail?
>>>>
>>>> The link I gave to you is for the project "Mean.js".
>>>>
>>>> Also, I can confirm to you that "make tests" always returns 0.
>>>>
>>>> The result is available in *junit.xml. Or you can simply grep for
>>>> "FAILED" in stdout.
>>>>
>>>> No FAILED string means it worked.
>>>>
>>>> >
>>>> > George
>>>> >
>>>> >
>>>> >
>>>> > George K. Thiruvathukal, PhD
>>>> >
>>>> > Professor of Computer Science, Loyola University Chicago
>>>> >
>>>> > Director, Center for Textual Studies and Digital Humanities
>>>> > Guest Faculty, Argonne National Laboratory, Math and Computer Science
>>>> Division
>>>> > Editor in Chief, Computing in
>>>> >  Science and Engineering (IEEE CS/AIP)
>>>> >
>>>> > (w) gkt.tv (v)
>>>> >  773.829.4872
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Thu, Nov 13, 2014 at 2:54 PM, Boisvert, Sebastien
>>>> > <boisvert at anl.gov> wrote:
>>>> >
>>>> > If you look at the following build:
>>>> > https://travis-ci.org/meanjs/mean/jobs/40710659
>>>> >
>>>> > it is written "passed" in green.
>>>> >
>>>> > Yet, if you look closely, the log file says:
>>>> >
>>>> >  13 passing (3s)
>>>> > 1 failing
>>>> >
>>>> > 1) Article Model Unit Tests: Method Save should be able to save
>>>> without problems:
>>>> > Error: timeout of 2000ms exceeded
>>>> > at null.<anonymous>
>>>> (/home/travis/build/meanjs/mean/node_modules/mocha/lib/runnable.js:158:19)
>>>> > at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)
>>>> >
>>>> >
>>>> >
>>>> > I am not sure what is going on there.
>>>> >
>>>> >
>>>> >
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cels.anl.gov/pipermail/biosal/attachments/20141113/c0e98880/attachment-0001.html>