[BIOSAL] Travis and failure
George K. Thiruvathukal
gkt at cs.luc.edu
Thu Nov 13 15:34:42 CST 2014
Séb,
I believe the project you showed me is an example of a broken project, so
that's why Travis is not detecting failure. When running unit tests,
failure is supposed to result not only in *junit.xml but also propagate an
error code back to the shell, not just rely on the *junit.xml (or standard)
output. The convention for this (I think you already know) is to return
non-zero.
Can you let me know what Thorium was intended to do when tests don't pass?
I just did a test, and it seems like something might be broken. I modified
one of the tests to fail (and it failed) but "make test" returns success to
the OS:
gkt at cs-nuc-wtc:~/Work/biosal$ make test
UnitTestSuite test_assembly_arc PASSED: 5 failed: 0
TOTAL: 5
UnitTestSuite test_assembly_vertex PASSED: 22 failed: 0
TOTAL: 22
UnitTestSuite test_dna_codec PASSED: 6 failed: 0
TOTAL: 6
UnitTestSuite test_dna_kmer PASSED: 6 failed: 0
TOTAL: 6
UnitTestSuite test_dna_sequence PASSED: 3 failed: 0
TOTAL: 3
UnitTestSuite test_dynamic_hash_table PASSED: 1020411 failed: 0
TOTAL: 1020411
UnitTestSuite test_fast_queue_iterator PASSED: 152 failed: 0
TOTAL: 152
UnitTestSuite test_fast_ring PASSED: 892 failed: 0
TOTAL: 892
UnitTestSuite test_hash_table_group PASSED: 43 failed: 0
TOTAL: 43
UnitTestSuite test_hash_table PASSED: 37921 failed: 0
TOTAL: 37921
UnitTestSuite test_map_delete PASSED: 400000 failed: 0
TOTAL: 400000
UnitTestSuite test_map PASSED: 2535104 failed: 0
TOTAL: 2535104
UnitTestSuite test_memory_pool PASSED: 60013 failed: 0
TOTAL: 60013
UnitTestSuite test_node PASSED: 14 failed: 0
TOTAL: 14
UnitTestSuite test_packer PASSED: 1 failed: 0
TOTAL: 1
UnitTestSuite test_queue PASSED: 33075 failed: 0
TOTAL: 33075
UnitTestSuite test_red_black_tree PASSED: 900002 failed: 0
TOTAL: 900002
Error File: tests/test_ring.c, Function: main, Line: 18
UnitTestSuite test_ring PASSED: 256 FAILED: 1
TOTAL: 257
UnitTestSuite test_ring_queue PASSED: 62139 failed: 0
TOTAL: 62139
UnitTestSuite test_set PASSED: 409052 failed: 0
TOTAL: 409052
UnitTestSuite test_statistics PASSED: 1 failed: 0
TOTAL: 1
UnitTestSuite test_string PASSED: 5 failed: 0
TOTAL: 5
UnitTestSuite test_vector PASSED: 706709 failed: 0
TOTAL: 706709
UnitTestSuite PASSED 6165832 / 6165833 FAILED 1 / 6165833
see unit-tests.junit.xml
gkt at cs-nuc-wtc:~/Work/biosal$ echo $?
0
Travis is not parsing the junit output (at least not for C projects). I can
obviously modify my script on Travis to ensure it does fail. But I think we
*should* return non-zero when any test fails.
Best,
George
George K. Thiruvathukal, PhD
*Professor of Computer Science*, Loyola University Chicago
*Director*, Center for Textual Studies and Digital Humanities
*Guest Faculty*, Argonne National Laboratory, Math and Computer Science
Division
Editor in Chief, Computing in Science and Engineering
<http://www.computer.org/portal/web/computingnow/cise> (IEEE CS/AIP)
(w) gkt.tv (v) 773.829.4872
On Thu, Nov 13, 2014 at 3:10 PM, George K. Thiruvathukal <gkt at cs.luc.edu>
wrote:
> Thanks for clarifying. I should have looked more closely at the project
> name. :-) Today has been busy since I arrived at the office.
>
> I need to read up more on Travis to understand its approach to tests in
> general. I will also ask my colleague, who uses it for more traditional
> xUnit testing projects, too.
>
> George
>
> George K. Thiruvathukal, PhD
> *Professor of Computer Science*, Loyola University Chicago
> *Director*, Center for Textual Studies and Digital Humanities
> *Guest Faculty*, Argonne National Laboratory, Math and Computer Science
> Division
> Editor in Chief, Computing in Science and Engineering
> <http://www.computer.org/portal/web/computingnow/cise> (IEEE CS/AIP)
> (w) gkt.tv (v) 773.829.4872
>
>
> On Thu, Nov 13, 2014 at 3:05 PM, Boisvert, Sebastien <boisvert at anl.gov>
> wrote:
>
>> > From: George K. Thiruvathukal [gkt at cs.luc.edu]
>> > Sent: Thursday, November 13, 2014 2:56 PM
>> > To: Boisvert, Sebastien
>> > Subject: Re: Travis and failure
>> >
>> >
>> >
>> >
>> > Hi there!
>> >
>> >
>> > I think the problem is that the failure is not being reflected as a
>> Unix error code (non-zero). Is it possible that your wrapper for the JUnit
>> results is not returning a non-zero code when tests fail?
>>
>> The link I gave to you is for the project "Mean.js".
>>
>> Also, I can confirm to you that "make tests" always returns 0.
>>
>> The result is available in *junit.xml. Or you can simply grep for
>> "FAILED" in stdout.
>>
>> No FAILED string means it worked.
>>
>> >
>> > George
>> >
>> >
>> >
>> > George K. Thiruvathukal, PhD
>> >
>> > Professor of Computer Science, Loyola University Chicago
>> >
>> > Director, Center for Textual Studies and Digital Humanities
>> > Guest Faculty, Argonne National Laboratory, Math and Computer Science
>> Division
>> > Editor in Chief, Computing in
>> > Science and Engineering (IEEE CS/AIP)
>> >
>> > (w) gkt.tv (v)
>> > 773.829.4872
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Nov 13, 2014 at 2:54 PM, Boisvert, Sebastien
>> > <boisvert at anl.gov> wrote:
>> >
>> > If you look at the following build:
>> > https://travis-ci.org/meanjs/mean/jobs/40710659
>> >
>> > it is written "passed" in green.
>> >
>> > Yet, if you look closely, the log file says:
>> >
>> > 13 passing (3s)
>> > 1 failing
>> >
>> > 1) Article Model Unit Tests: Method Save should be able to save without
>> problems:
>> > Error: timeout of 2000ms exceeded
>> > at null.<anonymous>
>> (/home/travis/build/meanjs/mean/node_modules/mocha/lib/runnable.js:158:19)
>> > at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)
>> >
>> >
>> >
>> > I am not sure what is going on there.
>> >
>> >
>> >
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cels.anl.gov/pipermail/biosal/attachments/20141113/ecd57403/attachment.html>
More information about the BIOSAL
mailing list