[BIOSAL] Travis and failure

George K. Thiruvathukal gkt at cs.luc.edu
Thu Nov 13 17:52:33 CST 2014 

To not keep the world waiting, I have pushed out a fix (return
$local_failures) for run-unit-tests.sh on ​granularity. I'll do some
nanocommits to address the others. I have not tested a failed test with
Travis directly, but I did temporarily modify a test to fail so I can
ensure that the error code propagates back to make, and it was successful.
Oddly, make is returning 2 when only 1 is returned by run-unit-tests.sh.
Will look into this further, but I did confirm that when all tests pass, I
get 0 as the result of make test.

Best,
George


George K. Thiruvathukal, PhD
*Professor of Computer Science*, Loyola University Chicago
*Director*, Center for Textual Studies and Digital Humanities
*Guest Faculty*, Argonne National Laboratory, Math and Computer Science
Division
Editor in Chief, Computing in Science and Engineering
<http://www.computer.org/portal/web/computingnow/cise> (IEEE CS/AIP)
(w) gkt.tv (v) 773.829.4872


On Thu, Nov 13, 2014 at 4:18 PM, George K. Thiruvathukal <gkt at cs.luc.edu>
wrote:

> Yes, of course! :) I was not speaking in terms of concrete syntax but just
> pointing out that you've already done the work to make it happen. (I forgot
> to put the word "with" after "exit".)
>
> Best,
> George
>
>
> George K. Thiruvathukal, PhD
> *Professor of Computer Science*, Loyola University Chicago
> *Director*, Center for Textual Studies and Digital Humanities
> *Guest Faculty*, Argonne National Laboratory, Math and Computer Science
> Division
> Editor in Chief, Computing in Science and Engineering
> <http://www.computer.org/portal/web/computingnow/cise> (IEEE CS/AIP)
> (w) gkt.tv (v) 773.829.4872
>
>
> On Thu, Nov 13, 2014 at 4:17 PM, Boisvert, Sebastien <boisvert at anl.gov>
> wrote:
>
>> > From: George K. Thiruvathukal [gkt at cs.luc.edu]
>> > Sent: Thursday, November 13, 2014 4:14 PM
>> > To: Boisvert, Sebastien
>> > Cc: biosal at lists.cels.anl.gov
>> > Subject: Re: Travis and failure
>> >
>> >
>> >
>> >
>> > Looks like you're right! I can probably just exit $total_failures and
>> be done with it.
>>
>> It is better if you use "return".
>>
>>
>> This works:
>>
>> $ cat test.sh
>> #!/bin/bash
>>
>> function main()
>> {
>>     return 1
>> }
>>
>> main
>>
>>
>> >
>> >
>> > I'll work on this at home.
>> >
>> > George
>> >
>> >
>> >
>> > George K. Thiruvathukal, PhD
>> >
>> > Professor of Computer Science, Loyola University Chicago
>> >
>> > Director, Center for Textual Studies and Digital Humanities
>> > Guest Faculty, Argonne National Laboratory, Math and Computer Science
>> Division
>> > Editor in Chief, Computing in
>> >  Science and Engineering (IEEE CS/AIP)
>> >
>> > (w) gkt.tv (v)
>> >  773.829.4872
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Nov 13, 2014 at 4:04 PM, George K. Thiruvathukal
>> > <gkt at cs.luc.edu> wrote:
>> >
>> >
>> > Yes, it's very quick. I will try to work on it before the weekend.
>> >
>> >
>> > Tomorrow I have to meet students to talk about their semester projects.
>> Then it should be smooth sailing (I hope).
>> >
>> >
>> > George
>> >
>> >
>> >
>> > George K. Thiruvathukal, PhD
>> >
>> > Professor of Computer Science, Loyola University Chicago
>> >
>> > Director, Center for Textual Studies and Digital Humanities
>> > Guest Faculty, Argonne National Laboratory, Math and Computer Science
>> Division
>> > Editor in Chief, Computing in
>> >  Science and Engineering (IEEE CS/AIP)
>> >
>> > (w) gkt.tv (v)
>> >  773.829.4872
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Nov 13, 2014 at 4:01 PM, Boisvert, Sebastien
>> > <boisvert at anl.gov> wrote:
>> >
>> > > From: George K. Thiruvathukal [gkt at cs.luc.edu]
>> > > Sent: Thursday, November 13, 2014 3:55 PM
>> > > To: Boisvert, Sebastien
>> > > Cc:
>> > biosal at lists.cels.anl.gov
>> > > Subject: Re: Travis and failure
>> > >
>> > >
>> > >
>> > >
>> > > Hi there,
>> > >
>> > >
>> > > Yes, now that I wrote my first Thorium program, I can help with this.
>> I may have to work on this over the weekend, but I will keep you posted.
>> >
>> > I created an issue:
>> >
>> > https://github.com/GeneAssembly/biosal/issues/818
>> >
>> > This should be fairly easy to implement since all 3 scripts already
>> have a variable containing the number of
>> > failures.
>> >
>> > Thanks.
>> >
>> >
>> > >
>> > > Best,
>> > > George
>> > >
>> > >
>> > >
>> > >
>> > > George K. Thiruvathukal, PhD
>> > >
>> > > Professor of Computer Science, Loyola University Chicago
>> > >
>> > > Director, Center for Textual Studies and Digital Humanities
>> > > Guest Faculty, Argonne National Laboratory, Math and Computer Science
>> Division
>> > > Editor in Chief, Computing in
>> > >  Science and Engineering (IEEE CS/AIP)
>> > >
>> > > (w)
>> > gkt.tv (v)
>> > >  773.829.4872
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Thu, Nov 13, 2014 at 3:52 PM, Boisvert, Sebastien
>> > > <boisvert at anl.gov> wrote:
>> > >
>> > > > From: George K. Thiruvathukal [gkt at cs.luc.edu]
>> > > > Sent: Thursday, November 13, 2014 3:37 PM
>> > > > To: Boisvert, Sebastien
>> > > > Cc:
>> > >
>> > biosal at lists.cels.anl.gov
>> > > > Subject: Re: Travis and failure
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > It looks like the run-unit-tests.sh script is actually detecting
>> errors but not returning an exit code anywhere in the code.
>> > >
>> > > If you want to add the return codes:
>> > >
>> > > - unit-tests: you already found the script
>> > > - example-tests: tests/run-examples.sh
>> > > - application-tests: tests/run-application-tests.sh
>> > >
>> > >
>> > >
>> > > >
>> > > >
>> > > > George
>> > > >
>> > > >
>> > > >
>> > > > George K. Thiruvathukal, PhD
>> > > >
>> > > > Professor of Computer Science, Loyola University Chicago
>> > > >
>> > > > Director, Center for Textual Studies and Digital Humanities
>> > > > Guest Faculty, Argonne National Laboratory, Math and Computer
>> Science Division
>> > > > Editor in Chief, Computing in
>> > > >  Science and Engineering (IEEE CS/AIP)
>> > > >
>> > > > (w)
>> > >
>> > gkt.tv (v)
>> > > >  773.829.4872
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Thu, Nov 13, 2014 at 3:35 PM, George K. Thiruvathukal
>> > > > <gkt at cs.luc.edu> wrote:
>> > > >
>> > > >
>> > > > FYI, when Scala runs unit tests using "sbt test", it returns 1 if
>> any test fails.
>> > > >
>> > > >
>> > > >
>> > > > George
>> > > >
>> > > >
>> > > >
>> > > > George K. Thiruvathukal, PhD
>> > > >
>> > > > Professor of Computer Science, Loyola University Chicago
>> > > >
>> > > > Director, Center for Textual Studies and Digital Humanities
>> > > > Guest Faculty, Argonne National Laboratory, Math and Computer
>> Science Division
>> > > > Editor in Chief, Computing in
>> > > >  Science and Engineering (IEEE CS/AIP)
>> > > >
>> > > > (w)
>> > >
>> > gkt.tv (v)
>> > > >  773.829.4872
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Thu, Nov 13, 2014 at 3:34 PM, George K. Thiruvathukal
>> > > > <gkt at cs.luc.edu> wrote:
>> > > >
>> > > >
>> > > > Séb,
>> > > >
>> > > >
>> > > >
>> > > > I believe the project you showed me is an example of a broken
>> project, so that's why Travis is not detecting failure. When running unit
>> tests, failure is supposed to result not only in *junit.xml but
>> > > >  also propagate an error code back to the shell, not just rely on
>> the *junit.xml (or standard) output. The convention for this (I think you
>> already know) is to return non-zero.
>> > > >
>> > > > Can you let me know what Thorium was intended to do when tests
>> don't pass? I just did a test, and it seems like something might be broken.
>> I modified one of the tests to fail (and it failed) but "make test" returns
>> success to the OS:
>> > > >
>> > > >
>> > > > gkt at cs-nuc-wtc:~/Work/biosal$ make test
>> > > >
>> > > > UnitTestSuite        test_assembly_arc   PASSED:       5   failed:
>>      0   TOTAL:       5
>> > > > UnitTestSuite     test_assembly_vertex   PASSED:      22   failed:
>>      0   TOTAL:      22
>> > > > UnitTestSuite           test_dna_codec   PASSED:       6   failed:
>>      0   TOTAL:       6
>> > > > UnitTestSuite            test_dna_kmer   PASSED:       6   failed:
>>      0   TOTAL:       6
>> > > > UnitTestSuite        test_dna_sequence   PASSED:       3   failed:
>>      0   TOTAL:       3
>> > > > UnitTestSuite  test_dynamic_hash_table   PASSED: 1020411   failed:
>>      0   TOTAL: 1020411
>> > > > UnitTestSuite test_fast_queue_iterator   PASSED:     152   failed:
>>      0   TOTAL:     152
>> > > > UnitTestSuite           test_fast_ring   PASSED:     892   failed:
>>      0   TOTAL:     892
>> > > > UnitTestSuite    test_hash_table_group   PASSED:      43   failed:
>>      0   TOTAL:      43
>> > > > UnitTestSuite          test_hash_table   PASSED:   37921   failed:
>>      0   TOTAL:   37921
>> > > > UnitTestSuite          test_map_delete   PASSED:  400000   failed:
>>      0   TOTAL:  400000
>> > > > UnitTestSuite                 test_map   PASSED: 2535104   failed:
>>      0   TOTAL: 2535104
>> > > > UnitTestSuite         test_memory_pool   PASSED:   60013   failed:
>>      0   TOTAL:   60013
>> > > > UnitTestSuite                test_node   PASSED:      14   failed:
>>      0   TOTAL:      14
>> > > > UnitTestSuite              test_packer   PASSED:       1   failed:
>>      0   TOTAL:       1
>> > > > UnitTestSuite               test_queue   PASSED:   33075   failed:
>>      0   TOTAL:   33075
>> > > > UnitTestSuite      test_red_black_tree   PASSED:  900002   failed:
>>      0   TOTAL:  900002
>> > > > Error File: tests/test_ring.c, Function: main, Line: 18
>> > > > UnitTestSuite                test_ring   PASSED:     256   FAILED:
>>      1   TOTAL:     257
>> > > > UnitTestSuite          test_ring_queue   PASSED:   62139   failed:
>>      0   TOTAL:   62139
>> > > > UnitTestSuite                 test_set   PASSED:  409052   failed:
>>      0   TOTAL:  409052
>> > > > UnitTestSuite          test_statistics   PASSED:       1   failed:
>>      0   TOTAL:       1
>> > > > UnitTestSuite              test_string   PASSED:       5   failed:
>>      0   TOTAL:       5
>> > > > UnitTestSuite              test_vector   PASSED:  706709   failed:
>>      0   TOTAL:  706709
>> > > > UnitTestSuite PASSED 6165832 / 6165833 FAILED 1 / 6165833
>> > > > see unit-tests.junit.xml
>> > > > gkt at cs-nuc-wtc:~/Work/biosal$ echo $?
>> > > > 0
>> > > >
>> > > >
>> > > > Travis is not parsing the junit output (at least not for C
>> projects). I can obviously modify my script on Travis to ensure it does
>> fail. But I think we *should* return non-zero when any test fails.
>> > > >
>> > > > Best,
>> > > > George
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > George K. Thiruvathukal, PhD
>> > > >
>> > > > Professor of Computer Science, Loyola University Chicago
>> > > >
>> > > > Director, Center for Textual Studies and Digital Humanities
>> > > > Guest Faculty, Argonne National Laboratory, Math and Computer
>> Science Division
>> > > > Editor in Chief, Computing in
>> > > >  Science and Engineering (IEEE CS/AIP)
>> > > >
>> > > > (w)
>> > >
>> > gkt.tv (v)
>> > > >  773.829.4872
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Thu, Nov 13, 2014 at 3:10 PM, George K. Thiruvathukal
>> > > > <gkt at cs.luc.edu> wrote:
>> > > >
>> > > >
>> > > > Thanks for clarifying. I should have looked more closely at the
>> project name. :-) Today has been busy since I arrived at the office.
>> > > >
>> > > >
>> > > > I need to read up more on Travis to understand its approach to
>> tests in general. I will also ask my colleague, who uses it for more
>> traditional xUnit testing projects, too.
>> > > >
>> > > >
>> > > > George
>> > > >
>> > > >
>> > > >
>> > > > George K. Thiruvathukal, PhD
>> > > >
>> > > > Professor of Computer Science, Loyola University Chicago
>> > > >
>> > > > Director, Center for Textual Studies and Digital Humanities
>> > > > Guest Faculty, Argonne National Laboratory, Math and Computer
>> Science Division
>> > > > Editor in Chief, Computing in
>> > > >  Science and Engineering (IEEE CS/AIP)
>> > > >
>> > > > (w)
>> > >
>> > gkt.tv (v)
>> > > >  773.829.4872
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Thu, Nov 13, 2014 at 3:05 PM, Boisvert, Sebastien
>> > > > <boisvert at anl.gov> wrote:
>> > > >
>> > > > > From: George K. Thiruvathukal [gkt at cs.luc.edu]
>> > > > > Sent: Thursday, November 13, 2014 2:56 PM
>> > > > > To: Boisvert, Sebastien
>> > > > > Subject: Re: Travis and failure
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > ​Hi there!
>> > > > >
>> > > > >
>> > > > > I think the problem is that the failure is not being reflected as
>> a Unix error code (non-zero). Is it possible that your wrapper for the
>> JUnit results is not returning a non-zero code when tests fail?
>> > > >
>> > > > The link I gave to you is for the project "Mean.js".
>> > > >
>> > > > Also, I can confirm to you that "make tests" always returns 0.
>> > > >
>> > > > The result is available in *junit.xml. Or you can simply grep for
>> "FAILED" in stdout.
>> > > >
>> > > > No FAILED string means it worked.
>> > > >
>> > > >
>> > > > >
>> > > > > George​
>> > > > >
>> > > > >
>> > > > >
>> > > > > George K. Thiruvathukal, PhD
>> > > > >
>> > > > > Professor of Computer Science, Loyola University Chicago
>> > > > >
>> > > > > Director, Center for Textual Studies and Digital Humanities
>> > > > > Guest Faculty, Argonne National Laboratory, Math and Computer
>> Science Division
>> > > > > Editor in Chief, Computing in
>> > > > >  Science and Engineering (IEEE CS/AIP)
>> > > > >
>> > > > > (w)
>> > > >
>> > >
>> > gkt.tv (v)
>> > > > >  773.829.4872
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Thu, Nov 13, 2014 at 2:54 PM, Boisvert, Sebastien
>> > > > > <boisvert at anl.gov> wrote:
>> > > > >
>> > > > > If you look at the following build:
>> > > > >
>> > > >
>> > >
>> > https://travis-ci.org/meanjs/mean/jobs/40710659
>> > > > >
>> > > > > it is written "passed" in green.
>> > > > >
>> > > > > Yet, if you look closely, the log file says:
>> > > > >
>> > > > >  13 passing (3s)
>> > > > > 1 failing
>> > > > >
>> > > > > 1) Article Model Unit Tests: Method Save should be able to save
>> without problems:
>> > > > > Error: timeout of 2000ms exceeded
>> > > > > at null.<anonymous>
>> (/home/travis/build/meanjs/mean/node_modules/mocha/lib/runnable.js:158:19)
>> > > > > at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)
>> > > > >
>> > > > >
>> > > > >
>> > > > > I am not sure what is going on there.
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cels.anl.gov/pipermail/biosal/attachments/20141113/b89fc211/attachment-0001.html>

More information about the BIOSAL mailing list