[BIOSAL] Travis and failure

Boisvert, Sebastien boisvert at anl.gov
Thu Nov 13 19:30:50 CST 2014 

> From: George K. Thiruvathukal [gkt at cs.luc.edu]
> Sent: Thursday, November 13, 2014 5:52 PM
> To: Boisvert, Sebastien
> Cc: biosal at lists.cels.anl.gov
> Subject: Re: Travis and failure
> 
> 
> 
> 
> To not keep the world waiting, I have pushed out a fix (return $local_failures) for run-unit-tests.sh on ​granularity. I'll do some nanocommits to address the others. I have not tested a failed test with Travis directly, but I did 

I pulled your branch. Thanks.

> temporarily modify a test
>  to fail so I can ensure that the error code propagates back to make, and it was successful. Oddly, make is returning 2 when only 1 is returned by run-unit-tests.sh.

This is make that is returning 2 when there were errors.

From "man make":

EXIT STATUS
       GNU make exits with a status of zero if all makefiles were successfully parsed and no targets that were built failed.  A status of one will be returned if the -q flag was used  and  make  determines
       that a target needs to be rebuilt.  A status of two will be returned if any errors were encountered.


> Will look into this further, but I did confirm that when all tests pass, I get 0 as the result
>  of make test.

Perfect !

> 
> 
> Best,
> George
> 
> 
> 
> 
> 
> George K. Thiruvathukal, PhD
> 
> Professor of Computer Science, Loyola University Chicago
> 
> Director, Center for Textual Studies and Digital Humanities
> Guest Faculty, Argonne National Laboratory, Math and Computer Science Division
> Editor in Chief, Computing in
>  Science and Engineering (IEEE CS/AIP)
> 
> (w) gkt.tv (v)
>  773.829.4872
> 
> 
> 
> 
> 
> 
> 
> On Thu, Nov 13, 2014 at 4:18 PM, George K. Thiruvathukal
> <gkt at cs.luc.edu> wrote:
> 
> 
> Yes, of course! :) I was not speaking in terms of concrete syntax but just pointing out that you've already done the work to make it happen. (I forgot to put the word "with" after "exit".)
> 
> 
> Best,
> George
> 
> 
> 
> 
> 
> George K. Thiruvathukal, PhD
> 
> Professor of Computer Science, Loyola University Chicago
> 
> Director, Center for Textual Studies and Digital Humanities
> Guest Faculty, Argonne National Laboratory, Math and Computer Science Division
> Editor in Chief, Computing in
>  Science and Engineering (IEEE CS/AIP)
> 
> (w) gkt.tv (v)
>  773.829.4872
> 
> 
> 
> 
> 
> 
> 
> 
> On Thu, Nov 13, 2014 at 4:17 PM, Boisvert, Sebastien 
> <boisvert at anl.gov> wrote:
> 
> > From: George K. Thiruvathukal [gkt at cs.luc.edu]
> > Sent: Thursday, November 13, 2014 4:14 PM
> > To: Boisvert, Sebastien
> > Cc: 
> biosal at lists.cels.anl.gov
> > Subject: Re: Travis and failure
> >
> >
> >
> >
> > Looks like you're right! I can probably just exit $total_failures and be done with it.
> 
> It is better if you use "return".
> 
> 
> This works:
> 
> $ cat test.sh
> #!/bin/bash
> 
> function main()
> {
>     return 1
> }
> 
> main
> 
> 
> 
> >
> >
> > I'll work on this at home.
> >
> > George
> >
> >
> >
> > George K. Thiruvathukal, PhD
> >
> > Professor of Computer Science, Loyola University Chicago
> >
> > Director, Center for Textual Studies and Digital Humanities
> > Guest Faculty, Argonne National Laboratory, Math and Computer Science Division
> > Editor in Chief, Computing in
> >  Science and Engineering (IEEE CS/AIP)
> >
> > (w) 
> gkt.tv (v)
> >  773.829.4872
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Nov 13, 2014 at 4:04 PM, George K. Thiruvathukal
> > <gkt at cs.luc.edu> wrote:
> >
> >
> > Yes, it's very quick. I will try to work on it before the weekend.
> >
> >
> > Tomorrow I have to meet students to talk about their semester projects. Then it should be smooth sailing (I hope).
> >
> >
> > George
> >
> >
> >
> > George K. Thiruvathukal, PhD
> >
> > Professor of Computer Science, Loyola University Chicago
> >
> > Director, Center for Textual Studies and Digital Humanities
> > Guest Faculty, Argonne National Laboratory, Math and Computer Science Division
> > Editor in Chief, Computing in
> >  Science and Engineering (IEEE CS/AIP)
> >
> > (w) 
> gkt.tv (v)
> >  773.829.4872
> >
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Nov 13, 2014 at 4:01 PM, Boisvert, Sebastien
> > <boisvert at anl.gov> wrote:
> >
> > > From: George K. Thiruvathukal [gkt at cs.luc.edu]
> > > Sent: Thursday, November 13, 2014 3:55 PM
> > > To: Boisvert, Sebastien
> > > Cc:
> > 
> biosal at lists.cels.anl.gov
> > > Subject: Re: Travis and failure
> > >
> > >
> > >
> > >
> > > Hi there,
> > >
> > >
> > > Yes, now that I wrote my first Thorium program, I can help with this. I may have to work on this over the weekend, but I will keep you posted.
> >
> > I created an issue:
> >
> > 
> https://github.com/GeneAssembly/biosal/issues/818
> >
> > This should be fairly easy to implement since all 3 scripts already have a variable containing the number of
> > failures.
> >
> > Thanks.
> >
> >
> > >
> > > Best,
> > > George
> > >
> > >
> > >
> > >
> > > George K. Thiruvathukal, PhD
> > >
> > > Professor of Computer Science, Loyola University Chicago
> > >
> > > Director, Center for Textual Studies and Digital Humanities
> > > Guest Faculty, Argonne National Laboratory, Math and Computer Science Division
> > > Editor in Chief, Computing in
> > >  Science and Engineering (IEEE CS/AIP)
> > >
> > > (w)
> > 
> gkt.tv (v)
> > >  773.829.4872
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Nov 13, 2014 at 3:52 PM, Boisvert, Sebastien
> > > <boisvert at anl.gov> wrote:
> > >
> > > > From: George K. Thiruvathukal [gkt at cs.luc.edu]
> > > > Sent: Thursday, November 13, 2014 3:37 PM
> > > > To: Boisvert, Sebastien
> > > > Cc:
> > >
> > 
> biosal at lists.cels.anl.gov
> > > > Subject: Re: Travis and failure
> > > >
> > > >
> > > >
> > > >
> > > > It looks like the run-unit-tests.sh script is actually detecting errors but not returning an exit code anywhere in the code.
> > >
> > > If you want to add the return codes:
> > >
> > > - unit-tests: you already found the script
> > > - example-tests: tests/run-examples.sh
> > > - application-tests: tests/run-application-tests.sh
> > >
> > >
> > >
> > > >
> > > >
> > > > George
> > > >
> > > >
> > > >
> > > > George K. Thiruvathukal, PhD
> > > >
> > > > Professor of Computer Science, Loyola University Chicago
> > > >
> > > > Director, Center for Textual Studies and Digital Humanities
> > > > Guest Faculty, Argonne National Laboratory, Math and Computer Science Division
> > > > Editor in Chief, Computing in
> > > >  Science and Engineering (IEEE CS/AIP)
> > > >
> > > > (w)
> > >
> > 
> gkt.tv (v)
> > > >  773.829.4872
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Nov 13, 2014 at 3:35 PM, George K. Thiruvathukal
> > > > <gkt at cs.luc.edu> wrote:
> > > >
> > > >
> > > > FYI, when Scala runs unit tests using "sbt test", it returns 1 if any test fails.
> > > >
> > > >
> > > >
> > > > George
> > > >
> > > >
> > > >
> > > > George K. Thiruvathukal, PhD
> > > >
> > > > Professor of Computer Science, Loyola University Chicago
> > > >
> > > > Director, Center for Textual Studies and Digital Humanities
> > > > Guest Faculty, Argonne National Laboratory, Math and Computer Science Division
> > > > Editor in Chief, Computing in
> > > >  Science and Engineering (IEEE CS/AIP)
> > > >
> > > > (w)
> > >
> > 
> gkt.tv (v)
> > > >  773.829.4872
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Nov 13, 2014 at 3:34 PM, George K. Thiruvathukal
> > > > <gkt at cs.luc.edu> wrote:
> > > >
> > > >
> > > > Séb,
> > > >
> > > >
> > > >
> > > > I believe the project you showed me is an example of a broken project, so that's why Travis is not detecting failure. When running unit tests, failure is supposed to result not only in *junit.xml but
> > > >  also propagate an error code back to the shell, not just rely on the *junit.xml (or standard) output. The convention for this (I think you already know) is to return non-zero.
> > > >
> > > > Can you let me know what Thorium was intended to do when tests don't pass? I just did a test, and it seems like something might be broken. I modified one of the tests to fail (and it failed) but "make test" returns success to the OS:
> > > >
> > > >
> > > > gkt at cs-nuc-wtc:~/Work/biosal$ make test
> > > >
> > > > UnitTestSuite        test_assembly_arc   PASSED:       5   failed:       0   TOTAL:       5
> > > > UnitTestSuite     test_assembly_vertex   PASSED:      22   failed:       0   TOTAL:      22
> > > > UnitTestSuite           test_dna_codec   PASSED:       6   failed:       0   TOTAL:       6
> > > > UnitTestSuite            test_dna_kmer   PASSED:       6   failed:       0   TOTAL:       6
> > > > UnitTestSuite        test_dna_sequence   PASSED:       3   failed:       0   TOTAL:       3
> > > > UnitTestSuite  test_dynamic_hash_table   PASSED: 1020411   failed:       0   TOTAL: 1020411
> > > > UnitTestSuite test_fast_queue_iterator   PASSED:     152   failed:       0   TOTAL:     152
> > > > UnitTestSuite           test_fast_ring   PASSED:     892   failed:       0   TOTAL:     892
> > > > UnitTestSuite    test_hash_table_group   PASSED:      43   failed:       0   TOTAL:      43
> > > > UnitTestSuite          test_hash_table   PASSED:   37921   failed:       0   TOTAL:   37921
> > > > UnitTestSuite          test_map_delete   PASSED:  400000   failed:       0   TOTAL:  400000
> > > > UnitTestSuite                 test_map   PASSED: 2535104   failed:       0   TOTAL: 2535104
> > > > UnitTestSuite         test_memory_pool   PASSED:   60013   failed:       0   TOTAL:   60013
> > > > UnitTestSuite                test_node   PASSED:      14   failed:       0   TOTAL:      14
> > > > UnitTestSuite              test_packer   PASSED:       1   failed:       0   TOTAL:       1
> > > > UnitTestSuite               test_queue   PASSED:   33075   failed:       0   TOTAL:   33075
> > > > UnitTestSuite      test_red_black_tree   PASSED:  900002   failed:       0   TOTAL:  900002
> > > > Error File: tests/test_ring.c, Function: main, Line: 18
> > > > UnitTestSuite                test_ring   PASSED:     256   FAILED:       1   TOTAL:     257
> > > > UnitTestSuite          test_ring_queue   PASSED:   62139   failed:       0   TOTAL:   62139
> > > > UnitTestSuite                 test_set   PASSED:  409052   failed:       0   TOTAL:  409052
> > > > UnitTestSuite          test_statistics   PASSED:       1   failed:       0   TOTAL:       1
> > > > UnitTestSuite              test_string   PASSED:       5   failed:       0   TOTAL:       5
> > > > UnitTestSuite              test_vector   PASSED:  706709   failed:       0   TOTAL:  706709
> > > > UnitTestSuite PASSED 6165832 / 6165833 FAILED 1 / 6165833
> > > > see unit-tests.junit.xml
> > > > gkt at cs-nuc-wtc:~/Work/biosal$ echo $?
> > > > 0
> > > >
> > > >
> > > > Travis is not parsing the junit output (at least not for C projects). I can obviously modify my script on Travis to ensure it does fail. But I think we *should* return non-zero when any test fails.
> > > >
> > > > Best,
> > > > George
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > George K. Thiruvathukal, PhD
> > > >
> > > > Professor of Computer Science, Loyola University Chicago
> > > >
> > > > Director, Center for Textual Studies and Digital Humanities
> > > > Guest Faculty, Argonne National Laboratory, Math and Computer Science Division
> > > > Editor in Chief, Computing in
> > > >  Science and Engineering (IEEE CS/AIP)
> > > >
> > > > (w)
> > >
> > 
> gkt.tv (v)
> > > >  773.829.4872
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Nov 13, 2014 at 3:10 PM, George K. Thiruvathukal
> > > > <gkt at cs.luc.edu> wrote:
> > > >
> > > >
> > > > Thanks for clarifying. I should have looked more closely at the project name. :-) Today has been busy since I arrived at the office.
> > > >
> > > >
> > > > I need to read up more on Travis to understand its approach to tests in general. I will also ask my colleague, who uses it for more traditional xUnit testing projects, too.
> > > >
> > > >
> > > > George
> > > >
> > > >
> > > >
> > > > George K. Thiruvathukal, PhD
> > > >
> > > > Professor of Computer Science, Loyola University Chicago
> > > >
> > > > Director, Center for Textual Studies and Digital Humanities
> > > > Guest Faculty, Argonne National Laboratory, Math and Computer Science Division
> > > > Editor in Chief, Computing in
> > > >  Science and Engineering (IEEE CS/AIP)
> > > >
> > > > (w)
> > >
> > 
> gkt.tv (v)
> > > >  773.829.4872
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Nov 13, 2014 at 3:05 PM, Boisvert, Sebastien
> > > > <boisvert at anl.gov> wrote:
> > > >
> > > > > From: George K. Thiruvathukal [gkt at cs.luc.edu]
> > > > > Sent: Thursday, November 13, 2014 2:56 PM
> > > > > To: Boisvert, Sebastien
> > > > > Subject: Re: Travis and failure
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ​Hi there!
> > > > >
> > > > >
> > > > > I think the problem is that the failure is not being reflected as a Unix error code (non-zero). Is it possible that your wrapper for the JUnit results is not returning a non-zero code when tests fail?
> > > >
> > > > The link I gave to you is for the project "Mean.js".
> > > >
> > > > Also, I can confirm to you that "make tests" always returns 0.
> > > >
> > > > The result is available in *junit.xml. Or you can simply grep for "FAILED" in stdout.
> > > >
> > > > No FAILED string means it worked.
> > > >
> > > >
> > > > >
> > > > > George​
> > > > >
> > > > >
> > > > >
> > > > > George K. Thiruvathukal, PhD
> > > > >
> > > > > Professor of Computer Science, Loyola University Chicago
> > > > >
> > > > > Director, Center for Textual Studies and Digital Humanities
> > > > > Guest Faculty, Argonne National Laboratory, Math and Computer Science Division
> > > > > Editor in Chief, Computing in
> > > > >  Science and Engineering (IEEE CS/AIP)
> > > > >
> > > > > (w)
> > > >
> > >
> > 
> gkt.tv (v)
> > > > >  773.829.4872
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Nov 13, 2014 at 2:54 PM, Boisvert, Sebastien
> > > > > <boisvert at anl.gov> wrote:
> > > > >
> > > > > If you look at the following build:
> > > > >
> > > >
> > >
> > 
> https://travis-ci.org/meanjs/mean/jobs/40710659
> > > > >
> > > > > it is written "passed" in green.
> > > > >
> > > > > Yet, if you look closely, the log file says:
> > > > >
> > > > >  13 passing (3s)
> > > > > 1 failing
> > > > >
> > > > > 1) Article Model Unit Tests: Method Save should be able to save without problems:
> > > > > Error: timeout of 2000ms exceeded
> > > > > at null.<anonymous> (/home/travis/build/meanjs/mean/node_modules/mocha/lib/runnable.js:158:19)
> > > > > at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)
> > > > >
> > > > >
> > > > >
> > > > > I am not sure what is going on there.
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> 
> 
> 
> 
> 
> 
> 
> 
>

More information about the BIOSAL mailing list