[BIOSAL] Travis and failure
Boisvert, Sebastien
boisvert at anl.gov
Thu Nov 13 16:17:25 CST 2014
> From: George K. Thiruvathukal [gkt at cs.luc.edu]
> Sent: Thursday, November 13, 2014 4:14 PM
> To: Boisvert, Sebastien
> Cc: biosal at lists.cels.anl.gov
> Subject: Re: Travis and failure
>
>
>
>
> Looks like you're right! I can probably just exit $total_failures and be done with it.
It is better if you use "return".
This works:
$ cat test.sh
#!/bin/bash
function main()
{
return 1
}
main
>
>
> I'll work on this at home.
>
> George
>
>
>
> George K. Thiruvathukal, PhD
>
> Professor of Computer Science, Loyola University Chicago
>
> Director, Center for Textual Studies and Digital Humanities
> Guest Faculty, Argonne National Laboratory, Math and Computer Science Division
> Editor in Chief, Computing in
> Science and Engineering (IEEE CS/AIP)
>
> (w) gkt.tv (v)
> 773.829.4872
>
>
>
>
>
>
>
> On Thu, Nov 13, 2014 at 4:04 PM, George K. Thiruvathukal
> <gkt at cs.luc.edu> wrote:
>
>
> Yes, it's very quick. I will try to work on it before the weekend.
>
>
> Tomorrow I have to meet students to talk about their semester projects. Then it should be smooth sailing (I hope).
>
>
> George
>
>
>
> George K. Thiruvathukal, PhD
>
> Professor of Computer Science, Loyola University Chicago
>
> Director, Center for Textual Studies and Digital Humanities
> Guest Faculty, Argonne National Laboratory, Math and Computer Science Division
> Editor in Chief, Computing in
> Science and Engineering (IEEE CS/AIP)
>
> (w) gkt.tv (v)
> 773.829.4872
>
>
>
>
>
>
>
>
> On Thu, Nov 13, 2014 at 4:01 PM, Boisvert, Sebastien
> <boisvert at anl.gov> wrote:
>
> > From: George K. Thiruvathukal [gkt at cs.luc.edu]
> > Sent: Thursday, November 13, 2014 3:55 PM
> > To: Boisvert, Sebastien
> > Cc:
> biosal at lists.cels.anl.gov
> > Subject: Re: Travis and failure
> >
> >
> >
> >
> > Hi there,
> >
> >
> > Yes, now that I wrote my first Thorium program, I can help with this. I may have to work on this over the weekend, but I will keep you posted.
>
> I created an issue:
>
> https://github.com/GeneAssembly/biosal/issues/818
>
> This should be fairly easy to implement since all 3 scripts already have a variable containing the number of
> failures.
>
> Thanks.
>
>
> >
> > Best,
> > George
> >
> >
> >
> >
> > George K. Thiruvathukal, PhD
> >
> > Professor of Computer Science, Loyola University Chicago
> >
> > Director, Center for Textual Studies and Digital Humanities
> > Guest Faculty, Argonne National Laboratory, Math and Computer Science Division
> > Editor in Chief, Computing in
> > Science and Engineering (IEEE CS/AIP)
> >
> > (w)
> gkt.tv (v)
> > 773.829.4872
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Nov 13, 2014 at 3:52 PM, Boisvert, Sebastien
> > <boisvert at anl.gov> wrote:
> >
> > > From: George K. Thiruvathukal [gkt at cs.luc.edu]
> > > Sent: Thursday, November 13, 2014 3:37 PM
> > > To: Boisvert, Sebastien
> > > Cc:
> >
> biosal at lists.cels.anl.gov
> > > Subject: Re: Travis and failure
> > >
> > >
> > >
> > >
> > > It looks like the run-unit-tests.sh script is actually detecting errors but not returning an exit code anywhere in the code.
> >
> > If you want to add the return codes:
> >
> > - unit-tests: you already found the script
> > - example-tests: tests/run-examples.sh
> > - application-tests: tests/run-application-tests.sh
> >
> >
> >
> > >
> > >
> > > George
> > >
> > >
> > >
> > > George K. Thiruvathukal, PhD
> > >
> > > Professor of Computer Science, Loyola University Chicago
> > >
> > > Director, Center for Textual Studies and Digital Humanities
> > > Guest Faculty, Argonne National Laboratory, Math and Computer Science Division
> > > Editor in Chief, Computing in
> > > Science and Engineering (IEEE CS/AIP)
> > >
> > > (w)
> >
> gkt.tv (v)
> > > 773.829.4872
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Nov 13, 2014 at 3:35 PM, George K. Thiruvathukal
> > > <gkt at cs.luc.edu> wrote:
> > >
> > >
> > > FYI, when Scala runs unit tests using "sbt test", it returns 1 if any test fails.
> > >
> > >
> > >
> > > George
> > >
> > >
> > >
> > > George K. Thiruvathukal, PhD
> > >
> > > Professor of Computer Science, Loyola University Chicago
> > >
> > > Director, Center for Textual Studies and Digital Humanities
> > > Guest Faculty, Argonne National Laboratory, Math and Computer Science Division
> > > Editor in Chief, Computing in
> > > Science and Engineering (IEEE CS/AIP)
> > >
> > > (w)
> >
> gkt.tv (v)
> > > 773.829.4872
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Nov 13, 2014 at 3:34 PM, George K. Thiruvathukal
> > > <gkt at cs.luc.edu> wrote:
> > >
> > >
> > > Séb,
> > >
> > >
> > >
> > > I believe the project you showed me is an example of a broken project, so that's why Travis is not detecting failure. When running unit tests, failure is supposed to result not only in *junit.xml but
> > > also propagate an error code back to the shell, not just rely on the *junit.xml (or standard) output. The convention for this (I think you already know) is to return non-zero.
> > >
> > > Can you let me know what Thorium was intended to do when tests don't pass? I just did a test, and it seems like something might be broken. I modified one of the tests to fail (and it failed) but "make test" returns success to the OS:
> > >
> > >
> > > gkt at cs-nuc-wtc:~/Work/biosal$ make test
> > >
> > > UnitTestSuite test_assembly_arc PASSED: 5 failed: 0 TOTAL: 5
> > > UnitTestSuite test_assembly_vertex PASSED: 22 failed: 0 TOTAL: 22
> > > UnitTestSuite test_dna_codec PASSED: 6 failed: 0 TOTAL: 6
> > > UnitTestSuite test_dna_kmer PASSED: 6 failed: 0 TOTAL: 6
> > > UnitTestSuite test_dna_sequence PASSED: 3 failed: 0 TOTAL: 3
> > > UnitTestSuite test_dynamic_hash_table PASSED: 1020411 failed: 0 TOTAL: 1020411
> > > UnitTestSuite test_fast_queue_iterator PASSED: 152 failed: 0 TOTAL: 152
> > > UnitTestSuite test_fast_ring PASSED: 892 failed: 0 TOTAL: 892
> > > UnitTestSuite test_hash_table_group PASSED: 43 failed: 0 TOTAL: 43
> > > UnitTestSuite test_hash_table PASSED: 37921 failed: 0 TOTAL: 37921
> > > UnitTestSuite test_map_delete PASSED: 400000 failed: 0 TOTAL: 400000
> > > UnitTestSuite test_map PASSED: 2535104 failed: 0 TOTAL: 2535104
> > > UnitTestSuite test_memory_pool PASSED: 60013 failed: 0 TOTAL: 60013
> > > UnitTestSuite test_node PASSED: 14 failed: 0 TOTAL: 14
> > > UnitTestSuite test_packer PASSED: 1 failed: 0 TOTAL: 1
> > > UnitTestSuite test_queue PASSED: 33075 failed: 0 TOTAL: 33075
> > > UnitTestSuite test_red_black_tree PASSED: 900002 failed: 0 TOTAL: 900002
> > > Error File: tests/test_ring.c, Function: main, Line: 18
> > > UnitTestSuite test_ring PASSED: 256 FAILED: 1 TOTAL: 257
> > > UnitTestSuite test_ring_queue PASSED: 62139 failed: 0 TOTAL: 62139
> > > UnitTestSuite test_set PASSED: 409052 failed: 0 TOTAL: 409052
> > > UnitTestSuite test_statistics PASSED: 1 failed: 0 TOTAL: 1
> > > UnitTestSuite test_string PASSED: 5 failed: 0 TOTAL: 5
> > > UnitTestSuite test_vector PASSED: 706709 failed: 0 TOTAL: 706709
> > > UnitTestSuite PASSED 6165832 / 6165833 FAILED 1 / 6165833
> > > see unit-tests.junit.xml
> > > gkt at cs-nuc-wtc:~/Work/biosal$ echo $?
> > > 0
> > >
> > >
> > > Travis is not parsing the junit output (at least not for C projects). I can obviously modify my script on Travis to ensure it does fail. But I think we *should* return non-zero when any test fails.
> > >
> > > Best,
> > > George
> > >
> > >
> > >
> > >
> > >
> > > George K. Thiruvathukal, PhD
> > >
> > > Professor of Computer Science, Loyola University Chicago
> > >
> > > Director, Center for Textual Studies and Digital Humanities
> > > Guest Faculty, Argonne National Laboratory, Math and Computer Science Division
> > > Editor in Chief, Computing in
> > > Science and Engineering (IEEE CS/AIP)
> > >
> > > (w)
> >
> gkt.tv (v)
> > > 773.829.4872
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Nov 13, 2014 at 3:10 PM, George K. Thiruvathukal
> > > <gkt at cs.luc.edu> wrote:
> > >
> > >
> > > Thanks for clarifying. I should have looked more closely at the project name. :-) Today has been busy since I arrived at the office.
> > >
> > >
> > > I need to read up more on Travis to understand its approach to tests in general. I will also ask my colleague, who uses it for more traditional xUnit testing projects, too.
> > >
> > >
> > > George
> > >
> > >
> > >
> > > George K. Thiruvathukal, PhD
> > >
> > > Professor of Computer Science, Loyola University Chicago
> > >
> > > Director, Center for Textual Studies and Digital Humanities
> > > Guest Faculty, Argonne National Laboratory, Math and Computer Science Division
> > > Editor in Chief, Computing in
> > > Science and Engineering (IEEE CS/AIP)
> > >
> > > (w)
> >
> gkt.tv (v)
> > > 773.829.4872
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Nov 13, 2014 at 3:05 PM, Boisvert, Sebastien
> > > <boisvert at anl.gov> wrote:
> > >
> > > > From: George K. Thiruvathukal [gkt at cs.luc.edu]
> > > > Sent: Thursday, November 13, 2014 2:56 PM
> > > > To: Boisvert, Sebastien
> > > > Subject: Re: Travis and failure
> > > >
> > > >
> > > >
> > > >
> > > > Hi there!
> > > >
> > > >
> > > > I think the problem is that the failure is not being reflected as a Unix error code (non-zero). Is it possible that your wrapper for the JUnit results is not returning a non-zero code when tests fail?
> > >
> > > The link I gave to you is for the project "Mean.js".
> > >
> > > Also, I can confirm to you that "make tests" always returns 0.
> > >
> > > The result is available in *junit.xml. Or you can simply grep for "FAILED" in stdout.
> > >
> > > No FAILED string means it worked.
> > >
> > >
> > > >
> > > > George
> > > >
> > > >
> > > >
> > > > George K. Thiruvathukal, PhD
> > > >
> > > > Professor of Computer Science, Loyola University Chicago
> > > >
> > > > Director, Center for Textual Studies and Digital Humanities
> > > > Guest Faculty, Argonne National Laboratory, Math and Computer Science Division
> > > > Editor in Chief, Computing in
> > > > Science and Engineering (IEEE CS/AIP)
> > > >
> > > > (w)
> > >
> >
> gkt.tv (v)
> > > > 773.829.4872
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Nov 13, 2014 at 2:54 PM, Boisvert, Sebastien
> > > > <boisvert at anl.gov> wrote:
> > > >
> > > > If you look at the following build:
> > > >
> > >
> >
> https://travis-ci.org/meanjs/mean/jobs/40710659
> > > >
> > > > it is written "passed" in green.
> > > >
> > > > Yet, if you look closely, the log file says:
> > > >
> > > > 13 passing (3s)
> > > > 1 failing
> > > >
> > > > 1) Article Model Unit Tests: Method Save should be able to save without problems:
> > > > Error: timeout of 2000ms exceeded
> > > > at null.<anonymous> (/home/travis/build/meanjs/mean/node_modules/mocha/lib/runnable.js:158:19)
> > > > at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)
> > > >
> > > >
> > > >
> > > > I am not sure what is going on there.
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
>
>
>
>
More information about the BIOSAL
mailing list