[BIOSAL] Travis and failure
George K. Thiruvathukal
gkt at cs.luc.edu
Thu Nov 13 16:18:47 CST 2014
Yes, of course! :) I was not speaking in terms of concrete syntax but just
pointing out that you've already done the work to make it happen. (I forgot
to put the word "with" after "exit".)
Best,
George
George K. Thiruvathukal, PhD
*Professor of Computer Science*, Loyola University Chicago
*Director*, Center for Textual Studies and Digital Humanities
*Guest Faculty*, Argonne National Laboratory, Math and Computer Science
Division
Editor in Chief, Computing in Science and Engineering
<http://www.computer.org/portal/web/computingnow/cise> (IEEE CS/AIP)
(w) gkt.tv (v) 773.829.4872
On Thu, Nov 13, 2014 at 4:17 PM, Boisvert, Sebastien <boisvert at anl.gov>
wrote:
> > From: George K. Thiruvathukal [gkt at cs.luc.edu]
> > Sent: Thursday, November 13, 2014 4:14 PM
> > To: Boisvert, Sebastien
> > Cc: biosal at lists.cels.anl.gov
> > Subject: Re: Travis and failure
> >
> >
> >
> >
> > Looks like you're right! I can probably just exit $total_failures and be
> done with it.
>
> It is better if you use "return".
>
>
> This works:
>
> $ cat test.sh
> #!/bin/bash
>
> function main()
> {
> return 1
> }
>
> main
>
>
> >
> >
> > I'll work on this at home.
> >
> > George
> >
> >
> >
> > George K. Thiruvathukal, PhD
> >
> > Professor of Computer Science, Loyola University Chicago
> >
> > Director, Center for Textual Studies and Digital Humanities
> > Guest Faculty, Argonne National Laboratory, Math and Computer Science
> Division
> > Editor in Chief, Computing in
> > Science and Engineering (IEEE CS/AIP)
> >
> > (w) gkt.tv (v)
> > 773.829.4872
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Nov 13, 2014 at 4:04 PM, George K. Thiruvathukal
> > <gkt at cs.luc.edu> wrote:
> >
> >
> > Yes, it's very quick. I will try to work on it before the weekend.
> >
> >
> > Tomorrow I have to meet students to talk about their semester projects.
> Then it should be smooth sailing (I hope).
> >
> >
> > George
> >
> >
> >
> > George K. Thiruvathukal, PhD
> >
> > Professor of Computer Science, Loyola University Chicago
> >
> > Director, Center for Textual Studies and Digital Humanities
> > Guest Faculty, Argonne National Laboratory, Math and Computer Science
> Division
> > Editor in Chief, Computing in
> > Science and Engineering (IEEE CS/AIP)
> >
> > (w) gkt.tv (v)
> > 773.829.4872
> >
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Nov 13, 2014 at 4:01 PM, Boisvert, Sebastien
> > <boisvert at anl.gov> wrote:
> >
> > > From: George K. Thiruvathukal [gkt at cs.luc.edu]
> > > Sent: Thursday, November 13, 2014 3:55 PM
> > > To: Boisvert, Sebastien
> > > Cc:
> > biosal at lists.cels.anl.gov
> > > Subject: Re: Travis and failure
> > >
> > >
> > >
> > >
> > > Hi there,
> > >
> > >
> > > Yes, now that I wrote my first Thorium program, I can help with this.
> I may have to work on this over the weekend, but I will keep you posted.
> >
> > I created an issue:
> >
> > https://github.com/GeneAssembly/biosal/issues/818
> >
> > This should be fairly easy to implement since all 3 scripts already have
> a variable containing the number of
> > failures.
> >
> > Thanks.
> >
> >
> > >
> > > Best,
> > > George
> > >
> > >
> > >
> > >
> > > George K. Thiruvathukal, PhD
> > >
> > > Professor of Computer Science, Loyola University Chicago
> > >
> > > Director, Center for Textual Studies and Digital Humanities
> > > Guest Faculty, Argonne National Laboratory, Math and Computer Science
> Division
> > > Editor in Chief, Computing in
> > > Science and Engineering (IEEE CS/AIP)
> > >
> > > (w)
> > gkt.tv (v)
> > > 773.829.4872
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Nov 13, 2014 at 3:52 PM, Boisvert, Sebastien
> > > <boisvert at anl.gov> wrote:
> > >
> > > > From: George K. Thiruvathukal [gkt at cs.luc.edu]
> > > > Sent: Thursday, November 13, 2014 3:37 PM
> > > > To: Boisvert, Sebastien
> > > > Cc:
> > >
> > biosal at lists.cels.anl.gov
> > > > Subject: Re: Travis and failure
> > > >
> > > >
> > > >
> > > >
> > > > It looks like the run-unit-tests.sh script is actually detecting
> errors but not returning an exit code anywhere in the code.
> > >
> > > If you want to add the return codes:
> > >
> > > - unit-tests: you already found the script
> > > - example-tests: tests/run-examples.sh
> > > - application-tests: tests/run-application-tests.sh
> > >
> > >
> > >
> > > >
> > > >
> > > > George
> > > >
> > > >
> > > >
> > > > George K. Thiruvathukal, PhD
> > > >
> > > > Professor of Computer Science, Loyola University Chicago
> > > >
> > > > Director, Center for Textual Studies and Digital Humanities
> > > > Guest Faculty, Argonne National Laboratory, Math and Computer
> Science Division
> > > > Editor in Chief, Computing in
> > > > Science and Engineering (IEEE CS/AIP)
> > > >
> > > > (w)
> > >
> > gkt.tv (v)
> > > > 773.829.4872
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Nov 13, 2014 at 3:35 PM, George K. Thiruvathukal
> > > > <gkt at cs.luc.edu> wrote:
> > > >
> > > >
> > > > FYI, when Scala runs unit tests using "sbt test", it returns 1 if
> any test fails.
> > > >
> > > >
> > > >
> > > > George
> > > >
> > > >
> > > >
> > > > George K. Thiruvathukal, PhD
> > > >
> > > > Professor of Computer Science, Loyola University Chicago
> > > >
> > > > Director, Center for Textual Studies and Digital Humanities
> > > > Guest Faculty, Argonne National Laboratory, Math and Computer
> Science Division
> > > > Editor in Chief, Computing in
> > > > Science and Engineering (IEEE CS/AIP)
> > > >
> > > > (w)
> > >
> > gkt.tv (v)
> > > > 773.829.4872
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Nov 13, 2014 at 3:34 PM, George K. Thiruvathukal
> > > > <gkt at cs.luc.edu> wrote:
> > > >
> > > >
> > > > Séb,
> > > >
> > > >
> > > >
> > > > I believe the project you showed me is an example of a broken
> project, so that's why Travis is not detecting failure. When running unit
> tests, failure is supposed to result not only in *junit.xml but
> > > > also propagate an error code back to the shell, not just rely on
> the *junit.xml (or standard) output. The convention for this (I think you
> already know) is to return non-zero.
> > > >
> > > > Can you let me know what Thorium was intended to do when tests don't
> pass? I just did a test, and it seems like something might be broken. I
> modified one of the tests to fail (and it failed) but "make test" returns
> success to the OS:
> > > >
> > > >
> > > > gkt at cs-nuc-wtc:~/Work/biosal$ make test
> > > >
> > > > UnitTestSuite test_assembly_arc PASSED: 5 failed:
> 0 TOTAL: 5
> > > > UnitTestSuite test_assembly_vertex PASSED: 22 failed:
> 0 TOTAL: 22
> > > > UnitTestSuite test_dna_codec PASSED: 6 failed:
> 0 TOTAL: 6
> > > > UnitTestSuite test_dna_kmer PASSED: 6 failed:
> 0 TOTAL: 6
> > > > UnitTestSuite test_dna_sequence PASSED: 3 failed:
> 0 TOTAL: 3
> > > > UnitTestSuite test_dynamic_hash_table PASSED: 1020411 failed:
> 0 TOTAL: 1020411
> > > > UnitTestSuite test_fast_queue_iterator PASSED: 152 failed:
> 0 TOTAL: 152
> > > > UnitTestSuite test_fast_ring PASSED: 892 failed:
> 0 TOTAL: 892
> > > > UnitTestSuite test_hash_table_group PASSED: 43 failed:
> 0 TOTAL: 43
> > > > UnitTestSuite test_hash_table PASSED: 37921 failed:
> 0 TOTAL: 37921
> > > > UnitTestSuite test_map_delete PASSED: 400000 failed:
> 0 TOTAL: 400000
> > > > UnitTestSuite test_map PASSED: 2535104 failed:
> 0 TOTAL: 2535104
> > > > UnitTestSuite test_memory_pool PASSED: 60013 failed:
> 0 TOTAL: 60013
> > > > UnitTestSuite test_node PASSED: 14 failed:
> 0 TOTAL: 14
> > > > UnitTestSuite test_packer PASSED: 1 failed:
> 0 TOTAL: 1
> > > > UnitTestSuite test_queue PASSED: 33075 failed:
> 0 TOTAL: 33075
> > > > UnitTestSuite test_red_black_tree PASSED: 900002 failed:
> 0 TOTAL: 900002
> > > > Error File: tests/test_ring.c, Function: main, Line: 18
> > > > UnitTestSuite test_ring PASSED: 256 FAILED:
> 1 TOTAL: 257
> > > > UnitTestSuite test_ring_queue PASSED: 62139 failed:
> 0 TOTAL: 62139
> > > > UnitTestSuite test_set PASSED: 409052 failed:
> 0 TOTAL: 409052
> > > > UnitTestSuite test_statistics PASSED: 1 failed:
> 0 TOTAL: 1
> > > > UnitTestSuite test_string PASSED: 5 failed:
> 0 TOTAL: 5
> > > > UnitTestSuite test_vector PASSED: 706709 failed:
> 0 TOTAL: 706709
> > > > UnitTestSuite PASSED 6165832 / 6165833 FAILED 1 / 6165833
> > > > see unit-tests.junit.xml
> > > > gkt at cs-nuc-wtc:~/Work/biosal$ echo $?
> > > > 0
> > > >
> > > >
> > > > Travis is not parsing the junit output (at least not for C
> projects). I can obviously modify my script on Travis to ensure it does
> fail. But I think we *should* return non-zero when any test fails.
> > > >
> > > > Best,
> > > > George
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > George K. Thiruvathukal, PhD
> > > >
> > > > Professor of Computer Science, Loyola University Chicago
> > > >
> > > > Director, Center for Textual Studies and Digital Humanities
> > > > Guest Faculty, Argonne National Laboratory, Math and Computer
> Science Division
> > > > Editor in Chief, Computing in
> > > > Science and Engineering (IEEE CS/AIP)
> > > >
> > > > (w)
> > >
> > gkt.tv (v)
> > > > 773.829.4872
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Nov 13, 2014 at 3:10 PM, George K. Thiruvathukal
> > > > <gkt at cs.luc.edu> wrote:
> > > >
> > > >
> > > > Thanks for clarifying. I should have looked more closely at the
> project name. :-) Today has been busy since I arrived at the office.
> > > >
> > > >
> > > > I need to read up more on Travis to understand its approach to tests
> in general. I will also ask my colleague, who uses it for more traditional
> xUnit testing projects, too.
> > > >
> > > >
> > > > George
> > > >
> > > >
> > > >
> > > > George K. Thiruvathukal, PhD
> > > >
> > > > Professor of Computer Science, Loyola University Chicago
> > > >
> > > > Director, Center for Textual Studies and Digital Humanities
> > > > Guest Faculty, Argonne National Laboratory, Math and Computer
> Science Division
> > > > Editor in Chief, Computing in
> > > > Science and Engineering (IEEE CS/AIP)
> > > >
> > > > (w)
> > >
> > gkt.tv (v)
> > > > 773.829.4872
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Nov 13, 2014 at 3:05 PM, Boisvert, Sebastien
> > > > <boisvert at anl.gov> wrote:
> > > >
> > > > > From: George K. Thiruvathukal [gkt at cs.luc.edu]
> > > > > Sent: Thursday, November 13, 2014 2:56 PM
> > > > > To: Boisvert, Sebastien
> > > > > Subject: Re: Travis and failure
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Hi there!
> > > > >
> > > > >
> > > > > I think the problem is that the failure is not being reflected as
> a Unix error code (non-zero). Is it possible that your wrapper for the
> JUnit results is not returning a non-zero code when tests fail?
> > > >
> > > > The link I gave to you is for the project "Mean.js".
> > > >
> > > > Also, I can confirm to you that "make tests" always returns 0.
> > > >
> > > > The result is available in *junit.xml. Or you can simply grep for
> "FAILED" in stdout.
> > > >
> > > > No FAILED string means it worked.
> > > >
> > > >
> > > > >
> > > > > George
> > > > >
> > > > >
> > > > >
> > > > > George K. Thiruvathukal, PhD
> > > > >
> > > > > Professor of Computer Science, Loyola University Chicago
> > > > >
> > > > > Director, Center for Textual Studies and Digital Humanities
> > > > > Guest Faculty, Argonne National Laboratory, Math and Computer
> Science Division
> > > > > Editor in Chief, Computing in
> > > > > Science and Engineering (IEEE CS/AIP)
> > > > >
> > > > > (w)
> > > >
> > >
> > gkt.tv (v)
> > > > > 773.829.4872
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Nov 13, 2014 at 2:54 PM, Boisvert, Sebastien
> > > > > <boisvert at anl.gov> wrote:
> > > > >
> > > > > If you look at the following build:
> > > > >
> > > >
> > >
> > https://travis-ci.org/meanjs/mean/jobs/40710659
> > > > >
> > > > > it is written "passed" in green.
> > > > >
> > > > > Yet, if you look closely, the log file says:
> > > > >
> > > > > 13 passing (3s)
> > > > > 1 failing
> > > > >
> > > > > 1) Article Model Unit Tests: Method Save should be able to save
> without problems:
> > > > > Error: timeout of 2000ms exceeded
> > > > > at null.<anonymous>
> (/home/travis/build/meanjs/mean/node_modules/mocha/lib/runnable.js:158:19)
> > > > > at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)
> > > > >
> > > > >
> > > > >
> > > > > I am not sure what is going on there.
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cels.anl.gov/pipermail/biosal/attachments/20141113/de84b530/attachment-0001.html>
More information about the BIOSAL
mailing list