Scrunitier offers four different obsevers that producve logs that write to (C++) streams: Those logs are designed so that they could be combined in different ways:
  • The brief session log and the verbose session log both interpret the option "-v". This flag disables the brief session log but it is not consumed (removed from the rag-list). The same flag enables the verbose session log, which also consumes it. So the "-v" flag could be used to switch between a brief an a verbose session output. Only the brief session log objects must be added before the verbose session log object to the test manager to ensure proper "-v" flag handling.
  • A session report could be easily combined with both brief and verbose session log. It summarises the exceptions at the end and produces a short test session statistics. It also prints the final verdict.
  • A second session report that is disabled by default could be combined with a report that is enabled by default. The one that that is enabled by default produces a screen log while the second one produces a log in a file if the "-ri" option is used.
  • An execution tracer could be combined with all kinds of logs. It is disabled by default. If the "-TX" option used, it generates a detailed execution trace log in the specified file. Useful when the session crashes during execution.

Brief Session Log

This test observer outputs - as the name suggests - only a quite brief output that informs only about the progress of a test session. Only one character is printed per test case.

  • In case the test runs without generating any exception, a dot is printed.
  • Otherwise a 'x' is printed.
  • When a single test is skipped, a backslash ("\") is printed.
  • When multiple test are skipped altogether, the number of tests skipped follows the backslash, i.e. "\7". 



.......xxxxxxxxxxxxxxxxxx................xxxxxxxxxxxxx.........
xxxxxxxxxxxxx..x\\3....

Verbose Trace Log

The verbose terminal log logs every test case and every test suite as one line where for each nested test suite the level of indentation is increased. That leads to a indent-tree like representation of the executed tests. First the name of the test is printed and then the status is logged. A line in the verbose session log has the following format: 
[test-type] [test-name]... [verdict]
  • The test type indicates the subtype of test: "-" means an ordinary test case is executed, "+" indicates a test suite. In this case, all following lines are promoted one level. When the suite is finished, the output is demoted one level.
  • The test name tells you what test case or suite is actually running.
  • For test cases, the verdict shows if the test was successful or not. 


+ Complete feature test suite
    - Auto-generated NOP test... OK
    - Created NOP test... OK
    - Full featured NOP test... OK
    - Binder1 test... OK
    - Binder2 test... OK
    - Binder3 test... OK
    - Binder4 test... OK
    - Expected failure in test... FAILED
    - Expected failure in set-up... FAILED
    - Expected failure in tear-down... FAILED
    - Expected failure via binder1... FAILED
    - Expected failure via binder2... FAILED
    - Expected failure via binder3... FAILED
    - Expected failure via binder4... FAILED
    - Expected error in test... FAILED
    - Expected error in set-up... FAILED
    - Expected error in tear-down... FAILED
    - Expected premature tear-down due to failure after failure... FAILED
    - Expected premature tear-down due to error after failure... FAILED
    - Expected premature tear-down due to failure after error... FAILED
    - Expected premature tear-down due to error after error... FAILED
    - Expected error via binder1... FAILED
    - Expected error via binder2... FAILED
    - Expected error via binder3... FAILED
    - Expected error via binder4... FAILED
    + Setup error suite
        + Take 5
            - Created NOP test... OK
            - Created NOP test... OK
            - Created NOP test... OK
            - Created NOP test... OK
            - Created NOP test... OK
        - Full featured NOP test... OK
    + Stress suite
        + Object test suite
            - Member function NOP test... OK
            - Member function condition assertion... OK
            - Member function euqal assertion... OK
            - Member function delta equal assertion... OK
            - Apply1 test... OK
            - Apply2 test... OK
            - Apply3 test... OK
            - Apply4 test... OK
            - Expected member function failure... FAILED
            - Expected member function condition assertion failure... FAILED
            - Expected member function equal assertion failure... FAILED
            - Expected member function delta equal assertion failure... FAILED
            - Expected failure via apply1... FAILED
            - Expected failure via apply2... FAILED
            - Expected failure via apply3... FAILED
            - Expected failure via apply4... FAILED
            - Expected member function error... FAILED
            - Expected error via apply1... FAILED
            - Expected error via apply2... FAILED
            - Expected error via apply3... FAILED
            - Expected error via apply4... FAILED
        + Object test suite
            - Member function NOP test... OK
            - Member function condition assertion... OK
            - Member function euqal assertion... OK
            - Member function delta equal assertion... OK
            - Apply1 test... OK
            - Apply2 test... OK
            - Apply3 test... OK
            - Apply4 test... OK
            - Expected member function failure... FAILED
            - Expected member function condition assertion failure... FAILED
            - Expected member function equal assertion failure... FAILED
            - Expected member function delta equal assertion failure... FAILED
            - Expected failure via apply1... FAILED
            - Expected failure via apply2... FAILED
            - Expected failure via apply3... FAILED
            - Expected failure via apply4... FAILED
            - Expected member function error... FAILED
            - Expected error via apply1... FAILED
            - Expected error via apply2... FAILED
            - Expected error via apply3... FAILED
            - Expected error via apply4... FAILED
    + Skipping test suite
        - Expected failure to trigger skipping tests... FAILED
        ! Created test B0
        ! Embedded test suite (3)
    - Shared test... OK
    - Shared test... OK

Session Report

Session reports generate a summary of the test session outcome. In case the session was successful (no failed or skipped tests), the time information, the number of tests and the final verdict is printed. In case of any problems, the list of exceptions is printed and a more detailed statistics about the number of exceptions and tests encountered is generated. An exception report record has the following format: 

[filename] [line-number] [exception-type] [phase-info]: [originator-name]
                                                        [message]
  • In case the filename and the line number is not known, "-- -" is printed instead. That could happen when the test exception was not generated by an assertion macro or by an test exception macro.
  • The exception type indicates the subtype. "f" means a test failure was generated, "E" indicates a test error.
  • The pase info tells you in what phase of the test execution the failure had happened.
    • "S": during set-up
    • "X": during test execution
    • "T": during tear-down
    • "P": during premature tear-down
    • "?": the exact phase is unknown
    If the phase indicator is prefixed with a "S" it means that this exception was thrown by a test suite.
  • Originator name identifies the test which generated the exception.
  • Message is the description of the exception cause. 


Xsample Error Report

 Started: 2000.08.06 10:55:39
 Stopped: 2000.08.06 10:55:48
Run-Time: 00:00:09.295


         --   - f  X: Expected failure in test:
                      Tag Failure
         --   - f  S: Expected failure in set-up:
                      Tag Failure
         --   - f  T: Expected failure in tear-down:
                      Tag Failure
         --   - f  X: Expected failure via binder1:
                      Tag Failure
         --   - f  X: Expected failure via binder2:
                      Tag Failure
         --   - f  X: Expected failure via binder3:
                      Tag Failure
         --   - f  X: Expected failure via binder4:
                      Tag Failure
         --   - E  X: Expected error in test:
                      Tag error
         --   - E  S: Expected error in set-up:
                      Tag error
         --   - E  T: Expected error in tear-down:
                      Tag error
         --   - f  X: Expected premature tear-down due to failure after failure:
                      Tag Failure
         --   - f  P: Expected premature tear-down due to failure after failure:
                      Tag Failure
         --   - f  X: Expected premature tear-down due to error after failure:
                      Tag Failure
         --   - E  P: Expected premature tear-down due to error after failure:
                      Tag error
         --   - E  X: Expected premature tear-down due to failure after error:
                      Tag error
         --   - f  P: Expected premature tear-down due to failure after error:
                      Tag Failure
         --   - E  X: Expected premature tear-down due to error after error:
                      Tag error
         --   - E  P: Expected premature tear-down due to error after error:
                      Tag error
         --   - E  X: Expected error via binder1:
                      Tag error
         --   - E  X: Expected error via binder2:
                      Tag error
         --   - E  X: Expected error via binder3:
                      Tag error
         --   - E  X: Expected error via binder4:
                      Tag error
         --   - f  X: Expected member function failure:
                      Tag Failure
DemoTest.cc 300 f  X: Expected member function condition assertion failure:
                      Expected failed assertion
DemoTest.cc 307 f  X: Expected member function equal assertion failure:
                      Expected failed equal assertion
DemoTest.cc 315 f  X: Expected member function delta equal assertion failure:
                      Expected failed epsilon equal assertion
         --   - f  X: Expected failure via apply1:
                      Tag Failure
         --   - f  X: Expected failure via apply2:
                      Tag Failure
         --   - f  X: Expected failure via apply3:
                      Tag Failure
         --   - f  X: Expected failure via apply4:
                      Tag Failure
         --   - E  X: Expected member function error:
                      Tag error
         --   - E  X: Expected error via apply1:
                      Tag error
         --   - E  X: Expected error via apply2:
                      Tag error
         --   - E  X: Expected error via apply3:
                      Tag error
         --   - E  X: Expected error via apply4:
                      Tag error
         --   - f  X: Expected member function failure:
                      Tag Failure
DemoTest.cc 300 f  X: Expected member function condition assertion failure:
                      Expected failed assertion
DemoTest.cc 307 f  X: Expected member function equal assertion failure:
                      Expected failed equal assertion
DemoTest.cc 315 f  X: Expected member function delta equal assertion failure:
                      Expected failed epsilon equal assertion
         --   - f  X: Expected failure via apply1:
                      Tag Failure
         --   - f  X: Expected failure via apply2:
                      Tag Failure
         --   - f  X: Expected failure via apply3:
                      Tag Failure
         --   - f  X: Expected failure via apply4:
                      Tag Failure
         --   - E  X: Expected member function error:
                      Tag error
         --   - E  X: Expected error via apply1:
                      Tag error
         --   - E  X: Expected error via apply2:
                      Tag error
         --   - E  X: Expected error via apply3:
                      Tag error
         --   - E  X: Expected error via apply4:
                      Tag error
         --   - f  X: Expected failure to trigger skipping tests:
                      Tag Failure

Exceptions:
  Failures.: 28
  Errors...: 21
Tests:
  Passed...: 31
  Failed...: 45
  Executed.: 76
  Skipped..: 4
  Total....: 80

>>> FAILED <<<

Execution Trace Log

An execution trace log is useful in situations where a test fails so that the whole test driver executable crashes (i.e. because of a segmentation fault). In such cases, having only the a terminal log might not be enough. The execution tracer logs every single step during test execution and immediately flushes the output stream buffer. Because this leads to high I/O load, the execution trace log should only be enabled when necessary. 



SESSION STARTED
TEST SUITE Complete feature test suite
  SETUP
  EXECUTION
TEST CASE  Auto-generated NOP test
  SETUP
  EXECUTION
  TEARDOWN
PASSED 
TEST CASE  Created NOP test
  SETUP
  EXECUTION
  TEARDOWN
PASSED 
TEST CASE  Full featured NOP test
  SETUP
  EXECUTION
  TEARDOWN
PASSED 
TEST CASE  Binder1 test
  SETUP
  EXECUTION
  TEARDOWN
PASSED 
TEST CASE  Binder2 test
  SETUP
  EXECUTION
  TEARDOWN
PASSED 
TEST CASE  Binder3 test
  SETUP
  EXECUTION
  TEARDOWN
PASSED 
TEST CASE  Binder4 test
  SETUP
  EXECUTION
  TEARDOWN
PASSED 
TEST CASE  Expected failure in test
  SETUP
  EXECUTION
    FAILURE DURING EXECUTION: Tag Failure
  PREMATURE_TEARDOWN
FAILED

... <Output skipped> ...

TEST CASE  Expected premature tear-down due to failure after failure
  SETUP
  EXECUTION
    FAILURE DURING EXECUTION: Tag Failure
  PREMATURE_TEARDOWN
    FAILURE IN PREMATURE TEARDOWN: Tag Failure
FAILED
TEST CASE  Expected premature tear-down due to error after failure
  SETUP
  EXECUTION
    FAILURE DURING EXECUTION: Tag Failure
  PREMATURE_TEARDOWN
    ERROR IN PREMATURE TEARDOWN: Tag error
FAILED

... <Output skipped> ...

TEST SUITE Skipping test suite
  SETUP
  EXECUTION
TEST CASE  Expected failure to trigger skipping tests
  SETUP
  EXECUTION
    FAILURE DURING EXECUTION: Tag Failure
  PREMATURE_TEARDOWN
FAILED
SKIP CASE  Created test B0
SKIP SUITE Embedded test suite (3)
  TEARDOWN
PASSED 
TEST CASE  Shared test
  SETUP
  EXECUTION
  TEARDOWN
PASSED 
TEST CASE  Shared test
  SETUP
  EXECUTION
  TEARDOWN
PASSED 
  TEARDOWN
PASSED 
SESSION COMPLETED

FAILURES: 28
ERRORS..: 21
PASSED..: 31
FAILED..: 45
EXECUTED: 76
SKIPPED.: 4
TOTAL...: 80
The output of the execution trace log is designed so that it could be read by humans and easily parsed by other programs.

The Scrunitizer C++ Unit Test Framework
by Bernd Linowski
[Scrunitizer]  [Overview]  [Cookbook]  [Download]  [Index]  [Linosphere]

Page generated: 1 Nov 2000
(C) by Bernd Linowski
scrunitizer@linosphere.de