Validator Tests can execute Lua function test points. A test author may use the Lua Test point in many number of ways, unit testing, state validation, end-to-end testing.
Validator Lua test scripts are located in your Storyboard application project under the tests/scripts directory. When Valiator is active, the files at the root of tests/scripts are loaded, in the same fashion as the standard script files are loaded on every initialization. When Validator is not active, the scripts will not be loaded into Storyboard by default.
Adding a Validator Lua test point can be done during test recording or later, from the Test View manually adding the option. During a test recording select New Action.
When the Validator Lua Test Point executes, Validator will search for the provided Lua function name and call the function. Since the function must exist at startup to be called, the Lua functions must be prepared prior to launching the test recording.
Similar to standard Lua function calls, Storyboard will call the function with a gre#context parameter, usually referred to as `mapargs`. The return value of the callback is used by Validator to determine the state of the test result. Validator supports different types of return types.
Table 9. Return Values
Type |
Value |
Result |
---|---|---|
#nil |
nil |
MISSING |
#boolean |
true |
PASS |
#boolean |
false |
FAIL |
#string |
"PASS" |
PASS |
#string |
"FAIL" |
FAIL |
#string |
"ERROR" |
ERROR |
#table |
|
Result is recursive. Parent result becomes that of highest importance: ERROR > FAIL > MISSING > PASS |
The following code snippets show Lua function return statements which output valid Validator Lua test results.
--- Interpreted by Validator as MISSING --@param gre#context mapargs function TestExample01(mapargs) return nil end --- Interpreted by Validator as PASS --@param gre#context mapargs function TestExample02() return true end --- Interpreted by Validator as FAIL --@param gre#context mapargs function TestExample03() return false end --- Interpreted by Validator as PASS --@param gre#context mapargs function TestExample04() return "PASS" end --- Interpreted by Validator as FAIL --@param gre#context mapargs function TestExample05() return "FAIL" end --- Interpreted by Validator as ERROR --@param gre#context mapargs function TestExample06() return "ERROR" end --- Interpreted by Validator as PASS --@param gre#context mapargs function TestExample07() return {name="Test Example 07", result="PASS"} end --- Interpreted by Validator as FAIL --@param gre#context mapargs function TestExample08() return {name="Test Example 08", result="FAIL", msg="This is the reason this test failed..."} end --- Interpreted by Validator as PASS --@param gre#context mapargs function TestExample09() return { {name="Test Example 09-01", result="PASS"}, {name="Test Example 09-02", result="PASS"}, {name="Test Example 09-03", result="PASS"} } end --- Interpreted by Validator as FAIL --@param gre#context mapargs function TestExample10() return { {name="Test Example 10-01", result="PASS"}, {name="Test Example 10-02", result="FAIL", msg="Expected result vs actual result..."}, {name="Test Example 10-03", result="PASS"} } end --- Interpreted by Validator as FAIL --@param gre#context mapargs function TestExample11() return { {name="Test Example 1-01", result="PASS"}, {name="Test Example 11-02", result="FAIL", msg="Expected result vs actual result..."}, { name="Test Example 11-03", result={ {name="Nested Test Example 1", result="PASS"}, {name="Nested Test Example 2", result="PASS"}, { name="Nested Test Example 3", result={ {name="Test ABC", result="PASS"}, {name="Test XYZ", result="FAIL", msg="This is the reason this test failed..."} } }, } } } end
As shown in the image below, Lua test results can be nested for organization.
Validator Lua Test Points can be used with open-source or handwritten test frameworks. Storyboard includes a Thermostat sample which includes Validator tests and an example of a modified open-source (BSD License) Lua unit testing framework called LuaUnit.
The following is a basic example of using the LuaUnit module with Validator.
In tests/scripts/callbacks.lua:
local lu = require("thirdparty.luaunit") lu:setVerbosity(lu.VERBOSITY_VERBOSE) --- @field thirdparty.luaunit#luaunit active_runner = lu.LuaUnit.new() active_runner:setOutputType("VALIDATOR") --@param gre#context mapargs function ValidatorUnitTests(mapargs) local tests = {} table.insert(tests, require("unit_tests.test_suite01")) table.insert(tests, require("unit_tests.test_suite02")) table.insert(tests, require("unit_tests.test_suite03")) return active_runner:runSuiteByInstances(tests, false) end
In tests/scripts/unit_tests/test_suite01.lua:
local lu = require("thirdparty.luaunit") local TestSuite01 = {} function TestSuite01:test01() lu.assertIsTrue(true, "This will pass") end function TestSuite01:test02() lu.assertIsTrue(false, "This will fail") end return {"TestSuite01", TestSuite01}
The Validator test point would be created to invoke ValidatorUnitTests function. The value returned by active_runner:runSuiteByInstances() will be processed by the Validator-Lua plugin and encoded as JSON and a new test artifact will be generated.