Validator Lua Test Point

February 20, 2024 15:52

Validator Tests can execute Lua function test points. A test author may use the Lua Test point in many number of ways, unit testing, state validation, end-to-end testing.

Validator Lua test scripts are located in your Storyboard application project under the tests/scripts directory. When Valiator is active, the files at the root of tests/scripts are loaded, in the same fashion as the standard script files are loaded on every initialization. When Validator is not active, the scripts will not be loaded into Storyboard by default.

Adding a Validator Lua test point can be done during test recording or later, from the Test View manually adding the option. During a test recording select New Action.

When the Validator Lua Test Point executes, Validator will search for the provided Lua function name and call the function. Since the function must exist at startup to be called, the Lua functions must be prepared prior to launching the test recording.

Similar to standard Lua function calls, Storyboard will call the function with a gre#context parameter, usually referred to as `mapargs`. The return value of the callback is used by Validator to determine the state of the test result. Validator supports different types of return types.

Table 9. Return Values

Type	Value	Result
#nil	nil	MISSING
#boolean	true	PASS
#boolean	false	FAIL
#string	"PASS"	PASS
#string	"FAIL"	FAIL
#string	"ERROR"	ERROR
#table	name #string result #boolean/#string/#table - a result, matching any elment from this Return Values table. msg #string - A message about the test, can include a reason for test failure	Result is recursive. Parent result becomes that of highest importance: ERROR > FAIL > MISSING > PASS

Example Return Results

The following code snippets show Lua function return statements which output valid Validator Lua test results.

--- Interpreted by Validator as MISSING
--@param gre#context mapargs
function TestExample01(mapargs)
    return nil
end

--- Interpreted by Validator as PASS
--@param gre#context mapargs
function TestExample02()
    return true
end

--- Interpreted by Validator as FAIL
--@param gre#context mapargs
function TestExample03()
    return false
end

--- Interpreted by Validator as PASS
--@param gre#context mapargs
function TestExample04()
    return "PASS"
end

--- Interpreted by Validator as FAIL
--@param gre#context mapargs
function TestExample05()
    return "FAIL"
end

--- Interpreted by Validator as ERROR
--@param gre#context mapargs
function TestExample06()
    return "ERROR"
end

--- Interpreted by Validator as PASS
--@param gre#context mapargs
function TestExample07()
    return {name="Test Example 07", result="PASS"}
end

--- Interpreted by Validator as FAIL
--@param gre#context mapargs
function TestExample08()
    return {name="Test Example 08", result="FAIL", msg="This is the reason this test failed..."}
end

--- Interpreted by Validator as PASS
--@param gre#context mapargs
function TestExample09()
    return {
        {name="Test Example 09-01", result="PASS"},
        {name="Test Example 09-02", result="PASS"},
        {name="Test Example 09-03", result="PASS"}
    }
end

--- Interpreted by Validator as FAIL
--@param gre#context mapargs
function TestExample10()
    return {
        {name="Test Example 10-01", result="PASS"},
        {name="Test Example 10-02", result="FAIL", msg="Expected result vs actual result..."},
        {name="Test Example 10-03", result="PASS"}
    }
end

--- Interpreted by Validator as FAIL
--@param gre#context mapargs
function TestExample11()
    return {
        {name="Test Example 1-01", result="PASS"},
        {name="Test Example 11-02", result="FAIL", msg="Expected result vs actual result..."},
        {
            name="Test Example 11-03", 
            result={
                {name="Nested Test Example 1", result="PASS"},
                {name="Nested Test Example 2", result="PASS"},
                {
                    name="Nested Test Example 3", 
                    result={
                        {name="Test ABC", result="PASS"},
                        {name="Test XYZ", result="FAIL", msg="This is the reason this test failed..."}
                    }
                },
            }
        }
    }
end

As shown in the image below, Lua test results can be nested for organization.

Lua Unit

Validator Lua Test Points can be used with open-source or handwritten test frameworks. Storyboard includes a Thermostat sample which includes Validator tests and an example of a modified open-source (BSD License) Lua unit testing framework called LuaUnit.

The following is a basic example of using the LuaUnit module with Validator.

In tests/scripts/callbacks.lua:

local lu = require("thirdparty.luaunit")

lu:setVerbosity(lu.VERBOSITY_VERBOSE)
--- @field thirdparty.luaunit#luaunit
active_runner = lu.LuaUnit.new()
active_runner:setOutputType("VALIDATOR")

--@param gre#context mapargs
function ValidatorUnitTests(mapargs)
  local tests = {}  
  table.insert(tests, require("unit_tests.test_suite01"))
  table.insert(tests, require("unit_tests.test_suite02"))
  table.insert(tests, require("unit_tests.test_suite03"))
  return active_runner:runSuiteByInstances(tests, false)
end

In tests/scripts/unit_tests/test_suite01.lua:

local lu = require("thirdparty.luaunit")
local TestSuite01 = {}

function TestSuite01:test01()
    lu.assertIsTrue(true, "This will pass")
end

function TestSuite01:test02()    
  lu.assertIsTrue(false, "This will fail")
end

return {"TestSuite01", TestSuite01}

The Validator test point would be created to invoke ValidatorUnitTests function. The value returned by active_runner:runSuiteByInstances() will be processed by the Validator-Lua plugin and encoded as JSON and a new test artifact will be generated.