Interacting with hardware on the i.MXRT10xx EVK using Storyboard IO


This document explains how to link the Storyboard application task running the ‘front-end’ GUI with the ‘back-end’ FreeRTOS and BSP driver code to accept, handle and display hardware button keypress events as well as controlling an LED via the UI.

It is assumed that Storyboard is already integrated with and running as part of an existing workspace project on the i.MXRT10xx SDK.

Storyboard IO Sample Application Overview

We will use a simple Storyboard application comprising of a splash screen and a main screen.

After a short pause the splash screen transitions to the main screen. The main screen contains an animated toggle switch led_control (ultimately for LED control) and a user_button image control that we will use to display the hardware push button state:

Exporting the Storyboard IO Sample Application

We can export the Storyboard sample application as a Storyboard Embedded Resource Header (C/C++) file named sbengine_model-SplashTransition_480x272.h using the Storyboard Application Export Configuration dialog as follows:

This new header file contains the Storyboard application model and all of the associated assets such as script, fonts and image files within arrays and structures organised as a Storyboard Virtual File System (SBVFS).

Add the sbengine_model-SplashTransition_480x272.h file to the project application source folder and replace the model include within sbengine_task.c to point to our new application.

EVK Hardware setup and configuration 

The application will make use of the USER_LED (D18, GPIO1 Pin 9) on the EVK for a visual output controlled from the GUI front-end as well as the USER_BUTTON (SW8, GPIO5 Pin 00) which will be handled by interrupt to send a key UP and key DOWN.

Before these IO pins can be used the pin mux configuration for the board must be modified to include and configure these pins.  This should be done using the NXP ConfigTools Pins utility and the following pin settings.

 - {pin_num: F14, peripheral: GPIO1, signal: 'gpio_io, 09', pin_signal: GPIO_AD_B0_09, pull_up_down_config: Pull_Up_100K_Ohm, pull_keeper_select: Pull}
- {pin_num: L6, peripheral: GPIO5, signal: 'gpio_io, 00', pin_signal: WAKEUP, direction: INPUT, software_input_on: Enable, pull_keeper_select: Pull}

Once the Pins configuration has been modified the pin_mux.c and pin_mux.h files should be updated automatically within the project with the correct initialization code.

Within board.h we will add the BOARD_InitUserGPIO() prototype for the function which can be called by our code to initialize these IO pins ready for use.

void BOARD_InitUserGPIO(void);

Within board.c we can implement the BOARD_InitUserGPIO() function which initializes these GPIO pins and interrupt sources. Note that the GPIO IRQ handler BOARD_USER_BUTTON_IRQ_HANDLER is configured to be triggered on both edges so that both the press and release states of the button can be handled:

/* initialise the IO pins for the input button and User LED */
void BOARD_InitUserGPIO(void)
/* Define the init structure for the output LED pin*/
    gpio_pin_config_t led_config = {
	kGPIO_DigitalOutput, 0, kGPIO_NoIntmode
    /* Init output LED GPIO. */
    /* initial pin state is high (LED off) */
    /* Define the init structure for the input switch pin */
    gpio_pin_config_t sw_config = {
    kGPIO_DigitalInput, 0,kGPIO_IntRisingOrFallingEdge,
    /* Init input switch GPIO. */
    /* Enable GPIO pin interrupt */

Again the BOARD_USER_BUTTON_IRQ_HANDLER is triggered on both edges so that both the press and release state of the button can be handled.  The IRQ handler can be implemented as follows which captures IO pin level and sets global variables to reflect state:

/* Whether the SW interrupt is triggered */
volatile bool g_KeyInputEvent = false;

/* The user button state */
volatile uint32_t g_btnPressed = 0; // released

* @brief Interrupt service function of switch.
/* Sample the USER BUTTON GPIO state on entry to determine press/release state, pin active low with a pull-up.
* interrupt is falling and rising edge triggered
*  if we were triggered by falling edge ^^\__ = button pressed
*  if we were triggered by rising edge  __/^^ = button released
g_btnPressed = 0;
g_btnPressed = 1;

/* clear the interrupt status */
/* Change state of switch. */
g_KeyInputEvent = true;

/* Add for ARM errata 838869, affects Cortex-M4, Cortex-M4F Store immediate overlapping
exception return operation might vector to incorrect interrupt */
#if defined __CORTEX_M && (__CORTEX_M == 4U)


Creating and configuring the Storyboard Events

The individual  IO events are defined in the Storyboard Model complete with parameters.

Here we have defined an event named controlLED which is of event type Outgoing (from front-end GUI to the back-end) and takes a single (unsigned char) byte as parameter led_state with value 0 or 1.

We trigger this event from the GUI toggle switch control and call the Lua function CBToggleLED() to set the desired LED state and post this ‘controlLED’ event to the back-end in order to control the hardware IO pin accordingly.

local gPower = 0

function CBToggleLED(mapargs)
local data = {}
local eventData = {}
local eventFormat = "1u1 state"

if (gPower == 0) then
gPower = 1
data["topLayerMain.led_power.image"] = "images/power_on.png"
gPower = 0
data["topLayerMain.led_power.image"] = "images/power_off.png"

eventData["state"] = gPower
gre.send_event_data("controlLED",eventFormat,eventData,"Outgoing") -- update LED state


local gButtonState = 0
--- @param gre#context mapargs

function CBHandleHardKey(mapargs)
local data = {}
local eventData = {}

if (mapargs.context_event == "gre.keydown") then
gButtonState = 1
elseif (mapargs.context_event == "gre.keyup") then
gButtonState = 0
elseif (mapargs.context_event == "gre.keyrepeat") then
if(gButtonState == 0) then gButtonState = 1 else gButtonState = 0 end

if ( gButtonState == 0 ) then
data["topLayerMain.user_button.image"] = "images/preset_off.png"
data["topLayerMain.user_button.image"] = "images/preset_on.png"



The application model events, formats, and their C equivalent data structures can be exported automatically from the Designer IDE in the form of a header file.

Select File->Export->Storyboard Development->Storyboard IO Event Header (C/C++) to create a header file splash_transition_events.h that can be added to the application project sources as below:

#define TIMER__TIMER_CALIBRATE_EVENT "timer.timer_calibrate"
#define TIMER__TIMER_SPLASH_EVENT "timer.timer_splash"
#define CONTROLLED_EVENT "controlLED"
#define CONTROLLED_FMT "1u1 led_state"

typedef struct {
uint8_t led_state;
} controlled_event_t;

#define OUTGOINGDATACHANGE_EVENT "outgoingDataChange"
#define OUTGOINGDATACHANGE_FMT "1u1 requestState"

typedef struct {
uint8_t requestState;
} outgoingdatachange_event_t;

#define UPDATEDATA_EVENT "UpdateData"
#define UPDATEDATA_FMT "4u1 myUInt32 4s1 myInt32 1s0 myString"

typedef struct {
uint32_t myUInt32;
int32_t    myInt32;
char       *myString;
} updatedata_event_t;

Configure the IO channel with the runtime engine

In the desktop and OS runtime environment the IO 'channel' is normally communicated on the sbengine command line whereas for RTExec this channel name is passed in as a greio argument to the app launch API call.

In a typical RTExec project the Storyboard task code is located in file sbengine_task.c . To enable the Storyboard runtime engine to access and use a named IO channel, add the name as an argument parameter ultimately via the API call gr_application_create_args().

We have again used the channel name 'Outgoing' as in the above example:

* This is the main runtime task for Storyboard.  It should be possible to call this
* task directly from within your FreeRTOS core application.
sbengine_main_task(void *arg) {
char *args[20];
int n = 0;  

args[n++] = "greio";
args[n++] = "channel=Outgoing";

// If using VFS, the Storyboard application model will be loaded via string data
// which is generally present in an sbengine_model.h file named 'sb_model'. This
// string data would be passed to gr_application_create_args with the GR_APP_LOAD_STRING
// option flag.
// In the case a filesystem is present, then the Storyboard model will be loaded
// from a file on the filesystem. The path to the file should be passed to
// gr_application_create_args with the GR_APP_LOAD_FILE option flag.

run_storyboard_app(sb_model, GR_APP_LOAD_STRING, args, n);

Implementing the ‘back-end’ Storyboard Event Handlers

We can now add these event definitions to the application by including the splash_transition_events.h header and our implementation code into the sbengine_task.c file.

#include #include "splash_transition_events.h"

With FreeRTOS the Storyboard IO API wraps a message queue object to which we can register listeners for each event to receive.

Within the run_storyboard_app() function, we add and register each of the event listeners (by event name) for our inbound events from the GUI process using the API call gr_application_add_event_listener().

In this example we have chosen to use a single, common event listener callback function CrankEventListenerCallback() which first dumps the content of the events to the serial port for debug and then matches the event by name:


static void
run_storyboard_app(const char *bundle, int flags, char * const *options, int option_count) {

app = gr_application_create_args(bundle, flags, options, option_count);
if(!app) {

// Sets the application logging verbosity. For more logging verbosities, refer to gre.h.
gr_application_debug(app, GR_DEBUG_CMD_VERBOSITY, GR_LOG_ACTION );//GR_LOG_ERROR);

// Register our debug console logger callback
gr_register_log_cb(app, &logger_cb, NULL, NULL);

// Add event listener for event 'outgoingDataChange' via channel 'Outgoing'
gr_application_event_listener_t *mEventListener = NULL;
mEventListener = gr_application_add_event_listener(app, OUTGOINGDATACHANGE_EVENT, &CrankEventListenerCallback, NULL);
if (mEventListener == NULL)
PRINTF("failed to add event listener\n");
// Add event listener for event 'controlLED' via channel 'Outgoing'
mEventListener = gr_application_add_event_listener(app, CONTROLLED_EVENT, &CrankEventListenerCallback, NULL);
if (mEventListener == NULL)
PRINTF("failed to add event listener\n");

// Start the application

// Free the application instance and close any resources.


We will now implement the handler function that will be invoked as a callback by Storyboard when one of the event listeners is triggered.

At the top of the sbengine_task.c file add a function prototype for the callback handler:

static void CrankEventListenerCallback(gr_application_t *app, gr_event_t *event, void *arg);

Within the CrankEventListenerCallback() function implementation below the CONTROLLED_EVENT here is handled by turning on or off the GPIO pin for the USER LED based on the state :

static void CrankEventListenerCallback(gr_application_t *app, gr_event_t *event, void *arg)
uint8_t *pData;
PRINTF("Received Event[%s]: %s, %d bytes;", event->name, event->format, event->nbytes);
  pData = (uint8_t *)event->data;

  for( int i=0; i< event->nbytes; i++)
  PRINTF(" 0x%X", *pData++ );

if( strncmp(event->name, CONTROLLED_EVENT, strlen(CONTROLLED_EVENT)) == 0 )
controlled_event_t *pLedEvent = (controlled_event_t *)event->data;
PRINTF("USER LED = %s\n\r", (pLedEvent->led_state)?"ON":"OFF");

    if (pLedEvent->led_state)

outgoingdatachange_event_t *pChangeEvent = (outgoingdatachange_event_t *)event->data;
    PRINTF("<TODO>Handle data change request\n\r");




Completing the GPIO initialization, event handling, and key event processing

In order to bring the ‘back-end’ code together we must first call the BOARD_InitUserGPIO() function we implemented previously in section EVK Hardware setup and configuration to configure the GPIO and interrupt behaviour before use.  We will also add the processing logic to emulate the posting of the keyboard press and release events to the ‘front-end’.

In this example, the hardware button behaviour will be emulating a keyboard ‘ENTER’ key which is subsequently handled in the Storyboard ‘front-end’ application via the Lua function CBHandleHardKey() we defined earlier in section Storyboard IO Sample Application.

A convenient place to insert this code is within the existing sbengine_input_task() function in the sbengine_task.c file.

We add the button state detection and appropriate calls to the Storyboard gr_application_send_event() API alongside the touch screen handling code which is polled every 100ms:

sbengine_input_task(void *arg) {
const int sleep_msec = 100;
greal_timespec_t sleep_time = {
.tv_sec = 0,
.tv_nsec = sleep_msec * 1000000

gr_key_event_t key_event = { 0, GR_KEY_ENTER, 0};

/* initialise GPIO for USER LED and Button */

   /* initialise touch */

  touch_poll_state_t previous_touch_state = {0};
 touch_poll_state_t touch_state;
  bool pressed = false;

  while (1) {
/* GPIO USER BUTTON interrupt key event is both falling and rising edge triggered
    *  We will send an ENTER key emulation
   if( g_KeyInputEvent == true )
   // button pressed (was a high to low transition)
gr_application_send_event(app, NULL, GR_EVENT_KEY_DOWN, GR_EVENT_KEY_FMT, &key_event, sizeof(key_event));
// button released (was a low to high transition)
gr_application_send_event(app, NULL, GR_EVENT_KEY_UP, GR_EVENT_KEY_FMT, &key_event, sizeof(key_event));

g_KeyInputEvent = false;

if (kStatus_Success != BOARD_Touch_Poll(&touch_state))
greal_nanosleep(&sleep_time, NULL);


Once the application has been successfully built by compiling, linking and deploying to the target board the application should demonstrate the following real-time hardware interactivity with a Storyboard application:

  • The state of the EVK hardware USER BUTTON (SW8) when pressed and released can be shown to be displayed on the GUI  ‘user_button’ control, implemented by changing the background image dynamically at runtime.
  • Toggling the ‘led_power’ switch control on the GUI will turn on and off the USER LED (D18) on the back of the EVK using GPIO driver calls.



Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request



Please sign in to leave a comment.