Storyboard and its runtime engine allow project teams to create and run dynamic GUI applications on a variety of embedded systems, from high-end microprocessors (MPUs) to tighter resources microcontrollers (MCUs). To enable the GUIs dynamic nature, the Storyboard Engine dynamically allocates and frees system resources.
On higher-end embedded systems, the ones that have megabytes of RAM and gigabytes of storage, this dynamic use of resources is typically something that does not concern most application developers. However, on lower-end embedded systems, the ones where RAM is measured in kilobytes or just a few megabytes and the storage space is small, careful consideration as to how the runtime engine will dynamically handle these elements becomes more vital.
Knowing which areas of the GUI application that need memory allocated and subsequently freed enables developers to obtain the all too important “big picture” required for GUI development on tight-resourced embedded systems.
Storyboard’s runtime model-view-controller (MVC) approach is a strength and asset for embedded GUI developers. By separating the GUI logic and backend logic into separate, yet interconnected elements, Storyboard enables developers to create a dynamic data structure that is independent of the user interface. This allows the application model to be viewed as a whole and for valuable predictive metrics, guidance, and monitoring to occur, enabling development teams to obtain the complete “big picture”.
Within GUI applications, images are stored using file formats, such as PNG or JPEG, which compress image data to make their storage size smaller. However, before they can be rendered to the screen, they must first be decompressed. This is done using a scratch area that is allocated in heap for systems that are not utilizing a GPU, or texture memory for those that have a GPU. A scratch area is used for decompression to help improve on system performance by removing the need for the image to be decompressed over and over each time it needs to be rendered.
The size of the image, in a heap or texture memory, can be calculated by multiplying the width (w) of the image by the height (h) of the image by the number of bytes that the color value of each pixel takes up (w x h x bytes). So an image that is 100 pixels wide by 50 pixels high that uses 4 bytes to store the color would use the following calculation:
100 x 50 x 4 = 19 KB
While this may not seem like much space, remember that the typical GUI application contains many images that quickly add up.
To assist with reducing the cost of images on memory, Storyboard provides an option to export images in a format that allows them to be drawn directly from storage. The following image shows the resource export configuration editor setup to export images in the DirectRGB8888 format. The comparison between the “Application Footprint Preview” and the “Metrics View” shows that the RAM usage has dropped from 3.11 MB to 14.18 KB.
 |
Storyboard’s Application Footprint Preview provides GUI developers with the memory being used by assets allowing for informed decisions to be made when creating the application.
By drawing images directly from flash, the memory normally required to first decompress the images in the heap or texture memory is removed, saving on memory consumption and speeding up the applications’ startup time. This space and time-saving option is available for virtual filesystem configurations of the Storyboard Engine when exporting a Storyboard built GUI project to a C/C++ header file.
When it comes to animated GIFs, it is important to note that they currently cannot be drawn directly from flash, but rather each frame in the GIF requires system memory to store each frame within. The memory calculation previously defined can be used to determine the memory used by the animated GIF.
Creating dynamic interactions in a Storyboard built application is achieved by adding events to the different GUI elements that are then associated with actions that interact with the application model in various ways. One of the actions that can be associated with events are user generated Lua functions. The use of Lua scripting as an action allows for more advanced dynamic interactions to be built into the application that is controlled via the Lua scripting interface to provide access to the Storyboard runtime engine. Through a library of different Lua-based functions, dynamic interactions that manipulate data can easily be added to the GUI application. However, the addition of Lua script within the GUI application does add to the amount of memory that is required.
Since each application typically uses a different mix of Lua code, to determine the amount of memory (or heap) that Lua is going to require involves running the application. When it is actively running, you can use the following command to uncover which functionality in the Lua code requires memory and report on how much memory Lua is using:
print(collectgarbage("count"))
To reduce the amount of memory required, several steps can be performed within Storyboard.
The first would be to set the gc=1 option in the Lua plugin. This is a command line option that is passed into the Storyboard Engine using the -olua,gc=1 option syntax. It can also be added to the simulation command line, located within the simulator configuration dialog, to use the option while performing a simulation test. The following shows what this would look like when added in the “Extra Engine Options:” text field:
 |
The gc=1 option is added to the Extra Engine Options line associated with the Storyboard Engine command line.
 |
To get to the simulator configuration dialog, simply click on the button that is highlighted in the following image.
The gc=1 option forces the Lua interpreter to perform a garbage collection cycle after each Lua action that is performed. This helps with reducing memory fragmentation, as objects that were just allocated and discarded will be freed immediately rather than remaining in memory for an undetermined amount of time.
Another measure that can help reduce the amount of memory required is to break the Lua code up into modules that are loaded only when required. The creation of directories, under the scripts directory, prevents the Lua interpreter from loading those scripts until the require operand is issued for the script file.
For example, if there was a module for each specific screen that was in the GUI, the following design pattern could be used to load and unload the Lua module. First, the screenshow.pre event to load the Lua module would be used:
function screenshow_pre()
first_screen = require("screens/first_screen")
end
Followed by the screenhide.post event to unload the Lua module:
function screenhide_post()
first_screen = nil
package.loaded["screens/first_screen"] = nil
collectgarbage("collect") --if you aren't using the gc option at engine level
end
The require operand is undone by the line first_screen = nil, and the package.loaded[“screens/first_screen”] = nil will remove the code from the internal Lua table.
For more information on how to structure your Lua code in modules, see the Modularizing Lua knowledge base article.
The last step that could be used to reduce memory is limiting the use Storyboard functions that allocate animations, controls, and timers dynamically. These include the following:
gre.animation_create
gre.clone_control
gre.timer_set_interval
gre.timer_set_timeout
When it comes to GUI development, we recommend allocating these sorts of resources upfront by creating them directly within the Storyboard project rather than creating them in Lua code. Doing so enables these elements to be built at startup, allowing the GUI to quickly reach a steady-state of memory consumption.
Font managers offer GUI developers the ability to incorporate rich typography into their designs that add personality to their GUI applications. However, this can come with a tradeoff in the form of memory being required to support and render it. When it comes to fonts, the Storyboard Engine offers support for two different types of font managers.
The first is called gre-plugin-freetype, which uses the FreeType font engine to open font files and extract the glyph data from them. The FreeType library has an internal cache to speed up the lookup of the glyph data for the FreeType font engine, so using this font manager will cause some use in the heap. The amount of memory that the cache will use depends on the font file(s) used and the size of the font(s) that is being rendered.
Use of the second supported font manager, the gre-plugin-sbfont manager, can help reduce memory usage for font rendering. This is because SBFont manager renders glyphs directly from the storage of the device, similar to drawing images directly from storage. When using the gre-plugin-sbfont manager, each glyph in the font file used will be pre-rendered at the sizes that are required. Please note that this can take up large amounts of storage space, especially if the font size being used is large.
Similar to drawing images directly from storage, the ability for rendering glyphs from storage is supported with the virtual filesystem configuration of the Storyboard Engine when exporting the project to a C/C++ header file.
Canvas is another method of drawing different graphics, such as images, shapes, etc., on the screen to create dynamic user experiences in GUI applications. In Storyboard, the canvas render extension enables custom drawing into a surface allowing the GUI to modify each pixel that resides in a buffer from Lua stored in the heap. This is useful if there is a UI that is needed in the design that cannot be constructed using the other render extensions that the Storyboard Engine provides.
The calculation for the buffer required for canvas rendering is similar to the calculation to determine the buffer required for an image. It is the width (w) of the canvas multiplied by the height (h) of the canvas multiplied by 4, as that is the number of bytes used to store each pixel. The reason that 4 bytes per pixel (ARGB8888) was chosen as the color format is that it allows for an alpha channel for drawing operations that require it.
For example, if the canvas is 200 pixels wide and 250 pixels high, it would require a buffer that uses 195 KB of heap to store the pixels:
200 x 250 x 4 = 195 KB
A canvas render extension can be reused multiple times within a Storyboard application reducing the amount of memory needed by the canvas . The following image illustrates an example of where a canvas, named “canvas1” is used on multiple screens. The memory for the “canvas1” canvas only needs to be allocated once.
 |
However if there are multiple canvas render extensions being used in the UI that have different names, for example “canvas1” and “canvas2”, then each canvas render extensions will require backing memory, as shown in the following illustration.
 |
Therefore, if the application requires the use of a canvas for its drawing operations, it is recommended to use only a single canvas within the application and manage what needs to be drawn into it.
The total cost of dynamic interactions on memory is not complete without looking at frame buffers. A frame buffer is the area of memory where what is to be displayed on the physical screen is kept. Regardless of the rendering technology used, all embedded devices with a physical screen need a place to store the pixels that are to be shown. Therefore, the cost of at least one frame buffer on memory consumption is a constant for embedded GUI app development.
However, if only one frame buffer is used, an effect known as tearing can occur. Tearing happens when the physical display is reading from the single frame buffer at the same time the GUI application is trying to write to it. The following illustration shows what the user experiences when tearing occurs.
 |
In the illustration, the speedometer needle has been torn in two due to using a single frame buffer to read and write.
To avoid the visual effect of tearing, GUI applications can be built to leverage two frame buffers. This way, while the screen is reading from one buffer, the app can be writing to the other. Of course the trade off is that the use of multiple buffers increases the amount of memory consumed.
There are two ways of managing frame buffers. The first is a “back buffer to front buffer” approach where the GUI always writes to the back buffer with the physical display reading from the front buffer. When the GUI is done updating content in the back buffer, it pushes the new content to the front buffer. This approach could cause tearing to occur; however, the chances are much lower, and steps, such as performing the update immediately after a display refresh, can decrease the chances of tearing occurring.
The other approach is to double buffer, where the physical display reads from one buffer while the GUI writes to the other; however, when the GUI is done updating content in its buffer, both the display and GUI app swap the buffers that they were using.
The amount of memory required for a single frame buffer is calculated by multiplying the device’s display width (in pixels) by the height (in pixels) by the number of bytes needed per pixel.
For example, for an embedded device with a display whose resolution is 480 pixels wide and 272 pixels in height and supports a color depth of two bytes per pixel (RGB565), a frame buffer of 255 kilobytes would be required:
480 x 272 x 2 = 255 KB
Keep in mind that if a double buffer or even triple buffer approach is used, this total would then be multiplied by the total amount of frame buffers that are used in the final application.
Animated screen transitions, such as a fade transition or slide transition, are not just flashy, they can help enhance the user experience better than their static counterparts. When it comes to the use of animated screen transitions on lower-end hardware, this is when their use is often discouraged.
When animated screen transitions occur, the current screen and the screen being transitioned to, need to be stored in memory so the transition can happen smoothly. This requires two extra memory allocations with each of those allocations the size of a frame buffer. It should be noted that the default screen transition, which is the screen replace transition, requires no extra memory to replace the current screen with the new screen.
The calculation is similar to the calculation used to determine the display cost, in that it is associated with the resolution (width and height in pixels) of the embedded device’s display multiplied by the number of bytes required to store a pixel. However, where this calculation differs is that you need to account for the multiple screens required for the transition.
For example, a display that has a resolution of 480 pixels in width and 272 pixels in height and a color depth of two bytes per pixel (RGB565) would look like this:
480 x 272 x 2 = 255 KB
To determine the memory cost for screen transitions, the number would be multiplied by two (for the two screens that will be stored in memory), making the total heap requirement for the screen transition 510 KB. On embedded systems that have a small amount of heap available, this could end up taking the majority of it. For example, an embedded system with a megabyte of the heap would have over half of its heap taken just to perform the screen transition.
This doesn’t mean you can’t achieve screen transition effects if the system does not have a lot of memory available; it merely means that a creative approach in GUI development needs to be taken to achieve a similar effect. For example, scrolling layers could be used instead of the screen path transition or alpha values applied to controls to create a fade-out or fade in effect.
Polygons and circles can be useful in adding dynamic rendering content to a GUI application; polygons for rendering data that is shown in trends and circles in showing progress in arched content.
When utilizing Polygons in GUI applications, memory is consumed in two ways.
The first is through the string that stores the points list. The longer the set of points, the more memory they will use. The following illustration shows a comparison between allocating for three points versus ten points.
 |
The second is that each point in the list needs to be converted from the string to a set of renderable coordinates which is also stored in the heap. Therefore, if the GUI application requires the use of a polygon, it is recommended to keep the number of points needed to display the polygon low as possible to save on memory.
There are a couple of ways to reduce the memory needed by polygons. One is to look at the resolution of the area where the polygon is being drawn and to limit the number of points to something that makes sense for that area. For example, if the area where the polygon is being rendered is one hundred pixels wide, yet the polygon data has five hundred x coordinates, then the data set can be reduced to only include every fifth x coordinate because any more than that is not going to be distinguishable in the final rendered output. Another method is to leverage the rectangle and rounded rectangle render extensions found within Storyboard. In particular, if the use of the polygon within the GUI application was to achieve that sort of shape rendering.
Similar to polygons, circles require a buffer to store the points that are going to be needed to draw the circle. When the properties of the circle change, such as its size, or the number of degrees the circle encompasses, the buffer that was associated with it is re-allocated.
Polygons and circles can be useful in adding dynamic rendering content to a GUI application; polygons for rendering data that is shown in trends and circles in showing progress in arched content.
When utilizing Polygons in GUI applications, memory is consumed in two ways.
The first is through the string that stores the points list. The longer the set of points, the more memory they will use. The following illustration shows a comparison between allocating for three points versus ten points.
|
The second is that each point in the list needs to be converted from the string to a set of renderable coordinates which is also stored in the heap. Therefore, if the GUI application requires the use of a polygon, it is recommended to keep the number of points needed to display the polygon low as possible to save on memory.
There are a couple of ways to reduce the memory needed by polygons. One is to look at the resolution of the area where the polygon is being drawn and to limit the number of points to something that makes sense for that area. For example, if the area where the polygon is being rendered is one hundred pixels wide, yet the polygon data has five hundred x coordinates, then the data set can be reduced to only include every fifth x coordinate because any more than that is not going to be distinguishable in the final rendered output. Another method is to leverage the rectangle and rounded rectangle render extensions found within Storyboard. In particular, if the use of the polygon within the GUI application was to achieve that sort of shape rendering.
Similar to polygons, circles require a buffer to store the points that are going to be needed to draw the circle. When the properties of the circle change, such as its size, or the number of degrees the circle encompasses, the buffer that was associated with it is re-allocated.
Cloned controls, tables, and scrolling layers all have a dynamic memory use associated with their implementation.
For example, when the number of elements in a table grows, then memory needs to be allocated to back the newly created elements.
Cloned controls are useful for dynamically laying out UI content. Each cloned control is a dynamically created model element that is created at runtime with the memory that backs the newly created control being allocated out of the heap. So if memory usage is a concern, it is recommended to use pre-laid out layers, or limit the number of cloned controls that are going to be created.
With tables, each element in the table needs to create space to store values for the dynamic aspects associated with the table. These dynamic aspects are the elements that will change from table cell to table cell with the content to be rendered in the table cell potentially requiring memory.
For example, if each cell contains a unique image that is stored in their native format, then each image will require memory in order to be decompressed before being rendered.
To help reduce this, the number of elements in the table can be limited and the data within the elements changed to use Lua or a C callback at runtime.
Finally, when it comes to layers, if they have the scrolling flag set memory will be required to store the information pertaining to the scroll state for the layer. To mitigate the cost of having large sets of controls within a scrolling layer, Storyboard provides an InfiniteList sample to offer GUI developers an example on how this cost can be reduced through the use of a viewport window.
 |
The infinite list sample can be imported into Storyboard Designer by selecting File-> Import, followed by selecting “Storyboard Sample” in the dialog box
 |
Select the InfiniteList entry within the list of samples to add it to a GUI project.