• 0

Hardware Security Modules

Category : security



Hardware Security Module (HSM) is a physical crypto device that is designed to creates, manages and protects in a safe manner crypto keys against tampering.

HSMs are used for different applications cases such as :

  • secure cryptographic key generation
  • secure key storage at least for master keys which other keys are derived
  • key management
  • encryption or digital signature
  • offloading of crypto operations (symetric/asymetric)

HSMs have different form-factor ranging from small USB sticks to server-like devices.

The difference between those devices is speed and storage capacity.

In this article we will show how to setup a HSM (Nitrokey) and how to use it perform a digital signature.


Nitrokey HSM

The Nitrokey HSM is a very affordable USB device:

It supports the following crypto operation:

  • RSA 1024-2048 Bit
  • ECC 192-521 Bit
  • AES 128-256 Bit


Software Installation

On an Ubuntu 18.04 the following packages are required:

$ sudo apt-get install opensc opensc-pkcs11 libengine-pkcs11-openssl

First let’s check if device is connected and available:

dmesg output after inserting the key:

[19264.240419] usb 1-1.3: New USB device found, idVendor=20a0, idProduct=4230
[19264.240426] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[19264.240430] usb 1-1.3: Product: Nitrokey HSM
[19264.240433] usb 1-1.3: Manufacturer: Nitrokey
[19264.240437] usb 1-1.3: SerialNumber: DENK01020420000

Everything is fine we can list the device using pkcs11-tool:

 $ pkcs11-tool --list-slots
Available slots:
Slot 0 (0x0): Nitrokey Nitrokey HSM (DENK01020420000 ) 00 00
token label : UserPIN (SmartCard-HSM)
token manufacturer : www.CardContact.de
token model : PKCS#15 emulated
token flags : login required, rng, token initialized, PIN initialized
hardware version : 24.13
firmware version : 3.1
serial num : DENK0102042
pin min/max : 6/15


PIN Setup

The HSM has factory default settings so needs to be initialized before it can be used in a secure way.


The Security Officer Pin is the admin user pin which can manage all the slots (in our case we have only one slot) 

For the Nitrokey HSM the default SO pin is 3537363231383830

  $ pkcs11-tool --login --login-type so --so-pin 3537363231383830 --change-pin --new-pin 3537363231383830



Each HSM Slot can be initilaized with a User-Pin, we can set it in the following way:

$ sc-hsm-tool --initialize --so-pin 3537363231383830
Using reader with a card: Nitrokey Nitrokey HSM (DENK01020420000 ) 00 00
Enter initial User-PIN (6 - 16 characters) :

*After Initialization all keys, certificates are erased.


Generate a RSA keypair

$ pkcs11-tool --keypairgen --key-type rsa:2048 --id 1 --login --usage-sign --label "testkey"
Using slot 0 with a present token (0x0)
Logging in to "UserPIN (SmartCard-HSM)".
Please enter User PIN:
Key pair generated:
Private Key Object; RSA
label: testkey
ID: 01
Usage: sign, unwrap
Public Key Object; RSA 2048 bits
label: testkey
ID: 01
Usage: verify, wrap

Check the slot:

$ pkcs15-tool -D
Using reader with a card: Nitrokey Nitrokey HSM (DENK01020420000 ) 00 00
PKCS#15 Card [SmartCard-HSM]:
Version : 0
Serial number : DENK0102042
Manufacturer ID: www.CardContact.de
Flags :
Object Flags : [0x3], private, modifiable
Auth ID : 02
ID : 01
Flags : [0x812], local, initialized, exchangeRefData
Length : min_len:6, max_len:15, stored_len:0
Pad char : 0x00
Reference : 129 (0x81)
Type : ascii-numeric
Path : e82b0601040181c31f0201::
Tries left : 3

Object Flags : [0x1], private
ID : 02
Flags : [0x9A], local, unblock-disabled, initialized, soPin
Length : min_len:16, max_len:16, stored_len:0
Pad char : 0x00
Reference : 136 (0x88)
Type : bcd
Path : e82b0601040181c31f0201::
Tries left : 15

Private RSA Key [testkey]
Object Flags : [0x3], private, modifiable
Usage : [0x2C], sign, signRecover, unwrap
Access Flags : [0x1D], sensitive, alwaysSensitive, neverExtract, local
ModLength : 2048
Key ref : 1 (0x1)
Native : yes
Auth ID : 01
ID : 01
MD:guid : 20f72851-1f5c-4436-0350-fbb2627fa1ce

Public RSA Key [testkey]
Object Flags : [0x0]
Usage : [0x51], encrypt, wrap, verify
Access Flags : [0x2], extract
ModLength : 2048
Key ref : 0 (0x0)
Native : no
ID : 01
DirectValue : <present>


OpenSSL Integration

OpenSSL can interact with different HSMs using the standard PKCS#11 API.

For the Nitrokey we can use the OpenSC implementation as follow:

$ cat > openssl.cnf << EOF
# OpenSSL config to use PKCS11 engine

openssl_conf = openssl_engine


pkcs11 = pkcs11_section

engine_id = pkcs11
MODULE_PATH = /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
init = 0

distinguished_name = req_distinguished_name



Now OpenSSL is ready to access the HSM.

To generate a certificate out of the testkey (ID: 1) we created before:

$ OPENSSL_CONF=openssl.cnf openssl req -engine pkcs11 -new -key 1 -keyform engine -out cert.pem -text -x509 -subj "/O=Embetrix/CN=HSM-Test/emailAddress=info@embetrix.com" 

Check the created certificate:

$ openssl x509 -in cert.pem -text
Version: 1 (0x0)
Serial Number:
Signature Algorithm: sha256WithRSAEncryption
Issuer: O = Embetrix, CN = HSM-Test, emailAddress = info@embetrix.com
Not Before: Jul 22 11:54:52 2020 GMT
Not After : Aug 21 11:54:52 2020 GMT
Subject: O = Embetrix, CN = HSM-Test, emailAddress = info@embetrix.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Exponent: 65537 (0x10001)
Signature Algorithm: sha256WithRSAEncryption

Let’s create and sign a file using OpenSSL command line:

$ dd if=/dev/urandom of=DATA bs=1M count=1

We use the testkey (ID: 1) to sign the file:

$ OPENSSL_CONF=openssl.cnf openssl cms -sign -signer cert.pem -engine pkcs11 -inkey 1 -keyform engine -binary -in DATA -outform der -out DATA.sig

Engine interaction with the HSM is also possible using libcrypto C native code, you can check our implementation example:


Finally let’s verify if the signature is correct:

 $ openssl cms -verify -binary -content DATA -in DATA.sig -certfile cert.pem -noverify -inform DER -outform DER -out /dev/null
Verification successful

  • 0

Build a GPS live tracking system

Category : IoT

GPS tracking systems are widely utilized in a multitude of applications such as fleet management, security, personal or merchandise remote monitoring.
In this tutorial we will show how to build and program a mobile GPS Tracking device that is capable of streaming location coordinates to an IoT cloud and enable a real-time map view from any location/device with a simple web-browser.

1. Hardware components

IMG_20170711_134936090 IMG_20170707_163651165 IMG_20170707_163658928_HDR IMG_20170711_191437023

For building the prototype we choose the following off-the-shelf components:

  • Raspberry Pi or any Linux embedded device
  • GPS Module or Dongle using standard NMEA protocol
  • 3G/4G Mobile broadband modem
  • Sim Card with data plan

2. Software components

The full code and build instructions are available on github.

2.1 IoT Cloud

PubNub cloud provides a realtime messaging API for building Mobile, Web, and IoT Applications. It has SDKs supporting a large set of platforms languages and Operating Systems.

To use PubNub, you should first register for an account, create an application and add a new Key set for our GPS data streaming. For more information you can consult the Quick Start guide.

After that you will get a pair of Keys one for publishing data to a channel and other one for subscribing to it.

2.2 Embedded Software

2.2.1 Broadband connection

Follow our previous guide on how to use Ofono to enable cellular modem connection.

2.2.2 GPS interfacing

GPS modules put out typically on the serial interface a series of strings of information called the National Marine Electronics Association (NMEA) protocol.

For our use case all we need is to fetch the position, speed and time. For that parsing the $GPRMC sentences is  enough. As parsing library we use minmea, a lightweight C library:

Parsing is restricted to only RMC sentences:

minmea_sentence_id(line, false) == MINMEA_SENTENCE_RMC

If input line is not empty pack it into a rmc frame :

struct minmea_sentence_rmc frame;
if (minmea_parse_rmc(&frame, line))

Get time, speed (m/s) and coordinates (latitude, longitude) :

minmea_gettime(&ts, &frame.date, &frame.time);

/*convert speed from knot to mps*/
speed = KNT2MPS*minmea_tofloat(&frame.speed);

Those parameters are then packed into a json array :

/*format json string*/
asprintf(&gps_json_string, gps_data, 

gps = json_tokener_parse(gps_json_string);

Here is an output example:

[ { "latlng": [ 47.648102, 18.327868 ], "speed": 20.011890, "time": "2017-07-03 17:36:42" } ]

2.2.3 Location publishing

PubNub C-SDK is used to used to publish the data on the cloud:

The connection initialization is done using the Publish/Subscribe keys:

struct pubnub_sync *sync = pubnub_sync_init();
struct pubnub *pubnb = pubnub_init(PUBKEY, SUBKEY, 
                                   &pubnub_sync_callbacks, sync);

Publishing the json GPS data:


*In our code, we choose to send data only if the target is moving (speed > Threshold)

Finally verify if data was correctly sent:

if (pubnub_sync_last_result(s) != PNR_OK)
    printf("pubnub publish error!\n");

2.3 Live View Web Interface

For tracking the device and display the position in real-time on a map, we will use Mapbox and Javascript powered EON Dashboard.

The full code can be found here.

To start you will need to create a MapBox account. Once this is done, you can either create a new map design or use an existing one for example Mapbox streets.

You will also get a Mapbox authentication token to be used to connect with your account.

The Javascript code will subscribe to the PuBNub corresponding channel, fetch the position and show it on the Map in real-time.

Initialize connection to PubNub:

var pn = new PubNub({
             //replace with your own sub-key
             subscribeKey: 'sub-c-xxxxxxxx-xxxxx-xxxxx-xxxxx-xxxxxxxxxx'            

Define the EON map function by providing the API access token (mbToken), map ID (mbId) and the channel used to get GPS data from:

var map = eon.map({
        pubnub: pn,
        id: 'map',
        mbId: 'mapbox.streets',
        mbToken: 'pk.eyxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
        channels: ['gps-location'],

Function setView is added to set the initial position from the last know value:

message: function (data) {

Zoom option is set a focus on the position:

options: {
    zoomAnimation: true,

Finally a following position marker parses the latlng coordinates and show them on the map:

marker: function (latlng) {
          return (new L.Marker(latlng));


In this tutorial we have implemented a real-time GPS tracking system using a Raspberrypi connected to the Cloud. In future articles we will show how to extend this setup using OBD2 adapter and a Bluetooth connection to stream more data from the car to the cloud.

  • 0

Add mobile broadband connectivity to Embedded Linux

In this article we will show how to extend an IoT Embedded Linux System with a broadband connection using 3G/4G networks.

Poky/Yocto is used as Linux distribution in combination with the mobile telephony application oFono.

As hardware example we have a raspberrypi, nevertheless the same setup was applied and verified on other standard development kits and custom boards.

For the broadband connectivity Huawei E173  3G USB stick is used:

If using a different modem, check first if it’s supported with oFono. Here is a list of supported hardware.

Kernel configuration

In order to support 3G USB modems, the following kernel configuration options
need to be enabled :



If those options are used, you should see the following in dmesg:

usbserial: USB Serial support registered for GSM modem (1-port)  
 GSM modem (1-port) converter detected          
usb 1-1.3: GSM modem (1-port) converter now attached to ttyUSB0  
option 1-1.3:1.1: GSM modem (1-port) converter detected          
usb 1-1.3: GSM modem (1-port) converter now attached to ttyUSB1  
option 1-1.3:1.2: GSM modem (1-port) converter detected          
usb 1-1.3: GSM modem (1-port) converter now attached to ttyUSB2


Yocto recipes

To support oFono, the image recipe should include the following packages:

IMAGE_INSTALL += "ofono ofono-tests"

To allow ofono integration with network manager connman, the following packages can be added:

IMAGE_INSTALL += "connman connman-client"

Connman shall be then enabled with 3g support:

PACKAGECONFIG_append_pn-connman = " 3g"


Ofono setup

oFono provides a mobile telephony (GSM/UMTS) application framework that includes consistent, minimal, and easy to use complete APIs. It offers a high-level D-Bus API for use and integrate with other applications.

The advantage of using oFono is that very simple to configure and you will not have to deal with any kind of AT commands.

Plug in your 3G modem and check if recognized by oFono:

root@raspberrypi:/usr/lib/ofono/test# ./list-modems

[ /huawei_0 ]
    Type = hardware
    Powered = 1                    
    Serial = 860051019861709
    Manufacturer = huawei
    Model = E173
    Emergency = 0
    Online = 0
    Features = sim 
    Lockdown = 0
    Revision =
    Interfaces = org.ofono.SimManager 
    [ org.ofono.SimManager ]
        BarredDialing = 0
        Present = 1
        CardIdentifier = 89492019165001742330
        LockedPins = pin 
        PinRequired = pin
        FixedDialing = 0
        PreferredLanguages = de en 
        Retries = [puk2 = 10] [pin2 = 3] [pin = 3] [puk = 10] 
        SubscriberNumbers =

Enable modem:

root@raspberrypi:/usr/lib/ofono/test# ./enable-modem
Connecting modem /huawei_0...

If SIM card is protected with pin, enter the code:

root@raspberrypi:/usr/lib/ofono/test# ./enter-pin pin 1234
Enter Pin for modem /huawei_0...

Depending on the country and Network provider, APN setting is to be configured:

root@raspberrypi:/usr/lib/ofono/test# ./create-internet-context internet.eplus.de
Found context /huawei_0/context1
Setting APN to internet.eplus.de

Now the modem can be set online and activated:

root@raspberrypi:/usr/lib/ofono/test# ./online-modem


Here is an example script of modem initialization:


#set -x



enable-modem $MODEM

if [ -n "$PIN" ]; then
    enter-pin $MODEM pin $PIN

create-internet-context $APN
online-modem  $MODEM

That’s it! the modem is up and the broadband connection is enabled:

root@raspberrypi:/usr/lib/ofono/test# ifconfig 
eth0      Link encap:Ethernet  HWaddr B8:27:EB:C4:02:82  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback  
          inet addr:  Mask:
          inet6 addr: ::1%694092/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:1120 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1120 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:81280 (79.3 KiB)  TX bytes:81280 (79.3 KiB)

ppp0      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          inet addr:  P-t-P:  Mask:
          RX packets:17 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:1762 (1.7 KiB)  TX bytes:1207 (1.1 KiB)

The cellular connection is also available in connman:

root@raspberrypi:/usr/lib/ofono/test# connmanctl technologies
  Name = Cellular
  Type = cellular
  Powered = True
  Connected = True
  Tethering = False
  Name = Wired
  Type = ethernet
  Powered = True
  Connected = False
  Tethering = False

we can for example enable cellular connection tethering over wifi:

root@raspberrypi:# sysctl -w net.ipv4.ip_forward=1

root@raspberrypi:# connmanctl tether wifi on EmbexuSpot 123456789

If having problem to connect with cellular network, you can use list-modem to check the connection status:

root@raspberrypi:/usr/lib/ofono/test# ./list-modems 
[ /huawei_1 ]
    Type = hardware
    Model = E173
    Powered = 1
    Interfaces = org.ofono.Phonebook org.ofono.ConnectionManager org.ofono.CellBroadcast org.ofono.NetworkRegistration org.ofono.SupplementaryServices org.ofono.CallBarring org.ofono.CallSettings org.ofono.CallF 
    Revision =
    Serial = 860051019861709
    Online = 1
    Features = gprs cbs net ussd sms rat sim 
    Emergency = 0
    Manufacturer = huawei
    Lockdown = 0
    [ org.ofono.Phonebook ]
    [ org.ofono.ConnectionManager ]
        Attached = 1
        Suspended = 0
        Powered = 1
        Bearer = umts
        RoamingAllowed = 0
    [ org.ofono.CellBroadcast ]
        Topics = 
        Powered = 0
    [ org.ofono.NetworkRegistration ]
        LocationAreaCode = 51906
        MobileCountryCode = 262
        CellId = 33184422
        Mode = auto
        Technology = umts
        Status = registered
        MobileNetworkCode = 03
        Name = Blau
        Strength = 41
    [ org.ofono.SupplementaryServices ]
        State = idle
    [ org.ofono.CallBarring ]
        VoiceOutgoing = disabled
        VoiceIncoming = disabled
    [ org.ofono.CallSettings ]
        VoiceCallWaiting = disabled
        HideCallerId = default
        ConnectedLineRestriction = unknown
        CallingNamePresentation = unknown
        CalledLinePresentation = disabled
        ConnectedLinePresentation = unknown
        CallingLineRestriction = off
        CallingLinePresentation = enabled
    [ org.ofono.CallForwarding ]
        VoiceNoReply = 
        VoiceNotReachable = +491793000400
        VoiceUnconditional = 
        ForwardingFlagOnSim = 0
        VoiceNoReplyTimeout = 20
        VoiceBusy = +491793000400
    [ org.ofono.MessageWaiting ]
        VoicemailMessageCount = 0
        VoicemailWaiting = 0
        VoicemailMailboxNumber = +491779911
    [ org.ofono.SmartMessaging ]
    [ org.ofono.PushNotification ]
    [ org.ofono.MessageManager ]
        Alphabet = default
        ServiceCenterAddress = +491770610000
        UseDeliveryReports = 0
        Bearer = cs-preferred
    [ org.ofono.RadioSettings ]
        GsmBand = any
        TechnologyPreference = any
        UmtsBand = any
    [ org.ofono.AudioSettings ]
        Active = 0
    [ org.ofono.VoiceCallManager ]
        EmergencyNumbers = 08 000 999 110 112 911 118 119 
    [ org.ofono.AllowedAccessPoints ]
    [ org.ofono.SimManager ]
        PreferredLanguages = de en fr 
        SubscriberIdentity = 262032735704422
        FixedDialing = 0
        MobileNetworkCode = 03
        MobileCountryCode = 262
        LockedPins = pin 
        CardIdentifier = 894921002875759225
        BarredDialing = 0
        SubscriberNumbers = 
        PinRequired = none
        Present = 1
        Retries = [pin2 = 3] [puk2 = 10] [puk = 10] [pin = 3]





  • 0

Setup a guest Wifi Hotspot using Yocto

In this tutorial we will show how to setup a guest Wifi hotspot and configure it to restrict the internet access to only web surfing.

We will showcase this setup using Poky/Yocto as Linux distribution and connman as network manager. As hardware we will use:

  • Raspberry-Pi
  • Beaglebone-black

For both boards we will use a Realtek RTL8192CU based Wifi USB dongle.


Yocto Layers Setup

First we clone Poky repository:

$ cd Projects
$ git clone -b master git://git.yoctoproject.org/poky

Add the meta-raspberrypi layer:

$ cd poky
$ git clone git://git.yoctoproject.org/meta-raspberrypi

Enable it in bblayers.conf:

$ source oe-init-build-env

$ echo 'BBLAYERS += "~/Projects/poky/meta-raspberrypi"' >> conf/bblayers.conf

Beaglebone Black machine configuration is already contained in Poky so no need for other layers unless you want to build it for other hardware not supported in those layers.


Kernel Configuration

In order to support tethering, the following kernel configuration options
need to be enabled either as modules (m) or builtin (y):


For routing and statistic support , the following options need to be enabled as modules (m) or builtin (y):


Finally the RTL8192Cu driver option need to be enabled as modules (m) or builtin (y):


You can use kernel fragments to set the configurations above.

Also make sure that ip_tables module is autoloaded by setting in kernel recipe or local.conf:

KERNEL_MODULE_AUTOLOAD_append = " ip_tables"


Yocto Recipes

Your image recipe must include connman and connmanctl packages:

require recipes-core/images/core-image-minimal.bb

IMAGE_INSTALL += "connman connman-client iptables"

Rtl8192cu firmware package must be also included:

IMAGE_INSTALL += "linux-firmware-rtl8192cu"

Ready to build an image either for the Bone or the Pi:

$ MACHINE=beaglebone bitbake hotspot-image
$ MACHINE=raspberrypi bitbake hotspot-image

Now we can write the output image to an SD card and start the corresponding board to setup tethering in connman.

Connman Setup

Plug in a Network cable on Ethernet interface and configure NAT(Network Address Translation):

$ sysctl -w net.ipv4.ip_forward=1

Enable Wifi:

$ connmanctl enable wifi

Finally activate tethering for Wifi using EmbexuSpot as SSID and 12345678 as password:

$ connmanctl tether wifi on EmbexuSpot 123456789


Firewall Setup

To restrict guest to browse only internet (No bittorrent, No nasty stuffs) we configure the firewall with the following rules:

# Flush existing tables
$ iptables -F
$ iptables -X

# Drop every connection by default
$ iptables -P INPUT DROP
$ iptables -P OUTPUT DROP
$ iptables -P FORWARD DROP

# Allow dns traffic on tcp/udp
$ iptables -A OUTPUT -p tcp --dport 53  -j ACCEPT
$ iptables -A INPUT  -p tcp --sport 53  -j ACCEPT
$ iptables -A OUTPUT -p udp --dport 53  -j ACCEPT
$ iptables -A INPUT  -p udp --sport 53  -j ACCEPT

# Allow traffic on the loopback interface
$ iptables -A INPUT -i lo -j ACCEPT
$ iptables -A OUTPUT -o lo -j ACCEPT

# Allow related connections
$ iptables -A INPUT  -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
$ iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Allow http traffic
$ iptables -A OUTPUT -p tcp -m tcp --dport 80  -j ACCEPT
$ iptables -A INPUT  -p tcp -m tcp --sport 80  -j ACCEPT

# Allow https traffic
$ iptables -A OUTPUT -p tcp -m tcp --dport 443  -j ACCEPT
$ iptables -A INPUT  -p tcp -m tcp --sport 443  -j ACCEPT

# Allow ping traffic from outside
$ iptables -A INPUT  -p icmp --icmp-type echo-request -j ACCEPT
$ iptables -A OUTPUT -p icmp --icmp-type echo-reply   -j ACCEPT

# Allow ping traffic from inside
$ iptables -A OUTPUT -p icmp --icmp-type echo-request -j ACCEPT
$ iptables -A INPUT  -p icmp --icmp-type echo-reply   -j ACCEPT

This will allow only http/https traffic and drop everything else.

  • 0

Booting Embedded Linux in One Second !

Booting a device as fast as possible is not only a requirement for time critical applications but also an important facet for improving the usability and user experience.
Most of the Embedded Linux distribution are designed to be generic and flexible to support variety of devices and use cases, therefore the boot-time aspect is not an important focus.
Thanks to its modularity and open source nature it is possible to reduce the boot-time and and achieve some spectacular results just using optimization techniques which does not require any considerable engineering effort.
we will cover in this article an ARM based systems and show a practical example of those tweaks applied to boot a Yocto based Linux on a Beaglebone Black in the blink of an eye:

Before starting any optimizations let’s get a closer look at a typical Embedded Linux boot-up sequence on an ARM processor and analyze how time is spent on each stage :

Boot Sequence on Sitara AM335x


Initial measurements

For our example on Beaglebone black the application is an In-vehicle infotainment (IVI) QT5 based connected application and the goal is to reduce the time from Power-On of the device (Cold-Boot) until the application shows up on the display and fully operable by a user.

To measure the time taken by the application to show its availability, we will use grabserial running on a Host (Ubuntu Linux) to measure time-stamps coming from the target on the serial console:

$ grabserial -d "/dev/ttyUSB0" -e 30 -t -m "U-Boot SPL*"

Important to note that grabserial cannot measure from power-on but starts counting time-stamps upon getting the first character on serial console. In the measurement above we set the time base to SPL using -m option.

Our application needed more than 12 seconds to start-up, from special markers present in the serial logs, we can deduce the time spent on each stage:

such time to start an Infotainment system in the car are unacceptable for an impatient end user.


As a recommendation, do not optimize things that reduce the ability to make measurements and hinder implementing further optimizations.

We start then from the last stage of the boot process, by optimizing the user-space and application start-up, then reduce kernel boot-time. Finally optimize the boot-loader(s).

User Space

Init Process:

One obvious optimization is to configure the Start of the critical application as soon as possible, of course after starting dependencies. In our case we use Systemd so we change the default target from multi-user to basic and remove dependencies to other services as follow:

Description=ConnectedCarIVI service

ExecStart=/usr/bin/ConnectedCarIVI -plugin Tslib


As Systemd has an overhead, specially if not running on a multi-core CPU, we can start our application before Systemd initialization by creating a wrapper to init:

#!/bin/busybox sh

echo "-> Start Application..."

#Initialize your time-critical application here !
/usr/bin/ConnectedCarIVI -plugin Tslib &

echo "-> Application started !"

# start real init (systemd/SysVinit)
exec /sbin/init

and instruct the kernel to use it instead of the default /sbin/init, by adding it to kernel command line:  init=/sbin/preinit

A drawback of this Setup is that your application loses some benefits of Systemd such as auto-restart after crash.

If there are many interdependent processes in play, systemd-analyze can be used to inspect those dependencies and reorder their priorities.


In our example, Qt application alone took almost 0,7s to run!

That could be definitely improved by:

  • choosing toolchains and compiler flags wisely, a new gcc build a faster code, compiler flags set with optimization flags: for example -O2 instead of -Os
  • compiling statically if possible. This will remove the overhead of using shared libraries
  • use prelink which reduce the time needed by dynamic linker to perform relocations
  • in case of a Qt QML based application, using QtQuickCompiler allows to compile QML source code into a binary that runs faster



Before running the init process, the Kernel needs first to mount the root Filesystem, therefore size and choice of the Filesystem have impact on startup time.

Filesystem Size

Size matters but in this case a smaller footprint will have less mount time. Here are some tweaks to reduce the footprint of a Yocto based Root Filesystem :

  • remove DISTRO features that are not used in local.conf:
    DISTRO_FEATURES_remove = "bluetooth"
    DISTRO_FEATURES_remove = "3g"
    DISTRO_FEATURES_remove = "opengl"
    DISTRO_FEATURES_remove = "wayland"
    DISTRO_FEATURES_remove = "x11"
    DISTRO_FEATURES_remove = "nfc"
    DISTRO_FEATURES_remove = "nfs"
    DISTRO_FEATURES_remove = "ext2"
  • remove unnecessary packages and dependencies from image recipes
  • finally use a lightweight C-library such as musl instead of default glibc:
    TCLIBC=musl MACHINE=my-machine bitbake my-image


Filesystem Type

Depending on the storage type an appropriate Filesystem can be used:

In case of eMMC/MMC, EXT3 or EXT4 are widely used but they have an overhead in compared to other Filesystems such as SquashFS (Read-only):

In Yocto this could be easily generated by selecting:

IMAGE_FSTYPE += "squashfs"

or if using wic kickstart :

part / --source rootfs --ondisk mmcblk --fstype=squashfs  --label root --size 150M

The kernel cmdline need to include:




This is an important part of the optimization since a big part of our boot process was spent at this stage.

here are few steps we performed to speed-up kernel loading and execution:

  • build everything that is not needed at boot time as a kernel module
  • reduce Kernel configuration to strict minimum drivers and features that the application need, this implies a lot of trial and error
  • remove from device tree redundant devices or set their status to disabled
  • avoid calibration of loop delay by presetting the value to kernel command line lpj=1990656
  • turn off console output by setting quiet option to command line or disabling  completely printk, which also significantly reduces the kernel size
  • benchmark compressed versus non-compressed Kernel, on our board the decompression went faster than loading an uncompressed image



We enabled falcon-mode to bypass u-boot and focused only on optimizing SPL startup: See our Article about how to enable falcon mode

We disabled in SPL all features that are not required for production such as Networking, USB, YModem, Environment, EFI and Filesystems support:


As we disabled Filesystems support to have less overhead, Boot-Rom code is loading SPL from Raw MMC partition using specific offsets.

We aslo avoided slow bus initialization such as I2C, for example in the board file we removed the code responsible for board detection using I2C/EEPROM and hard-coded the board type to beaglebone black:

index 48c139a..18c7942 100644
--- a/board/ti/am335x/board.h
+++ b/board/ti/am335x/board.h
@@ -26,27 +26,27 @@
 static inline int board_is_bone(void)
-       return board_ti_is("A335BONE");
+       return 0;
 static inline int board_is_bone_lt(void)
-       return board_ti_is("A335BNLT");
+       return 1;
 static inline int board_is_bbg1(void)
-       return board_is_bone_lt() && !strncmp(board_ti_get_rev(), "BBG1", 4);
+       return 0;
 static inline int board_is_evm_sk(void)
-       return board_ti_is("A335X_SK");
+       return 0;
 static inline int board_is_idk(void)
-       return !strncmp(board_ti_get_config(), "SKU#02", 6);
+       return 0;
 static inline int board_is_gp_evm(void)
@@ -56,13 +56,12 @@ static inline int board_is_gp_evm(void)
 static inline int board_is_evm_15_or_later(void)
-       return (board_is_gp_evm() &&
-               strncmp("1.5", board_ti_get_rev(), 3) <= 0);
+       return 0;
 static inline int board_is_icev2(void)
-       return board_ti_is("A335_ICE") && !strncmp("2", board_ti_get_rev(), 1);
+       return 0;

All changes made for SPL can be found here.



Last but not least, hardware settings can have an impact on boot time. For example the Boot Rom may lose precious time by trying to fetch software from wrong media if the bootstrap pins configuration is not correctly set .

On our board, we also noticed that boot up from internal eMMC configured in SLC Mode is a bit faster than default MLC mode configuration, and even faster than using a fast SD-Card(Class 10).



we succeeded in reducing the boot time from 12 second to one second with optimizing different components of the software. The startup time could be further shortened but at cost of the system flexibility.

  • 0

Fast Boot Linux with u-Boot Falcon Mode

Falcon mode is a feature in u-Boot that enables fast booting by allowing SPL directly to start Linux kernel and skip completely u-boot loading and initialization.

To understand how Falcon mode works let’s first have a quick look at a typical Linux boot-up sequence on an ARM processor:

Standard Linux Boot Process

1. First stage – Boot ROM

This is the primary program loader residing on a read-only flash memory (ROM) integrated directly into the processor chip.
It contains the very first code which is executed on power-on or reset.
Depending on the configuration of the bootstrap pins or internal fuses it may decide from which media to load and run the next piece of software. In case of a Secure Boot processor it will also verify the code authenticity before its execution.
At this stage, Boot ROM code is not aware about memory type and different interconnected peripherals.
The main goal here is to perform basic peripherals initialization such as PLLs, system clocks setup then find a boot device from which load a bootloader such as u-Boot.

2. Second stage – SPL

A typical u-Boot image is around few hundreds KB size (~300KB) which does not fit inside internal SRAM of most ARM processor. They are typically less than 100KB.
To handle this limitation, u-Boot adopted the SPL (Secondary Program Loader) approach which consists of creating a very small pre-loader that after configuring and initializing peripherals and the main system memory can load the full blown u-Boot.
It shares the same u-Boot’s sources but with a minimal set of code.
So when u-Boot is built for a platform that requires SPL, it generate two binaries : SPL (MLO file) and u-Boot image.

3. Third stage – u-Boot

Das u-Boot aims to offer a flexibel way to load and start the Linux Kernel from a different type of devices, it also provides rich features for a bootloader, such as a command line interface, Shell Scripting, Support of a variety of Filesystems, networking and other options that are very helpful during initial Hardware Bring-Up and development process, but can be bypassed for the production by enabling the Falcon-Mode and save by the way some precious seconds of the boot time !

Falcon Mode

Configure and enable Falcon-Mode

We will use a Beaglebone Black  as hardware example to showcase the setup, booting either from an eMMC or SD Card. Nevertheless the procedure should be almost identical to other ARM based boards supporting the SPL framework.

If Boot Rom Code support it, we recommend to store and boot the SPL from raw partition and by this mean also u-Boot and Linux Kernel to skip the overhead of using a Filesystem. As result the boot is even  faster !

Partition # Name Description Offset range (Bytes) Offset range (Blocks*) Size
1 MBR Master Boot Record 0x000000 – 0x010000 0x0000 – 0x007F 64KB
2 FDT Device Tree + ARGS 0x010000 – 0x040000 0x0080 – 0x01FF 192KB
4 SPL*   SPL 0x040000 – 0x060000 0x0200 – 0x02FF 128KB
5 U-Boot Full Bootloader 0x060000 – 0x0e0000 0x0300 – 0x06FF 512KB
6 U-Boot Env U-Boot environment 0x0e0000 – 0x120000 0x0700 – 0x08FF 256KB
7 Kernel Linux Kernel 0x120000 -0x1000000 0x0900 – 0x28FF 14MB


SPL* offset is the address from which the Boot ROM can fetch bootloader. This address is hard coded in the Boot ROM and specific to processor.

In case of AM335x there are 4 possibilities at 0x0, 0x20000,0x40000, 0x60000 [chapter in the technical refrence manual ].

[1 x Block is 512 Bytes]

We are going to use u-boot  v2017.05-rc3:

$ git clone git://git.denx.de/u-boot.git

$ git checkout v2017.05-rc3

U-boot offset location is defined in the sources by the following config :


Kernel offset location is defined in the sources by the following config :


Environment offset location and size are defined by the following configs:

#define CONFIG_ENV_OFFSET 0x0e0000

#define CONFIG_ENV_SIZE (128 << 10)

Configs above are default in u-Boot apart from environment config which can be set in the board config file as follow:

diff --git a/include/configs/am335x_evm.h b/include/configs/am335x_evm.h
index fc8a08f..c1408e7 100644
--- a/include/configs/am335x_evm.h
+++ b/include/configs/am335x_evm.h
@@ -340,9 +340,8 @@
#elif defined(CONFIG_EMMC_BOOT)
-#define CONFIG_ENV_OFFSET 0x0
+#define CONFIG_ENV_OFFSET 0x0e0000

Make sure that Falcon Mode config is enabled :


Let’s configure now u-boot for the beaglebone black :

$ make ARCH=arm am335x_boneblack_defconfig
  HOSTCC  scripts/basic/fixdep
  HOSTCC  scripts/kconfig/conf.o
  SHIPPED scripts/kconfig/zconf.tab.c
  SHIPPED scripts/kconfig/zconf.lex.c
  SHIPPED scripts/kconfig/zconf.hash.c
  HOSTCC  scripts/kconfig/zconf.tab.o
  HOSTLD  scripts/kconfig/conf
# configuration written to .config

Build it using an arm cross-compiler for example yocto toolchains:

$ make ARCH=arm CROSS_COMPILE=arm-poky-linux-gnueabi-

If everything went well, MLO and u-boot.img files should generated in the top directory.

Now we are ready to write them adding the Kernel and Device Tree to SD card using the address table above:

dd if=am335x-boneblack.dtb of=/dev/mmcblk0 bs=1 seek=65536 (offset 0x010000)
dd if=MLO of=/dev/mmcblk0 bs=1 seek=262144 (offset 0x040000)
dd if=u-boot.img of=/dev/mmcblk0 bs=1 seek=393216 (offset 0x060000)
dd if=uImage of=/dev/mmcblk0 bs=1 seek=1179648 (offset 0x120000)

In case of using Yocto we can easily generate an image to flash on SD Card/eMMC using the following wic kickstart file:

part fdt    --source rawcopy  --sourceparams="file=uImage-am335x-boneblack.dtb" --ondisk mmcblk --no-table --align 64
part spl    --source rawcopy  --sourceparams="file=MLO" --ondisk mmcblk --no-table --align 256
part uboot  --source rawcopy  --sourceparams="file=u-boot.img" --ondisk mmcblk --no-table --align 384
part kernel --source rawcopy  --sourceparams="file=uImage" --ondisk mmcblk --no-table --align 1152

part / --source rootfs --ondisk mmcblk --fstype=ext4 --label root --align 16384

Note that Falcon-Mode supports only uImage Kernel format !

Now let’s start our board using the previous image and configure it to use falcon-mode:

we set first the bootargs:

U-Boot SPL 2017.05-rc3-dirty (May 04 2017 - 22:54:01)
Trying to boot from MMC1

U-Boot 2017.05-rc3-dirty (May 04 2017 - 22:12:24 +0200)

CPU : AM335X-GP rev 2.1
I2C: ready
DRAM: 512 MiB
Net: cpsw, usb_ether
Press SPACE to abort autoboot in 2 seconds
=> setenv args_mmc 'setenv bootargs console=${console}\
 ${optargs} root=/dev/mmcblk0p1 ro rootfstype=${mmcrootfstype}'

overwrite loadfdt and loadimage macros to use raw partitions:

=> setenv loadfdt 'mmc read ${fdtaddr} 80 180'
=> setenv loadimage 'mmc read ${loadaddr} 900 2000'

then change the bootcmd to reflect our changes:

=> setenv bootcmd 'run args_mmc; run loadfdt; run loadimage;\
bootm ${loadaddr} - ${fdtaddr}'

=> saveenv

we run spl export command from u-Boot to prepare for the SPL everything that should be in place as if bootm to be executed:

=> run args_mmc
=> run loadimage

MMC read: dev # 0, block # 2304, count 8192 ... 8192 blocks read: OK
=> run loadfdt

MMC read: dev # 0, block # 128, count 384 ... 384 blocks read: OK

=> spl export fdt ${loadaddr} - ${fdtaddr}
## Booting kernel from Legacy Image at 82000000 ...
 Image Name: Linux-4.4.61-beagleboard.org
 Created: 2017-05-04 10:55:09 UTC
 Image Type: ARM Linux Kernel Image (uncompressed)
 Data Size: 2641896 Bytes = 2.5 MiB
 Load Address: 80008000
 Entry Point: 80008000
 Verifying Checksum ... OK
## Flattened Device Tree blob at 88000000
 Booting using the fdt blob at 0x88000000
 Loading Kernel Image ... OK
 Loading Device Tree to 8ffee000, end 8ffff310 ... OK
subcommand not supported
subcommand not supported
 Loading Device Tree to 8ffd9000, end 8ffed310 ... OK
Argument image is now in RAM: 0x8ffd9000
The spl export command does not persist to media so we have to overwrite
the prepared FDT from RAM offsets 8ffd9000 to 8ffed310 into the FDT partition:
mmc write ${fdtaddr} 80 180

Finally we are ready to switch-on the falcon mode :

=> setenv boot_os 1
=> saveenv
=> reset

On the next boot we see that SPL jumped directly to Linux kernel :

[0.000011 0.000011] 
[0.000206 0.000195] U-Boot SPL 2017.05-rc3-dirty (May 04 2017 - 22:01:09)
[0.093199 0.092993] Trying to boot from MMC1
[1.449622 0.015281] Detected architecture arm.
[1.452503 0.002881] 
[1.452528 0.000025] Welcome to Embexus-Linux 1.0 (Guacamole)!

In other articles we will cover further techniques used to achieve a Faster Linux Boot, stay tuned !

  • 0

Create a custom Linux Distribution using Yocto

Category : Yocto

The Yocto Project is an open source collaboration project that provides templates, tools and methods to help creating a custom Linux-based systems for embedded products regardless of the hardware architecture.

Yocto Project uses Poky as a reference distribution but it can also creates a custom one. The purpose of this article is to show how to create, configure and build an alternative Yocto based embedded Linux Distribution.

Our custom distribution example mydistro extends the basic settings of Poky and uses alternate distro features and configurations such as systemd as init system and ipk as package manager.

A good practice is to isolate the distro configuration into a separate layer meta-mydistro:

First step is to checkout a local copy of the poky project :

$ mkdir -p ~/Projects/mydistro-oe

$ git clone -b master git://git.yoctoproject.org/poky ~/Projects/mydistro-oe

$ cd ~/Projects/mydistro-oe

$ source oe-init-build-env

Let’s create the corresponding distro layer meta-mydistro using yocto-layer tool:

$ yocto-layer create meta-mydistro -o ~/Projects/-oe/meta-mydistro

Please enter the layer priority you'd like to use for the layer: [default: 6]
Would you like to have an example recipe created? (y/n) [default: n] n
Would you like to have an example bbappend file created? (y/n) [default: n] n

New layer created in ~/Projects/mydistro-oe/meta-mydistro

Now we can define our distro settings by creating a configuration file:


# Distro Layer configuration
# include and overwrite default poky distro
include conf/distro/poky.conf
DISTRO = "mydistro"
DISTRO_NAME = "MyDistro-Linux"
SDK_VENDOR = "-mydistro-sdk"
MAINTAINER = "info@mydistro.com"

TARGET_VENDOR = "-mydistro"

# Override these in poky based distros
MYDISTRO_DEFAULT_DISTRO_FEATURES = "bluetooth ext2 usbgadget usbhost wifi xattr nfs zeroconf 3g"
MYDISTRO_DEFAULT_EXTRA_RDEPENDS = "packagegroup-core-boot"
MYDISTRO_DEFAULT_EXTRA_RRECOMMENDS = "kernel-module-af-packet"



PACKAGE_CLASSES = "package_ipk"

# Use systemd as init manager
DISTRO_FEATURES_append = " systemd"
VIRTUAL-RUNTIME_init_manager = "systemd"
VIRTUAL-RUNTIME_initscripts = "systemd-compat-units"

Settings provided in meta-mydistro/conf/distro/mydistro.conf override similar settings that BitBake finds in the conf/local.conf file in the Build Directory.

To enable meta-mydistro layer we need to add it first to the bblayers.conf :

$ mkdir -p meta-mydistro/conf/samples
$ cp conf/bblayers.conf meta-mydistro/conf/samples/bblayers.conf.sample

and fill bblayers.conf.sample with the following:

##OEROOT##/meta \
##OEROOT##/meta-yocto-bsp \
##OEROOT##/meta-poky \
##OEROOT##/meta-yocto-bsp \

Then select mydistro as DISTRO either from bitbake or in local.conf :

$ cp conf/local.conf meta-mydistro/conf/samples/local.conf.sample

Point DISTRO variable in local.conf or local.conf.sample to use mydistro :

DISTRO ?= "mydistro"

Finally we are able to build an image using mydistro as distribution:

$ cd ~/Projects/mydistro-oe
$ TEMPLATECONF=meta-mydistro/conf/samples/ source oe-init-build-env
$ MACHINE=beaglebone bitbake core-image-minimal