The most common way for LoRaWAN devices to establish sessions is to use the Over-The-Air Activation, aka "OTAA".  It uses a join-accept handshake to negotiate session keys with the network. It also does reset the session parameters such as the frame counter (FCnt), the data rate configuration (ADR) or the network Id (NetId).

On the other hand, devices have the possibility to use the Activation By Personalization, aka "ABP" for joining the network. At high level, ABP can be seen as a pre-agreed and frozen set of configuration parameters, meaning that there is no need for the initial handshake. However, ABP is not state-less. It is still possible to dynamically change the configure. The TTN console provides an option for reseting the session ABP devices.

This is the description provided by TTN:

Resetting the session context and MAC state will reset the end device to its initial (factory) state. This includes resetting the frame counters and any other persisted MAC setting on the end device. [...]. Activation-by-personalization (ABP) end devices will only reset the MAC state, while preserving up/downlink queues."

There are many fields defined in the MAC state, and here a few of the relevant ones:

There is also a definition for the Session state:

Our focus in the post are on the frame counters (fcnt). The reason is that, when restarting the device with ABP, the device needs to remember the last frame counter. But, is it enough? What about other counters, such as downlink counters, and other MAC parameters?

In the case of OTAA, when a the device restarts from a blank memory, then the session is automatically reset after re-joining the network. This means that the device only need to store in a "permanent storage" the keys and nonce. Other parameters such as the frame counter and session keys can be kept in the "battery powered storage", eg the RTC memory. If the RTC memory gets flushed, (eg if the battery is removed or flat), then the OTAA enabled device can get fresh new paramters by re-joining the network.  But for ABP, this is not the case, and frame counters must be kept in the permanent storage. In practice, this is not an issue, but one may want to limit flash wearing by having to write into the flash each time a packet is sent or received.

Practical Implementation: RadioLib

Let's dig into the RadioLib LoRaWAN implementation to better understand how sessions are restored. The function is LoRaWANNode::setBufferSession(uint8_t* buffer), and located in File "src/protocols/LoRaWAN/LoRaWAN.cpp".

Restoring Sessions

The first part of the code check the integrity of the buffer.

// the Nonces buffer holds a checksum signature - compare this to the signature that is in the session buffer
uint16_t signatureNonces = LoRaWANNode::ntoh<uint16_t>(&this->bufferNonces[NONCES_SIGNATURE]);
uint16_t signatureInSession = LoRaWANNode::ntoh<uint16_t>(&session[SESSION_NONCES_SIGNATURE]);
if(signatureNonces != signatureInSession) {
  RADIOLIB_DEBUG_PROTOCOL_PRINTLN("The Session buffer (%04x) does not match the Nonces buffer (%04x)", signatureInSession, signatureNonces);
  return(RADIOLIB_ERR_SESSION_DISCARDED);
}

Then gets the value of the device Id, Network ID,  sessions keys. This code is actually designed of OTAA, since for ABP, those parameters are all frozen.

// pull all authentication keys from persistent storage
this->devAddr = LoRaWANNode::ntoh<uint32_t>(&session[SESSION_DEV_ADDR]);
memcpy(this->appSKey,     &session[SESSION_APP_SKEY],      RADIOLIB_AES128_BLOCK_SIZE);
memcpy(this->nwkSEncKey,  &session[SESSION_NWK_SENC_KEY],  RADIOLIB_AES128_BLOCK_SIZE);
memcpy(this->fNwkSIntKey, &session[SESSION_FNWK_SINT_KEY], RADIOLIB_AES128_BLOCK_SIZE);
memcpy(this->sNwkSIntKey, &session[SESSION_SNWK_SINT_KEY], RADIOLIB_AES128_BLOCK_SIZE);

The code then restores the various frame counters, as well as network Id and revision (eg LoraWan 1.1 or  1.0.x). In case of ABP, the network Id and revision is frozen, while other parameters are dynamic.

// restore session parameters
this->rev          = ntoh<uint8_t>(&session[SESSION_VERSION]);
RADIOLIB_DEBUG_PROTOCOL_PRINTLN("LoRaWAN session: v1.%d", this->rev);
this->homeNetId    = ntoh(&session[SESSION_HOMENET_ID]);
this->aFCntDown    = ntoh(&session[SESSION_A_FCNT_DOWN]);
this->nFCntDown    = ntoh(&session[SESSION_N_FCNT_DOWN]);
this->confFCntUp   = ntoh(&session[SESSION_CONF_FCNT_UP]);
this->confFCntDown = ntoh(&session[SESSION_CONF_FCNT_DOWN]);
this->adrFCnt      = ntoh(&session[SESSION_ADR_FCNT]);
this->fCntUp       = ntoh(&session[SESSION_FCNT_UP]);

The rest of the code is used to restore the MAC state and parameters:

uint8_t cid; // Command ID
uint8_t cLen = 0; // Command Length
uint8_t cOcts[14] = { 0 }; // Command options buffer

// setup the default channels
if(this->band->bandType == BAND_DYNAMIC) {
  this->selectChannelPlanDyn();
} else { ... }

// for dynamic bands,  additional channels must be restored per-channel
if(this->band->bandType == BAND_DYNAMIC) { ... }

// restore the state - ADR needs special care, other is straight default
cid = MAC_LINK_ADR;
cLen = 14; // special internal ADR command
memcpy(cOcts, &session[SESSION_LINK_ADR], cLen);
(void)execMacCommand(cid, cOcts, cLen);

uint8_t cids[6] = {
  MAC_DUTY_CYCLE,          MAC_RX_PARAM_SETUP, 
  MAC_RX_TIMING_SETUP,     MAC_TX_PARAM_SETUP,
  MAC_ADR_PARAM_SETUP,     MAC_REJOIN_PARAM_SETUP
};
uint16_t locs[6] = {
  SESSION_DUTY_CYCLE,      SESSION_RX_PARAM_SETUP,
  SESSION_RX_TIMING_SETUP, SESSION_TX_PARAM_SETUP,
  SESSION_ADR_PARAM_SETUP, SESSION_REJOIN_PARAM_SETUP
};

for(uint8_t i = 0; i < 6; i++) {
  (void)this->getMacLen(cids[i], &cLen, DOWNLINK);
  memcpy(cOcts, &session[locs[i]], cLen);
  (void)execMacCommand(cids[i], cOcts, cLen);
}

// set the available channels
uint16_t chMask = LoRaWANNode::ntoh<uint32_t>(&session[SESSION_AVAILABLE_CHANNELS]);
this->setAvailableChannels(chMask);

// copy uplink MAC command queue back in place
memcpy(this->fOptsUp, &session[SESSION_MAC_QUEUE], FHDR_FOPTS_MAX_LEN);
memcpy(&this->fOptsUpLen, &session[SESSION_MAC_QUEUE_LEN], 1);

This last part of the code looks a bit barbarian, but what it does in practice is only to call the following MAC commands:

• LINK_ADR (LinkADRReq, 0x3): Sets the data rate, transmit power, repetition rate or channel.
• DUTY_CYCLE (DutyCycleReq, 0x4): Sets the maximum aggregated transmit duty-cycle of a device
• RX_PARAM_SETUP (RXParamSetupReq, 0x5): Sets the reception slots parameters
• RX_TIMING_SETUP ( RXTimingSetupReq, 0x8): Sets the timing of the of the reception slots
• TX_PARAM_SETUP (TxParamSetupReq, 0x9): Sets the maximum allowed dwell time and Max EIRP of end-device, based on local regulations
• ADR_PARAM_SETUP (ADRParamSetupReq, 0xc): Sets the limit and delay parameters defining the ADR back-off algorithm.
• REJOIN_PARAM_SETUP (RejoinParamSetupReq, 0xf): With this command, the network may request the device to periodically send a RejoinReq Type 0 message with a custom periodicity defined as a time or a number of uplinks
• NEW_CHANNEL, DL_CHANNEL: In case of dynamic band configuration (which is the case of EU868/IN865/AS923/KR920 but not US915/AU915). Sets sets the center frequency of the new channel and the range of uplink data rates usable on this channel.

Voila, restoring a session seems simpler now. But, let's also have a looks at the code for initializing a session from scratch. The function is void LoRaWANNode::createSession(uint16_t lwMode, uint8_t initialDr) from the same LoRaWAN.cpp file.

Creating New Sessions

The structure is similar to the restore function. First it takes care of the bands:

this->clearSession();

// setup JoinRequest uplink/downlink frequencies and datarates
if (this->band->bandType == RADIOLIB_LORAWAN_BAND_DYNAMIC) {... }

// on fixed bands, the first OTAA uplink (JoinRequest) is sent on fixed datarate
if (this->band->bandType == RADIOLIB_LORAWAN_BAND_FIXED && lwMode == RADIOLIB_LORAWAN_MODE_OTAA) { ... }
else { ... }

And then executes the necessary MAC commands:

uint8_t cOcts[5]; // 5 = maximum downlink payload length
uint8_t cid = RADIOLIB_LORAWAN_MAC_LINK_ADR;
uint8_t cLen = 1;       // only apply Dr/Tx field
cOcts[0] = (drUp << 4); // set uplink datarate
cOcts[0] |= 0;          // default to max Tx Power
(void)execMacCommand(cid, cOcts, cLen);

cid = RADIOLIB_LORAWAN_MAC_DUTY_CYCLE;
this->getMacLen(cid, &cLen, RADIOLIB_LORAWAN_DOWNLINK);
uint8_t maxDCyclePower = 0;
switch (this->band->dutyCycle)
{
case (3600):  maxDCyclePower = 10; break;
case (36000): maxDCyclePower = 7;  break;
}
cOcts[0] = maxDCyclePower;
(void)execMacCommand(cid, cOcts, cLen);

cid = RADIOLIB_LORAWAN_MAC_RX_PARAM_SETUP;
(void)this->getMacLen(cid, &cLen, RADIOLIB_LORAWAN_DOWNLINK);
cOcts[0] = (RADIOLIB_LORAWAN_RX1_DR_OFFSET << 4);
cOcts[0] |= this->channels[RADIOLIB_LORAWAN_DIR_RX2].dr; // may be set by user, otherwise band's default upon initialization
LoRaWANNode::hton<uint32_t>(&cOcts[1], this->channels[RADIOLIB_LORAWAN_DIR_RX2].freq, 3);
(void)execMacCommand(cid, cOcts, cLen);

cid = RADIOLIB_LORAWAN_MAC_RX_TIMING_SETUP;
(void)this->getMacLen(cid, &cLen, RADIOLIB_LORAWAN_DOWNLINK);
cOcts[0] = (RADIOLIB_LORAWAN_RECEIVE_DELAY_1_MS / 1000);
(void)execMacCommand(cid, cOcts, cLen);

cid = RADIOLIB_LORAWAN_MAC_TX_PARAM_SETUP;
(void)this->getMacLen(cid, &cLen, RADIOLIB_LORAWAN_DOWNLINK);
cOcts[0] = (this->band->dwellTimeDn > 0 ? 1 : 0) << 5;
cOcts[0] |= (this->band->dwellTimeUp > 0 ? 1 : 0) << 4;
uint8_t maxEIRPRaw;
switch (this->band->powerMax)
{
case (12):  maxEIRPRaw = 2; break;
case (14):  maxEIRPRaw = 4; break;
...
}
cOcts[0] |= maxEIRPRaw;
(void)execMacCommand(cid, cOcts, cLen);

cid = RADIOLIB_LORAWAN_MAC_ADR_PARAM_SETUP;
(void)this->getMacLen(cid, &cLen, RADIOLIB_LORAWAN_DOWNLINK);
cOcts[0] = (RADIOLIB_LORAWAN_ADR_ACK_LIMIT_EXP << 4);
cOcts[0] |= RADIOLIB_LORAWAN_ADR_ACK_DELAY_EXP;
(void)execMacCommand(cid, cOcts, cLen);

cid = RADIOLIB_LORAWAN_MAC_REJOIN_PARAM_SETUP;
(void)this->getMacLen(cid, &cLen, RADIOLIB_LORAWAN_DOWNLINK);
cOcts[0] = (RADIOLIB_LORAWAN_REJOIN_MAX_TIME_N << 4);
cOcts[0] |= RADIOLIB_LORAWAN_REJOIN_MAX_COUNT_N;
(void)execMacCommand(cid, cOcts, cLen);

The MAC commands are actually, and unsurprisingly, the same as for the restore function.

ABP Sessions

So, what about ABP sessions? What should actually be restored compared to OTAA. Well, just about every single MAC parameter, since it defines, for example, how much time the gateway has for transmitting a downlink to the device, and on which channel. If the gateway does not talk to the device on the same channels and at the same time, then that won't work. And in this case, only an ADR back-off to the initial setting, or a MAC reset in the TTN console can help to restore the connection. This is probably one of significant drawback from ABP compared to OTAA, since using OTAA, the device can dynamically reset the MAC state just by rejoining the network, while for ABP, the device must wait for the server to ADR back-off.

One question remains, about the session parameters. The above discussion is mainly about MAC parameters, which we now know must be restored according to the session. But what about the frame counters? What happens if there are not restored properly, can the server still receive the frames from the device? The answer is no - and to make it worse, this is a common issue with TTN/ABP, referring to the discussion "ABP device packets in gateway view but not arriving at application". And when looking at if "there is a way to monitor why the network server drops certain frames", then answer is pretty straightforward:

Question: Is there a way to monitor on the LNS why it is rejecting those frames ?

Answer: For ABP the classic is the frame counters. In this situation it’s unlikely to be as they are incrementing. So the refined answer is no, there are no further logs for us to see. If you want to write your own stack, how about doing it against a copy of TTS OS on a local server, then you can poke under the hood as much as you like.

In short, if there is an issue with ABP, you are on your own. And, while using the reset MAC state in the TTN console is "acceptable" for developpement configuration, it is definitely not for a production environment. And as for the "advice" to write one's own TTN stack, that's honestly the worst recommendation.

Of course, one may wonder why this is important, since restoring the full MAC and Session states using the setBufferSession should be enough to restore the frame counters? Well, in actual fact,  that sometime works, but quite often, the devices gets into the "seen on the gateway but not in the app" state. And when this happens, the only solution that works is to reset the MAC state. Definitelty not a suitable solution for production.

Conclusion

It will take a bit more time to find out how to properly restore ABP session without getting in the "seen on the gateway but not in the app" issue, and I will write later about the possible approaches and solutions.

Meanwhile, using OTAA is the correct solution. And on a side topic, one question to investigate is why OTAA only needs to 2 way handshake, while TCP needs 3 handshakes. Something for a yet another future post!