IV Index and LPN devices that leave mesh

RE: IV Index and LPN devices that leave mesh

user99042 — Thu, 02 Jun 2022 17:43:15 GMT

Hello,

It's worth noting that we took the advice from another thread and set NETWORK_SEQNUM_FLASH_BLOCK_SIZE from 8192 and changed it to 512.

I added in the sequence number/IV index

I'll repeat this with units with a fresh IV index and see if there's any hints there.

Let me know if there's anywhere I could add debug messages to better see the error.

Right now the error coming back is timeout so it might be that the server is rejecting messages from the phone

Here's the nRF Mesh app for Android log while doing it:
test-devzone.nordicsemi.com/.../app-log-only-8049.txt

Here's the server/proxy receiving an 8049 opcode to node 0x0066 (I don't see it here)

test-devzone.nordicsemi.com/.../1680.8049-server_5F00_proxy.txt

RE: IV Index and LPN devices that leave mesh

user8164 — Thu, 02 Jun 2022 13:22:32 GMT

Hi,

[quote user="aross11"]I overrode a client's IV index to be 42 (within). This caused by server to update to IV index 42 as well[/quote]

This would be expected behavior, and is explained by client sharing secure beacon which server receives, server noticing it is behind and therefore performing IVI Recovery.

[quote user="aross11"]however, even the client can't send a message to the server anymore[/quote]

Where does it fail, is it in the sending, or that the server never seem to receive anything? What is the sequence number and IV Index bits of those packets?

Do you have minimal examples for reproducing this on a DK?

Based on your descriptions, it sounds like for instance sequence number not being reset could be an explanation, or if the replay protection is not working properly (not reset) when doing IVI Recovery. From what I recall, IV Index is not fully stored neither in packets sent nor in replay list, rather only the least significant bit is used (since the node knows what IV Indexes are possible for the current IV Index and IV Index update state.) If there's a bug somewhere, that sounds like a possible place.

[quote user="aross11"]Does the nRF Mesh app for iOS have it's own IV index?[/quote]

I do not quite understand what youi mean by that. The nRF Mesh apps (both Android and iOS) follows the Bluetooth mesh specification, which means they should also be able to participate in IV Index Update and IV Index Recovery. I will double check if the phone apps do indeed have IVI Recovery functionality.

You mention in a separate thread (nRF Mesh App stops workings after IV update) that you think this is a sequence number issue. I agree. Can you share the logs (or other information)I from your testing on this?

Regards,
Terje

RE: IV Index and LPN devices that leave mesh

user99042 — Tue, 31 May 2022 15:41:37 GMT

Should I open up a new issue for this?

RE: IV Index and LPN devices that leave mesh

user99042 — Mon, 30 May 2022 17:29:09 GMT

I'm pretty convinced me screwing with the IV index is messing with the sequence number and now nothing gets through anywhere

RE: IV Index and LPN devices that leave mesh

user99042 — Thu, 26 May 2022 18:00:21 GMT

Okay so now I'm getting somewhere.

I overrode a client's IV index to be 42 (within). This caused by server to update to IV index 42 as well, now I can see all of my nodes again (friendship succeeded, I see them all), however, even the client can't send a message to the server anymore. When they're both fresh, no problem, however once the IV update happens, the two are unable to send messages to each other anymore (I have verified with a log output message that the IV index is correct and matches.)

The other funny part is that once I have the IV index "hacked" I can no longer get the TTL information from the app either. That is to say:

-Start with default IV on server, TTL can be loaded in nRF Mesh App
-Provision, things look good, TTL refresh still works
-"Hack" IV to match locked up friends
-Now the server is friends with the locked up clients, however button presses aren't working
-TTL loading doesn't work either
-It seems as though it's stuck in NET_STATE_IV_UPDATE_IN_PROGRESS mode as well

Basically I need a way to hack the IV index in the most expected of ways such that the TTL refresh in the app still works. I also have the client in the same configuration, and again, I can't load the TTL. It almost seems as though an IV index of any non zero value and the app doesn't work with it.

Does the nRF Mesh app for iOS have it's own IV index?

Any suggestions?

RE: IV Index and LPN devices that leave mesh

user99042 — Wed, 25 May 2022 16:26:40 GMT

"I highly doubt there could be any issue there, but if you could sniff the network traffic (e.g. nRF Sniffer) or provision a DK into the network with a mesh applicaiton logging network activity."

I'll try both of these things. I can add another client and output what it sees. Can you tell me what function triggers the secure update so I can add a log output?

"This sounds a bit weird. How are the messages addressed? Are the proper appkeys bound to the models and used for the publication? I agree, if messages go one way this is most likely not an IV Index issue.

For confirmation of my assumptions, the LPN is what you refer to as the client, and the gateway is what you refer to as the server? What mesh models are you using?"

Correct, client = LPN.

In the nRF Mesh app I can refresh TTL, and I've verify they've subscribed to the correct groups and publish to the correct group. I even see it add itself as a friend to the server, however nothing I do seems to make it's way down to it.

I'll try out the nRF Sniffer and see what I can find. My other thought was to override the IV index to something higher and see if I can trigger an IV update in test mode that way.

The model is based off of the generic on off but modified for an 8 byte message.

It's worked pretty well for a while now, it just seems occasionally something really weird happens and the messages get rejected in some fashion.

RE: IV Index and LPN devices that leave mesh

user8164 — Wed, 25 May 2022 16:10:01 GMT

Hi,

[quote user="aross11"]How can I be sure I'm sending out a secure beacon?[/quote]

I highly doubt there could be any issue there, but if you could sniff the network traffic (e.g. nRF Sniffer) or provision a DK into the network with a mesh applicaiton logging network activity.

[quote user="aross11"]Essentially I have a switch that when triggered, the client sends a notification over mesh to the server. This works, I can see the message come in on the server. However, sending messages down to the client are being ignored.[/quote]

This sounds a bit weird. How are the messages addressed? Are the proper appkeys bound to the models and used for the publication? I agree, if messages go one way this is most likely not an IV Index issue.

For confirmation of my assumptions, the LPN is what you refer to as the client, and the gateway is what you refer to as the server? What mesh models are you using?

Regards,
Terje

RE: IV Index and LPN devices that leave mesh

user99042 — Wed, 25 May 2022 12:26:17 GMT

So I actually have some units now having the issue I've seen before.

Essentially I have a switch that when triggered, the client sends a notification over mesh to the server. This works, I can see the message come in on the server. However, sending messages down to the client are being ignored.

How can I be sure I'm sending out a secure beacon? Shouldn't the fact that I can receive a message from the client mean that the IV indexes are good?

The client is sealed and potted so there's not a lot of physical debugging I can do I'm afraid. (I really need to make a DFU backdoor)

RE: IV Index and LPN devices that leave mesh

user8164 — Wed, 25 May 2022 12:24:30 GMT

Hi,

I got some feedback from our mesh team.

[quote user="tesc"]If the LPN is not in a friendship, it will try to initiate one. If there has been IV Index updates, this will fail repeatedly, at which point the LPN should check Secure Network Beacons to see if an IV Index Recovery is needed. Some action may be needed from the application.[/quote]

The IV Index Recovery will be triggered if receiving a secure beacon with an IV Index higher than the current one. Beacons are sent through the network every 10 seconds (on average), which means there is some waiting time, and the node must be in a scanning state in order to receive the beacon. The recovery procedure itself is instantaneous (it updates a few variables in memory.)

It does however depend on a couple of prerequisites: It can only happen after a timeout of 192 hours, and only if the node is not in the mimddle of an IVI Update. In order for the LPN to know that 192 hours has passed, it must have been on for that amount of time since the timer was last reset. Timer status is stored periodically, so the total "on time" might be divided in several sessions.

[quote user="tesc"]Similarly if the LPN is in a friendship, but has not polled through an IV Index Update, some action may be needed from the application.[/quote]

Staying in friendship beyond one IV Index update is not possible, since the maximum PollTimeout corresponds to 96 hours. In other words: In order to keep the friendship, one must have one successful poll at least once every IV Index. I did the calculations wrong when checking max PollTimeout for that previous answer, and found an erroneous max PollTimout that was longer than 96 hours. Correct number is 96 hours max.

Regards,
Terje

RE: IV Index and LPN devices that leave mesh

user8164 — Tue, 24 May 2022 16:02:02 GMT

Hi,

I noticed you had a second set of questions.

The beacon is a collective effort of the network, where at any one place in the network topology one is expected to get, on average, one beacon every ten seconds. I.e. with two nodes, each node would send a beacon every 20 seconds, for a 10 second average. All nodes participate in this, as mandated by spec, and there is no setting controlling it.

[quote user="aross11"]I'm in a situation now where I'm sending out a replacement unit to a client and I'm realizing that there's no real gaurentee that the two will talk now.[/quote]

It must be provisioned into the network, and as such "synced" with the IVI, yes. Does the system have provisioning support at the customer location, or is it "pre-provisioned" (so that provisioning is out-of-scope for the customer?)

You will likely need some action from the application in order to recover the IVI. As mentioned in my previous reply, I will get back to you, hopefully before week-end (i.e. tomorrow.)

Regards,
Terje

RE: IV Index and LPN devices that leave mesh

user8164 — Tue, 24 May 2022 15:56:09 GMT

Hi,

[quote user="aross11"]I am a little hesitant to switch to nRF Connect SDK because of the project development timeline.[/quote]

I have no issues understanding that rationale, especially if far in devlopment (and with devices out there already.) We are keeping some track of key values for the new SDK (memory requirements, power usage, etc.) although I do not currently have LPN power usage numbers at hand.

[quote user="aross11"]I assume the phone app compatibility hasn't changed?[/quote]

Phone app is unchanged, as is the mesh specification, mesh models, etc. In fact nRF Connect SDK has slighlty better model coverage (and is the place where all new functionality will be developed.)

[quote user="aross11"]Do you have any suggestions on how to create a scenario where the IV index is bad so I can see what's going wrong?[/quote]

For testing purposes you could lift the restriction of 192 hours between IV Index Updates, or maybe even hard code some IV Index "jump" to a new value, for instance set back the IVI for a node to emulate it being "behind".

I am yet to hear back from the team, but will check with them and (hopefully) get back to you tomorrow.

Please note Thursday is public holiday here in Norway, and many take Friday off as well for the long weekend.

Regards,
Terje

RE: IV Index and LPN devices that leave mesh

user99042 — Tue, 24 May 2022 14:50:47 GMT

Hello,

I'm really hoping there's a clear why to validate that the secure beacon update is actually being sent.

I called for a client:

    status = net_state_iv_index_set(30, p_prov_data->flags.iv_update);
    //status = net_state_iv_index_set(p_prov_data->iv_index, p_prov_data->flags.iv_update);

And left it at default for the server.

At no point was the server able to communicate to the client, even when trying to set up publishing and subscription in nRF Mesh. (Updating TTL fails)

I'm in a situation now where I'm sending out a replacement unit to a client and I'm realizing that there's no real gaurentee that the two will talk now.

Does the server send out the secure beacon by default or is it only enabled by the setting in the app?

RE: IV Index and LPN devices that leave mesh

user99042 — Thu, 19 May 2022 14:26:35 GMT

Thank you for the detailed response!

First off I am using the nRF 5 Mesh SDK/17 (co-existence). I am a little hesitant to switch to nRF Connect SDK because of the project development timeline. I'll spend time investigating the new SDK and see what the transition would look like. Power consumption is the biggest concern and something I hope having an RTOS running won't be impacted by- are there numbers for LPN current draw? I assume the phone app compatibility hasn't changed? E.g. the iOS nRF Mesh has worked well for us. The Android version has not, but that's a seperate issue.

"If the LPN is not in a friendship, it will try to initiate one. If there has been IV Index updates, this will fail repeatedly, at which point the LPN should check Secure Network Beacons to see if an IV Index Recovery is needed. Some action may be needed from the application. Similarly if the LPN is in a friendship, but has not polled through an IV Index Update, some action may be needed from the application. I have reached out to our mesh development team for further details, and will get back to you when I know more."

I think is this what I'm seeing. I don't have any units displaying the behavior currently. Do you have any suggestions on how to create a scenario where the IV index is bad so I can see what's going wrong? I was thinking of just doing a test where the init value sets the IV index to something that would trigger the secure update process which is where I think things aren't working right.

RE: IV Index and LPN devices that leave mesh

user8164 — Thu, 19 May 2022 12:36:06 GMT

Hi,

First of all, which SDK are you using? For new projects we recommend the nRF Connect SDK, while the older nRF5 SDK for Mesh is in maintenance mode and will not see new development. See our nRF Connect SDK and nRF5 SDK statement. The thread you refer to is about the nRF5 SDK for Mesh, which is the old solution.

Are you currently experiencing issues with nodes losing track of the network, or is it a theoretical exercise? If you have issues in practice, then we can look further into those. For now I will assume it is a theoretical matter.

[quote user="aross11"]If the whole mesh was powered off for up to 6 months at a time, would this pose any issue?[/quote]

If all nodes, including provisioners, relay nodes, everything is off, then there will be no IV Index Updates. The updates are triggered by network activity. Messages sent from a node are tagged with a sequence number, for replay protection, and IVI Update is triggered when a node gets close to depleting sequence numbers. With no activity, there should be no updates triggered. Similarly, if network traffic is low, the updates may be much less frequent than once every 192 hours.

[quote user="aross11"]
What could happen with our device is that it goes out of the mesh for 3 days, comes back, *maybe* talks back and forth with the server (sending stuff like battery level back), and then goes out of the mesh.

A schedule could look like this (and why the 8 days is concerning)

Day 1: Out of the mesh
Day 3: Back on mesh briefly,leaves again
Day 6: Back on mesh briefly,leaves again
--Day 8: IV update happens--
Day 9: Back on mesh but can no longer communciate as IV is now out of date[/quote]

Thanks for elaborating. In order to perform the IV Index Recovery procedure, the node must listen for the Secure Network beacon, which is (on average) transmitted once every 10 seconds. It contains the network ID and the current IV Index. I see now that according to the Bluetooth Mesh Profile specification, in order to follow the current state of the IV Index, the LPN must poll its Friend node at least once every 96 hours (half the frequency of IV Index updates.) This is most likely due to the update procedure comprising of two main steps, each (at least) 96 hours long. If doing so, the node will keep track of the IV Index.

If the LPN is not in a friendship, it will try to initiate one. If there has been IV Index updates, this will fail repeatedly, at which point the LPN should check Secure Network Beacons to see if an IV Index Recovery is needed. Some action may be needed from the application. Similarly if the LPN is in a friendship, but has not polled through an IV Index Update, some action may be needed from the application. I have reached out to our mesh development team for further details, and will get back to you when I know more.

Regards,
Terje

RE: IV Index and LPN devices that leave mesh

user99042 — Wed, 18 May 2022 13:37:17 GMT

"If you experience issues after a week away from the network, then there might be some issues with the IV Index Recovery procedure."

If this event happens what can be done to recover the IV index? Is there a method of manually triggering it? I've noticed some units stop talking all together to us after a couple of weeks separated but some messages do manage to come back through (mainly those sent by the LPN, not the other way). The only fix I've found is unprovisioning and reprovisioning once it starts up the advertisement. This likely won't be acceptable to our users so I'm hoping there's a safe way to send out the secure beacon, or increase the rate it goes out such that the scenario is less likely to happen.

From the link I posted they make a few suggestions about updating some defines.

NETWORK_MIN_IV_RECOVERY_INTERVAL_MINUTES or even set it to zero for debugging purposes so that a node won't wait for timeout to run the IV Index Recovery procedure. (will this prevent said scenario from happening?)

Our units are battery powered and have seasons where they're in use and not in use. If the whole mesh was powered off for up to 6 months at a time, would this pose any issue? I'm thinking we need a "storage mode" that would prevent the IV updates from happening.

I think I need to clarify the scenario a bit better:

"IV Index update can only happen once every 192 hours (8 days). This means you should have no IV Index related issues if away from the network for 3 days."

What could happen with our device is that it goes out of the mesh for 3 days, comes back, *maybe* talks back and forth with the server (sending stuff like battery level back), and then goes out of the mesh.

A schedule could look like this (and why the 8 days is concerning)

Day 1: Out of the mesh
Day 3: Back on mesh briefly,leaves again
Day 6: Back on mesh briefly,leaves again
--Day 8: IV update happens--
Day 9: Back on mesh but can no longer communciate as IV is now out of date

RE: IV Index and LPN devices that leave mesh

user8164 — Wed, 18 May 2022 08:59:23 GMT

Hi,

IV Index update can only happen once every 192 hours (8 days). This means you should have no IV Index related issues if away from the network for 3 days. If the IV Index of the node is lagging too far behind for an IV Index update, the IV Index Recovery procedure will kick in, up to around 40 IV Updates behind. If further behind the node is unable to rejoin the network and must be reprovisioned.

If you experience issues after a week away from the network, then there might be some issues with the IV Index Recovery procedure. After around 10 months you risk enough IV Index updates to have passed for the node to have completely lost the network.

Regards,
Terje

RE: IV Index and LPN devices that leave mesh

user106736 — Mon, 16 May 2022 11:29:31 GMT

Our Mesh experts are out of office until Wednesday due to a national holiday.

We will return to this case later this the week.

Regards,
Sigurd Hellesvik