Mesh DFU not terminating after update is complete

Hello there,

Recently I have added and been testing DFU into my existing application. I have conducted a few DFU updates and have noticed a semi-reoccurring issue. I use the nRF SDK for Mesh 5.0.0, and followed the "Configuring and performing DFU" page in the infocenter

I will give some background on my project first: currently the mesh network consists of 2 types of devices (so it is a mixed-device mesh network), and to update the mesh network, I simply just get a nrf52840 DK board with the mesh_DFU example with serial, and use that as the initial point to update all other devices in the network (not sure if this DFU updater can be considered a third device, as it is not provisioned into the bluetooth mesh network, but I did not need to since we use the OpenMesh protocol for DFU updates). 

The Mesh DFU update itself goes well - there are no issues with the update. The problem comes after the update is completed. After the updated devices reset, the devices still are stuck in the DFU state - what I mean is that after they have been updated, they are in a loop where they endlessly relay the firmware for basically forever as far as I can tell. The main application and functionality is completely fine and intact though, its just that they just never stop relaying the firmware even though all appropriate devices in the network have completed their DFU update. 

To hopefully make myself more clear on what the issue is, I have provided 4 RTT logs of a DFU I have performed where this issue happened when I tested DFU in a 4-device mesh network (including the DFU_example board itself). The "updater.log" file is the DFU-example from the SDK that connects via serial to the computer to receive the DFU package so that it can broadcast it to the rest of the mesh network, and it has an application id of 3. The "client.log" file is 1 device in the mesh network with an application id of 1. The two "server.log" files are the devices the devices that actually underwent the update in this test and have an application id of 2. You can see that they successfully update to a newer version, but then get stuck in an endless cycle of relaying the DFU firmware after their update and subsequent reset. For the interest of readability and size, I have removed the log-messages that are repetitive (since I left the DFU update to be conducted for an entire night).

Note: I forgot to turn on outputting RTT-log saving for the client.log until two hours after I begun the DFU process, hence why its beginning timestamp is different. My apologies, but as far as I can tell, there was not anything abnormal with the client device during the DFU process

Fortunately, since the main application functionality is completely unaffected (as expected though), so it is not the end of the world. The issue is that, if I do not manually reset or power off the device for a couple of times to get rid of this "relaying DFU firmware" state, then a subsequent DFU will fail, which is could either be a minor inconvenience or a huge inconvenience depending on the circumstances.Does anyone have any clues to why this happens?

Parents
  • Hey Jeffe! 

    I'm looking into it, but I will have to get back to you.

    Though the DFU is being relayed to make sure that all nodes gets it, and they wouldn't have much of a way of knowing that every node already does. Normally you could limit the TTL if you would like the relaying to stop faster, though the proprietary DFU in our Mesh SDK uses the IETF Trickle algorithm instead, so it might not be that easy.

    Best regards,

    Elfving

Reply
  • Hey Jeffe! 

    I'm looking into it, but I will have to get back to you.

    Though the DFU is being relayed to make sure that all nodes gets it, and they wouldn't have much of a way of knowing that every node already does. Normally you could limit the TTL if you would like the relaying to stop faster, though the proprietary DFU in our Mesh SDK uses the IETF Trickle algorithm instead, so it might not be that easy.

    Best regards,

    Elfving

Children
  • Alright, thank you for looking into this! 

    In the meantime I will conduct more DFU tests to see if I can figure out if there is a way to reliably replicate this issue, since at the moment it only happens sometimes

  • Ok I have done a couple of DFU on a test mesh network (same network as the one in the initial post), and so far I have not been able to replicate it (i did about 7-8 DFU updates with the application). It seems like this issue is not common although the uncertainty is not ideal. 

    I would like to recompile the bootloader so that it can print to RTT, however, I am not sure how to compile it. I have followed the instructions in the infocenter and I am able to build the examples with cmake without issues. However, when attempting to build the bootloader, nothing happens at all (i run the makefile generated by cmake in the mesh bootloader directory, and it did nothing). How do you compile the mesh bootloader?

  • Hi Jeffe,
    My name is Hung. I'm taking over the case from Håkon.

    We are suspecting your issue is similar to this case: https://devzone.nordicsemi.com/f/nordic-q-a/56151/mesh-dfu-relay-problem

    But it should be solved from SDK v4.2 and it's a corner case where there is no application on one of the board (just Softdevice and Bootloader) in your case it's a bit different. For some reason the FWID beaconing has been relayed forever in the network. This should not happen normally. If you observed the issue again please let us know. We are thinking of having to introduce a timeout before starting new relaying at: https://github.com/NordicSemiconductor/nRF5-SDK-for-Mesh/blob/master/examples/dfu/src/main.c#L149


    Regarding your question, the problem with RTT logging for the bootloader is that we have an issue that the RTT logging for application will take over and that's why you don't see any log from the bootloader. We haven't got a clear solution for that. You can have a look at the answer here.


    But if you want to be easier to debug the bootloader you can think of using the Segger Embedded Studio project for bootloader. We provided one here for SDK v4.x: https://devzone.nordicsemi.com/f/nordic-q-a/51534/stepping-through-nrf-mesh-bootloader-code-in-segger-embedded-studio

  • Hi Hung,

    Apologies for the late reply. 

    So far the issue is fairly uncommon and difficult to reproduce. I had no luck reproducing it so far. Let me know if the Mesh Developers decide to introduce a timeout before starting new relaying as I will be interested in such a patch. At the moment, however, I have become busy with other things so I will come back to this issue in the future and investigate at a much later date. 

    The link for the bootloader RTT logging problem is broken (I get "Access Denied")? But anyways it is a bummer that RTT logging does not work for the bootloader. On the other hand, I figured out how to build the bootloader with cmake thanks to your links. 

    Also thank you for the segger embedded studio project file, I will keep a bookmark on that when I come back to this. Thank you for your support!  

  • Hi Jeffe, 
    Thanks for the info. Please keep us updated if you observe the issue again. I suspect that it might a corner case that hard to reproduce. 


    The link to the RTT case was unfortunately a private case. So that's why you couldn't read it. But it's about an issue of RTT memory conflict between application and bootloader. 

Related