• State Not Answered
  • Replies 13 replies
  • Subscribers 27 subscribers
  • Views 596 views
  • Users 0 members are here
Attachments (0)
Nordic Case Info
  • Case ID: 287870
Options

Saving coredump to flash

user115422

I'm trying to enable saving of coredump to flash. I'm running on nrf52840 with NRF SDK 1.6.1.

I have defined the following:

CONFIG_DEBUG_COREDUMP=y
CONFIG_DEBUG_COREDUMP_BACKEND_FLASH_PARTITION=y
CONFIG_DEBUG_COREDUMP_MEMORY_DUMP_MIN=y
I set up a partition using pm_static.yml:
coredump_partition:
  address: 0xff000
  size: 0x1000
  region: flash_primary
and added the entry in the dts file under &flash0:
coredump_partition: partition@ff000 {
  label = "coredump_partition";
  reg = <0x000ff000 DT_SIZE_K(4)>;
};
I then added a crash shell command that invokes a crash using either k_oops(); or __asm__ volatile("udf #0" : : : );. In either case, I verified that a coredump is generated if using LOGGING.
However, when I tried to use the BACKEND_FLASH_PARTITION it first complained about not being able to find an MSPL timeslot. There were no instructions regarding MPSL but I checked and saw that the default settings has 0 timeslots. I set it to 1 timeslot. Then I'm getting the following error when it's trying to erase the flash partition in preparation for coredump. 
<err> flash_sync_mpsl: timeout
<err> coredump: Cannot start coredump!
I did verify that I can successfully erase the coredump using coredump erase
< coredump erase
00> coredump erase
00> Stored coredump erased.
There seems to be some issue with accessing flash during a coredump. There are no examples specific to nrf52 anywhere.
Parents
  • 0

    Hi,

    You only see this issue on samples that uses Bluetooth?

    Could you try setting these configs? (taken from this .config file)

    CONFIG_LOG=y
CONFIG_LOG_MODE_MINIMAL=y
CONFIG_DEBUG_COREDUMP=y
CONFIG_DEBUG_COREDUMP_BACKEND_FLASH_PARTITION=y
CONFIG_MP_NUM_CPUS=1
CONFIG_FLASH=y
CONFIG_DEBUG_COREDUMP_MEMORY_DUMP_MIN=y
CONFIG_DEBUG_COREDUMP_MEMORY_DUMP_LINKER_RAM=n

  • 0 in reply to user15146

    Our application does use Bluetooth. I don't have a sample application without Bluetooth that I can try on our board or even the DK board.

    We already had most of these set. The only difference was LOG_MODE. I updated LOG_MODE to be minimal but that did not make any difference. Still getting:

    00> =====> Crashing system... (this is expected crash)
    00> E: r0/a1: 0x00000003 r1/a2: 0x20013680 r2/a3: 0x00000040
    00> E: r3/a4: 0x00000000 r12/ip: 0x00000000 r14/lr: 0x0002532b
    00> E: xpsr: 0x41000000
    00> E: Faulting instruction address (r15/pc): 0x00042352
    00> E: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 0
    00> E: Current thread: 0x200031b0 (shell_rtt)

    // This is where it's trying to dump the core to flash but is failing
    00> E: timeout
    00> E: Cannot start coredump!

  • 0 in reply to user15146

    The code is ETIMEDOUT 116.

    Interestingly, if I disable Bluetooth:

    CONFIG_BT=n
    CONFIG_BT_PERIPHERAL=n
    CONFIG_BT_CTLR=n
    Then I no longer see the timeout errors. It then complains about invalid parameter. Tracing that I found that because I put the coredump partition at the end of flash:
    coredump_partition:
    address: 0xff000
    size: 0x1000
    region: flash_primary
    coredump_flash_backend_start() adds an offset for the header but does not adjust the size but the same amount. This causes stream_flash_init() to fail with invalid parameter because the start address (which now includes the offset) plus size of partition is past the end of flash. 

    offset = backend_ctx.flash_area->fa_off;
    offset += ROUND_UP(sizeof(struct flash_hdr_t),
    FLASH_WRITE_SIZE);
    After adjusting the size by the offset (backend_ctx.flash_area->fa_size - 0x10,), I do see the coredump being written to flash. However, printing coredump fails because it fails CRC check. I'm currently trying to figure out why CRC in the header doesn't match read CRC.
    In any case, we need to figure out why there is a timeout if BT is enabled. There also needs to be a bug filed to fix the size adjustment due to offset.
  • 0 in reply to user115422

    OK, the CRC error was being caused because my defined partition size of 4k and the shell thread stack size (where I was triggering the crash from) was also 4k. This caused incomplete coredump from being written to flash. In such a case, the CRC is not being calculated correctly, it's always off by 0x86. This probably should be fixed, even if incomplete coredump is saved.

    Once I increased the partition to be 8k, I was able to successfully read it using coredump print.

    So we just need to figure out how to make this work with BT enabled.

  • 0 in reply to user115422
    NRF SDK 1.6.1.
    ST-Kon said:
    coredump_flash_backend_start() adds an offset for the header but does not adjust the size but the same amount

    I see there was a fix here recently(fixed in NCS v1.9.x), see these 2 commits:

    https://github.com/nrfconnect/sdk-zephyr/commit/1dc74d70f265ab753b9ef3f00450426bc2ccf296

    https://github.com/nrfconnect/sdk-zephyr/commit/d5520f2e59b7d6d65bb8f4b6e848df1d46e95836

  • 0 in reply to user15146
    user15146 said:
    Seems like it's stream_flash_init() that returns an error-code. Could you print the error-code, and post it here? E.g. change the line to something like this to print the value:

    I traced the issue down to nrf_flash_sync_exe() in flash_sync_mpsl.c. The error happens when that function tries to obtain the semaphore (with my slight modification to print the status):

    int status = k_sem_take(&_context.timeout_sem, K_MSEC(FLASH_TIMEOUT_MS));
    LOG_ERR("Sem take result: %d (%ld)\n", status, FLASH_TIMEOUT_MS);
    if (status < 0) {
      LOG_ERR("timeout");
      atomic_set(&_context.timeout_occured, 1);
    }
    This results in EAGAIN:
     <err> flash_sync_mpsl: Sem take result: -11 (22963)
    I tried to play with the number of slots. I've tried 0, 1, and 2. All result in the same error.
    CONFIG_MPSL_TIMESLOT_SESSION_COUNT=2
  • 0 in reply to user115422

    After more digging, it appears that the issue is with how nrf uses k_sem_take(). According to Zephyr docs (https://docs.zephyrproject.org/3.0.0/reference/kernel/synchronization/semaphores.html), if k_sem_take is called from ISR, it must use K_NO_WAIT for timeout.

    However, NRF SDK uses K_FOREVER or actual timeout values in various places that are involved in the coredump. Because coredump is happening in the ISR, those timeout values are not valid. An assert gets triggered, or if asserts are disabled, I imagine Zephyr is unhappy, which is why we fail to take the semaphore.

    __ASSERT(((arch_is_in_isr() == false) ||
    K_TIMEOUT_EQ(timeout, K_NO_WAIT)), "");
    Same issue exists with mutexes. nrf_flash_sync_exe() acquires a mutex but mutex lock has this assert
    __ASSERT(!arch_is_in_isr(), "mutexes cannot be used inside ISRs");
Reply
  • 0 in reply to user115422

    After more digging, it appears that the issue is with how nrf uses k_sem_take(). According to Zephyr docs (https://docs.zephyrproject.org/3.0.0/reference/kernel/synchronization/semaphores.html), if k_sem_take is called from ISR, it must use K_NO_WAIT for timeout.

    However, NRF SDK uses K_FOREVER or actual timeout values in various places that are involved in the coredump. Because coredump is happening in the ISR, those timeout values are not valid. An assert gets triggered, or if asserts are disabled, I imagine Zephyr is unhappy, which is why we fail to take the semaphore.

    __ASSERT(((arch_is_in_isr() == false) ||
    K_TIMEOUT_EQ(timeout, K_NO_WAIT)), "");
    Same issue exists with mutexes. nrf_flash_sync_exe() acquires a mutex but mutex lock has this assert
    __ASSERT(!arch_is_in_isr(), "mutexes cannot be used inside ISRs");
Children
Related
Terms of service