MSC exception

RE: MSC exception

user7377 — Thu, 16 Jun 2022 06:28:45 GMT

Hi,

Unfortunately not. The team was looking into it, but have not concluded. I will talk to the team and let you know if they have anything to share, but it does not seem like it.

RE: MSC exception

user85932 — Wed, 15 Jun 2022 01:02:20 GMT

Hi Einar

Is there any good news on this issue so far?

RE: MSC exception

user7377 — Wed, 03 Nov 2021 16:49:20 GMT

Hi,

This has turned out to be a tricky problem. R&D is still looking into it, and it has not been forgotten. Unfortunately I do not have any new information to share yet, though.

RE: MSC exception

user85932 — Wed, 03 Nov 2021 04:25:59 GMT

Hi Einar

Is there any progress on this issue?

RE: MSC exception

user7377 — Thu, 05 Aug 2021 08:42:52 GMT

Hi,

[quote user="greyorbit"]Yes, it seems so[/quote]

Good. Then we see the same.

[quote user="greyorbit"]Do you mean ejecting the drive like the below picture? if yes, I am not sure, because we always unplug the USB directly.[/quote]

Yes, that is what I mean.

RE: MSC exception

user85932 — Thu, 05 Aug 2021 07:27:24 GMT

[quote userid="7377" url="~/f/nordic-q-a/73389/msc-exception/323439#323439"]Corruption only seems to happen after writing to the USB drive from the PC?[/quote]

Yes, it seems so, and it also may happen after formating the mass storage via the f_mkfs function(It seems to make it more likely to happen after formating).

[quote userid="7377" url="~/f/nordic-q-a/73389/msc-exception/323439#323439"]Corruption never seems to happen if properly ejecting the drive before disconnecting it?[/quote]

Do you mean ejecting the drive like the below picture? if yes, I am not sure, because we always unplug the USB directly.

RE: MSC exception

user7377 — Thu, 05 Aug 2021 06:50:18 GMT

Hi,

[quote user="greyorbit"]I still can reproduce this issue with the patch using the below method.[/quote]

I was probably just unlucky when I tested, then. And in any case I did get complaints from Windows, so I agree there is corruption there.

[quote user="greyorbit"]So there is still an unexpected write operation from the PC, whether we can see the corruption depends on if there is data in the address of the unexpected write.[/quote]

Yes, that makes sense.

Just to be clear on a few things: Do you also see that:

Corruption only seems to happen after writing to the USB drive from the PC?
Corruption never seems to happen if properly ejecting the drive before disconnecting it?

(The above findings come after several thousand iterations on an automated test setup)

RE: MSC exception

user85932 — Thu, 05 Aug 2021 04:25:33 GMT

Hi Einar,

[quote userid="7377" url="~/f/nordic-q-a/73389/msc-exception/323045#323045"]but no corruption was seen and no issues logged. (with your patched file I got a lot of "Cannot uninit because QSPI is busy" and "Cannot init because QSPI is busy", but this is addressed by the diff below (see line 438-445).[/quote]

Actually, we have applied this patch in our product code, please see below.

[quote userid="85932" url="~/f/nordic-q-a/73389/msc-exception"]

There are two major modifications.

1.the issue was reported in this link https://devzone.nordicsemi.com/f/nordic-q-a/36654/usbd_msd-disk-initialization-fails-in-usb-unplug-with-sdk15-0

the difference was I added a timeout for the wait_for_idle() function because I found it would hang on this function sometimes.

[/quote]

I still can reproduce this issue with the patch using the below method.

[quote userid="85932" url="~/f/nordic-q-a/73389/msc-exception/320156#320156"]I have found a way to reproduce this issue, I tested it on the below project that I attached before.[/quote]

But need to modify the main.c file, create more folders in the root directory.

--- D:/Project/Nordic SDK/nRF5_SDK_17.0.2_d674dde/examples/peripheral/usbd_msc/main.c
+++ D:/Project/Nordic SDK/nRF5_SDK_17.0.2_d674dde/examples/peripheral/usbd_msc/main.c
@@ -505 +505 @@ const char *dir_list[] = {
-    "Activi", "Totals", /*"Tracks", "Workout",*/
+    "Activi", "Totals", "Tracks", "Work", "Sport", "sys"

So there is still an unexpected write operation from the PC, whether we can see the corruption depends on if there is data in the address of the unexpected write.

RE: MSC exception

user7377 — Tue, 03 Aug 2021 10:09:48 GMT

Hi Jason,

I am sorry for the late reply. I have been OoO.

I see the same as you. I have only been able to reproduce when following these instructions exactly, writing and not ejecting before removing the USB cable. If I eject first no error happens. Also, I did not do extensive testing today, but I did not see corruptions with the attached patch for nrf_block_dev_qspi.c instead of using yours. There were still errors from Windows though, so apparently there was an issue, but no corruption was seen and no issues logged. (with your patched file I got a lot of "Cannot uninit because QSPI is busy" and "Cannot init because QSPI is busy", but this is addressed by the diff below (see line 438-445).

test-devzone.nordicsemi.com/.../nrf_5F00_block_5F00_dev_5F00_qspi.c.diff

I did extensive testing before the summer with several thousands of write cycles, where the computer wrote a file, ejected, disconnected USB, reconnected and verified, etc. This did not fail using the nrf_block_dev_qspi.c from the diff in this post. So it seem to me like the only way this can be reproduced after applying currently known fixes is by writing from a computer and not ejecting before removing the drive.

RE: MSC exception

user85932 — Thu, 15 Jul 2021 07:11:40 GMT

Hi Einar,

I have found a way to reproduce this issue, I tested it on the below project that I attached before.

[quote userid="85932" url="~/f/nordic-q-a/73389/msc-exception/306456#306456"]

Please replace the nrf_block_dev_qspi.c of the SDK, I use RTT for logging output and test on the nRF5_SDK_17.0.2.

2626.Project.zip

[/quote]

1. press button 3 to format the mass storage

2. press button 1 to create an ACTIVI folder and TOTALS folder

3. connect the USB to the PC, and create a new file as below

3.disconnect the USB

4.connect the USB to the PC again, and then the MS-Windows will popup a message as below

5.click this window and then popup another window as below

6.click "Scan and fix", popup the next window

7.click "Repair drive", and wait for repairs completely.

8.create a new file again and then repeat step 3 through step 8

9.Repeat it one or several times, you will find the file system was corrupted. The Totals folder change to a file.

RE: MSC exception

user7377 — Mon, 05 Jul 2021 15:23:31 GMT

That is good, thank you. That will be very useful input for the team looking into this.

RE: MSC exception

user85932 — Mon, 05 Jul 2021 05:41:30 GMT

[quote userid="7377" url="~/f/nordic-q-a/73389/msc-exception/318403#318403"]

We are wondering if it is possible to rule out the MSC class as responsible for the issue or not. In your test, where you sometimes manage to get the corrupt data, we wonder if you can do some adjustments in order to narrow it down:

prepare/modify a FW that reproduces the issue
connect the DK to the PC via the JLink USB port
flash the FW
let the FW run until the issue occurs - so basically FW needs to have some way of validation when the corruption happens. Never connect the nRF52840 USBD port to the PC so MSC class never drives the block_dev.

Is that possible?

[/quote]

Yes, that is possible.

RE: MSC exception

user7377 — Fri, 02 Jul 2021 10:40:47 GMT

prepare/modify a FW that reproduces the issue
connect the DK to the PC via the JLink USB port
flash the FW
let the FW run until the issue occurs - so basically FW needs to have some way of validation when the corruption happens. Never connect the nRF52840 USBD port to the PC so MSC class never drives the block_dev.

Is that possible?

RE: MSC exception

user7377 — Tue, 29 Jun 2021 12:05:34 GMT

I see. The team is continuously looking into this and we have some ideas. I will update this thread when we have something worth sharing.

RE: MSC exception

user85932 — Tue, 29 Jun 2021 08:25:51 GMT

Hi Einar,

[quote userid="7377" url="~/f/nordic-q-a/73389/msc-exception/314629#314629"]We have not got to the bottom of this yet, but we have a fix that may be relevant. This is a change in app_usbd_msc.c:[/quote]

Unfortunately, I have applied the patch, but it could still happen.

RE: MSC exception

user7377 — Thu, 10 Jun 2021 13:17:52 GMT

I do not immediately see that it would be needed, but it should not hurt either provided the timeout is not too short.

RE: MSC exception

user85932 — Thu, 10 Jun 2021 10:04:39 GMT

Hi Einar,

Thanks for the update.

Do I need to add a timeout for the while loop?

I have encountered many times that the device hangs in the while loop of the USB module.

In my current code, I added a timeout for the while loop.

--- nrf_block_dev_qspi.c-rev7574.svn001.tmp.c	 
+++ nrf_block_dev_qspi.c-rev7575.svn001.tmp.c
@@ -289,2 +289,2 @@ static void qspi_handler(nrf_drv_qspi_evt_t event,
-
-static void wait_for_idle(nrf_block_dev_qspi_t const * p_qspi_dev)
+#include "rtc.h"
+static ret_code_t wait_for_idle(nrf_block_dev_qspi_t const * p_qspi_dev)
@@ -291,0 +292,2 @@ static void qspi_handler(nrf_drv_qspi_evt_t event,
+	time_t startTick, currentTick;
+	startTick =  RTC_getTimeTick();
@@ -294,0 +297 @@ static void qspi_handler(nrf_drv_qspi_evt_t event,
+		/*The watchdog timer stop in WFI mode, it will be wakeup by any interrupt*/
@@ -295,0 +299,6 @@ static void qspi_handler(nrf_drv_qspi_evt_t event,
+		currentTick =  RTC_getTimeTick();
+		/*wait for 3 seconds*/
+		if(currentTick > (startTick + 3))
+		{
+			return NRF_ERROR_TIMEOUT;
+		}
@@ -296,0 +306 @@ static void qspi_handler(nrf_drv_qspi_evt_t event,
+	return NRF_SUCCESS;

--- diskio_blkdev.c-rev7578.svn001.tmp.c	
+++ diskio_blkdev.c-rev7579.svn001.tmp.c	
@@ -43 +43,4 @@
-
+#define DISKIO_MSC_LOG
+#ifdef DISKIO_MSC_LOG
+#include "log.h"
+#endif
@@ -82 +85,2 @@ static void block_dev_handler(struct nrf_block_dev
-static void default_wait_func(void)
+#include "rtc.h"
+static INT default_wait_func(BYTE drv)
@@ -84 +88,14 @@ static void block_dev_handler(struct nrf_block_dev
-    __WFE();
+	time_t startTick, currentTick;
+	startTick =  RTC_getTimeTick();
+    while (m_drives[drv].busy)
+    {
+		/*The watchdog timer stop in WFI mode, it will be wakeup by any interrupt*/
+        __WFI();
+		currentTick =  RTC_getTimeTick();
+		/*wait for 5 seconds*/
+		if(currentTick > (startTick + 5))
+		{
+			return 1;
+		}
+    }
+	return 0;

RE: MSC exception

user7377 — Thu, 10 Jun 2021 08:39:55 GMT

Hi Jason,

We have not got to the bottom of this yet, but we have a fix that may be relevant. This is a change in app_usbd_msc.c:

test-devzone.nordicsemi.com/.../app_5F00_usbd_5F00_msc.c.diff

RE: MSC exception

user85932 — Thu, 27 May 2021 10:14:23 GMT

Hi,

[quote userid="7377" url="~/f/nordic-q-a/73389/msc-exception/312018#312018"]I was not able to reproduce the issue with warnings on Windows on my side[/quote]

I test on the Windows 10 Pro. I can also reproduce this problem in another product that is not developed by us, and it also uses NRF52840.

RE: MSC exception

user7377 — Thu, 27 May 2021 09:48:56 GMT

Hi,

[quote user="greyorbit"]What I understand so far, this issue likely occurs while connecting to the PC and copying files to the MSC from the PC side. Our products can be upgraded by copying the firmware to MSC. I have asked our test team to check the MSC before and after the upgrade, it's still being tested.[/quote]

It will be interesting to learn the result of that test.

[quote user="greyorbit"]Have you fixed these problems? They can be reproduced on the Nordic DK. I think if these are fixed, this issue will probably be fixed.[/quote]

I was not able to reproduce the issue with warnings on Windows on my side, but I managed once to get a corrupt file while coping new files, disconnect, reconnect, copy new file, etc. I agree it seems reasonable that the same root cause can be the cause of all this though we cannot say for sure at this point.

I will let you know when the team investigating this have any progress, if they have any questions and when they have some code we would like you to test.

RE: MSC exception

user85932 — Wed, 26 May 2021 08:15:16 GMT

Hi Einar,

Thanks for the update.

[quote userid="7377" url="~/f/nordic-q-a/73389/msc-exception/311649#311649"]Have you had any progress on this issue on your side?[/quote]

No，we haven't made any progress.

[quote userid="7377" url="~/f/nordic-q-a/73389/msc-exception/311649#311649"]I had a meeting with the R&D team responsible for this yesterday, and the current state is that after a thorough inspection of the code we cannot see an issue with the blockdevice code that would result in the observed corruption. In other words, we cannot see by analysis, a code path that would result in this. This does not mean that we have ruled out that there is an issue.[/quote]

What I understand so far, this issue likely occurs while connecting to the PC and copying files to the MSC from the PC side. Our products can be upgraded by copying the firmware to MSC. I have asked our test team to check the MSC before and after the upgrade, it's still being tested.

[quote userid="85932" url="~/f/nordic-q-a/73389/msc-exception/306456#306456"]After the patch was applied, although this issue couldn't be reproduced, I found some unexpected write operation while connect to the PC. [/quote][quote userid="85932" url="~/f/nordic-q-a/73389/msc-exception/307176#307176"]

One more finding.

1.Plug the USB into the PC and create a new file from the PC

[/quote]

Have you fixed these problems? They can be reproduced on the Nordic DK. I think if these are fixed, this issue will probably be fixed.

[quote userid="7377" url="~/f/nordic-q-a/73389/msc-exception/311649#311649"]As we are not able to reproduce the corruption itself, we suggest that we provide you with a modified library that has instrumentation to detect some unexpected changes/states/behaviors. Would you be willing to use that in our testing to help narrow down the issue?[/quote]

Yes, of course.

[quote userid="7377" url="~/f/nordic-q-a/73389/msc-exception/311649#311649"]Another hypothesis that came up was that there could be a reentrancy issue where you call from multiple interrupt contexts (into the blockdevice code. Could that be the case?[/quote]

I don't think so. We do not use any RTOS, nor do we do any file operations in the interrupt service.

RE: MSC exception

user7377 — Wed, 26 May 2021 07:11:51 GMT

Hi,

Have you had any progress on this issue on your side?

I had a meeting with the R&D team responsible for this yesterday, and the current state is that after a thorough inspection of the code we cannot see an issue with the blockdevice code that would result in the observed corruption. In other words, we cannot see by analysis, a code path that would result in this. This does not mean that we have ruled out that there is an issue.

As we are not able to reproduce the corruption itself, we suggest that we provide you with a modified library that has instrumentation to detect some unexpected changes/states/behaviors. Would you be willing to use that in our testing to help narrow down the issue?

Another hypothesis that came up was that there could be a reentrancy issue where you call from multiple interrupt contexts (into the blockdevice code. Could that be the case?

RE: MSC exception

user7377 — Thu, 29 Apr 2021 10:58:48 GMT

Thanks for the update. I do not have any news from my side atm.

RE: MSC exception

user85932 — Wed, 28 Apr 2021 09:06:27 GMT

Hi Einar,

One more finding.

1.Plug the USB into the PC and create a new file from the PC

2. Disconnect the USB, and plug the USB into the PC again, the PC pops up the following message.

RE: MSC exception

user7377 — Fri, 23 Apr 2021 12:04:50 GMT

Hi,

I have asked the maintainers to review the logs and provide input. I will let you know when we have something.

Einar