MPU fault on performing fifo operations

[karthikprabhu17]

Describe the bug
Whenever I try to insert a fifo element inside mqtt async evt callback, I get a a MPU fault on my sam_e70_board.

***** MPU FAULT *****
Data Access Violation
MMFAR Address: 0x401779
***** Hardware exception *****
Current thread ID = 0x20409254
Faulting instruction address = 0x41cbd8
Fatal fault in ISR! Spinning...

Steps taken for diagnosis:
I ran k_isr_in_isr() inside the client->evt_cb. It returns a 1 which means it runs with interrupt context.
I tried offloading from this ISR to systemworkqueue thread and that did not give MPU fault. But the receiving thread(main thread) still did not get my fifo item

To Reproduce
Steps to reproduce the behavior:

1. mkdir build;
2. cd build
3. cmake -DWEST_DIR=/Users/kprabhuv/new_zephyr/west ../
4. make flash
5. Trigger a MQTT_PUBLISH evt, the client->evt_cb is called but inside that a fifo_put operation fails.

Expected behavior
Since as per documentation, an ISR or any other thread should be able to easily perform k_fifo_put operations and receiving thread should be able to receive it. There should no MPU faults.

Impact
It is stopping us from moving our mqtt application to a MQTT_SOCK implementation. I didnt face these issues with MQTT legacy. Also, legacy has been removed. So we are in a race against time.

Screenshots or console output
If applicable, add a screenshot (drag-and-drop an image), or console logs
(cut-and-paste text and put a code fence (```) before and after, to help
explain the issue.

Environment (please complete the following information):

• OS: MacOs
• Toolchain : gnuarmemb
• Commit SHA or Version used: da44946 on master and then applied drivers: eth: sam-e70: revision B has more queues #12711 on top of it.

Additional context
Add any other context about the problem here.

[andrewboie]

How confident are you that you are passing valid data to the k_fifo APIs?

A fifo is just a k_queue, which is just a set of memory blocks whose first 32-bits are reserved for linked list pointers, which creates the ordered queue.

It looks to me like this is a memory corruption issue where the pointers are being modified incorrectly somehow. Depending on how they are corrupted, it could truncate the queue (if it gets zeroed) or cause invalid memory to be accessed as the next queue item when the API is called.

You might want to look at the queue items in a debugger and see where the 0x401779 address is coming from.

It's possible, but seems very unlikely that the culprit is somewhere in kernel/, I would exhaustively investigate any garbage-in-garbage-out scenarios first.

[karthikprabhu17]

@andrewboie: @andyross helped me diagnose the issue. I was able to solve this by bumping the CONFIG_ISR_STACK_SIZE from 2048 to 4096.

The instruction was trying to write to an address that was not writable

For future, is gdb the only tool I need to use diagnose such issues?

I will close this. we still need #12711

[ioannisg]

@andrewboie: @andyross helped me diagnose the issue. I was able to solve this by bumping the CONFIG_ISR_STACK_SIZE from 2048 to 4096.

The instruction was trying to write to an address that was not writable

For future, is gdb the only tool I need to use diagnose such issues?

I will close this. we still need #12711

@andrewboie it would have been nice to be able to detect ISR stack overflow on ARM. Is there a way to do it?
The stack overflow detection in ARMv8 is available for ARMv8-M, though :)

[andrewboie]

@andrewboie it would have been nice to be able to detect ISR stack overflow on ARM. Is there a way to do it?

On X86, we have a non-present guard page immediately preceding the interrupt stack.
Should be doable to have something similar on ARM, but without doing something clever it will eat up an MPU region