I remember the old times. Debugging was mostly made using printf()-output that was redirected to a UART channel and a serial cable hooked up to a terminal window on the PC. This technique was however too slow when debugging interrupt handlers, so I had to toggle a LED or set a digital I/O pin and let the oscilloscope visualize its value, to see what happened inside the interrupt handler.
Sometimes, a rudimentary debugger with run-stop-step debugging was available, but not always. In a historical perspective, this was not so long ago, although younger developers will probably think it was during the medieval times. Luckily, modern developers now enjoy much better tools to look into the target system and debug it. In this blog post, I summarize my best advice for ARM Cortex-M debugging. I have saved my best advice to the very end!
Cortex-M hardware support
The Cortex-M processors from ARM Ltd have excellent hardware support for debugging.
To use system analysis and real-time event tracing in compatible Cortex-M processors, a number of different technologies interact; Serial Wire Viewer (SWV), Serial Wire Debug (SWD) and Serial Wire Output (SWO). Their respective roles will be explained below.
- Serial Wire Debug (SWD) is a debug port similar to JTAG and provides the same debug capabilities (run, stop on breakpoints, single-step) but with fewer pins. It replaces the JTAG connector with a 2-pin interface (one clock pin and one bi-directional data pin). The SWD port itself does not provide for real-time tracing.
- The Serial Wire Output (SWO) pin can be used in combination with SWD and is used by the processor to emit real-time trace data, thus extending the two SWD pins with a third pin. The combination of the two SWD pins and the SWO pin enables Serial Wire Viewer (SWV) real-time tracing in compatible ARM processors.
- Serial Wire Viewer is a real-time trace technology that uses the Serial Wire Debugger (SWD) port and the Serial Wire Output (SWO) pin. Serial Wire Viewer provides advanced system analysis and real-time tracing without the need to halt the processor to extract certain types of debugging information.
Additionally, some Cortex-M cores and devices have support for instruction tracing, using the Instruction Trace Macrocell (ITM), the Embedded Trace Buffer (ETB) or the Micro Trace Buffer (MTB) technologies.
- Embedded Trace Macrocell (ETM) tracing requires an ETM trace-enabled debugger probe, such as SEGGER J-Trace. Such debugger probes typically have a large on-probe trace buffer to store the recorded execution history, or even uses real-time streaming to the host PC, and are thus more expensive compared to standard debugger probes without ETM trace support. Trace buffers used with ETM tracing are typically many MB or GB in size. To use ETM tracing, a 20-pin JTAG connector is used where 5 pins are dedicated for ETM tracing (1 clock pin and 4 data pins).
- Embedded Trace Buffer (ETB) tracing do not need any particular trace-enabled debugger probe, but instead uses a (very much) smaller trace buffer in RAM of the Cortex-M device. ETB can thus be used without the more expensive trace-enabled debugger probes with compatible Cortex-M devices, but the execution time that can be recorded is a lot shorter. Trace buffers used with ETB are typically very small, only a couple of KB, as they use the RAM of the microcontroller device as a trace buffer. To use ETB tracing, no particular extra JTAG pins are needed, as the trace buffer is just uploaded from the target RAM just like any other array of bytes.
- Micro Trace Buffer (MTB) is similar to ETB but a low-end solution for the Cortex-M0+ cores. The (minimal) trace buffer is filled into a small on-chip memory buffer, that can be uploaded to the developer PC for analysis.
And finally, ARM added support for hard fault crash analysis. This enables powerful debuggers to work out why the software crashed the CPU and went into the hard fault state.
All these capabilities, when taken together, is a very useful set of hardware capabilities in terms of debugger support. Now, what can you do with it?
1. The SFR register view
I know, I know. This is not a “shiny new object” you haven’t played with before. But even so, having the capability to look into the peripheral SFR registers and see how the hardware is configured, and its status, is a must-have for embedded developers worth their salt.
2. Hard fault crash analysis
Does your system crash once every week? Are you still clueless as to why it happens? Use a debugger with a hard fault crash analyzer. It can tell you exactly why the CPU went into the hard fault crash state, what code line made it happen, and the CPU context when it happened.
3. Serial Wire Viewer real-time event tracing and system analysis
This is the single most powerful debugger feature of compatible Cortex-M processors and debuggers.Serial Wire Viewer (SWV) and compatible debuggers can provide the following types of target information (in real-time, as the target executes at full speed):
- Statistical profiling
- Memory access history log
- Live variable view
- Data tracing, including live oscilloscope graphs
- Event notification on exception entry and exit
- Software instrumented trace
The developer can configure SWV in many different ways, to make it emit various combinations of information, e.g. values of memory locations, events, PC values etc. As more types of trace data are enabled for transmission, the more trace packages are needed to be sent from the CPU to the PC debugger. It is possible in certain circumstances that all of the data may not be received by the front end debugging software on the PC.
There are several reasons for this. The target CPU running at full speed can exceed bandwidth on the one-pin SWO output. In addition, trace data makes its way to the SWO via internal buffers which are limited in depth. If the CPU generates more trace packages than the SWO pin can transmit, there will be resulting packet loss. This is normal.
The reason is that Cortex-M microcontrollers are designed to meet aggressive cost targets. The resource limitation is a byproduct of the tradeoff between resources and cost, and is to some extent, “by design”. While this imposes minor limitations on what can be done, it is also true that SWV, used with care, are capable of providing very powerful debug information that in some cases are otherwise unobtainable information without very expensive debug hardware.
In either case, SWV is one of the most powerful debugging tools you have in your toolbox.
4. Instruction tracing
To crack really tough bugs, you might have to resort to instruction tracing, where the execution history is recorded for later off-line analysis. Using a trace log, you can work out what the CPU made before a particular bug – revealing the cause of most problems. With an instruction trace log, every C code line and assembler instruction the CPU executed up to the bug has been recorded. Advanced trigger conditions and filtering capabilities are usually available to aid the developer, who would otherwise be drenched in enormous amounts of trace data.
5. Live variable watch
Using certain debugger probes (Segger J-Link, for example), powerful debuggers can visualize the variable values “live” as the target system executes at full speed. With Atollic TrueSTUDIO, for example, this is possible also with complex data types, like pointers to an array of structs. You can get the complete data structure visualized with live data.
6. Kernel-aware RTOS debugging
If you are using an RTOS, such as FreeRTOS, you really need to have a debugger that is RTOS-aware. You want the debugger to visualize all the RTOS objects, for example, tasks/threads, semaphores, timers, mutexes, message queues, etc. I am sorry, but using an RTOS and not having an RTOS-aware debugger is a very poor choice, in my opinion.
7. Memory analyzer
Have you ever experienced a bug that was almost impossible to find? You were studying the source code for days, and everything looked perfectly fine and there absolutely was no bug there? And still, the system didn’t work as expected? I have. It is very frustrating.
As it turns out, some bugs cannot be found by studying the source code. Why is that? Because it is not caused by the software code itself, but rather by the way the system was built together. Either, the makefile included the wrong version of the source code files (and so you were trying to find the bug in the wrong files), or the linker has placed some code or data in the wrong memory location. Or, the linker optimizer inadvertently removed the code, as the case can be if you use function pointers instead of static calls to the function.
Either way, you will not find this type of bugs by studying the source code. You may have to analyze the memory layout of the generated binary flash image file. Only then can you detect that the interrupt vector table is misconfigured, or the linker has removed the function you call only using a function pointer.
If you have a modern tool like Atollic TrueSTUDIO, use the visual build analyzer. If not, wrap up your sleeves and get dirty using the obscure MAP file.
8. Don’t introduce bugs
This is probably my best advice. You want a debugger with all the powerful debug features to help you out when you run into problems – and you will get into problems, sooner or later. But having great debugger functionalities should not be taken as an excuse to behave like a cowboy hacker and write error-prone spaghetti code.
What separates professional developers from the crowd is the experience that makes them write better code in the first place. They can thus avoid many bugs, saving themselves from grief later. While this list is by no means exhaustive, you can deploy these techniques to prevent bugs from being introduced in your code:
- Review the source code using source code review team meetings
- Follow a best-practice coding standard, such as MISRA-C
- Format your code in a way that makes your code easier to understand and maintain
- Create a robust software architecture with independent modules and clearly defined interfaces between them – no spaghetti code and informal dependencies everywhere, thank you very much.
- Code defensively and instrument the code to detect unexpected behaviors. Letting the code detect problems in itself may be one of your best strategies.
And so, these are the 8 tips I have to minimize the bugs you introduce into your design and find the ones you do introduce.
Do you want to learn more on ARM Cortex development and debugging? Read this whitepaper: