Monday, February 8, 2016^4, life, death, and debug

I have been trying to maintain a tradition of blogging on my work anniversary Given that this is a firmware blog, I will try to craft some interesting message on firmware. But ahead of the bits and bytes, another year has led me to reflect on life. The first reflection occurred while taking the train into Seattle and hearkens back to my boyhood in Houston.

When I was in early elementary school during those humidity-filled hours, I remember a particular 'Go Western Day.' The teacher would have us each share our cowboy attire. One particular student always appeared decked out in the cowboy regalia and was quite enthusiastic about the rodeo that occurred this time of year. While we were milling about showing off our western gear, the school principal came to the classroom and whispered something in the teacher's ear. After the principal departed, the teacher gave the as-always excited student the feather out of her hat, which thrilled the student even more. He was beaming ear-to-ear. While riding that wave of glee, the teacher then told the student there was a message waiting for him in the office. After he left, the teacher told us that the student's mother had just passed away.

A first view at the stark dichotomies attending death - happiness and pathos.

Fast forward twenty years. My hiring manager and his manager from 1997 both passed away while on the job.  When the 2nd line manager had a memorial service at the work facility, his son showed up and said to us “I hope you enjoyed my father’s company since he spent more time with you than he did with us.”

Another dichotomy - the hard-driving world of technology and missed hours at home.

So this isn't what you visited this blog to read, I suspect. The page-rank, keyword filter undoubtedly chanced upon some terms such as 'UEFI' or 'debug.' Lest I disappoint, I will talk a bit about debugging firmware and some of the stark dichotomies there, too.

I recall hearing recently about various classes of bugs - hindenbugs, mandelbugs, heisenbugs, .... The first is a bug that makes the system go 'boom' like the Hindenburg.  The second is a bug that when solved leads to series of similar bugs, like exploring one of Mandelbrot's fractals. The final bug term is an homage to Heisenberg and his uncertainty principle wherein you cannot measure a particle's position and location at the same time - measuring one property will perturb the other. In the case of debugging, the act of debugging can often perturb the device under test (DUT).

How is a heisenbug related to debugging? One of the reasons this can occur is that host-based software debuggers, such as an implementation of the EFI_DEBUG_SUPPORT and EFI_DEBUG_PORT protocol, are done as drivers within the UEFI firmware. The former serves as a means to interact with a debug agent, or some entity that intends to provide a debug messaging channel to a remote debugger. The latter interface abstracts a simple messaging interface in the platform hardware, such as the USB debug port, to allow for sending messages using the respective debugger's 'wire protocol.' provides an example of the host based tools to install into Windows or Linux, and provides the debug agent code to build into the platform firmware.

For the debug agent, the source provides instance of using simple messaging hardware, such as a serial port and USB debug port. The way that the debug agent and the communications libraries that comprise the debug port protocol avoid heisenbugs, or interference with the rest of the system, is to make the libraries of type base, or self-contained code flows that do not depend upon other protocols or PPI's. This means that there is a better chance of observing failures in the rest of the system and not having the debugging infrastructure interfere with or perturb the rest of the services.

This non-interference property is important, but it can be aggravated when richer communications services are required, namely moving from simple serial interfaces to a richer network-based communications stack. To achieve debugging over a wider area network, it is appealing to consider using the extant UEFI working services and then applying the debug port abstraction atop the IP-based comms stack. The problem with this approach is that the debug agent is now intertwined with the network stack, has to follow the TPL calling restrictions of the latter, and may in fact induce errors based upon this interaction. The other approach is to use an isolated stack, such as that in a baseboard management controller (BMC) of a server, or duplicate the host networking stack with an independent one that has its own network interface controller (NIC) or somehow safely multiplexes uses of the primary NIC.

Again, this is tricky stuff. It is provocative trying to reuse the host firmware stack, but there is no 'free lunch', I fear. As such, the best approach is to have dedicated hardware like the USB debug port and a dedicated UART (with simple UART hardware or proxied to an embedded subsystem like the BMC or smart networking device) so that the debug agent can be coded without depending upon the full UEFI I/O stack running upon the host. This model of dedicated debug hardware will at least avoid one of the dichotomies in system software - debug observability and the behavior of the DUT.

Before I forget, if you are performing TCG Measured Boot and you have the UEFI Debugger enabled, don't forget to record "UEFI Debug Mode" into PCR[7]

As for the the other dichotomies of life mentioned earlier, there is no simple answer, free lunch, silver bullet, or technology palliative (no offense Ray).

No comments: