October firmware events

Apropos of the 25 year anniversary of the first IBI/EFI/UEFI boot services at week ago, as commemorated at the 20 year milestone in https://vzimmer.blogspot.com/2018/10/  

                                                        "Ken Reneris     Oct-14-1998"  

there was a question of the coding standards for UEFI https://twitter.com/bexcran/status/1701071379171619236 recently which made me think of Ken. I mentioned how that since Ken had come over to Intel from the Windows NT team at Microsoft it was natural to adopt the NT coding standard https://computernewb.com/~lily/files/Documents/NTDesignWorkbook/coding.pdf for the original IBI/EFI/UEFI work. This includes the EFI_ERROR macros, inf's, CR macro, TPLs as simplified IRQLs https://vzimmer.blogspot.com/2015/06/guids-revisions-interrupts.html, etc. This made IBI/EFI/UEFI a sort of a cultural 'Windows BIOS' given that lineage. 

And Ken as the MS HAL owner continually interfaced with the platform. In fact, standards like ACPI pre-dated IBI/EFI/UEFI by starting in the mid-90's. Ken was deep into that work, too, as evidenced by his solo inventor ship of the now-expired S4 resume patent https://patents.google.com/patent/US6209088B1/en. 

These UEFI technology elements serve a counterpoint with coreboot, which was LinuxBIOS in 99. The latter have technology elements https://www2.cs.arizona.edu/~mccann/cstyle.html, KConfig, etc.  

https://link.springer.com/book/10.1007/978-1-4842-0070-4. So just as we have had parallel growth of Windows and Linux, there have been threads of 'Windows BIOS' (e.g., EDKII UEFI based) and 'LinuxBIOS' (e.g., coreboot and maybe less so u-boot?).

But then again, who uses 'BIOS' https://basicinputoutput.com/2014/08/will-i-be-jailed-for-saying-uefi-bios.html? 

So the blog title is firmware events. Over the last couple of weeks there was a UEFI plugfest in the Portland area and a Open Compute Project (OCP) event in San Jose. As I drove back from the former, I stopped by a bookstore and saw https://www.amazon.com/Beyond-BIOS-Developing-Extensible-Interface/dp/1501514784

on the shelf.

On that same shelf I saw 

and

Though not the immediate book neighbor, the nearby https://www.amazon.com/ARM-Architecture-Reference-Manual-2nd/dp/0201737191 reminded me of Dave Jaggar. Back in the mid-1990's we were development hardware RAID controllers at Compaq and using custom ASICs alongside an AMD 29k RISC CPU. I recall Jaggar from ARM coming to visit and explaining the ARM architecture. Part of the discourse included showing a silicon die with a small portion highlighted for the ARM core itself. It was quite a shock for me to so such small mm-squared for a CPU. My firmware lead (lead == sole design & developer) work on the SMART-2SL 

https://www.amazon.com/Compaq-242777-001-WIDE-SCSI-CONTROLLER-242777001/dp/B0002FDXLQ is but a memory to be rekindled occasionally by seeing aftermarket examples of this device. In addition to being the first device to not have a non-volatile post write cache, this work gave me the opportunity to do some interesting firmware performance innovations with a colleague https://patents.google.com/patent/US6341342B1/en. 

Speaking of CPQ, it has been a strange migration of companies for me. My original internship was with Texaco, which in turn was purchased by Chevron. Then I did my first full-time firmware development at Daniel Industries, later acquired by Emerson Electric. My first foray into PC BIOS was at Texas Microsystems (TMI) working on industrial computers, which then was acquired by Radisys. Finally, my pre-Intel employer Compaq server group was in turn acquired by Hewlett-Packard, and then split into the enterprise side HPe. Luckily Intel is still Intel.

So speaking of the development event, it spanned 3 days. I presented on the first day (lower left image)

https://uefi.org/sites/default/files/resources/Tuesday_02_Kubacki%20and%20Zimmer_Final.pdf and final

day (rightmost image) https://uefi.org/sites/default/files/resources/Firmware%20Configuration%20%E2%80%93%20Past%2C%20Present%2C%20and%20Future_Zimmer.pdf. The first day of the event included a description of SPDM https://www.dmtf.org/standards/spdm and its support introduced into the UEFI 2.10 specification. SPDM is homed at the DMTF but has associated work-product having in groups like UEFI and the Trusted Computing Group (TCG). 

In the Insyde talk Tim Lewis mentioned the growth of attacks on the UEFI network stack after many years of battering SMM. This reminded me of a recent posting I saw https://www.linaro.org/blog/ledge-blogs-uefi-http-and-https-boot-in-u-boot/ where the U-Boot community discussing using https://savannah.nongnu.org/projects/lwip/ which is a similar approach taken in https://github.com/tianocore/edk2-staging/tree/MpNetworkStack mentioned in https://uefi.org/sites/default/files/resources/7_Maciej%20Vincent_INTEL_network%20stack%20performance.pdf. From the days of discussing HTTP booting in 2009 https://dblp.org/rec/conf/csreaSAM/Zimmer09.html?view=bibtex and having the URL boot option mentioned in https://www.rfc-editor.org/rfc/rfc5970.txt has come a long way. Just as EDKII consumes cryptography as a submodule from another community, maybe it's time to do so for the basic networking capabilities.

Given that AWS started in 2006, it may have a bit short-sighted of me to have said "...emergent compute models such as cloud computing." in that SAM 09 paper.

Maybe I was keying off the 09 date of what I thought was the definitive paper on the cloud, viz., https://www2.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-28.pdf.

So back to the topic of this blog, namely firmware events. On that topic, the next week came the Open Compute Project (OCP) event in San Jose. I first presented at OCP in 2015 with Mallik Bulusu https://cdrdv2.intel.com/v1/dl/getContent/671185 

and then again in 2016 on firmware updates http://files.opencompute.org/oc/public.php?service=files&t=1f7831234dce58bb875b1b5b24f7154d. 

This session last week was on the universal payload

I have been engaged on server platforms since I was hired into Intel in February 1997 to lead the IA-64 (Merced, Itanium) firmware. Along the way we devised ways to facilitate ease of firmware development, like multi-socket cache-as-RAM https://patents.google.com/patent/US7254676B2/en 

and similar for IA-32 https://patents.google.com/patent/US8078862B2/en. Palsamy and I also collaborated on UEFI and ACPI for RAS and error support https://cdrdv2.intel.com/v1/dl/getContent/671067. 

And all of these threads come together in these recent talks. The SPDM prezo mentioned at the top of the posting here entail facing the post-quantum cryptographic migrations like all of the other standards, including UEFI https://uefi.org/, TCG https://trustedcomputinggroup.org/, and others. The proposal for augmenting UEFI is mentioned in https://bugzilla.tianocore.org/show_bug.cgi?id=4087 and some prezos on the topic can be found at https://uefi.org/sites/default/files/resources/Post%20Quantum%20Webinar.pdf and https://uefi.org/sites/default/files/resources/USF_Security_Webinar_Final.pdf. The specific study on SPDM was posted as a pre-print to https://eprint.iacr.org/2022/1049 and then a journal submission in https://www.mdpi.com/2410-387X/6/4/48. 

Since SPDM has fixed message sizes, the concept of 'chunking' or breaking up the larger payloads demanded by these post-quantum algorithms is a common concern other hardware-based messaging interfaces will face, like the Trusted Computing Group's Trusted Platform Module (TPM). Interestingly this topic of post-quantum impacts on firmware standards that motivated the SPDM paper

 listed above was inspired by the study https://eprint.iacr.org/2021/041 from Cisco.

And during Q/A with the CISA presentation during the UEFI developer event I asked about formal methods for this domain. Since this talk followed the SPDM talk one recommendation was to use formal for the SPDM wire protocol. It turns out the above MDPI SPDM paper is referenced by a few formal studies of SPDM, including https://www.usenix.org/conference/usenixsecurity23/presentation/cremers-spdm and https://ieeexplore.ieee.org/abstract/document/10149352/. 

And speaking of security and standards, this upcoming week is the TCG members meeting at the Google campus in Kirkland, WA. I am Intel's representative on the Technical Committee, assuming that role after Kirk Brannock retired. 

 

But the TCG is not unfamiliar to me. I have been engaged with TCG https://en.wikipedia.org/wiki/Trusted_Computing_Group even when it was called TCPA, as evidenced by work-product like https://patents.google.com/patent/US7200758B2/en and delivering the Itanium and EFI API and platform specification, respectively.

When reading https://research.tue.nl/en/publications/communication-in-a-world-of-pervasive-surveillance-sources-and-me from the mention of Jacob in https://www-theregister-com.cdn.ampproject.org/c/s/www.theregister.com/AMP/2023/09/19/marvell_disputes_claim_that_cavium/, I hearkened back to my first ToorCamp presentation 

Based upon the slide

and discussion around the TPM, Jacob told me that we should design a TPM so that it could quickly be removed from board and chewed up/swallowed if someone tries to take your computer. Quite the privacy-preserving dietary strategy.

It's interesting to see people in person after the years of COVID seclusion and event cancellation or wholly-virtual events. Just like the psychic shock and fatigue of crowds, continual interaction and noise, it is taking a bit of effort to get re-acclimated. Part of the symptomology is falling behind on blogging, I guess, since I had intended to post this entry on the day of the EFI 25-year anniversary, not a week later. Oh well. Only so many free moments on the weekends these days.

Open platforms snapshot - May 2023

From https://github.com/tianocore/edk2-platforms/tree/master/Platform/Intel there are a rich set of platforms for EDKII in the open. 

Other than emulation platforms like SIMICS or 32-bit IOT Quark, which are all open source, the rest of the platforms are based upon variants of the Intel Firmware Support Package (FSP) https://github.com/intel/fsp

 

But how did this journey begin?

The first public discussion of the need to have a simpler, more open set of platform code commenced in 2014. I described this 'min tree' effort in the following 2015 prezo  https://github.com/vincentjzimmer/Documents/blob/master/OSTS-2015.pdf

 

 

 

 with guidance around how to take a large internal, closed source corpus into something smaller

 

 The strategy included the following elements

 

 

I described the value of open source for security assurance in another venue https://www.intel.com/content/dam/develop/external/us/en/documents/stts003-sf15-stts003-100f-820238.pdf

including the Baytrail-based MinnowBoard and Quark

I was a strong proponent of openness allowing for better security. One caveat I missed was that once the code is out there and bugs are found, then others will all lean in to help fix, at least for shared 'core' code, not https://en.wikipedia.org/wiki/Tragedy_of_the_commons

The challenge at the time was how to structure the code. We used Quark, since it was fully open, to model some of the software practices https://github.com/tianocore-docs/Docs/raw/master/White_Papers/A_Tour_Beyond_BIOS_Open_Source_IA_Firmware_Platform_Design_Guide_in_EFI_Developer_Kit_II.pdf

 

 

 

 

The approach entailed a decomposition of the workflow for configuration, porting, and feature addition.

At this point we only had Atom-based Minnow in the open. During this time we worked w/ the business units to get permission to open up a big core based platform code, namely Kaby Lake, and a Xeon big core server, namely Purley. The work is described in https://github.com/tianocore/edk2-platforms/blob/devel-MinPlatform/Platform/Intel/MinPlatformPkg/Docs/A_Tour_Beyond_BIOS_Open_Source_IA_Firmware_Platform_Design_Guide_in_EFI_Developer_Kit_II%20-%20V2.pdf

 

 

 

 

These studies provided a decomposition both logically

 

 

and in the source code

 

 

We updated the ecosystem on this work in https://www.platformsecuritysummit.com/2018/speaker/

This provided an overview of the server work and also a description of the software stacking.

Another view provided the workflow of the open source core on the left, the silicon packages in the middle, and the open source platform code on the right in order to curate a complete solution.

The best description of the overall workflow with a given SOC was described in https://www.intel.com/content/dam/develop/external/us/en/documents/uefi-firmware-enabling-guide-for-the-intel-atom-processor-e3900-series-820238.pdf in 2018
 

Just as we explicated security best practices in  https://link.springer.com/book/10.1007/978-1-4842-6106-4, we did describe some of this platform work that is now embodied as the 'min platform architecture' (MPA) https://github.com/tianocore/tianocore.github.io/wiki/Minimum-Platform-Architecture--MinPlatform.

Specifically, the book https://link.springer.com/book/10.1007/978-1-4842-7974-8 and its associated site https://github.com/Apress/Firmware-Development touch on this topic

 

With the evolution of a min-tree and min platform

and the MPA stack itself.

This work is also included in training material https://github.com/tianocore-training/PlatformBuildLab_MinPlatform_FW.

 

The idea behind 'min tree' was to right-size things. One of the concurrent investigations was to subset the main edk2 upstream, including breaking down some of the existing packages https://github.com/tianocore/edk2/compare/master...jyao1:edk2:ReOrg. Many companies do git-filtering to pull a subset or curate a smaller list of packages, like https://microsoft.github.io/mu/ with https://github.com/microsoft/mu_tiano_plus. The platform clean-up was easier since we historically cloned-and-updated platform code per generation, whereas the core packages are often reused, with figures like below from http://masters.donntu.ru/2020/fknt/yakubov/library/article6.pdf

Then there is also the 'Tragedy of the Commons' allusion wrt core I mentioned above.

As such, the feedback from the ecosystem at the time was that convincing management types to change a code base, albeit for a slimmer one, was a hard sell since it would invariably lead to new issues, including potential bugs. The older core had the advantage of years of flight-testing, namely evidence-of-use. But I was told that if new capabilities were added to vet the quality, such as testing / unit testing, then it would be an easier argument. That was a tough story to sell 5+ years ago but today falls on more friendly ears with the community rallying around things like https://github.com/tianocore/edk2/tree/master/UnitTestFrameworkPkg. Even though there are a paucity of unit tests written today, the above package forms a foundation to get that trust such that more radical refactoring can be done with higher confidence. I remember the TPM TSS project lead telling me that they had 90+% code coverage and any potential commit that broke a unit test or dropped that % was automatically rejected. 

The same story holds for evolution of new techniques like memory-safe languages like Rust into a large extant code base. Although the rewrite of critical libraries in Rust https://cfp.osfc.io/media/osfc2020/submissions/SLFJTN/resources/OSFC2020_Rust_EFI_Yao_Zimmer_NDK4Dme.pdf showed efficacy

given history like https://www.blackhat.com/presentations/bh-usa-09/WOJTCZUK/BHUSA09-Wojtczuk-AtkIntelBios-SLIDES.pdf

the cost of integrating the language in mainline and the hardening since 2009 make it a hard sell. But for 'new code' with tricky attacker-controlled data types, perhaps the dialectic can lean toward embracing language-based security?

Speaking of wisdom of others, some of this work for a min* was motivated by a comment I heard from https://en.wikipedia.org/wiki/Window_Snyder, namely 'the security defender's best weapon is the "delete" key.' Specifically, deleted code cannot have CVE's or a lack of test coverage. So 'min' as a tactic for 'less code' is often a salubrious path to pursue.

Regarding the thread on platforms above, mentioning MPA doesn't mean that there are not other open opportunities beyond EDKII.  There are examples such as https://github.com/slimbootloader/slimbootloader/tree/master/Platform

and https://github.com/coreboot/coreboot/tree/master/src/mainboard/intel

 

 

You can find info in https://link.springer.com/book/10.1007/978-1-4842-7939-7 https://github.com/Apress/System-Firmware

 

for slim bootloader

 

 

and coreboot, respectively.

Speaking of the various platform code embodiment's, efforts like Universal Scalable Firmware Architecture (intel.com) https://universalscalablefirmware.github.io/documentation/ work to find reuse across these different environments

 This work also mentions accommodating Rust-based designs, including https://github.com/oreboot/oreboot, which is mentioned in the first of the two books above https://link.springer.com/book/10.1007/978-1-4842-7974-8, too.

 and https://github.com/u-boot/u-boot, too

 including its use in https://github.com/openbmc/openbmc

PS
Sometimes my blog posts are like Joyce's Finnegan's Wake where the last sentence and the first sentence of the book overlap.  Or maybe a snake eating its tail?  Whatever the literary motif or allusion, another of the slides from the 2015 prezo at the top of this post, namely

reminded me of more recent happening in the industry, namely the 2021 prezo from AMI https://www.youtube.com/watch?v=SeiigV8PJPE and the Aptio Open edition. It describes their variant of this 2015 vision for OCP realized in practice in 2021

This leverages the work out of the Open System Firmware (OSF) https://www.opencompute.org/wiki/Open_System_Firmware

https://docs.google.com/document/d/1pvb4cyQIdXzOtmCb1y6o_6JBnJOx9DkdrMHeYHb5LU0/edit#heading=h.6iuk77tyg1a

Sidebar - what a change between 2018-2023. Ron now at Samsung. David at Amazon. Isaac recently retired from Intel. And Gundrala sadly passed

https://ocp-all.groups.io/g/main/topic/in_memory_of_gundrala/87592963?p=,,,20,0,0,0::recentpostdate/sticky,,,20,1,80,87592963,previd%3D1611775656365391031,nextid%3D1638983329612768441&previd=1611775656365391031&nextid=1638983329612768441. Gundrala and I collaborated quite a bit when he was at Intel, including

and many others

Gundrala is missed. It was nice having an opportunity to roll down 156th with him and grab Indian Food across from Crossroads here in Bellevue during his MS (post-Intel) years.

Ok. Back to the topics of open source platform code. So this 'open platform code/mintree' is another 'recent-to-past' binding. In general the 2015 to 2021 hang time of 6 years is not unreasonable. I once recall a software VP in early 2000's telling me that 'nothing significant happens in less than 10 years.' In the world of internet time and AI I suspect the timelines are a bit accelerated, but I have definitely observed that looking for quick wins in the infrastructure space rarely occurs. 

PPS

https://twitter.com/kirkbrannock/status/1665121431519367169

 

saddened me.  Kirk this year and Sham last year

have left the building to retirement, or as I sometimes joke "had a sharp enough spoon to dig out of Shawshank." https://en.wikipedia.org/wiki/The_Shawshank_Redemption

Good times working on BIOS with both, Sham being a huge mentor to me starting in 1997. He was sort of the "BIOS Yoda" when I joined Intel. Sham's the guy who did the first SMM BIOS enabling on the 386SL, etc. And I met Kirk in 1998 starting on workstation BIOS, EFI, and then many things platform thereafter, including great design oppty's with both like https://patents.google.com/patent/US7392371B2/en

which ended up landing in https://uefi.org/specs/PI/1.8/V3_Design_Discussion.html#firmware-file-system-format

Or with guru Sham on SMM on https://patents.google.com/patent/US6775728B2/en

that lives on today with things like https://uefi.org/specs/PI/1.8/V4_MM_Protocols.html#mm-mp-protocol.

And Sham also dropped one of my favorite farewell messages, viz.,

            As many of you have already heard - after 32+ years at Intel, I have decided to retire and start the next phase of my life’s exciting journey. I had prepared a very long farewell message for all of you, filled with my sage wisdom and list of accomplishments etc. etc. but then, I looked at the address field and it struck me that I still really like most of you on the list – so here is a short version.

 

            For past three decades, I watched technologies change. An amazing Technological Symphony was being played at Intel and I was fortunate enough to be hired into the orchestra as a bell ringer. I hope my cow bell went “ting” at the right time to make the symphony even more brilliant and melodious to the world, and that it helped to harmonize and amplify the brilliant voices around me. Now, a time has come for me to pass on the bell to steadier and younger hands that would ring the bell even more vigorously and timely than I could. As for me, I would gracefully join the audience and keep nodding as you play along taking this complex orchestra to dizzying heights.