100 posts

This marks my 100th blog posting, spanning 3 decades of blogs 1999-2022 https://vzimmer.blogspot.com/. 

This corresponds with 3 decades of publications https://dblp.org/pid/34/5641.html.

 

I start spilling across another decade, though, with 4 decades of patents https://en.wikipedia.org/wiki/List_of_prolific_inventors 1999—2022 , which generally tracks my Intel career of February 1997 to this date in 2022 touching 4 decades, too, namely 1990's, 2000's, 2010's, 2020's, respectively.

These are some pretty rough blogs hosted on a free blogging platform comprised mostly of rambling thoughts, with a few repeats, and little to no editing.

This effort is definitely not part of the day job. For after hours there are already a lot of other non-day job writing activities queued up historically, including patents, papers, books.  As such, this blog has often been often log-jammed behind those. It's hard to priority boost, other than signaling events like IDF STM unleashed http://vzimmer.blogspot.com/2015/08/smi-transfer-monitor-stm-unleashed.html I posted right after delivering https://www.intel.com/content/dam/develop/external/us/en/documents/stts003-sf15-stts003-100f-820238.pdf in San Francisco in 2015. Latter took a winding path. Showed up in coreboot  https://review.coreboot.org/c/coreboot/+/33234 https://www.platformsecuritysummit.com/2018/speaker/myers/ and a variant in ppam https://www.intel.com/content/dam/www/central-libraries/us/en/documents/drtm-based-computing-whitepaper.pdf. OG info can still be found at https://www.intel.com/content/dam/develop/external/us/en/documents/a-tour-beyond-bios-launching-stm-to-monitor-smm-in-efi-developer-kit-ii-819978.pdf  and  https://cdrdv2.intel.com/v1/dl/getContent/671411 and   I guess that I should go back and update the 'released' blog posting, too since so many of the referenced sites have come and gone over time. I can easily see how much history (I was about to type 'wisdom' but I cannot claim to discern wisdom signal from information glut noise easily) is lost during our age of everything online. 

While running into a colleague during the first visit to Oregon after COVID quarantine ended I was given a recommendation to read https://www.amazon.com/Inside-Intel-Worlds-Powerful-Company/dp/0452276438. The book ended in 1997, the year I joined Intel, so I cannot claim to have experienced the environment first hand. But some aspects of the culture described did map to my early Intel years. One quote I especially liked, though, was about how to work with an long-lived culture of technology, namely:

"Respect the past, be honest about the present, be optimistic about the future"

Or something like that. The stories about Hungarian Grove, whose signature I picked up as a recent hire shown in http://vzimmer.blogspot.com/2014/02/, reminded me of the manager in Oregon with whom I interviewed in October 1996, Mil Travniceck and then offered me the job, although I hemmed and hawed a few months while still working at Compaq prior to ultimately joining Intel in February 1997. From the first edition of https://www.amazon.com/Beyond-Bios-Implementing-Extensible-Interface/dp/0974364908

Mil coached me on many things, including life at Intel. He would always have the paper notebook and record meticulous details of our 1:1's, following the style described in Grove's https://www.amazon.com/High-Output-Management-Andrew-Grove/dp/0679762884. Mil was also from eastern Europe and had a deep accent. I never asked him his origin story, though. He was pretty legendary in the Intel BIOS community at the time and I appreciate the opportunity to have been hired by, learned from, and mentored by him in those early years in DuPont, WA. I still recall the last conversation I had with him. Although I was hired by him to lead the 64-bit Merced firmware in the High End Server Division (HESD), I had since moved on to the EFI team in the Microcomputer Software Labs (MSL) of the microprocessor division MD6. Mil was always a bit skeptical of the move from legacy BIOS, and in the hallway that day he challenged me "Is this move to EFI a good idea. A move away from legacy? You know legacy, we're really good at that here at Intel." 

Those are some words I'd like to carry forward, too. In this blog, though, maybe I'm starting to repeat myself.  As I close blog posting 100, maybe the universe is telling me that this blog stream is 'done'. Or maybe it's still an opportunity to practice respect, honesty, and optimism.

fin (100)

And then there were 99

One of the more satisfying areas in tech I’ve done involved creating a component based firmware model for early, space-constrained firmware execution, namely the Pre-EFI Initialization (PEI) architecture. I created both the specification, design and initial implementation. 

The journey to creation of PEI began in the late 1990’s. At that time the Extensible Firmware Interfaced (EFI) specification was defined, but the EFI ‘sample’ code was layered on top of a PC/AT BIOS or other precursor firmware. In 2000 there was a request to have a component-based firmware model for interoperable industry enabling. 

To that end, the PEI infrastructure includes a small core that is responsible for memory management, service registration, service discovery, and executable module dispatch. PEI Modules, or “PEIM’s”, can be PE/COFF executables or a subset of the PE/COFF referred to as a “Terse Executable”, or TE Image. The input of the PEI phase entails the machine reset and the output of the phase is a set of usable system memory that can be persist into the operating system runtime, along with the cause of the restart, or ‘boot mode.’

[from beyond bios book]

The problem being addressed was to allow for composing executable modules from different business entities, such as the CPU vendor for host CPU initialization, board vendor for platform specific logic, reusable generic infrastructure, and 3rd party silicon providers. These executables could range from a simple platform-specific PEIM that samples a general purpose I/O and publishes a flash map up into a module responsible for link and memory initialization built from 10’s of thousands of lines of ANSI C code. Given that the system on a chip vendor exposing the memory and fabric initialization can be different than the platform vendor, the ability to segregate the logic into separate executable modules was a requirement.  Correspondingly, some of the PEIM’s may only be delivered in binary, thus requiring a strong contract between the various units of execution.  The latter contract entailed the use of PEIM-to-PEIM interfaces, or sets of interfaces named by a Globally Unique Identifier (GUID). The PEI modules are stored in a container known as a Firmware Volume (FV). Typically a system will have a least two FV’s: one for the PEI Modules and another for the larger UEFI and DXE modules.

Compounding the need for module interoperability, the PEI phase had to eXecute-In-Place (XIP) from a memory-mapped store immediately after the machine reset. During this early phase the main DRAM complex has not been initialized, and the execution path to the SPI NOR-attached storage is typically slow, so the PEI core and modules had to be optimized to minimize both code size and memory usage.  Specifically, the memory for this early phase entailed the use of the process Cache As RAM (CAR).

The usage of CAR, both for single and later parallel PEI instances, was another innovation I drove from the PEI definition. This usage of PEI allowed for writing C code for even the earliest flows, especially the notoriously complex DDR initialization sequences. Prior to CAR the original design of PEI entailed an assembly-language, register-based software model with ‘call levels’ mapped to various sets of the CPU register set. We discovered that writing code with these ‘level’s’ instead of common idioms like a call stack was untenable, as was factoring large algorithmic routines in assembly. The latter made writing portable, re-usable code difficult. Also, as IP blocks were shared between different CPU cores, not having re-usable C code compounded the software development costs.

The PEI infrastructure started with the register-based model that was ultimately abandoned and then moved to the CAR-based software model in an Intel-defined capability in 2001, and finally culminated in the UEFI Platform Initialization (PI) specification http://www.uefi.org/sites/default/files/resources/PI_Spec_1_6.pdf, volume 1, on http://www.UEFI.org. There is a corresponding open source implementation of the EFI Developer Kit II (EDKII)-based PEI core at https://github.com/tianocore/edk2/tree/master/MdeModulePkg/Core/Pei with some sample platform and silicon PEIM’s at https://github.com/tianocore/edk2-platforms/tree/devel-MinPlatform/Platform/Intel/MinPlatformPkg/PlatformInit/PlatformInitPei. The UEFI Platform Initialization (PI) specific provides guidelines on how the PEI core operates, its core services, the TE definition, and the output state of the PEI phase, or the Hand-off-blocks (HOBs).  The latter decouples the PEI core from the later state of platform initialization.

PEI has bindings to several architectures, including Itanium, IA32, x64, 32-bit ARM and Aarch64. Today the ARM licensees and ARM Ltd. for 32-bit and Aarch64 servers and client, AMD client and servers, and all of Intel products provide PEI-based silicon enabling. Given the BSD licensing of the EDKII, commercial BIOS vendors and many OEM’s have their own instance of the PEI core. Silicon providers deliver PEI modules as binaries and/or source as either licensed closed source or open source. Regrettably, most of the critical silicon specific flows for memory initialization are either in a proprietary boot ROM or binary blob, not open source. In order to accommodate the various open source host firmware communities, Intel provides a variant of a PEI Firmware Volume with a set of additional API’s called the Intel Firmware Support Package (FSP) https://www.intel.com/fsp.  The latter specification has a set of simple services that allow for coreboot https://www.apress.com/us/book/9781484200711 or EDKII https://www.intel.com/content/dam/develop/external/us/en/documents/a-tour-beyond-bios-using-the-intel-firmware-support-package-with-the-efi-developer-kit-ii-fsp2-0-820293.pdf to ingest the PEI modules. 

Given that Intel uses UEFI PI EDKII based PEI modules to validate and enable its SOC’s, having the same critical code usable by various open or closed source host firmware frameworks, from U-boot to coreboot to UEFI to custom RTOS loading argues for the business value of container strategies like Intel FSP to deliver the PEIM’s.

PEI and the PI specification are really a business-to-business technology, namely silicon to OEM.  This is distinct from the UEFI specification which has both business-to-business, Original Equipment Manufacturer (OEM) to independent hardware vendor (IHV) and operating system vendor (OSV), and business-to-consumer, such as OEM to enthusiast or writing UEFI applications.

PEI isn’t universal, though. A native U-Boot SPL or coreboot romstage may not resemble PEI, or the alternate firmware boot flow such as OpenPower and its PEI-equivalent called HostBoot, or the RISC-V Freedom Unleashed’s FSBL. Also, the business-to-business interoperability or richness of PEI are not required for certain classes of platform designs.  Given the above caveats, though, I would say PEI has been a success given the mainstream CPU vendor usage, although it isn’t the only way to perform this initialization.

And I remember working on this tech in the early 2000's. A patent attorney helping to draft some of the material said 'no one is patenting this type of material.' Some work like expired https://patents.google.com/patent/US7103529B2/en filed in 2001 cited by https://scholar.google.com/scholar?oi=bibs&hl=en&cites=8788045269349566754&as_sdt=5&as_ylo=2022&as_yhi=2022 for their work on firmware sand boxing in products 20 years later.  Perhaps this, and PEI during 2001, can fall under the theme of 'value of working on unpopular things.' Once they become popular, the jfk quote kicks in about 'success has many patents but failure is an orphan.'

I hear other strange history, namely some parties outside of Intel working on EFI in 1997 and 1996 when IBI, the EFI predecessor, didn't commence until 1998. FSP started in 2012, but some folks mention it as being 14 years old.

Helping finish and scale is valuable. No need to claim inventorship/creation. Sometimes tech companies encourage that pernicious behavior as part of the promotion process. This leads to the 'pump and dump' phenomena I mentioned earlier http://vzimmer.blogspot.com/2019/02/tiano-147-and-22-or-anniversarynext7.html. And I'm still amazed by the tech market. From the feast-to-famine on hiring to the interesting organization psychology. For example, reading https://www.warp.dev/blog/problems-with-promotion-oriented-cultures reminds me of my PDD reference in March posting http://vzimmer.blogspot.com/2022/03/ that shared similar sentiment with this May article

The dark side of 'finishing' is the loose ends.  mention the capsule/variable tweet w nikolaj and tianocore infosec.  https://www.goodreads.com/quotes/276615-another-flaw-in-the-human-character-is-that-everybody-wants, which remind me of https://twitter.com/pietrushnic/status/1514741511203794945 and discussion with https://twitter.com/vincentzimmer/status/1501826510738522113.

Another reminder of finishing was helping stand up the EDKII infosec group https://tianocore.org/security. The journey went from chasing items like https://invisiblethingslab.com/resources/misc09/Quest%20To%20The%20Core%20(public).pdf and its ilk, curating bespoke list at https://edk2-docs.gitbook.io/security-advisory/, including epic fixes like https://edk2-docs.gitbook.io/security-advisory/boot_failure_related_to_uefi_variable_usage#description. Then leveraging the notification mechanism of https://uefi.org/security mantis tickets to until https://tianocore.org/security was ready. Some of the process

 was inspired by other open source projects like Xen https://xenproject.org/developers/security-policy/. The latest was ensuring that EDKII could issue its own CVE's as a CNA https://www.ami.com/ami-and-industry-partners-drive-acceptance-of-tianocore-as-cve-numbering-authority-cna-by-the-common-vulnerabilities-and-exposures-cve-program/#:~:text=The%20acceptance%20of%20TianoCore%20as%20a%20CNA%20by%20the%20CVE,%2Frequest_id.html%23cna_participants, including learning how to curate the json for issues and being tested by Mitre prior to achieving this milestone. There are still rough edges to work out, such as commitment from community, tragedy of commons, permissive license where folks can fix and not give back, such as marc jones mentions in http://vzimmer.blogspot.com/2020/12/musings-about-firmware-cultures.html

fin (99)

Re-use - ideas, et al.

This is a quick note on 're-use'.  Recall how UEFI was derived from EFI, which in turn started as IBI, as mentioned in in http://vzimmer.blogspot.com/2018/10/ghosts-of.html and various histories of EFI in books http://vzimmer.blogspot.com/2021/01/books-and-computers.html and papers https://www.intel.com/content/dam/www/public/us/en/documents/research/2011-vol15-iss-1-intel-technology-journal.pdf. 

As a background, the original Intel Boot Initiative had its own filesystem called "IBIFS". It was a simple log-structured filesystem that made for easy parsing. But the challenge with this filesystem was how to provision boot media from other operating systems, such as Windows. In the 1990's MS-FAT was the prevailing filesystem although it was much more complex than IBIFS. The IBI/EFI team was encourage to adopt MS-FAT https://download.microsoft.com/download/1/6/1/161ba512-40e2-4cc9-843a-923143f3456c/fatgen103.doc, along with PE/COFF https://docs.microsoft.com/en-us/windows/win32/debug/pe-format executables, as the filesystem and executable format, respectively. This was supported by having a license to use these specifications for production of EFI firmware. 

The other advantage of using MS filesystem included MS providing tools to format, check and provision disks https://www.intel.com/content/www/us/en/download/714351/uefi-shell-disk-utilities.html whose sources were derived from MS Installable Filesystem (IFS) C++ kit provided to OEM's. This was my first dive into C++-on-a-C-framework (i.e., no global constructors, etc).

And PE/COFF allowed for using Visual Studio and commercial linkers. At the time we had less luck in getting PDB format and KD wire protocol made public. We wouldn't open source the EFI Debug support protocol and nub based upon reversing that infrastructure, of course. 

Of course using ANSI C with PE/COFF was more advantageous than using non-standard art like the coreboot romcc that proved brittle to maintain long-term, although it was a clever solution prior to the advent of cache-as-ram (CAR) https://www.coreboot.org/data/yhlu/cache_as_ram_lb_09142006.pdf. Prior to CAR we had to perform unnatural acts using MMX registers and other on-processor resources (with the peak weirdness of PEI CIS 0.3 and its register 'call levels') in order to implement pre-permanent memory flows like DRAM initialization in the host firmware. 

I was reminded of this logic of re-use based upon a few recent topics, including USF https://github.com/universalscalablefirmware and EDK2 GSoC conversation. On the USF topic, there is a discussion of using a self-describing format like CBOR https://www.rfc-editor.org/rfc/rfc8949.html in lieu of the UEFI HOB's https://universalscalablefirmware.groups.io/g/discussion/files/New%20data%20format%20for%20Universal%20Payload%20Interface.pdf. Even though I helped create HOB's, I am one of the first to support moving to a more standard format, especially given the existence of work like https://github.com/intel/tinycbor. To me this is the same spirit of using known filesystem formats, etc.

When asked about this I often cite the economics of host firmware development. Imagine a 5,000 person host firmware producing community across open source participants, OEM's, ODM's, firmware vendors, first party device creators, etc.  Recall the supply chain picture from erstwhile 2015 CanSecWest prezo

As such, compare this to the application and OS producing community that is easily orders of magnitude larger. Given the developer community size disparity it is difficult to justify the R&D and validation expense for bespoke host firmware-only solutions. Given NIH sometimes in engineering communities I also quote "Good artists invent, great artists steal" Picasso  https://medium.com/ben-shoemate/what-does-it-mean-good-artists-copy-great-artists-steal-ee8fd85317a0.

And given challenges of HOB's I understand the critique of Terse Executable (TE) during gsoc thread "[edk2-discuss] GSoC Proposala replacement for the TE format (it’s buggy and most platforms mostly abandoned it over various issues)," I'm not sure where he received his telemetry on 'abandoned' for working systems. Scraping FD images from the internet? Similar to some of the weird history on those GSoC threads like "PEI and DXE in 1998" when the latter were created in early 2000's as derivatives of EFI into the earlier boot flow. This TE assertion is close to home given the TE image format signature has 'vz' as opposed to PE/COFF 'mz', as originally described in https://www.amazon.com/Beyond-Bios-Implementing-Extensible-Interface/dp/0974364908

So has the TE image been relevant in the industry? Let's do some rough back-of-the-envelope calculations. To begin, the TE image format is easily over 20 years old. In the spirit of the last blog on statistics I'd sat we started to scale Framework EDKII in 2005

Imagine in the ensuing years it's tough to calculate the number of PC's, servers, and clients that shipped using UEFI PI-based firmware, including #'s like https://www.statista.com/statistics/576151/unit-shipments-pcs-united-states/. For Windows8 UEFI was required so you can argue 100% at that point. 

To take a swag lets assume 250 million units / year over the last 17 years, yielding 4.25 billion units. Let's also assume the fabrication of the PEI phase pre-memory uses TE's and that each fv/ibb/fvrecovery has ~ 50 peim's compiled with TE. So that would yield the market has shipped 50 * 4.25  =  212.5 billion TE images into the market. And if platform updated 5 times during life, 1 trillion TE's may have been produced. So even if we go 'Beyond TE', 1T has been a good run. And I look forward to the next wave of firmware technologists' proposals - the critique is a pre-condition to invention/innovation - but the next wave needs to close the loop by proposing, documenting, coding, delivering, validating, and scaling their alternative. 

I still recall meeting one of the original developers of MS-FAT in Windows at a driver dev-con in the early 2000's in Redmond. He mentioned that 'writing the code was only 10% of the effort.' The latter class of activities were the remaining 90% for a professional software engineer. Or the old trope 'the last 10% of a project takes 90% of the time.'

Finally, these cardinality of the produced TE image set calculations remind me of a colleague at Cornell during my undergrad. He worked part-time in the astrophysics department, whereas I spent time modeling and mocking up particle beams for a professor who did microwave and plasma research as part of SDI https://en.wikipedia.org/wiki/Strategic_Defense_Initiative. My friend noted that the astrophysics researchers there would reckon calculations plus or minus a couple orders of magnitude. I won't argue host firmware is of the same scale but it does note the challenges in precision in various domains.

Sometimes folks believe in bespoke and not studying the past arc of history in tech. I recommend spending some time on bitsavers.org, for example, to folks. But it reminds me of the advice given to me by a retired manager - "Sometimes you see someone put their hand in the chipper. You can say 'there's a chipper, there's a chipper', but they ignore you. Once they are beyond their elbow you need to jump in, at least." Or the dual of child rearing, which was "Sometimes the young child needs to bump her knees on the coffee table while learning to walk."  For those not in the PNW a chipper is https://www.familyhandyman.com/list/best-wood-chippers/.

BTW
I just sampled some of the OCP Security https://www.opencompute.org/wiki/Security tech talks. Work to avoid e-waste such as ownership transfer https://docs.google.com/document/d/1oANhjvv_R7E5n8w1RroN8l8-0jdYlfdQDp_3RqGV66k/edit# for servers presented by Google datacenter engineer nicely complements their work on Chromebooks with their developer mode https://www.chromium.org/chromium-os/chromiumos-design-docs/developer-mode/ and other art to allow for unsupported devices to be over-flashed by community firmware for continued life & usage. Nice to see work to avoid aged hardware from only have a fate of visiting the refuse bin/salvage yard.

It might be a stretch to analogize re-use of ideas in this blog post and re-use of hardware in the circular economy from this OCP talk. The former stands on the shoulders of giants and leverages information ecosystems (knowledge, tools, remediated flaws) whereas the latter leverages extant hardware for appropriate use-cases. But interesting stuff nevertheless.

fin (98)