Re-use - ideas, et al.

This is a quick note on 're-use'.  Recall how UEFI was derived from EFI, which in turn started as IBI, as mentioned in in http://vzimmer.blogspot.com/2018/10/ghosts-of.html and various histories of EFI in books http://vzimmer.blogspot.com/2021/01/books-and-computers.html and papers https://www.intel.com/content/dam/www/public/us/en/documents/research/2011-vol15-iss-1-intel-technology-journal.pdf. 

As a background, the original Intel Boot Initiative had its own filesystem called "IBIFS". It was a simple log-structured filesystem that made for easy parsing. But the challenge with this filesystem was how to provision boot media from other operating systems, such as Windows. In the 1990's MS-FAT was the prevailing filesystem although it was much more complex than IBIFS. The IBI/EFI team was encourage to adopt MS-FAT https://download.microsoft.com/download/1/6/1/161ba512-40e2-4cc9-843a-923143f3456c/fatgen103.doc, along with PE/COFF https://docs.microsoft.com/en-us/windows/win32/debug/pe-format executables, as the filesystem and executable format, respectively. This was supported by having a license to use these specifications for production of EFI firmware. 

The other advantage of using MS filesystem included MS providing tools to format, check and provision disks https://www.intel.com/content/www/us/en/download/714351/uefi-shell-disk-utilities.html whose sources were derived from MS Installable Filesystem (IFS) C++ kit provided to OEM's. This was my first dive into C++-on-a-C-framework (i.e., no global constructors, etc).

And PE/COFF allowed for using Visual Studio and commercial linkers. At the time we had less luck in getting PDB format and KD wire protocol made public. We wouldn't open source the EFI Debug support protocol and nub based upon reversing that infrastructure, of course. 

Of course using ANSI C with PE/COFF was more advantageous than using non-standard art like the coreboot romcc that proved brittle to maintain long-term, although it was a clever solution prior to the advent of cache-as-ram (CAR) https://www.coreboot.org/data/yhlu/cache_as_ram_lb_09142006.pdf. Prior to CAR we had to perform unnatural acts using MMX registers and other on-processor resources (with the peak weirdness of PEI CIS 0.3 and its register 'call levels') in order to implement pre-permanent memory flows like DRAM initialization in the host firmware. 

I was reminded of this logic of re-use based upon a few recent topics, including USF https://github.com/universalscalablefirmware and EDK2 GSoC conversation. On the USF topic, there is a discussion of using a self-describing format like CBOR https://www.rfc-editor.org/rfc/rfc8949.html in lieu of the UEFI HOB's https://universalscalablefirmware.groups.io/g/discussion/files/New%20data%20format%20for%20Universal%20Payload%20Interface.pdf. Even though I helped create HOB's, I am one of the first to support moving to a more standard format, especially given the existence of work like https://github.com/intel/tinycbor. To me this is the same spirit of using known filesystem formats, etc.

When asked about this I often cite the economics of host firmware development. Imagine a 5,000 person host firmware producing community across open source participants, OEM's, ODM's, firmware vendors, first party device creators, etc.  Recall the supply chain picture from erstwhile 2015 CanSecWest prezo

As such, compare this to the application and OS producing community that is easily orders of magnitude larger. Given the developer community size disparity it is difficult to justify the R&D and validation expense for bespoke host firmware-only solutions. Given NIH sometimes in engineering communities I also quote "Good artists invent, great artists steal" Picasso  https://medium.com/ben-shoemate/what-does-it-mean-good-artists-copy-great-artists-steal-ee8fd85317a0.

And given challenges of HOB's I understand the critique of Terse Executable (TE) during gsoc thread "[edk2-discuss] GSoC Proposala replacement for the TE format (it’s buggy and most platforms mostly abandoned it over various issues)," I'm not sure where he received his telemetry on 'abandoned' for working systems. Scraping FD images from the internet? Similar to some of the weird history on those GSoC threads like "PEI and DXE in 1998" when the latter were created in early 2000's as derivatives of EFI into the earlier boot flow. This TE assertion is close to home given the TE image format signature has 'vz' as opposed to PE/COFF 'mz', as originally described in https://www.amazon.com/Beyond-Bios-Implementing-Extensible-Interface/dp/0974364908

So has the TE image been relevant in the industry? Let's do some rough back-of-the-envelope calculations. To begin, the TE image format is easily over 20 years old. In the spirit of the last blog on statistics I'd sat we started to scale Framework EDKII in 2005

Imagine in the ensuing years it's tough to calculate the number of PC's, servers, and clients that shipped using UEFI PI-based firmware, including #'s like https://www.statista.com/statistics/576151/unit-shipments-pcs-united-states/. For Windows8 UEFI was required so you can argue 100% at that point. 

To take a swag lets assume 250 million units / year over the last 17 years, yielding 4.25 billion units. Let's also assume the fabrication of the PEI phase pre-memory uses TE's and that each fv/ibb/fvrecovery has ~ 50 peim's compiled with TE. So that would yield the market has shipped 50 * 4.25  =  212.5 billion TE images into the market. And if platform updated 5 times during life, 1 trillion TE's may have been produced. So even if we go 'Beyond TE', 1T has been a good run. And I look forward to the next wave of firmware technologists' proposals - the critique is a pre-condition to invention/innovation - but the next wave needs to close the loop by proposing, documenting, coding, delivering, validating, and scaling their alternative. 

I still recall meeting one of the original developers of MS-FAT in Windows at a driver dev-con in the early 2000's in Redmond. He mentioned that 'writing the code was only 10% of the effort.' The latter class of activities were the remaining 90% for a professional software engineer. Or the old trope 'the last 10% of a project takes 90% of the time.'

Finally, these cardinality of the produced TE image set calculations remind me of a colleague at Cornell during my undergrad. He worked part-time in the astrophysics department, whereas I spent time modeling and mocking up particle beams for a professor who did microwave and plasma research as part of SDI https://en.wikipedia.org/wiki/Strategic_Defense_Initiative. My friend noted that the astrophysics researchers there would reckon calculations plus or minus a couple orders of magnitude. I won't argue host firmware is of the same scale but it does note the challenges in precision in various domains.

Sometimes folks believe in bespoke and not studying the past arc of history in tech. I recommend spending some time on bitsavers.org, for example, to folks. But it reminds me of the advice given to me by a retired manager - "Sometimes you see someone put their hand in the chipper. You can say 'there's a chipper, there's a chipper', but they ignore you. Once they are beyond their elbow you need to jump in, at least." Or the dual of child rearing, which was "Sometimes the young child needs to bump her knees on the coffee table while learning to walk."  For those not in the PNW a chipper is https://www.familyhandyman.com/list/best-wood-chippers/.

BTW
I just sampled some of the OCP Security https://www.opencompute.org/wiki/Security tech talks. Work to avoid e-waste such as ownership transfer https://docs.google.com/document/d/1oANhjvv_R7E5n8w1RroN8l8-0jdYlfdQDp_3RqGV66k/edit# for servers presented by Google datacenter engineer nicely complements their work on Chromebooks with their developer mode https://www.chromium.org/chromium-os/chromiumos-design-docs/developer-mode/ and other art to allow for unsupported devices to be over-flashed by community firmware for continued life & usage. Nice to see work to avoid aged hardware from only have a fate of visiting the refuse bin/salvage yard.

It might be a stretch to analogize re-use of ideas in this blog post and re-use of hardware in the circular economy from this OCP talk. The former stands on the shoulders of giants and leverages information ecosystems (knowledge, tools, remediated flaws) whereas the latter leverages extant hardware for appropriate use-cases. But interesting stuff nevertheless.

fin (98)

Tiano, '147, and 22 or Anniversary.Next^7

This covers my 7th blog aligned with my work anniversary, a successor to http://vzimmer.blogspot.com/2018/03/open-platforms-and-21-or.html.  I'm now passing the 22 year milestone.

I try to land this blog posting on the anniversary day. Luckily this year I received an email reminding me that I'm already one year into my 3 year sabbatical eligibility. As far as topics go, replying to Juan's https://twitter.com/juanrodpfft/status/1099896998704996352 brought up a couple of milestones in time, namely Intel Achievement Awards (IAA's).  Some background on the IAA can be found at https://blogs.intel.com/jobs/2012/05/why-the-intel-achievement-awards-ceremony-was-amazing/#gs.13ww1ppY.

My first of 2 IAA's was delivered in 2004 and read on the Tiano architecture.

Tiano is the code name of what became the Intel Framework Specifications and the EFI Developer Kit (EDK), which has since evolved into the EDKII project and the UEFI Platform Initialization (PI) specifications. Even though this was about 15 years ago, progress in this space is continuing apace, including the open platform work. The last decade and a half was laying the foundations of the host firmware, and the next will be scaling how it's delivered.

To that end, the open Kaby Lake (KBL) platform has a newly added system, namely the Clevo board
https://github.com/tianocore/edk2-platforms/tree/devel-MinPlatform/Platform/Intel/ClevoOpenBoardPkg for System76 KBL laptops. This builds upon the infrastructure detailed in https://github.com/tianocore/edk2-platforms/blob/devel-MinPlatform/Platform/Intel/MinPlatformPkg/Docs/A_Tour_Beyond_BIOS_Open_Source_IA_Firmware_Platform_Design_Guide_in_EFI_Developer_Kit_II%20-%20V2.pdf.

So this means that I'm still building upon the project recognized by the first IAA, but what about the second one? The award from 2012 was on deploying signed updates across Intel and the industry, building upon the NIST 800-147 standards effort https://csrc.nist.gov/publications/detail/sp/800-147/final.

In the last 7 years work continues in this space and follows a similar arc, namely standards and scaling an implementation. A scaling of implementations of signed updates can be found in work like https://github.com/tianocore-docs/Docs/raw/master/White_Papers/A_Tour_Beyond_BIOS_Capsule_Update_and_Recovery_in_EDK_II.pdf and additional standards, such as https://csrc.nist.gov/publications/detail/sp/800-193/final for resiliency. The latter is important because one of the challenges of deploying '147 style updates includes fear of a machine become bricked, or not successfully completing the update.

Awards are an interesting thing. One school of thought is that an award should only be delivered after an idea is delivered to market for several generations, thus ensuring that the originator of the idea carries the ball to the goal line. Without this tracking you end up with possible technologist patterns like a 'pump and dump', namely evangelize an idea and achieve rewards, but move on to the 'next big thing' prior to enabling and scaling the concept, or 'dumping it' on others prior to delivering market success. At the same time, though, technology is often a funnel that starts small with a few pioneering parties at the beginning, or mouth of the funnel, and telescopes to much larger sets of contributors by the time it appears in the market. So if you wait until the end, it's harder to reward smaller classes.

A colleague once suggested a potential solution to this 'pump and dump' risk for senior technologists, namely extend their review cycle from twelve months to a couple of years. The logic being that the annual review cycles encourages 'pump' periodicity of 12 months in order to optimize the remuneration calculus of annual reviews.

I personally don't know if there's a magic bullet other than the have a culture of each individual having the 'business first' mindset of one of the below quotations. Also, perhaps you can get away with a random 'pump and dump', but your personal brand and reputation will ultimately suffer for this type of behavior. The industry is relatively small, and 'trust is earned in droplets but lost in buckets.' A dump or two can lead to spilling that bucket.

Speaking of quotations and beyond those couple of awards milestones, a few other items came to mind during this anniversary posting. These include memory quotes from mostly-former colleagues, such as DM's "If anyone knew true cost of a project, nothing would be funded" or BP's "It has always been this bad.  you are just now higher up in the organization to see more." From there we have more valuable life and career advice, such as "Never lie, but don't tell all of the truth" or GC's "Two sisters never got along with - 'Polly' and her sister 'Ticks.'" Another one that helped me empathize with the machine was RH's "Moving higher in management ranks means making successively more impactful decisions with diminishing amounts of information."

In addition to the quotations I can source, the next ones are ones I find myself dispensing that I cannot recall if they originated from a party inside or outside, including "Prioritize your work with business first, team second, career third, "Be kind to people because you don't know what crisis they have going on personally," and finally "Your career is like archery of Zen - the harder you focus on the target of just 'success' the more difficult it will be to achieve."

OK. Awards and quotations. That's enough for noting the passing of 264 months.

This one is for 20, or Anniversary.Next^5

As I reach my 20 year anniversary with Intel today, I reflect upon advice that resonates with me. I especially like the posting http://perspectives.mvdirona.com/2016/11/advice-to-someone-just-entering-our-industry/, including the admonition to "Play the Long Game."

20 years. And all doing firmware. Several different firmware architectures and many instances of EFI-style firmware (e.g., Release 1-Release 8.1/2/3/4/5/6, Release 9 "aka EDKII").....

Hopefully this won't encourage me to abuse logical fallacies like argument from authority, saying 'In my 20 years at Intel we.....' Instead you're only as good as the last game you've played, not your record of games.

Or having a Whiggish view of tech history. Instead it's more Kolmogorov probability that monotonically increasing (or decreasing) progress and determinism.

Speaking of history, my original badge from February 24, 1997 can be found below, with the drop-e logo and, gasp, a suit and tie.

And now

Ah, the thick head of hair that I had in the 90's. And my Harry Potter glasses. I recall visiting Shanghai and Suzhou in '01. In the latter city the locals pointed at me in those crazy glasses and a scratch on my forehead from my two year old daughter (that resembled the lightning bolt), reinforcing the Potter doppelganger experience. Pre-SARs in Shanghai, so it was still possible to eat snake, drunken shrimp, and dining colleague from the south China province whose restaurant jaunt truly lived up to the saying "... the Chinese eat anything with four legs except a table, and anything that flies that isn't an airplane..." http://simonlesser.blogspot.com/2009/06/food.html.

So my journey at Intel started in 1996 after contact from an Intel recruiter while I lived in Houston,TX. He exhorted me to join Intel, especially given the 'imminent' Merced CPU development. I interviewed in Hillsboro, OR in October 1996 and was told that I could go to Oregon for IA32 Xeon, or DuPont, WA for IA-64 Merced. Having grown up in Houston, Texas and not realizing that the Pacific Northwest even existed prior to this conversation, I naturally chose DuPont in order to be part of the 64-bit revolution.

Fast forward to February 1997. My wife and I moved to Olympia, WA. Given some of the, er, delay in Merced, I had the opportunity to pick up a Masters at the University of Washington

and at the same time work on developing getting our Itanium firmware ready. This included working on the System Abstraction Layer (SAL) http://www.intel.com/content/www/us/en/processors/itanium/itanium-system-abstraction-layer-specification.html with my BIOS hero/guru Sham D. in Hillsboro, along with Mani and Suresh in Santa Clara. The original boot flow entailed SAL-A for memory initialization, SAL-B for platform initialization and the "SAL_PROC" for the OS-visible API's to enable boot-loaders. The loader API into the firmware was a direct mapping of the PC/AT BIOS calls, with examples including instances like SAL_PROC 0x13 having a similar command set to int13h https://en.wikipedia.org/wiki/BIOS_interrupt_call.

As an arbitrary pedantic sidebar, you definitely see a pattern in firmware for 'phases' that typically include 'turn on memory,' 'turn on platform', and 'provide the boot loader environment.' Itanium had SAL-A, SAL-B, EFI. UEFI PI has SEC, PEI, DXE, BDS/TSL/UEFI API's. coreboot has bootblock, rom stage, ram stage, payload (including Seabios or UEFI or Depthcharge or ...). Power8 has hostboot, skiboot, and Petitboot (or EDKII UEFI). The workstation BIOS for IA-32 below had VM0, VM1, VM2, Furball. PC/AT BIOS has bootblock, POST, BIOS runtime. You see a pattern here?

Writing SAL_PROC code was pretty exciting. It could be invoked in virtual or physical mode. With hand-crafted Itanium assembly it was pretty reasonable to write position independent code (PIC) and use the GP register to discern where to find global data. But in moving to C, writing portable C code to abstract the SAL services was quite a feat. This is distinct from the UEFI runtime where were are callable in 1:1 mapping and then non-1:1 after the invocation of the SetVirtualAddress call by the OS kernel.

Regarding gaps with SAL_PROC as a boot firmware interface, as chronicled in page 8 of http://www.intel.com/content/dam/www/public/us/en/documents/research/2011-vol15-iss-1-intel-technology-journal.pdf, Intel created the Intel Boot Initiative (IBI) as a C-callable alternative. The original IBI specification looked a lot like ARC http://www.netbsd.org/docs/Hardware/Machines/ARC/riscspec.pdf. Ken R., a recent join to Intel from an MS (where he had a lot of DNA for ACPI), helped turn IBI into what we know as EFI 1.02, namely evolving IBI to have discoverable interfaces like protocols (think COM IUnknown::QueryInterface) and Task Priority Levels (think NT IRQLs), and of course the Camelcase coding style and use of CONTAINING_RECORD macro for information hiding of private data in our public protocol interface C structures. Many thanks to Ken.

Building out EFI was definitely evolutionary. It started from the 'top down' with EFI acting as that final phase/payload in the first instances with alternative platform initialization instances underneath. This view even informed the thema of 'booting from the top down' that informed how we sequenced the chapters in the 2006 Beyond BIOS book, for example. The initial usage of EFI was the 'sample implementation' built on top of the reference SAL code and a PC/AT BIOS invoked by the EFI 'thunk' drivers.

As we moved into the 2000's, the Intel Framework Specifications were defined in order to replace the SAL for Itanium and PC/AT BIOS for Itanium and IA-32, respectively. We internally referred to things like SAL + BIOS + EFI Sample as a "Franken-BIOS." The associated code base moved from the EFI Sample to the EFI Developer Kit, or EDKI, to distinguish it from the EDKII done in the later 2000's. This internal code-base was called 'Tiano', thus the name of community sites like http://www.tianocore.org.  Someone said the name came from the sailor with Columbus who first noticed America, but the only citation I could find publicly is the "Taino" tribe with whom Columbus engaged.

As a funny sidebar, I do recall the meeting where someone found "Tiano Island" http://pf.geoview.info/tiano,4033365 on the web. At the time it cost some number of millions of dollars. The original director of our team, numbering just a few engineers in the room, said 'let's each pool a couple percent of our stock options and buy the island.' I guess Stu had a much more significant equity position than I did, as a lowly grade 7 engineer.

In late 90's at DuPont, SAL and EFI sample were not the only code base activities. While in DuPont the erstwhile workstation group also created a clean-room replacement for the early boot flow. This started on IA32 and the OS interface was the PC/AT BIOS. For this effort we didn't have an image loader and instead just used non-1:1 GDT settings in order to run the protected mode code. For booting the protected mode code provisioned the 16-bit BIOS blob with information like the disk parameters, etc, so that the 16-bit code was just the 'runtime interface.' The 16-bit BIOS blob was called the 'furball' since we hoped to 'cough it up' once the industry had transitioned into a modern bootload erenvironment, such as EFI.

I still recall colleagues in the traditional business units yelling 'you'll never pass WHQL' with the above solution, but it did work. In fact, the work informed the subsequent interfaces and development with the Intel Framework Compatibility Support Module http://www.intel.com/content/www/us/en/architecture-and-technology/unified-extensible-firmware-interface/efi-compatibility-support-module-specification-v096.html.

We then ported the workstation code to boot the first Itanium workstation. I left that effort and joined the full EFI effort afterward. I recall the specific event which precipitated the decision. I was chatting with Sham and the workstation BIOS lead in the latter's cube. The lead said 'Now that we have our BIOS in modular code code "Plug-In Modules" (PIM's) we can tackle the option ROM problem. I thought to myself that just refactoring code into separate entities isn't the challenge in moving from PC/AT 16-bit option ROM's into a native format, it's all about the 'interfaces, namely how would a 'new' option ROM snap into a modern firmware infrastructure. IBI (now called EFI) was on that path to a solution, whereas a chunk of 'yet another codebase with PIM's' wasn't. Thus I was off to chatting with my friend Andrew, then lunch with Mark D, and onto the EFI quest in 1999. Quite the firmware long-game.

Next we're of finishing the first EFI, going from IBI to EFI .98 to EFI 1.02.

Next we're off on a cross-divisional team to create the '20 year BIOS replacement' called Tiano and the Intel Framework Specifications are born.

Next we solve the option ROM and driver problem with EFI 1.10.  Along the way between 1.10 and UEFI 2.0 we incubate a lot of future technology with the never release 'EFI 1.20' work.

Next Andrew Fish and I ported EDK to Intel 64. And I had fun with a port to XScale back in 2001. I have always enjoyed firmware bring-up on new CPU's.

Fast forward to 2005. The EFI specification became the UEFI 2.0 specification, and many of the Intel Framework Specifications became the UEFI Platform Initialization specification. Wrote the first EFI interface and platform spec for TPM measured boot https://github.com/vincentjzimmer/Documents/blob/master/EFIS004Fall06.pdf.

Fast forward to the 2010's.  More open source. More device types. More CPU ports. Continue to evolve network booting, such as IPV6 https://tools.ietf.org/rfc/rfc5970.txt and HTTP https://github.com/vincentjzimmer/Documents/blob/master/UEFI-Recovery-Options-002-1.pdf. Good stuff. Helped deliver UEFI Secure Boot https://github.com/vincentjzimmer/Documents/blob/master/SAM4542.pdf https://github.com/vincentjzimmer/Documents/blob/master/UEFI-Networking-and-Pre-OS-Security.pdf.

In parallel, I often had side firmware engagements, including a fun tour of duty helping our the solid state disk (SSD) team on firmware.

I still believe in better living through tools, too, whether they have landed in the community http://www.uefi.org/sites/default/files/resources/2014_UEFI_Plugfest_04_Intel.pdf, almost made it https://github.com/vincentjzimmer/Documents/blob/master/Vij_KRHZRWL_13.pdf https://github.com/termite2/Termite, or are in incubation https://www.usenix.org/system/files/conference/woot15/woot15-paper-bazhaniuk.pdf.

Fast forward to 2017. Year 20. It's still a lot of fun solving crossword puzzles with hardware and firmware.

During my time at Intel I've also appreciated the wisdom of others, whether through the mentoring of direct interaction or the written word. For the latter I heartily recommend keeping the following close at hand.

So am I done this morning? Let's do a final rewind to February 1992 when I jumped into industry in Houston. First I wrote firmware for embedded systems attached to natural gas pipelines http://www.emerson.com/resource/blob/133882/bb9c3232256dfab98cc6a20a27d43c1f/document-3-9000-309-data.pdf - sensors, serial protocols with radio interfaces to SCADA host, control algorithms, I2c pluggable expansion cards, loaders in microcontroller mask ROM's, porting a lot of evil assembly to C code...  fun stuff. The flow computer/Remote Telemetry Unit (RTU) work was an instance of the Internet of Things before the IOT was invented. Then on to industrial PC BIOS and management controller firmware. Then on to hardware RAID controllers and server BIOS. And then Intel in February 1997. 5 years of excitement in Houston prior to my Intel journey.

So I guess that sums out to 25. Now I feel tired. Time to stop blogging and playing the rewinding history game. Here's looking to the next 25.

Cheers.