"Advances in Platform Firmware 'Beyond BIOS'" - 20 years later.....

 A decade has passed since the posting https://vzimmer.blogspot.com/2014/01/advances-in-platform-firmware-beyond.html describing the journey of Tiano https://www.researchgate.net/publication/377810413_TechnologyIntel_Magazine_-_Advances_in_Platform_Firmware_Beyond_BIOS_and_Across_all_Intel_R_Silicon, which at the time was Intel Framework based specifications and the EDK implementation. As such, this month makes 20 years since the article landed on the internet.

The write-up was curiously available in several languages:

    English - https://github.com/vincentjzimmer/Documents/blob/master/it01043_eng.pdf

    Portuguese - https://github.com/vincentjzimmer/Documents/blob/master/it01043_pg.pdf

    Japanese - https://github.com/vincentjzimmer/Documents/blob/master/it01043_j.pdf

    Russian - https://github.com/vincentjzimmer/Documents/blob/master/it01043_ru.pdf

    Spanish - https://github.com/vincentjzimmer/Documents/blob/master/it01043_sp.pdf

    Chinese - https://github.com/vincentjzimmer/Documents/blob/master/it01043_cn.pdf 

Today the EDK https://sourceforge.net/projects/edkit/ has become https://github.com/tianocore/edk2 and the Framework https://www.intel.com/content/www/us/en/architecture-and-technology/unified-extensible-firmware-interface/efi-specifications-general-technology.html corpus is UEFI PI https://uefi.org/specs/PI/1.8/index.html. The OS-visible interface has gone from EFI 1.10 https://www.intel.com/content/dam/doc/product-specification/efi-v1-10-specification.pdf to UEFI https://uefi.org/specs/UEFI/2.10/, too. 

Given the amount of change over this interval, I always have a tough time refreshing the graphic 

Many interesting twists and turns on this journey that started in the late 90's and still continues as we step into the year 2024.

October firmware events

Apropos of the 25 year anniversary of the first IBI/EFI/UEFI boot services at week ago, as commemorated at the 20 year milestone in https://vzimmer.blogspot.com/2018/10/  

                                                        "Ken Reneris     Oct-14-1998"  

there was a question of the coding standards for UEFI https://twitter.com/bexcran/status/1701071379171619236 recently which made me think of Ken. I mentioned how that since Ken had come over to Intel from the Windows NT team at Microsoft it was natural to adopt the NT coding standard https://computernewb.com/~lily/files/Documents/NTDesignWorkbook/coding.pdf for the original IBI/EFI/UEFI work. This includes the EFI_ERROR macros, inf's, CR macro, TPLs as simplified IRQLs https://vzimmer.blogspot.com/2015/06/guids-revisions-interrupts.html, etc. This made IBI/EFI/UEFI a sort of a cultural 'Windows BIOS' given that lineage. 

And Ken as the MS HAL owner continually interfaced with the platform. In fact, standards like ACPI pre-dated IBI/EFI/UEFI by starting in the mid-90's. Ken was deep into that work, too, as evidenced by his solo inventor ship of the now-expired S4 resume patent https://patents.google.com/patent/US6209088B1/en. 

These UEFI technology elements serve a counterpoint with coreboot, which was LinuxBIOS in 99. The latter have technology elements https://www2.cs.arizona.edu/~mccann/cstyle.html, KConfig, etc.  

https://link.springer.com/book/10.1007/978-1-4842-0070-4. So just as we have had parallel growth of Windows and Linux, there have been threads of 'Windows BIOS' (e.g., EDKII UEFI based) and 'LinuxBIOS' (e.g., coreboot and maybe less so u-boot?).

But then again, who uses 'BIOS' https://basicinputoutput.com/2014/08/will-i-be-jailed-for-saying-uefi-bios.html? 

So the blog title is firmware events. Over the last couple of weeks there was a UEFI plugfest in the Portland area and a Open Compute Project (OCP) event in San Jose. As I drove back from the former, I stopped by a bookstore and saw https://www.amazon.com/Beyond-BIOS-Developing-Extensible-Interface/dp/1501514784

on the shelf.

On that same shelf I saw 

and

Though not the immediate book neighbor, the nearby https://www.amazon.com/ARM-Architecture-Reference-Manual-2nd/dp/0201737191 reminded me of Dave Jaggar. Back in the mid-1990's we were development hardware RAID controllers at Compaq and using custom ASICs alongside an AMD 29k RISC CPU. I recall Jaggar from ARM coming to visit and explaining the ARM architecture. Part of the discourse included showing a silicon die with a small portion highlighted for the ARM core itself. It was quite a shock for me to so such small mm-squared for a CPU. My firmware lead (lead == sole design & developer) work on the SMART-2SL 

https://www.amazon.com/Compaq-242777-001-WIDE-SCSI-CONTROLLER-242777001/dp/B0002FDXLQ is but a memory to be rekindled occasionally by seeing aftermarket examples of this device. In addition to being the first device to not have a non-volatile post write cache, this work gave me the opportunity to do some interesting firmware performance innovations with a colleague https://patents.google.com/patent/US6341342B1/en. 

Speaking of CPQ, it has been a strange migration of companies for me. My original internship was with Texaco, which in turn was purchased by Chevron. Then I did my first full-time firmware development at Daniel Industries, later acquired by Emerson Electric. My first foray into PC BIOS was at Texas Microsystems (TMI) working on industrial computers, which then was acquired by Radisys. Finally, my pre-Intel employer Compaq server group was in turn acquired by Hewlett-Packard, and then split into the enterprise side HPe. Luckily Intel is still Intel.

So speaking of the development event, it spanned 3 days. I presented on the first day (lower left image)

https://uefi.org/sites/default/files/resources/Tuesday_02_Kubacki%20and%20Zimmer_Final.pdf and final

day (rightmost image) https://uefi.org/sites/default/files/resources/Firmware%20Configuration%20%E2%80%93%20Past%2C%20Present%2C%20and%20Future_Zimmer.pdf. The first day of the event included a description of SPDM https://www.dmtf.org/standards/spdm and its support introduced into the UEFI 2.10 specification. SPDM is homed at the DMTF but has associated work-product having in groups like UEFI and the Trusted Computing Group (TCG). 

In the Insyde talk Tim Lewis mentioned the growth of attacks on the UEFI network stack after many years of battering SMM. This reminded me of a recent posting I saw https://www.linaro.org/blog/ledge-blogs-uefi-http-and-https-boot-in-u-boot/ where the U-Boot community discussing using https://savannah.nongnu.org/projects/lwip/ which is a similar approach taken in https://github.com/tianocore/edk2-staging/tree/MpNetworkStack mentioned in https://uefi.org/sites/default/files/resources/7_Maciej%20Vincent_INTEL_network%20stack%20performance.pdf. From the days of discussing HTTP booting in 2009 https://dblp.org/rec/conf/csreaSAM/Zimmer09.html?view=bibtex and having the URL boot option mentioned in https://www.rfc-editor.org/rfc/rfc5970.txt has come a long way. Just as EDKII consumes cryptography as a submodule from another community, maybe it's time to do so for the basic networking capabilities.

Given that AWS started in 2006, it may have a bit short-sighted of me to have said "...emergent compute models such as cloud computing." in that SAM 09 paper.

Maybe I was keying off the 09 date of what I thought was the definitive paper on the cloud, viz., https://www2.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-28.pdf.

So back to the topic of this blog, namely firmware events. On that topic, the next week came the Open Compute Project (OCP) event in San Jose. I first presented at OCP in 2015 with Mallik Bulusu https://cdrdv2.intel.com/v1/dl/getContent/671185 

and then again in 2016 on firmware updates http://files.opencompute.org/oc/public.php?service=files&t=1f7831234dce58bb875b1b5b24f7154d. 

This session last week was on the universal payload

I have been engaged on server platforms since I was hired into Intel in February 1997 to lead the IA-64 (Merced, Itanium) firmware. Along the way we devised ways to facilitate ease of firmware development, like multi-socket cache-as-RAM https://patents.google.com/patent/US7254676B2/en 

and similar for IA-32 https://patents.google.com/patent/US8078862B2/en. Palsamy and I also collaborated on UEFI and ACPI for RAS and error support https://cdrdv2.intel.com/v1/dl/getContent/671067. 

And all of these threads come together in these recent talks. The SPDM prezo mentioned at the top of the posting here entail facing the post-quantum cryptographic migrations like all of the other standards, including UEFI https://uefi.org/, TCG https://trustedcomputinggroup.org/, and others. The proposal for augmenting UEFI is mentioned in https://bugzilla.tianocore.org/show_bug.cgi?id=4087 and some prezos on the topic can be found at https://uefi.org/sites/default/files/resources/Post%20Quantum%20Webinar.pdf and https://uefi.org/sites/default/files/resources/USF_Security_Webinar_Final.pdf. The specific study on SPDM was posted as a pre-print to https://eprint.iacr.org/2022/1049 and then a journal submission in https://www.mdpi.com/2410-387X/6/4/48. 

Since SPDM has fixed message sizes, the concept of 'chunking' or breaking up the larger payloads demanded by these post-quantum algorithms is a common concern other hardware-based messaging interfaces will face, like the Trusted Computing Group's Trusted Platform Module (TPM). Interestingly this topic of post-quantum impacts on firmware standards that motivated the SPDM paper

 listed above was inspired by the study https://eprint.iacr.org/2021/041 from Cisco.

And during Q/A with the CISA presentation during the UEFI developer event I asked about formal methods for this domain. Since this talk followed the SPDM talk one recommendation was to use formal for the SPDM wire protocol. It turns out the above MDPI SPDM paper is referenced by a few formal studies of SPDM, including https://www.usenix.org/conference/usenixsecurity23/presentation/cremers-spdm and https://ieeexplore.ieee.org/abstract/document/10149352/. 

And speaking of security and standards, this upcoming week is the TCG members meeting at the Google campus in Kirkland, WA. I am Intel's representative on the Technical Committee, assuming that role after Kirk Brannock retired. 

 

But the TCG is not unfamiliar to me. I have been engaged with TCG https://en.wikipedia.org/wiki/Trusted_Computing_Group even when it was called TCPA, as evidenced by work-product like https://patents.google.com/patent/US7200758B2/en and delivering the Itanium and EFI API and platform specification, respectively.

When reading https://research.tue.nl/en/publications/communication-in-a-world-of-pervasive-surveillance-sources-and-me from the mention of Jacob in https://www-theregister-com.cdn.ampproject.org/c/s/www.theregister.com/AMP/2023/09/19/marvell_disputes_claim_that_cavium/, I hearkened back to my first ToorCamp presentation 

Based upon the slide

and discussion around the TPM, Jacob told me that we should design a TPM so that it could quickly be removed from board and chewed up/swallowed if someone tries to take your computer. Quite the privacy-preserving dietary strategy.

It's interesting to see people in person after the years of COVID seclusion and event cancellation or wholly-virtual events. Just like the psychic shock and fatigue of crowds, continual interaction and noise, it is taking a bit of effort to get re-acclimated. Part of the symptomology is falling behind on blogging, I guess, since I had intended to post this entry on the day of the EFI 25-year anniversary, not a week later. Oh well. Only so many free moments on the weekends these days.

And the world keeps changing

 I'm a long-time fan on Twitter. I still recall merging onto I-5 years ago and listening to some tech podcast about this new 'micro-blogging service', namely Twitter. I still keep up with https://twitter.com/vincentzimmer and don't worry so much about the vagaries of ownership or management. But I was intrigued by the discussion of other services like https://mastodon.social/explore. As such I setup an account and posting some material akin to what I saw from another Twitter person regarding some personal metrics for 2022 https://mas.to/@vincentzimmer/109549960566794774, including a reply to myself with another random observation this week https://mas.to/@vincentzimmer/109550056099262201.

This is my first post to mastodon: 

"Interesting 2022. First journal pub https://www.mdpi.com/2410-387X/6/4/48 although waiting for red box on https://dblp.uni-trier.de/pid/34/5641.html. I also had oppty to collaborate on an IEEE conference http://www.ieee-smart-world.org/2022/uic/index.php mentioned in http://www.ieee-smart-world.org/2022/uic/uic-2022.htm. Then a couple of unrefereed items https://embeddedcomputing.com/technology/security/software-security/understanding-uefi-firmware-update-and-its-vital-role-in-keeping-computing-systems-secure and https://eprint.iacr.org/2022/1049. Then 2 books https://link.springer.com/book/10.1007/978-1-4842-7939-7 and https://link.springer.com/book/10.1007/978-1-4842-7974-8. A podcast https://www.youtube.com/watch?v=wqcUWAEHcVg. 6 US patent filings and 7 issued.  Whew."

and

"Speaking of https://eprint.iacr.org/2022/1724, it was referenced by https://eprint.iacr.org/2022/1724. 

Latter doesn't read in on PQC aspects of SPDM but provides a formal verification of extant SPDM protocol using a theorem prover https://github.com/tamarin-prover/tamarin-prover with public proofs https://github.com/AnalysisSPDM/FormalModel. Perhaps this is a use-case to leverage for achieving a long-held ambition, namely axiomatize and 'prove' some tricky parts of UEFI, viz https://uefi.org/specs/UEFI/2.10/08_Services_Runtime_Services.html?highlight=authenticated#setvariable and EFI_VARIABLE_TIME_BASED_AUTHENTICATED_WRITE_ACCESS?"

I try to be pragmatic about formal. I recall reading once that even in the days of the Orange books with A-level certification requiring 'formal proof' it was hard to achieve, but the attempt at least 'provided better documentation.' And at a professional level I've always mutated the 'culture trumps strategy' aphorism to be 'implementation trumps architecture.' A painful reminder of this dichotomy is having studied https://www.cl.cam.ac.uk/~lp15/papers/Auth/tls.pdf back in the day and then meeting Marsh Ray and learning of his work https://troopers.de/events/troopers10/242_history_of_the_tls_authentication_gap_bug/. Networking has been close to my heart for a couple decades as I helped grow our EFI network stack from its nascent PXE equivalent in 1999 through IPV6 https://www.rfc-editor.org/rfc/rfc5970.html and into HTTP and TLS (e.g., HTTP-S boot). I still recall reading the Rescorla book on SSL/TLS while waiting for my older daughter to be born in 1999 at the University of Washington hospital.

Perhaps if I have the opportunity in the future to use my sabbatical

I can do some thinning of the archives. Rescorla and Rudin have aged well, but I suspect I can sunset the Java duo of Aglets and JavaOS. On the other hand, though, I'm a fan of keeping around some documents of 'old tech' since there is often wisdom to be gleaned from the past.  Hmm. Decisions, decision.

Well, to close on formal, specifications, and networking, I should tie off that thought at least with a re-invocation of as always, there is no 'silver bullet.' 

These mastodon posts feel like my usual terse blog posts. Perhaps my posts are 2xM or 3xM (i.e., 2 or 3 times the text length of a Mastodon post).

This blog is usually pretty low traffic, too. The sources of eyes are either google searches or link via https://www.amazon.com/stores/Vincent-Zimmer/author/B002I6IW4A (which included the blog links) or blogroll like https://blogs.coreboot.org/. 

With the removal of the former, I suspect there will be less traffic. That should be fine. I don't do sponsored ads or other revenue-generating activities here. It's more of a personal set of footprints in the sand to chronicle the journey.  

And luckily https://en.wikipedia.org/wiki/List_of_prolific_inventors has been curated to just the top 100, so watching my slow descent down the leader board, such as http://vzimmer.blogspot.com/2022/09/new-milestones.html and its earlier ilk, are no long a nagging distraction.

Ah well, so much for a mastodon experiment and clone+amplification to my OG blog. Back to work. Need to figure out how to add an abstraction service to code I wrote 20 years ago before 9am tomorrow....

Cheers

Re-use - ideas, et al.

This is a quick note on 're-use'.  Recall how UEFI was derived from EFI, which in turn started as IBI, as mentioned in in http://vzimmer.blogspot.com/2018/10/ghosts-of.html and various histories of EFI in books http://vzimmer.blogspot.com/2021/01/books-and-computers.html and papers https://www.intel.com/content/dam/www/public/us/en/documents/research/2011-vol15-iss-1-intel-technology-journal.pdf. 

As a background, the original Intel Boot Initiative had its own filesystem called "IBIFS". It was a simple log-structured filesystem that made for easy parsing. But the challenge with this filesystem was how to provision boot media from other operating systems, such as Windows. In the 1990's MS-FAT was the prevailing filesystem although it was much more complex than IBIFS. The IBI/EFI team was encourage to adopt MS-FAT https://download.microsoft.com/download/1/6/1/161ba512-40e2-4cc9-843a-923143f3456c/fatgen103.doc, along with PE/COFF https://docs.microsoft.com/en-us/windows/win32/debug/pe-format executables, as the filesystem and executable format, respectively. This was supported by having a license to use these specifications for production of EFI firmware. 

The other advantage of using MS filesystem included MS providing tools to format, check and provision disks https://www.intel.com/content/www/us/en/download/714351/uefi-shell-disk-utilities.html whose sources were derived from MS Installable Filesystem (IFS) C++ kit provided to OEM's. This was my first dive into C++-on-a-C-framework (i.e., no global constructors, etc).

And PE/COFF allowed for using Visual Studio and commercial linkers. At the time we had less luck in getting PDB format and KD wire protocol made public. We wouldn't open source the EFI Debug support protocol and nub based upon reversing that infrastructure, of course. 

Of course using ANSI C with PE/COFF was more advantageous than using non-standard art like the coreboot romcc that proved brittle to maintain long-term, although it was a clever solution prior to the advent of cache-as-ram (CAR) https://www.coreboot.org/data/yhlu/cache_as_ram_lb_09142006.pdf. Prior to CAR we had to perform unnatural acts using MMX registers and other on-processor resources (with the peak weirdness of PEI CIS 0.3 and its register 'call levels') in order to implement pre-permanent memory flows like DRAM initialization in the host firmware. 

I was reminded of this logic of re-use based upon a few recent topics, including USF https://github.com/universalscalablefirmware and EDK2 GSoC conversation. On the USF topic, there is a discussion of using a self-describing format like CBOR https://www.rfc-editor.org/rfc/rfc8949.html in lieu of the UEFI HOB's https://universalscalablefirmware.groups.io/g/discussion/files/New%20data%20format%20for%20Universal%20Payload%20Interface.pdf. Even though I helped create HOB's, I am one of the first to support moving to a more standard format, especially given the existence of work like https://github.com/intel/tinycbor. To me this is the same spirit of using known filesystem formats, etc.

When asked about this I often cite the economics of host firmware development. Imagine a 5,000 person host firmware producing community across open source participants, OEM's, ODM's, firmware vendors, first party device creators, etc.  Recall the supply chain picture from erstwhile 2015 CanSecWest prezo

As such, compare this to the application and OS producing community that is easily orders of magnitude larger. Given the developer community size disparity it is difficult to justify the R&D and validation expense for bespoke host firmware-only solutions. Given NIH sometimes in engineering communities I also quote "Good artists invent, great artists steal" Picasso  https://medium.com/ben-shoemate/what-does-it-mean-good-artists-copy-great-artists-steal-ee8fd85317a0.

And given challenges of HOB's I understand the critique of Terse Executable (TE) during gsoc thread "[edk2-discuss] GSoC Proposala replacement for the TE format (it’s buggy and most platforms mostly abandoned it over various issues)," I'm not sure where he received his telemetry on 'abandoned' for working systems. Scraping FD images from the internet? Similar to some of the weird history on those GSoC threads like "PEI and DXE in 1998" when the latter were created in early 2000's as derivatives of EFI into the earlier boot flow. This TE assertion is close to home given the TE image format signature has 'vz' as opposed to PE/COFF 'mz', as originally described in https://www.amazon.com/Beyond-Bios-Implementing-Extensible-Interface/dp/0974364908

So has the TE image been relevant in the industry? Let's do some rough back-of-the-envelope calculations. To begin, the TE image format is easily over 20 years old. In the spirit of the last blog on statistics I'd sat we started to scale Framework EDKII in 2005

Imagine in the ensuing years it's tough to calculate the number of PC's, servers, and clients that shipped using UEFI PI-based firmware, including #'s like https://www.statista.com/statistics/576151/unit-shipments-pcs-united-states/. For Windows8 UEFI was required so you can argue 100% at that point. 

To take a swag lets assume 250 million units / year over the last 17 years, yielding 4.25 billion units. Let's also assume the fabrication of the PEI phase pre-memory uses TE's and that each fv/ibb/fvrecovery has ~ 50 peim's compiled with TE. So that would yield the market has shipped 50 * 4.25  =  212.5 billion TE images into the market. And if platform updated 5 times during life, 1 trillion TE's may have been produced. So even if we go 'Beyond TE', 1T has been a good run. And I look forward to the next wave of firmware technologists' proposals - the critique is a pre-condition to invention/innovation - but the next wave needs to close the loop by proposing, documenting, coding, delivering, validating, and scaling their alternative. 

I still recall meeting one of the original developers of MS-FAT in Windows at a driver dev-con in the early 2000's in Redmond. He mentioned that 'writing the code was only 10% of the effort.' The latter class of activities were the remaining 90% for a professional software engineer. Or the old trope 'the last 10% of a project takes 90% of the time.'

Finally, these cardinality of the produced TE image set calculations remind me of a colleague at Cornell during my undergrad. He worked part-time in the astrophysics department, whereas I spent time modeling and mocking up particle beams for a professor who did microwave and plasma research as part of SDI https://en.wikipedia.org/wiki/Strategic_Defense_Initiative. My friend noted that the astrophysics researchers there would reckon calculations plus or minus a couple orders of magnitude. I won't argue host firmware is of the same scale but it does note the challenges in precision in various domains.

Sometimes folks believe in bespoke and not studying the past arc of history in tech. I recommend spending some time on bitsavers.org, for example, to folks. But it reminds me of the advice given to me by a retired manager - "Sometimes you see someone put their hand in the chipper. You can say 'there's a chipper, there's a chipper', but they ignore you. Once they are beyond their elbow you need to jump in, at least." Or the dual of child rearing, which was "Sometimes the young child needs to bump her knees on the coffee table while learning to walk."  For those not in the PNW a chipper is https://www.familyhandyman.com/list/best-wood-chippers/.

BTW
I just sampled some of the OCP Security https://www.opencompute.org/wiki/Security tech talks. Work to avoid e-waste such as ownership transfer https://docs.google.com/document/d/1oANhjvv_R7E5n8w1RroN8l8-0jdYlfdQDp_3RqGV66k/edit# for servers presented by Google datacenter engineer nicely complements their work on Chromebooks with their developer mode https://www.chromium.org/chromium-os/chromiumos-design-docs/developer-mode/ and other art to allow for unsupported devices to be over-flashed by community firmware for continued life & usage. Nice to see work to avoid aged hardware from only have a fate of visiting the refuse bin/salvage yard.

It might be a stretch to analogize re-use of ideas in this blog post and re-use of hardware in the circular economy from this OCP talk. The former stands on the shoulders of giants and leverages information ecosystems (knowledge, tools, remediated flaws) whereas the latter leverages extant hardware for appropriate use-cases. But interesting stuff nevertheless.

fin (98)

wither network boot?

I thought about pre-OS networking recently after giving a talk to other engineers about the value of EQ + IQ recently.  For the latter, what I called "Technical IQ" in http://vzimmer.blogspot.com/2013/03/a-technical-career-path.html, I use the T-shaped model https://careeredge.bentley.edu/blog/2015/04/21/how-gaining-t-shaped-skills-will-give-you-an-edge/. For this model networking was in the depth or vertical bar of my T since evolving the EFI network stack from a simple legacy PXE & UNDI thunk wrapper in 1999 to the native mode implementation in the early 2000's, certifying a clean-room IPV6 stack, and further into scalable use cases with wireless and HTTP boot a half decade ago. In the late days of 2021 perhaps it's not as prominent in the vertical bar but it's an important capability nevertheless.

As part of this survey, this blog is an elaboration upon some arbitrary points on the timeline of UEFI firmware network boot history, in the spirit of older postings like http://vzimmer.blogspot.com/2013/02/anniversary-daynext-arch-ps-and-some.html. This latter posting mentioned the 1.2 work https://github.com/vincentjzimmer/Documents/blob/master/EFIS001_100_2004.pdf, including 

client authentication w/ EAP https://www.semanticscholar.org/paper/Cloud-Net-Booting-Beyond-BIOS-Using-the-Unified-Zimmer/8f158dd172ca406095d051cc9bb5a7f5cc09435b and the folly of trying to create bespoke authentication methods. 

This same work was described in the 2009 https://github.com/vincentjzimmer/Documents/blob/master/SAM6560.pdf  paper https://dblp.uni-trier.de/rec/conf/csreaSAM/Zimmer09.html?view=bibtex in 2009, the same year as Berkeley Cloud paper that it cited https://www2.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-28.pdf. At the time is was becoming increasingly obvious that a HTTP-based boot was necessary. We in fact left provision for the same in https://tools.ietf.org/rfc/rfc5970.txt. 

Page 97 of https://www.intel.com/content/dam/www/public/us/en/documents/research/2011-vol15-iss-1-intel-technology-journal.pdf notes the full move to standard network authentication protocols.

Another place I captured some history of pre-OS UEFI networking was in https://github.com/vincentjzimmer/Documents/raw/master/A-Quick-History-of-UEFI-Networking.pdf. This was just a quick background that I ultimately integrated into https://github.com/vincentjzimmer/Documents/blob/master/UEFI-Recovery-Options-002-1.pdf in 2016. This folds in advances such as UEFI wireless support and HTTP boot that I led through the UEFI Network (UNST) & Security (USST) subteams https://uefi.org/sites/default/files/resources/UEFI_Plugfest_VZimmer_Fall_2016.pdf, respectively, in the early portion of that decade.  Commercial usage of HTTP boot was described by HP enterprise, too

https://www.uefi.org/sites/default/files/resources/FINAL%20Pres4%20UEFI%20HTTP%20Boot.pdf.

Although https://www.uefi.org/sites/default/files/resources/UEFI_Plugfest_VZimmer_Fall_2016.pdf updates  after wireless UEFI boot, the suggested next paradigm of pre-OS network bootstrap of  'NVME over Fabric UEFI boot' has been evolved in the NVME standards group versus my UNST in UEFI  https://linuxplumbersconf.org/event/7/contributions/737/attachments/531/944/LPC_2020_-_NVMe-oF_Boot_from_Ethernet_-_Final.pdf. I discovered that working the standard closer to the subject matter experts is best, just as I created the EFI TCG Protocols http://people.eecs.berkeley.edu/~kubitron/courses/cs194-24-S14/hand-outs/SF09_EFIS001_UEFI_PI_TCG_White_Paper.pdf in the Trusted Computing Group versus the UEFI Forum.  

At other times it is satisfying to see evolution of requirements that support important use cases. There has been a spate of activity in enabling https://csrc.nist.gov/publications/detail/sp/800-193/final over the last several years, including the low level infrastructure code like https://github.com/vincentjzimmer/Documents/blob/master/A_Tour_Beyond_BIOS_Capsule_Update_and_Recovery_in_EDKII.pdf. The idea therein entails having the platform having some degree of fault tolerance on the system board firmware and device firmware.

Beyond 193, though, there are scenarios where the operating system can become corrupted, such as infamous https://www.tomshardware.com/news/malware-shamoon-virus-security-disttrack,16988.html. There are some early thoughts about options here https://www.it-scc.org/uploads/4/7/2/3/47232717/requirements_for_recoverable_systems.pdf that happen to nicely match some of the OS recovery infrastructure defined in https://github.com/vincentjzimmer/Documents/blob/master/UEFI-Recovery-Options-002-1.pdf, especially figure 7 of the latter mapped to figure 2 of the former

which not surprisingly bear resemblance to 

During that effort above on OS recovery it was noted that although we defined UEFI network support in the UEFI standard and some infrastructure for the same in https://github.com/tianocore/edk2/blob/master/MdePkg/Include/Protocol/Supplicant.h, great support for multi device and protocol was available in https://ipxe.org. In fact, ipxe has historically been a favorite of datacenter operators using firmware based networking, including the ipxe scripting https://ipxe.org/scripting. In fact after talking with some hyperscalers in the mid 2000's, I tried to create a 'safe shell' subset to use .nsh in lieu of ipxe scripting for deployment w/ UEFI network stack to provide coequal, and in the limit script-compatible behavior between ipxe and UEFI. This didn't work out for various technical and community reasons.

This usage of ipxe in the pre-OS speaks to the challenge of OS-absent networking. There has always been a complain that UEFI didn't solve one of the fundamental systems problems, namely needing to write drivers twice: once for the boot environment, and once for the OS. Vertically integrated folks like Intel UEFI Apple Macs can have a OS kernel engineer and firmware engineer who is a domain expert do both, but in the horizontal industry where the OS providers and platform provider are different business entities, this is more difficult. And historically the OS folks eschewed using firmware at runtime, even for safe mode, which led to deprecating UNDI at runtime.

In the past some of the DEC Alpha platform manufacturers wrapped Windows NT NDIS drivers in the Arc firmware, but this was only the port level, datagram interface.  The higher level networking primitives had to be curated for each domain (pre-OS, OS runtime). Other challenges in pre-OS networking include security - the support of packets on the wire (or air with wireless) are a huge attack surface since the incoming packets are ostensibly attacker-controlled data. The use of hardware VPN's can ameliorate some of this concern for deployments in the traditional enterprise, but the world of borderless/zero trust architectures moots that argument. 

One potential approach to more robustness may include using language-based security, such as Rust and implementations such as smol https://github.com/smoltcp-rs/smoltcp? In the early 2000's I explored with Linux kernel EFI maintainer Matt Fleming on some options about encapsulating Linux in the pre-OS for the network stack and then returning to the UEFI environment to boot the downloaded image. There were a lot of intricacies about state of the platform in absence of exit boot services and other blockers that put that exploration on the shelf.  At the time there were already a lot of usages of embedding Linux in the flash as the boot environment, but it was a one-way gate from UEFI.  UEFI->Linux.  Never a UEFI->Linux->UEFI->Final OS.

It is nice to see that latter use-case having been re-invigorated in the last few years with https://www.linuxboot.org/ in the datacenter. Instead of UEFI netboot->Linux or the chain loading of UEFI netboot->ipxe->Linux, you have the platform firmware, UEFI or coreboot or slimboot or....based, directly launch Linux and its integrated networking support from the system board flash. 

In addition to the interesting works Trammel Hudson has done in firmware security, I am happy to see his work in this space of using Linux for pre-OS networking https://trmm.net/LinuxBoot_34c3/. This includes a recent patch to run interleave UEFI and Linux execution, or the design point Fleming and I explored years ago, viz.,

The other historical irony I have to mention about LinuxBoot and firmware relates to a conversation I had on the showcase floor at the Ubuntu Developer Summit in Oakland, CA 2012. My colleague brought Mark Shuttleworth over to have a conversation, ostensibly about UEFI secure boot and some of the other recent features I had been working on in this space. Shuttleworth nodded his head in response to the overview and then left me with the single question: "Why don't you just use Linux as the firmware to boot the operating system?"  Of course supply chain, horizontal industry COTs, .... and the myriad of other social, technical and business reasons from my confirmation bias vantage of working on UEFI and EDKII didn't really empower me to have a quick answer :)

So we mention LinuxBoot above which obviates the need for the UEFI network stack in some use-cases, especially Cloud. That doesn't mean that UEFI networking isn't important. In fact enterprise and client devices still deploy the capability today, and it definitely has challenges, including performance and robustness. On the former, we noted some of the glass jaws on the polled UEFI driver model and its impact on perf in https://uefi.org/sites/default/files/resources/7_Maciej%20Vincent_INTEL_network%20stack%20performance.pdf and the associated implementation using LWIP and multiprocessing https://github.com/tianocore/edk2-staging/tree/MpNetworkStack. For security we can go beyond just the forklift upgrade of a Rust port and near-term isolate the network stack drivers in ring 3 or user mode https://github.com/jyao1/SecurityEx/tree/master/UserModePkg or put into a VM https://www.intel.com/content/dam/develop/external/us/en/documents/a-tour-beyond-bios-launching-vmm-in-efi-developer-kit-ii-0-819978.pdf. In fact after the 2007 paper laying the ground work for UEFI secure boot in SAM07 https://www.semanticscholar.org/paper/Platform-Trust-Beyond-BIOS-Using-the-Unified-Zimmer/0bd3bdeb6dcadf088137e13c00adc7e4390fa0de was 2008 https://www.semanticscholar.org/paper/System-Isolation-Beyond-BIOS-using-the-Unified-Zimmer/cf0261fe8d8dc078fb389dc04a56188695581949 including VMM and rings for firmware. 

Firmware device interrupts http://vzimmer.blogspot.com/2015/06/guids-revisions-interrupts.html, ring isolation, multiprocessing, etc all seem to argue for just diving into a full operating system, but a deft hand can still apply some of these design precepts into pre-OS firmware. Driving change is hard, though. I recall a quote from a colleague about another engineer 'he spent a lot of time working on a problem that engineers (at the company) didn't want solved.' In other words, a good business problem isn't enough. For various historical and / or cultural reasons the existing engineering community may not want to pursue a solution. A mental model I sometimes use for this engineering reluctance is WWI/WWII. Leaders who came up in some technology era understand the guardrails and technologies for success there - mapping to WWI they are great trench builders and trench warfare combatants. And the ultimate example of trench warfare skills was the Maginot line https://en.wikipedia.org/wiki/Maginot_Line.

Technology changes. Say going from mainframe to mini, mini to PC, PC to Cloud/post-PC, anything to mobile, ...... A phase and its successor may represent a different corpus of technology or business constraints. Let's call one of these transitions going from a WWI to WWII based technology environment. When WWII commenced, airplanes just flew over the Maginot line. Just like your real competition is yourself professionally, not others, the way to leverage this insight is to avoid trying to leverage your trench-digging skills when the world war precepts have changed, and more importantly continually evolve skills and practices. As for the sentiment in the quotation above, a technologists should take a constructive approach that includes providing data on the environment change and technical options that can inform the organization of potentially evolve the bench from WWI to WWII class readiness.

OK. Enough on this topic of the past. If there are any more postings in '22 they will catch up to more recent events and commentary in the firmware space. 

Different boot and testing schemes

In the news, interesting to see the details of the M1-based Apple devices. As a firmware person, I was curious about how the machines boot. 

They mention the omission of bootcamp https://support.apple.com/boot-camp https://discussions.apple.com/thread/252021896 or the support of 3rd party OS's https://www.zdnet.com/article/top-apple-exec-native-windows-10-on-m1-macs-is-a-choice-microsoft-needs-to-make/.

Does that mean the boot paradigm of the Macs has changed from UEFI http://refit.sourceforge.net/info/boot_process.html? Apple was never fully UEFI compliant and often eschewed more complex capabilities like SMM-protected Authenticated Variables and other rich runtime services, although they did support ACPI.

The answer was formally revealed in the video https://developer.apple.com/videos/play/wwdc2020/10686 at 16:22

which mentions that the boot process is leveraged from the iPhone/iPad, namely iBoot https://en.wikipedia.org/wiki/IBoot. 

The native iBoot launch was confirmed by Apple UEFI engineer https://twitter.com/NikolajSchlej/status/1275951574200709120, along with using device tree https://elinux.org/Device_Tree_What_It_Is in lieu of ACPI https://twitter.com/NikolajSchlej/status/1275951827754774528. Flattened Device Tree predates ACPI and was part of how the original PowerPC MACs booted, with OpenFirmware Forth-based FCode and FDT as the equivalent of UEFI/ACPI for the programmatic and declarative runtime tables interfaces, respectively. Open Firmware was circa 1994 https://en.wikipedia.org/wiki/Open_Firmware whereas ACPI was 1996 https://en.wikipedia.org/wiki/Advanced_Configuration_and_Power_Interface#:~:text=First%20released%20in%20December%201996,Play%20BIOS%20(PnP)%20Specification.&text=ACPI%20defines%20a%20hardware%20abstraction,UEFI)%20and%20the%20operating%20systems., respectively. EFI, the precursor the UEFI, is the youngest of all three, dating its earliest appearance in 1998 https://en.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface.

As Apple showed in slide 14, the T2-based Mac's already support the bridgeOS kernel going from iBoot to UEFI. This sort of layering of UEFI implementations on top of a non-PI based platform code is pretty common, with the https://github.com/tianocore/edk2/tree/master/UefiPayloadPkg providing an EDK2-based implementation of UEFI for coreboot https://doc.coreboot.org/ and slim bootloader https://slimbootloader.github.io/index.html. We discussed this in figure 6-17 of https://link.springer.com/chapter/10.1007/978-1-4842-0070-4_6 and we are trying to generalize the use of payloads in https://github.com/universalpayload. We compare a T2-based secure boot chain to other implementations in chapter 4 of https://www.amazon.com/Building-Secure-Firmware-Armoring-Foundation/dp/1484261054, too.

I suspect the disk layout is still based upon the GUIDed Partition Table (GPT), which was an invention of EFI. You can support GPT even without performing a non-UEFI OS boot, as showed by Chromebooks https://www.chromium.org/chromium-os/chromiumos-design-docs/disk-format, for example. To support a non-UEFI OS you simple can omit the EFI System Partition (ESP) and have your boot loader load a kernel from some arbitrary GUID-defined partition type.

We also show how the concept of a phase-based boot and clean interfaces to payloads enables more radical firmware stacks, such as slide 24 of https://cfp.osfc.io/media/osfc2020/submissions/SLFJTN/resources/OSFC2020_Rust_EFI_Yao_Zimmer_NDK4Dme.pdf. 

 

Another interesting example of stacking a Rust-based UEFI implementation on an alternate platform initialization layer can be found in https://github.com/retrage/rust-hypervisor-firmware/tree/coreboot-support, too, that I saw on the open source firmware's oreboot slack channel. 

Speaking of another A-prefixed company that's not Apple, Amazon has been doing some pretty interesting things in firmware. Former Intel colleague Mark Tuttle's work in mentioned in the write-up https://www.freertos.org/2020/02/ensuring-the-memory-safety-of-freertos-part-1.html about using the C Bounded Model Checker (CBMC) https://www.cprover.org/cbmc/ to ensure memory safety of FreeRTOS. This is a nice balance of providing better assurance with an existing type-weak language like C versus the forklift upgrade to something like Rust. We discuss CBMC a bit in chapter 21 of https://www.amazon.com/Building-Secure-Firmware-Armoring-Foundation/dp/1484261054, including an example of usage in listing 21-2. 

This work in the rebar-embossed book is nothing like the fully operationalized example of the real-time OS listed above, though. Beyond this specific work, I appreciate how Amazon endeavors to make formal methods more developer friendly, as described in the work by Nathan, Mark, and their colleagues in https://assets.amazon.science/d0/de/cbec0b4547e3ae7ff077f8aa978f/code-level-model-checking-in-thesoftware-development-workflow.pdf. This type of analysis is much more relevant to my day-job than the former treatise on FM they wrote regarding TLA https://cacm.acm.org/magazines/2015/4/184701-how-amazon-web-services-uses-formal-methods/fulltext, although some of Hillel Wayne's https://www.hillelwayne.com/post/why-dont-people-use-formal-methods/ work may help build a bridge for me. More examples of the good stuff published by their engineers/'builders' that I also referenced in my last blog posting.

And formal verification of system software is popping up in other places, such as the mention "Formal verification of the code running at EL2" in https://linuxplumbersconf.org/event/7/contributions/780/attachments/514/925/LPC2020_-_Protected_KVM_.pdf. I haven't dug into this too much, but I wonder if it is aligned with Google Project Oak, including the Hafnium hypervisor https://hafnium.googlesource.com/hafnium/+/HEAD/docs/Architecture.md and its Coq based verification https://github.com/project-oak/hafnium-verification/tree/hfo2/coq-verification? Or some other KVM-focused verification? 

As a parting note, I'd like to thank the work of Tim Lewis https://uefi.blogspot.com/ as the technical reviewer of https://link.springer.com/book/10.1007/978-1-4842-6106-4 and Leendert Van Doorn https://www.paramecium.org:4443/~leendert/ for writing the preface. I have appreciated the opportunity to work with these two technical leaders in various roles since the 2000's.

Beware Experts, Beware Disruptors

This blog entry hearkens back to a tale of expertise versus disruption. The scene is DuPont, WA in the late 1990's. Prior to Tiano (2000+) there was an effort to write a BIOS from scratch in the erstwhile workstation product group (WPG) at Intel. The code name of the effort was "Kittyhawk" (1998). Like all boot firmware, the initialization was broken up into phases. Instead of the tuple of {SEC, PEI, DXE, BDS} of UEFI PI of {BOOTBLOCK, ROMSTAGE, RAMSTAGE, PAYLOAD) of coreboot, Kittyhawk had {VM0, VM1, VM2, BOOTLOAD} phases.

Although the platform initialization of Kittyhawk was in native code protected mode C code (and cross-compiled to Itanium for the latter boot variant), this was prior to emergence of IA-32 EFI and it's OS's, which was a parallel effort in DuPont. Instead, the 32-bit version of Kittyhawk relied upon a PC/AT BIOS boot interface for MBR-based operating systems. To make this happen an engineer decomposed an existing PC/AT BIOS from it's POST (Power-On Self Test) component from it's 'runtime', or the 16-bit code that provided the Int-callable BIOS API's. The 32-bit Kittyhawk code did device discovery and initialization, and then the 32-bit code provided the device information in a data block into the generic 16-bit BIOS 'runtime.' This 16-bit code was called 'the furball' by management in anticipation of a day when it could be 'coughed up' in lieu of a native 32-bit operating system load.

This Kittyhawk effort on IA-32 was definitely deemed disruptive at the time. The DuPont Intel site was more of a rebellious effort, with the larger product teams in Oregon constituting the existing 'experts'. There were cries from Oregon that the disruptors in Washington would never boot Windows, and if they did, they'd never pass WHQL, or the suite of compliance tests for Windows. The furball booted OS's and passed compliance tests. It turned out that having a non-BIOS engineer look at the problem didn't entail the prejudices of what's possible, and the work about having a clean interface into a BIOS runtime was used in the subsequent Tiano Framework work such as the Compatibility Support Module (CSM) csm https://www.intel.com/content/dam/www/public/us/en/documents/reference-guides/efi-compatibility-support-module-specification-v098.pdf API design.

So this part of the story entailed providing some caution in listening to the experts, but....

You sometimes need to beware disruptors. Specifically, riding high upon the success of building a common C code based to support IA-32 and Itanium, with the furball providing 32-bit OS support and the Intel Boot Initiative (IBI)/EFI 0.92 sample provide 64-bit Itanium OS support, the next challenge was scaling to other aspects of the ecosystem. Part of this scaling entailed support of code provided by other business entities. The EFI work at the time had the EFI images based upon PE/COFF, so the Kittyhawk team decided to decompose the statically linked Kittyhawk image into a set of Plug In Modules (PIM's).

After the Kittyhawk features were decomposed into PIM's, I remember standing in the cubicle of one of the Kittyhawk engineers and a BIOS guru visiting from Oregon helping with the 64-bit bring-up, the topic ranged over to option ROM's. The Kittyhawk engineer said "let's just put our PIM's into option ROM containers." I was a bit surprised, since to me the problem with option ROM's wasn't carving out code to run in the container but more entailed 'how' to have the option ROM's interoperate with the system BIOS. That latter point is where standards like EFI came into play by having a 'contract' between the Independent Hardware Vendors (IHV's) that built the adapter cards and the OEM's building the system board BIOS.

So the work of the disruptors was laudable in showing that you could have shared, common code to initialize a machine, but to scale to new paradigms like OS and OpROM interfaces you really needed a broader paradigm switch. And as would be shown, it took another decade to solve this interoperability issue.

As such, sometimes you have to be a bit wary of the disruptors, although the disruptive Kittyhawk did provide many learning's beyond the furball/CSM concept, including how up to today EDKII builds its ACPI tables via extracting data tables from C code files. Alas the Kittyhawkfort was shut down as part of the shuttling of the Workstation Group, and efforts to build a native code BIOS solution fell into the hands of the EFI team as part of the emergent (1999+) Tiano effort. At this time there was the EFI sample that eventually grew into the EFI Development Kit (aka the first Tiano implementation), now referred to as EDKI versus today's EDKII. EDK leveraged a lot of the learning's of EFI to inform the Intel Framework specifications, including the phases of SEC, PEI, DXE, and BDS we know today.

This original PEI phase differed somewhat from the PEI you can find in the UEFI PI specification, though. Instead of the concept of C-based PEIM's and PEI core, the original instantiation of the PEI Core Interface Specification (PEI-CIS) was based upon processor registers. There was still the business requirement of separate binary executables and the solution of GUID's for naming interfaces was used. But instead of a PPI database, the PEIM's exposed exported interfaces through a Produced GUID List (PGL) and imported interfaces via a Consumed GUID List (CGL). During firmware construction and update the CGL and PGL were reconciled and linked together. And in order to having services that interoperate, the concept of a 'call level' was introduced. The call level detailed which registers could be used by services in a PEIM since there was no memory for a call stack with this early PEI-CIS 0.3.

At the same time we were defining PEI 0.3, there was another approach to writing pre-DRAM code without memory, namely the Intel® Applied Computing System Firmware Library V1.0 (hereafter referred to as Intel® ACSF Library).

ACSFL was described in a Intel Update article (same dev update magazine https://github.com/vincentjzimmer/Documents/blob/master/it01043.pdf)

that provided 32-bit callable libraries for boot-loaders. This effort came from the embedded products team and entailed delivering a series of libraries built as .a files. This simplified initialization code addressed the lack of a memory call stack by using the MMX registers to emulate a call stack. This differed from the PEI 0.3 model of reserving a set of registers for each call level. The ACSFL concept was smarter in that constraining PEIM's to a given call level really impacted the composition of these modules into different platform builds.

I do enjoy the quotation 'requires only 15KB of flash' when I wander over to https://github.com/IntelFsp/FSP/blob/master/KabylakeFspBinPkg/Fsp.fd with its size of 592k, or 39x size dilation of silicon initialization code over the last 20 years. This is similar to the 29x scaling in the number of transistors between the 7.5 million in a Pentium II https://en.wikipedia.org/wiki/Pentium_II and the 217 million in a Kaby Lake https://www.quora.com/How-many-transistors-are-in-i3-i5-and-i7-processors.

The same article provides the software layering. One can see some similarity of ACSF Library to the Intel Firmware Support Package (FSP). This is no accident since the Intel FSP was intended to originally target the same embedded market, although Intel FSP in 2012 had the embedded mantle being carried by the Intel of Things Group. As another point of irony, the original FSP built w/ the Consumer Electronics Firmware Development Kit (CEFDK). The CEFDK was in fact the evolution of the ACSF library, going through various instances, like FDK. The latter were used to enable phones and tablets beyond just embedded.

So ACSF Library provided some learning's, and at the same time the LinuxBIOS (prior name of coreboot) solved this issue of supporting C code without initialized DRAM via the ROMCC tool.  ROMCC was a custom compiler that used processor registers to emulate a stack so that you could write things like DRAM initialization in C code.

The initial implementations of PEI-CIS 0.3 https://www.intel.co.kr/content/dam/www/public/us/en/documents/reference-guides/efi-pei-cis-v09.pdf with just registers proved difficult to deploy, and since part of the Tiano effort entailed a requirement to use standard tools, techniques like ROMCC were deemed not tenable. As such, as the PEI CIS went from 0.3 to 0.5, we moved PEI to the concept of temporary memory, with using the processor cache as RAM (CAR) as the 'temp ram.' Regrettably we were asked to not disclose how we implemented temp RAM, and the technique and initialization code were not made open source or published (and a modern instance of this reluctance can be found in the FSP-T abstraction of FSP 2.0). The coreboot community, though, didn't have this constraint and https://www.coreboot.org/images/6/6c/LBCar.pdf https://www.coreboot.org/data/yhlu/cache_as_ram_lb_09142006.pdf provided details on how to use this technique.

As time progresses, I am amused about the various recollections. During the late 2000's someone told me 'you rewrote PEI many times' when in fact the only substantive change was going from registers in 0.3 to temporary memory in 0.5. Although not necessarily a full re-write, the EFI implementations did have a long history:

IBI Sample -> EFI Sample -> Tiano Release 1..8, Tiano Release 8.1..8.7,  EDK1117, ...

...UDK2015, UDK2016, .......

Also, some fellow travelers mentioned to me their fond recollections of EFI discussions in 1996. I smile because the first discussions of EFI (in the form of IBI) weren't until 1998. But I suspect everyone's memory gets hazy over time, with this blog post having its own degree of fog and haze.

And these efforts also remind me that changes in this space take time. A recent reminder that api support takes time, such as the discussion of the EFI Random Number protocol in Linux
https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.5-EFI-RNG-x86. This is an API that was defined in 2010 by Microsoft for Windows 8 and then introduced in the UEFI standard in 2013. Or features like UEFI secure boot from 2011 UEFI 2.3.1 showing up in Debian Buster https://wiki.debian.org/SecureBoot just in mid 2019.

The other interesting perspective is https://fs.blog/2019/12/survivorship-bias/, namely you often hear about the success stories, not failures. Only for extreme cases like https://en.wikipedia.org/wiki/Normal_Accidents do you find a rich journal of failures. So in portions of this blog and the EFI1.2 discussion in http://vzimmer.blogspot.com/2013/02/anniversary-daynext-arch-ps-and-some.html I have tried to shed light upon some of the not so successful paths.