October firmware events

Apropos of the 25 year anniversary of the first IBI/EFI/UEFI boot services at week ago, as commemorated at the 20 year milestone in https://vzimmer.blogspot.com/2018/10/  

                                                        "Ken Reneris     Oct-14-1998"  

there was a question of the coding standards for UEFI https://twitter.com/bexcran/status/1701071379171619236 recently which made me think of Ken. I mentioned how that since Ken had come over to Intel from the Windows NT team at Microsoft it was natural to adopt the NT coding standard https://computernewb.com/~lily/files/Documents/NTDesignWorkbook/coding.pdf for the original IBI/EFI/UEFI work. This includes the EFI_ERROR macros, inf's, CR macro, TPLs as simplified IRQLs https://vzimmer.blogspot.com/2015/06/guids-revisions-interrupts.html, etc. This made IBI/EFI/UEFI a sort of a cultural 'Windows BIOS' given that lineage. 

And Ken as the MS HAL owner continually interfaced with the platform. In fact, standards like ACPI pre-dated IBI/EFI/UEFI by starting in the mid-90's. Ken was deep into that work, too, as evidenced by his solo inventor ship of the now-expired S4 resume patent https://patents.google.com/patent/US6209088B1/en. 

These UEFI technology elements serve a counterpoint with coreboot, which was LinuxBIOS in 99. The latter have technology elements https://www2.cs.arizona.edu/~mccann/cstyle.html, KConfig, etc.  

https://link.springer.com/book/10.1007/978-1-4842-0070-4. So just as we have had parallel growth of Windows and Linux, there have been threads of 'Windows BIOS' (e.g., EDKII UEFI based) and 'LinuxBIOS' (e.g., coreboot and maybe less so u-boot?).

But then again, who uses 'BIOS' https://basicinputoutput.com/2014/08/will-i-be-jailed-for-saying-uefi-bios.html? 

So the blog title is firmware events. Over the last couple of weeks there was a UEFI plugfest in the Portland area and a Open Compute Project (OCP) event in San Jose. As I drove back from the former, I stopped by a bookstore and saw https://www.amazon.com/Beyond-BIOS-Developing-Extensible-Interface/dp/1501514784

on the shelf.

On that same shelf I saw 

and

Though not the immediate book neighbor, the nearby https://www.amazon.com/ARM-Architecture-Reference-Manual-2nd/dp/0201737191 reminded me of Dave Jaggar. Back in the mid-1990's we were development hardware RAID controllers at Compaq and using custom ASICs alongside an AMD 29k RISC CPU. I recall Jaggar from ARM coming to visit and explaining the ARM architecture. Part of the discourse included showing a silicon die with a small portion highlighted for the ARM core itself. It was quite a shock for me to so such small mm-squared for a CPU. My firmware lead (lead == sole design & developer) work on the SMART-2SL 

https://www.amazon.com/Compaq-242777-001-WIDE-SCSI-CONTROLLER-242777001/dp/B0002FDXLQ is but a memory to be rekindled occasionally by seeing aftermarket examples of this device. In addition to being the first device to not have a non-volatile post write cache, this work gave me the opportunity to do some interesting firmware performance innovations with a colleague https://patents.google.com/patent/US6341342B1/en. 

Speaking of CPQ, it has been a strange migration of companies for me. My original internship was with Texaco, which in turn was purchased by Chevron. Then I did my first full-time firmware development at Daniel Industries, later acquired by Emerson Electric. My first foray into PC BIOS was at Texas Microsystems (TMI) working on industrial computers, which then was acquired by Radisys. Finally, my pre-Intel employer Compaq server group was in turn acquired by Hewlett-Packard, and then split into the enterprise side HPe. Luckily Intel is still Intel.

So speaking of the development event, it spanned 3 days. I presented on the first day (lower left image)

https://uefi.org/sites/default/files/resources/Tuesday_02_Kubacki%20and%20Zimmer_Final.pdf and final

day (rightmost image) https://uefi.org/sites/default/files/resources/Firmware%20Configuration%20%E2%80%93%20Past%2C%20Present%2C%20and%20Future_Zimmer.pdf. The first day of the event included a description of SPDM https://www.dmtf.org/standards/spdm and its support introduced into the UEFI 2.10 specification. SPDM is homed at the DMTF but has associated work-product having in groups like UEFI and the Trusted Computing Group (TCG). 

In the Insyde talk Tim Lewis mentioned the growth of attacks on the UEFI network stack after many years of battering SMM. This reminded me of a recent posting I saw https://www.linaro.org/blog/ledge-blogs-uefi-http-and-https-boot-in-u-boot/ where the U-Boot community discussing using https://savannah.nongnu.org/projects/lwip/ which is a similar approach taken in https://github.com/tianocore/edk2-staging/tree/MpNetworkStack mentioned in https://uefi.org/sites/default/files/resources/7_Maciej%20Vincent_INTEL_network%20stack%20performance.pdf. From the days of discussing HTTP booting in 2009 https://dblp.org/rec/conf/csreaSAM/Zimmer09.html?view=bibtex and having the URL boot option mentioned in https://www.rfc-editor.org/rfc/rfc5970.txt has come a long way. Just as EDKII consumes cryptography as a submodule from another community, maybe it's time to do so for the basic networking capabilities.

Given that AWS started in 2006, it may have a bit short-sighted of me to have said "...emergent compute models such as cloud computing." in that SAM 09 paper.

Maybe I was keying off the 09 date of what I thought was the definitive paper on the cloud, viz., https://www2.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-28.pdf.

So back to the topic of this blog, namely firmware events. On that topic, the next week came the Open Compute Project (OCP) event in San Jose. I first presented at OCP in 2015 with Mallik Bulusu https://cdrdv2.intel.com/v1/dl/getContent/671185 

and then again in 2016 on firmware updates http://files.opencompute.org/oc/public.php?service=files&t=1f7831234dce58bb875b1b5b24f7154d. 

This session last week was on the universal payload

I have been engaged on server platforms since I was hired into Intel in February 1997 to lead the IA-64 (Merced, Itanium) firmware. Along the way we devised ways to facilitate ease of firmware development, like multi-socket cache-as-RAM https://patents.google.com/patent/US7254676B2/en 

and similar for IA-32 https://patents.google.com/patent/US8078862B2/en. Palsamy and I also collaborated on UEFI and ACPI for RAS and error support https://cdrdv2.intel.com/v1/dl/getContent/671067. 

And all of these threads come together in these recent talks. The SPDM prezo mentioned at the top of the posting here entail facing the post-quantum cryptographic migrations like all of the other standards, including UEFI https://uefi.org/, TCG https://trustedcomputinggroup.org/, and others. The proposal for augmenting UEFI is mentioned in https://bugzilla.tianocore.org/show_bug.cgi?id=4087 and some prezos on the topic can be found at https://uefi.org/sites/default/files/resources/Post%20Quantum%20Webinar.pdf and https://uefi.org/sites/default/files/resources/USF_Security_Webinar_Final.pdf. The specific study on SPDM was posted as a pre-print to https://eprint.iacr.org/2022/1049 and then a journal submission in https://www.mdpi.com/2410-387X/6/4/48. 

Since SPDM has fixed message sizes, the concept of 'chunking' or breaking up the larger payloads demanded by these post-quantum algorithms is a common concern other hardware-based messaging interfaces will face, like the Trusted Computing Group's Trusted Platform Module (TPM). Interestingly this topic of post-quantum impacts on firmware standards that motivated the SPDM paper

 listed above was inspired by the study https://eprint.iacr.org/2021/041 from Cisco.

And during Q/A with the CISA presentation during the UEFI developer event I asked about formal methods for this domain. Since this talk followed the SPDM talk one recommendation was to use formal for the SPDM wire protocol. It turns out the above MDPI SPDM paper is referenced by a few formal studies of SPDM, including https://www.usenix.org/conference/usenixsecurity23/presentation/cremers-spdm and https://ieeexplore.ieee.org/abstract/document/10149352/. 

And speaking of security and standards, this upcoming week is the TCG members meeting at the Google campus in Kirkland, WA. I am Intel's representative on the Technical Committee, assuming that role after Kirk Brannock retired. 

 

But the TCG is not unfamiliar to me. I have been engaged with TCG https://en.wikipedia.org/wiki/Trusted_Computing_Group even when it was called TCPA, as evidenced by work-product like https://patents.google.com/patent/US7200758B2/en and delivering the Itanium and EFI API and platform specification, respectively.

When reading https://research.tue.nl/en/publications/communication-in-a-world-of-pervasive-surveillance-sources-and-me from the mention of Jacob in https://www-theregister-com.cdn.ampproject.org/c/s/www.theregister.com/AMP/2023/09/19/marvell_disputes_claim_that_cavium/, I hearkened back to my first ToorCamp presentation 

Based upon the slide

and discussion around the TPM, Jacob told me that we should design a TPM so that it could quickly be removed from board and chewed up/swallowed if someone tries to take your computer. Quite the privacy-preserving dietary strategy.

It's interesting to see people in person after the years of COVID seclusion and event cancellation or wholly-virtual events. Just like the psychic shock and fatigue of crowds, continual interaction and noise, it is taking a bit of effort to get re-acclimated. Part of the symptomology is falling behind on blogging, I guess, since I had intended to post this entry on the day of the EFI 25-year anniversary, not a week later. Oh well. Only so many free moments on the weekends these days.

Deprecation and introduction of interfaces

Specifications may appear to be static codifications set in stone, but they are often evolutionary species. One of the challenges in evolving a specification includes when, if ever, to deprecate content in lieu of new additions. Sometimes technology may fall out of use or be deemed by the market not to be the most viable. Often, though, this class of information is not elided from specifications unless they are hard scientific reasons, like elision of MD5 or SHA1 from specifications 

because of pre-image attacks https://en.wikipedia.org/wiki/Preimage_attack. I recall one person telling me that about 40% of the Wifi specification was relevant; the key was knowing 'which' 40% merited attention.

This blog will talk about a few specifications, such as the UEFI, PI and Intel SDM, that weigh in at several thousands of pages. To be with the UEFI and PI specification, one area that has potential for deprecation is Itanium support. The EDKII upstream has already removed Itanium related code from the various packages. There are vestiges of Itanium in chapter 2 of the UEFI Specification https://uefi.org/specs/UEFI/2.10/02_Overview.html#intelitanium-based-platforms for the calling conventions, though. Similarly, support for the Itanium reset paths https://uefi.org/specs/PI/1.8/V4_MCA_INIT_PMI_Protocol.html# and extended SAL https://uefi.org/specs/PI/1.8/V4_Sal.html services can be found in the PI specification. The latter of which was the EDKII adaptation of calling the Itanium System Abstraction Layer (SAL) (SAL) https://redirect.cs.umbc.edu/portal/help/architecture/24535901.pdf interfaces from a UEFI environment. 

As a quick background, Itanium had a platform scoped SAL and processor scoped Processor Abstraction Layer (PAL) set of firmware layers designed to provide both boot and runtime services. The RISC-V Supervisor Binary Interface (SBI) https://github.com/riscv-non-isa/riscv-sbi-doc is sort of an amalgam of SAL and PAL since it provides both core, SOC and (potential) platform capabilities. SAL is interesting in that unlike UEFI, that goes into a virtual-only calling mode after SetVirtualAddressMap(), the SAL calls could be called in either physical or virtual mode throughout the life of the platform. This posed some challenges for writing UEFI code since position independent code (PIC) options for C compilers haven't been universally supported across all of the EDK toolchains, especially in the early days with Visual Studio, or for IA32 with its inability to read the instruction pointer address as possible in other architectures. For assembly-language Itanium code it was pretty simple to write PIC code. So the Extended SAL (ESAL) of the PI spec and EDKII provided a way to have non-fixed up and fixed up C code that would use a common data area. 

In addition to the SAL support, another fascinating aspect of Itanium was the support for floating point exceptions in the Floating-Point Software Assist (FPSWA) https://redirect.cs.umbc.edu/portal/help/architecture/24541501.pdf driver, as described in https://www.amazon.com/IA-64-Linux-Kernel-Design-Implementation/dp/0130610143 

This driver was loaded from the EFI System Partition during boot and would provide runtime support for floating point exceptions. Regrettably soft-loading critical flows from disk like the FPSWA have not been pervasive, even in the face of relatively expensive $/byte of semiconductor NOR SPI flash. This stems from the supply chain challenge where the motherboard and fixed disk media may be provisioned, secured, and recovered by alternate parts of the ecosystem, namely OEM/ODM for the board versus OEM/ODM/integrator/VAR/IT for the disk and its bootloader and OS images, respectively.  

Another area that is an interesting artifact from the past is the BIOS interface in the Intel Software Developer Manual https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-vol-3a-part-1-manual.pdf. This interface is an Int15h API to manage microcode patches.

Starting in the early 1980s, the PC/AT BIOS exposed services through 16-bit 'int' calls or software traps, which the most famous being Int13h for disk access, Int 10h for video, etc. These API's all have correlatives in UEFI, with Int13h having the equivalent in EFI_BLOCK_IO_PROTOCOL, for example. The Intel Framework Compatibility Support Model (CSM) specification https://www.intel.com/content/dam/www/public/us/en/documents/reference-guides/efi-compatibility-support-module-specification-v098.pdf provided a bridge from EFI into BIOS calls in the early days where there were few EFI native drivers, for example. But the CSM support was not introduced into the UEFI PI specification since the idea with PI commencing in 2006 would have UEFI-spec-defined APIs. In fact, Intel declared the PC/AT BIOS interfaces to be end-of-life in 2020 https://www.phoronix.com/news/Intel-Legacy-BIOS-EOL-2020 

and http://www.uefi.org/sites/default/files/resources/Brian_Richardson_Intel_Final.pdf.

That's where the Int15h API mentioned above is interesting. It can either go away as its the only BIOS API in the SDM, or it could be complemented by/replaced with a UEFI equivalent. To that end, the https://raw.githubusercontent.com/tianocore-docs/Docs/master/White_Papers/A_Tour_Beyond_BIOS_Capsule_Update_and_Recovery_in_EDK_II.pdf design reads on this capability 

The mapping of the UEFI interfaces to the Int15h included

with the specific capsule itself having the following layout

The generic capsule overview flow is described in the UEFI specification https://uefi.org/specs/UEFI/2.10/08_Services_Runtime_Services.html#update-capsule 

and other write-ups https://embeddedcomputing.com/technology/security/software-security/understanding-uefi-firmware-update-and-its-vital-role-in-keeping-computing-systems-secure and https://archive.fosdem.org/2020/schedule/event/firmware_culisfu/attachments/slides/3709/export/events/attachments/firmware_culisfu/slides/3709/FOSDEM_2020_Intel_Capsule_Update.pdf. As the code-base has been re-arranged in the open, the most recent location to find the FMP DXE Microcode Capsule support is https://github.com/tianocore/edk2-platforms/tree/master/Silicon/Intel/IntelSiliconPkg/Feature/Capsule and of course https://github.com/tianocore/edk2/tree/master/SignedCapsulePkg. 

This builds upon the generic FMP 

and capsule update flow 

widely deployed today. Maybe avoiding putting something 'post-Int15h' in the SDM is a wise move, though, considering the plurality of interfaces for firmware updates, from https://slimbootloader.github.io/security/firmware-update.html to https://uefi.org/sites/default/files/resources/OCPsummit2016_Towards%20a%20Firmware%20Update%20Standard.pdf to https://uefi.org/sites/default/files/resources/PRM_Platform_Runtime_Mechanism_1_1_release_candidate.pdf to https://uefi.org/sites/default/files/resources/Intel_MM_OS_Interface_Spec_Rev100.pdf to.... It's a veritable embarrassment of riches.

Co-authors then and now

 So I mentioned in my last blog that I could always talk about https://www.cisa.gov/news-events/news/call-action-bolster-uefi-cybersecurity-now.  Well, it turns out the UEFI Forum posted a pretty good read this week on the topic of UEFI and ecosystems, namely the top posting at https://uefi.org/learning_center/papers, in the document https://uefi.org/sites/default/files/resources/Decoding%20UEFI%20Firmware-Aug24-2023-Final_v2_0.pdf. 

The authors are folks I have enjoyed working with across various teams and decades. Some of the collaborations have spanned companies and different venues, such as papers, books, and presentations. 

To begin, I think Dong Wei https://www.crunchbase.com/person/dong-wei and I first appeared in print together with the original Beyond BIOS book in 2006, 

with Dong providing the forward

Next up was the 2008 UEFI Shell book

with preface

This 2008 book was followed with a 2009 whitepaper https://uefi.org/sites/default/files/resources/A_Tale_of_Two_Standards_0.pdf

I was a bit excited about IPV6 at the time as I was in the throes of getting https://datatracker.ietf.org/doc/html/rfc5970 through an unfamiliar standards body, namely the IETF.

Next was the presentation circuit with 2010 presentation in Shanghai

and San Francisco

Shanghai https://en.sjtu.edu.cn/ was nice to visit since so many of my long-time collaborators in our Intel Shanghai office, like Jiewen Yao, are alumni of this institution.

Then the 2nd edition of the Beyond BIOS book in 2010 came next

with Dong reprising his preface-writing skills with

Finally a couple of proposals for UEFI and RISC-V for the 2015 

and 2016 conference

were created with Dong. 

All of the collaborations before 2023 with Dong were when he was at HP (and then HPe after the split). 

And now in 2023 with Dong as ARM Ltd's chief standards architect and an ARM Fellow.  The total collaboration instances makes nine per my accounting.

Speaking of 'chief's', Insyde Software's Chief Technology Officer  (CTO) https://www.insyde.com/company/executive-management was another rich engagement.

The collaborations with Tim commenced in 2008 with the UEFI Shell book first edition

and an IDF presentation that same year presented at both the Taipei and Shanghai events

The Intel Developer Forum (IDF) was an annual event in California show-casing various Intel and industry advancements. Today I believe it has been superseded by venues such as Intel Ignite. 

I co-authored an Intel Technology Journal https://www.intel.com/content/dam/www/public/us/en/documents/research/2011-vol15-iss-1-intel-technology-journal.pdf article with Tim in 2011

Tim and I also jointly presented in 2011 at the Intel Developer Forum in San Francisco

All of the preceding presentations were done when Tim was with Phoenix Technology. Tim had joined Insyde when the joint book-authorship was reprised with the 2nd edition of the UEFI Shell book in 2017 https://www.degruyter.com/document/doi/10.1515/9781501505751/html

Tim was also gracious to serve as the technical reviewer for Firmware Security https://link.springer.com/book/10.1007/978-1-4842-6106-4 book in 2020 

These collaborations with Tim look like they sum to seven.

Tim and Dong have made seminal contributions to the ACPI, UEFI, and PI specifications. In fact as late as 2010 Tim was the chair of the UEFI Security Subteam (started with the UEFI Forum in 2010). From the 2nd edition of Beyond BIOS

 

 

before I took over that subteam. In fact I drafted this recent overview of the UEFI Forum Subteam's

From there you can see the various working groups.  I believe Dong and Mark Doran co-chair ASWG, Mark PIWG, Mark USWG, etc. Among the co-authors of this blog's showcase paper. Dick Wilkins is the Phoenix Board of Directors (BOD) rep, Bill Keown for Lenovo, and Dong for ARM, respectively. Co-author Brian Mullen chairs the new Software Bill of Material (SBOM) subteam, Dick chairs the UEFI Security Response Team, and I chair the UEFI Security Subteam. I elided subteams like graphics, configuration, and networking from the infographic as they are mostly dormant these last few years. I created the above image derived from earlier https://uefi.org/sites/default/files/resources/UEFI_Plugfest_VZimmer_Fall_2016.pdf as a companion to another mutation (from image in https://embeddedcomputing.com/technology/security/software-security/understanding-uefi-firmware-update-and-its-vital-role-in-keeping-computing-systems-secure) I crafted to give folks an idea about how SBOM's impact EDKII-style system firmware, viz.,

Well, that's it for today.  This is my small gesture to leave a bit of history since a lot of this system firmware work I've done will be unlikely to land in more esteemed repos like http://www.bitsavers.org/bits/ and is surely aging off the internet. I even recall that a request was required to get the Intel Technology Journal back on intel.com. Regrettably the same audible wasn't called for other publications like "Technology at Intel" magazine or most back-dated Intel Developer Forum prezos. An example of what gets lost on the internet can be found in citation curation sites like https://dblp.uni-trier.de/pid/34/5641.html; here I only see Tim mentioned once, for example.

UW PMP, Patents, T-shape, Job ladders

I haven't blogged in a while. Given things in the news like generative AI you'd think I'd dive into thoughts there, especially interesting aspects like the use of probability, with my usual meanderings to related topics like Terrence Fine, Cornell, and formalists versus frequentists, etc. Or posts like https://www.cisa.gov/news-events/news/call-action-bolster-uefi-cybersecurity-now where I could possibly opine for another 900pp+ (like https://link.springer.com/book/10.1007/978-1-4842-6106-4).

Instead I was reminded of late 90's when reading https://www.quantamagazine.org/complexity-theorys-50-year-journey-to-the-limits-of-knowledge-20230817/#, specifically my time of the University of Washington (UW) in the UW PMP (Professional Masters Program) https://www.cs.washington.edu/academics/pmp. The PMP started in 1997 and I was accepted prior to the 2nd quarter having commenced, or the '2nd wave of inductees'). This was fortuitous for many reasons.  One, Intel was the largest recruiter from UW at the time, including donating a lot of equipment, and two, the other regional software companies hadn't flooded the application pool with talent form Microsoft, UW, Boeing, etc. If I had waited a few more quarters, the probability of acceptance as an 5-year-in-industy-EE-undergrad may have been tough going.  

Also, I had my first daughter during that last quarter. New tech job + around-the-clock-diaper-change + after-work-hours-studying-and-projects nearly killed me.

My 6-month-old little Husky didn't look so happy in her purple onesie on that hot day back in 1999.

Fast forward to 2023

My 6-month-old turned out to eschew the UW path, but her younger sister ended up choosing UW and is in her 3rd year

and my 2023 Husky next to her older sister

and a couple of my favorite things my UW daughter gave me include a box years ago

and then this coin when she entered UW

that I keep in the box.

As part of that program, the computer science department hosted a welcome event at the Pacific Science Center in downtown Seattle next to the Space Needle. There were many UW professors in attendance, included a newly minted graduate from Caltech named Chis Diorio who had worked with Carver Mead on hardware neural nets. Part of his research entailed work on multi-level cell storage, and Intel had a significant history and products around NOR storage at the time, so Chris asked if I could connect him with any other Intel folks. Diorio has done pretty well in the intervening years https://www.impinj.com/library/blog/impinj-named-a-finalist-in-two-categories-2023-geekwire-awards with Impinj, too.  My learning from this interaction is as follows: always assess if the value others see in the conversation with you are because of the 'person' or the 'platform'. In this case I suspect it was the 'platform', namely my employer.

Speaking of Carver Mead, it's interesting to see the wisdom of pioneers potentially resurface. Namely, reading https://www.techradar.com/pro/nvidia-beware-ibm-has-a-new-analog-ai-chip-that-could-give-the-h100-a-run-for-its-money made me think of https://www.amazon.com/gp/product/0201059924/. This is definitely a contrast to the digital state-of-the-art sort of read I found in https://www.amazon.com/Efficient-Processing-Deep-Neural-Networks/dp/168173835X, for example.

So after a couple paragraphs, how does this relate to the Quantum magazine article? Well, another professor was Richard Karp, viz., " In addition, the American complexity theorist Richard Karp proved that the universality property identified by Cook (and Levin, though Karp and Cook didn’t know of Levin’s work until years later) was itself all but universal".  More details on Karp can be found in https://en.wikipedia.org/wiki/Richard_M._Karp, including "Apart from a 4-year period as a professor at the University of Washington, he has remained at Berkeley. From 1988 to 1995 and 1999 to the present he has also been a research scientist at the International Computer Science Institute in Berkeley, where he currently leads the Algorithms Group."  And what was my interaction with this esteemed Turing Award winner?  After Karp introduced himself I replied, "ah, I guess you get to work on things like compilers and stuff." My learning from this interaction 20 years later is as follows:  always do your homework on the group with which you are interacting ahead of time.

The late 1990's had other great interactions. Having come up to Seattle from Houston, Texas, I couldn't but help to name my proposed Test Executive, or test framework for firmware, 'TEX", when working with Eric Hudnell. Or the spirited discussions with Andrew Fish in 1997 about ASIM (Advanced Server Information Modules), or a way to decompose our server firmware prior to the next of many, many 'modularization' initiatives, such as Intel Boot Initiative (IBI) and Davinci/Kittyhawk BIOS in 1998, EFI in 1999 Framework in 2000, ...... My retrospective learning: infrastructure evolution takes time. There are few 'short wins.'

A lot of those interactions above also entailed wisdom from others. I still recall a director who told me about the plan/clue matrix for engineers. Namely, you aspire to working with someone who has a 'Plan + Clue.'  They are effective and directed.  The 'Clue but no plan' are often visionaries/strategists in that they know where to go but not how. The 'Plan but no clue' are the scariest since they are akin to a bull in a china shop- they have a lot of kinetic energy to expel but perform the drunkard's random walk in plotting a path. Finally, the easiest to handle are the 'No plan and no clue' since they are largely inert/ignorable. Cook reminded me of this early 2000 wisdom in his posting https://www.johndcook.com/blog/2010/12/27/dumb-and-gets-things-done/. 

Speaking of wisdom, I was asked to give some patent training last week. One of the pieces of wisdom I always offer to folks on patents is to have a sense of urgency, especially with the normalization of invention primacy with 'first to file' and the American Inventors Act a few years back. I often put this urgency in the context of the industry, namely invoking Joy's Law "no matter who you are, most of the smartest people work for someone else,” https://en.wikipedia.org/wiki/Joy%27s_law_(management). What that means for inventors is that given similar problem statements and the frontiers of knowledge, others may come up with similar ideas. But what you have within a large company is perhaps earlier access to the problem and/or the frontier of knowledge since Fortune 100's like Intel often 'create' said knowledge. 

I also try to temper the patent presentation to put patenting in context. First the positive where I mention the ability to get patents as a software person in a hardware company (data also mentioned a year ago in http://vzimmer.blogspot.com/2022/09/new-milestones.html), viz.,

but then I remind the audience that patents are part of delivering value, not a means in-and-of-themselves (including quoting this blog from 10 years back http://vzimmer.blogspot.com/2013/12/invention-and-innovation.html), viz.,

From the patent milestones you see the title "Fellow." This is one of the highest designations of a technical-track employee, as distinct from management-track, employee at Intel. Many tech companies have similar designations, and I culled a comparison of the Seattle area shops below https://www.levels.fyi/?compare=Amazon,Intel,Google,Microsoft&track=Software%20Engineer

NB
Someone might question why I continually show an image from a website that others can find themselves. I guess I have two answers. One, I learned that content on the web ages out or evolves, such as top inventor Wikipedia started to subset its listing. And two, for better or worse a colleague introduced me to the MS Windows 'snippet' tool and the barrier to image creation was significantly lowered for me :)

Speaking of old blogs in 2013, I should probably reprise http://vzimmer.blogspot.com/2013/03/a-technical-career-path.html since I discovered the T-shaped model https://collegeinfogeek.com/become-t-shaped-person/ and have used it a lot since in mentoring people. Then the other part of me notes that I also usually advise people that 3-7 years is optimal occupancy time in a grade level, say 9, prior to becoming a 10. Too short a time and you don't have time to generate sufficient evidence to prove you delivered at that last level, and too long you become stale and people ask 'why now?' As I roll into year 10 as a Senior Principal Engineer or grade 11, I am in the 'why now' camp where people can dismiss earlier accomplishments in the decade as not fresh or of insufficient merit since if they were, I would have been acknowledged then. As such, I challenge myself as to whether I am the right person to be dispensing 'career growth wisdom' if my career isn't growing. 

There are many great engineers in the Sr. PE community, including the two I mentioned in the last blog who retired, Sham and Kirk. And as the company matures, this cohort continues to grow in cardinality. But it does make me wonder if it's a terminal level, or sort of a tech elephant's graveyard

 

I couldn't help but use that metaphor since found the image arresting from reading Tarzan? or some such as a youth.

More likely it's the other career advice I give folks, namely that getting promoted is a lot like real-estate opportunities, namely 'timing and location.' Since promotions are a meld of business impact, technical acumen, and leadership, not all role or organizations afford opportunities for the same, or that triple may not be as business relevant as the market moves or the exigencies of the employer change.

I like to stand outside of the phenomena and observe it sometimes. Thanks Marcus Aurelius and Seneca, too, for the classical doses of introspection and stoicism, respectively.

Maybe part of it is the reality expressed by Corey Quinn

Corey Quinn (@QuinnyPig) / X

Twitter

https://twitter.com › QuinnyPig

From what I can see, getting promoted at a big tech company is roughly four times
harder than getting that job at another company.

(I apologize for the rough copy-paste of a Google-search-of-a-Tweet).  The gist of the sentiment above is that the new company wants to believe in you and leans in, whereas your present employer knows all of the defects of an internal candidate and perhaps subconsciously over-indexes on them. Another area where I have seen Quinn's assertion in practice is that if things are not going well in a company, outsiders from successful companies are seen as more attractive since the internal candidates are the ones who 'contributed to the company and its ills' (i.e., why promote the arsonists?); of course this confuses correlation with causation on both internal and external candidates to a first order since rarely is a single individual responsible for the ills or gains of any large company. Finally, one reason that Quinn's logic holds is that as the business evolves, it is necessary to important new skills for the new business domains, as distinct for taking the time or risk of 'up-skilling' the present bench; I still recall in my early days why hiring was prolific that the 2nd line manager told us 'we need the exact skills. There is no slack in the schedule to train someone up over three months.' 

Or maybe it's related to the business cycle? Just as Ben Horowitz opined about 'wartime versus peacetime' CEO's https://www.linkedin.com/pulse/wartime-vs-peacetime-ceo-insights-from-hard-thing-things-pandey/, maybe there's a similar dynamic that applies down the stack, including technologists? The market dynamic might argue for loading the bench with more 'wartime'-minded folks? Or does the company need more 'referees' versus 'risk takers?'

So the magic bullet is job flipping employers then?  Maybe, or maybe not. I've always found the promotion-through-job-change risky in that the grass isn't always greener. You'll see from zoom in of above 

that my present job code overlaps with Microsoft 'Partner,' for example,  I recall one Partner Architect at Microsoft, now retired, telling me that at MS the title 'Partner' is also code for "throw the other Partner under the bus" job description. Namely, it's an understandably competitive peer group when total compensations can easily exceed $1million/year. Similar for my slight overlap with Amazon Sr. PE. I recall asking an Amazon executive what happened when someone was hired at too high a grade level. His glib reply was 'Oh, that?  We take care of those situations quickly.' Woof. 

So what's my parting advice on this blog postings sidetrack on careers? Enjoy the trip. Add business value. Learn. Appreciate contributing to and receiving wisdom from others. And promotions are a Faustian deal, too. As Goethe expressed in this epic, be careful what you wish for. namely the pursuit of power can entail a deal with the devil, etc. The $1million/year compensation isn't because the employer admires your bio and portrait as a 'new partner.' Instead, it represents an expectation of output against both business goals and your new peers commensurate with this compensation. And these days it's delivering value on 'internet time.' 

I've never worked in the high-flying world of silicon valley or startups either, but I suspect that there's no 'rest and vest' on a California tech building roof like https://silicon-valley.fandom.com/wiki/Big_Head in almost all cases. But if you enjoy the trip, moving up the tech job ladder can be awesome. You will be at a technical level aligned with Vice-President+'s of the management track and the role should provide more of an opportunity to make a dent in the universe as a technologist as you advocate for and drive insanely cool technical progress.