Geoff Chappell - Software Analyst
On a sufficiently advanced processor, executing the cpuid instruction with 13 as the leaf number in eax and with a sub-leaf number in ecx produces well-defined output in all four of eax, ebx, ecx and edx. For sub-leaf 0 the output in all four registers is meaningful. For others, mostly depending on the output from sub-leaf 0, the output in all four registers is either meaningful or zero.
The purpose to the whole leaf is to report on extended processor features. Unlike the features that are reported by bits in edx from cpuid leaf 1, many of which tell of the availability of just one instruction, these extended features are large enough to have their own sets of registers. If operating systems (such as Windows) are to provide multiple execution contexts (such as threads) with the illusion that each has independent use of these features, then the processor must provide an efficient means for the system to save and restore the registers when switching contexts. What Intel eventually came up with is generalised, configurable and extensible. It is generalised in the sense that the operating system learns of each feature’s support and storage requirements in essentially the same way. It is configurable in that the operating system can choose which features are enabled for saving and restoring, and can then select from these when doing each save or restore. It is extensible in that an old operating system on a new processor can learn of new features, and although the operating system won’t know how to use these features for itself, it can at least not get in the way of their use by new applications, since the system’s old code for the vital work of context-switching can still be good for features both old and new.
Registers that need to be saved and restored are organised into state components. An extended feature may spread over multiple state components. What cpuid leaf 13 enumerates are the state components. What the xsetbv instruction enables and what the xsave instruction saves is a selection of state components. Intel’s literature tends to talk of the state components and Microsoft’s of extended features, though both sometimes seek to have it all ways with such constructions as “processor extended states” and even “extended processor feature states”.
State components are numbered from 0 up to and including 62 (if only for now). Which state components exist for any one processor, which are enabled by the operating system and which are included when asking the xsave and xrstor family of instructions to save and restoring state is conveniently represented by the set bits in a 64-bit register or in a pair of 32-bit registers. Such a collection is a state-component bitmap. The first two bits represent legacy components which the scheme manages compatibly with the fxsave and fxrstor instructions.
State is saved to and restored from an XSAVE area. Each state component has its own storage in the XSAVE area. The first two sub-leaves of cpuid leaf 13 report generally about state components and the XSAVE area. The higher sub-leaves each report about the corresponding state component, including to describe the component’s storage within the XSAVE area. The two legacy components do not need sub-leaves because their properties are fixed by compatibility.
The entry for cpuid in the Instruction Set Reference, A-M of the Intel® 64 and IA-32 Architectures Software Developer’s Manual first lists leaf 13 in Order Number 253666-026, dated February 2008. The Windows kernel first uses cpuid leaf 13 in version 6.1, which was built for release in July 2009.
For testing whether cpuid leaf 13 is even available, the kernel makes a rare exception to comparing against the maximum leaf number reported by eax from cpuid leaf 0. What it checks instead is the 0x04000000 bit (XSAVE) in ecx from cpuid leaf 1. In this, Microsoft follows the first of Intel’s directions in what was then the newly written guidance on Detection, Enumeration, Enabling Processor Extended State Support in the System Programming Guide, Part 1 of the Intel® 64 and IA-32 Architectures Software Developer’s Manual (Order Number 253668-026, dated February 2008). The thinking is apparently that the XSAVE bit ensures the existence not just of the xsave instruction family but of the new control register xcr0 which is in turn “architected to allow software to use CPUID leaf function 0DH”.
Intel’s first listing of cpuid leaf 13 in the Instruction Set Reference is plain that the “Leaf 0DH output depends on the initial value in ECX.” It also proceeds to describe sub-leaves in three sets. Sub-leaf 0 has its own ouput. Sub-leaf 1 has too, though with all its output documented as Reserved. Sub-leaves 2 and higher are documented together because they all have the same output except for describing different components. The architecture seems clear from the start, then: one sub-leaf for each of the possible state components, with the exception that sub-leaves 0 and 1, which would represent the legacy components, are free for wider purposes.
It is perhaps only a historical curiosity now, but some indications survive that cpuid leaf 13 had an early design that anticipated sub-leaves but with no correspondence yet between sub-leaves and state components. The first few appearances of cpuid leaf 13 in Intel® Processor Identification and the CPUID Instruction, also known as Application Note 485, from November 2008 (Order Number 241618-033) through to August 2009 (Order Number 241618-036), present 0 as the only valid sub-leaf, i.e., with no mention of others except to say they are invalid and therefore return zero in all the applicable registers. Since cpuid leaf 13 had by then been documented in the Instruction Set Reference for most of a year and since the next revision of the Application Note (Order Number 241618-037, dated January 2011) was a long time coming, Intel was arguably just not giving this note enough attention for it to count as definitive, whether for programmers at the time or for historians now. Also possibly stale is the guidance in the System Programming Guide, Part 1. Not until Order Number 253668-037, also dated January 2011, does it mention any sub-leaf other than 0.
In some sense, the difference between an early architecture with only the one defined sub-leaf and the finished architecture that defines sub-leaves for each state component is just a theoretical point of architecture. The whole scheme of cpuid leaf 13, the xcr0 register and the xsave instruction family predates by several years the (public) existence of any processor that had any state component other than the two legacy components. January 2011 is not just when Intel’s various manuals and notes all get onto the one page, so to speak, about sub-leaves: it’s also when these manuals start documenting that a state component 2 exists. Before then, while nobody’s code could encounter a processor whose cpuid leaf 13 has a sub-leaf 2 that’s worth executing, would it matter if a note here or some guidance there hadn’t been updated from a time when only sub-leaf 0 was yet architected? In another sense, though, the whole point was the advance planning.
The only interest that any version of the Windows kernel has in the result of executing cpuid with 13 in eax and 0 in ecx is for the state-component bitmap that is produced in edx:eax.
The kernel requires that the two bits for legacy components are both set. If they are not, the kernel dismisses the whole of cpuid leaf 13 as unreliable, in effect as if XSAVE had been clear. Whether this defence was Microsoft’s from observation or was supported by Intel from the start is unclear. Intel nowadays guarantees that “Every processor that supports the XSAVE feature set will set” both these bits, but the earliest I have yet found of this comes from when Intel moved its guidance on xsave to Volume 1: Basic Architecture (Order Number 253665-048, dated September 2013).
Of the sizes that sub-leaf 0 produces in ebx and ecx, the kernel ignores one entirely and treats the other as immaterial. The kernel instead learns and remembers the storage requirements for each component (see below) and aggregates them according to which features end up being enabled.
Right from version 6.1, released in 2009, the kernel knows that sub-leaf 1 of cpuid leaf 13 loads at least one bit flag in to eax. This is notable because even as late as Order Number 235666-037, dated January 2011, Intel’s documentation of cpuid in the Instruction Set Reference, A-M had it that eax, ebx, ecx and edx as output from sub-leaf 1 are all Reserved.
The particular interest that the Windows 7 kernel has in sub-leaf 1 is whether the 0x00000001 bit (XSAVEOPT) gets set in eax. This indicates that the xsaveopt instruction is available. If it is, and no configurable option tells the kernel to disregard it, then the kernel uses it in preference to plain ordinary xsave when switching threads.
The xsaveopt instruction is first listed in the Instruction Set Reference, N-Z of the Intel® 64 and IA-32 Architectures Software Developer’s Manual (Order Number 253667-037, dated January 2011). That it is said, both there and in the contemporaneous System Programming Guide, Part 1 (Order Number 253668-037), to need the XSAVEOPT bit in eax from sub-leaf 1 surely means just that this bit’s omission from the cpuid documentation was an oversight. It was anyway put right in the next revision (Order Number 235666-038, dated April 2011). Still, do not miss that Microsoft had coded for both the bit and the instruction a year and a half before either appeared in Intel’s published manuals.
Starting with Windows 10, the kernel knows of two more bits in eax and also that edx:ecx is loaded with another state-component bitmap. These enhancements to sub-leaf 1 had been in Intel’s published manuals for a while. They are not in the 048 revisions, dated September 2013, but are in the 050 revisions, dated February 2014. (The page Missing Intel SDM Documents by Michal Necasek of the OS/2 Museum notes that revision 049 was “short lived”.)
The 0x00000002 (XSAVEC) and 0x00000008 (XSS) bits in eax indicate that the processor has two more xsave variants: xsavec and xsaves, respectively. As far as Windows is concerned, both may as well be absent unless both are present. Right from these bits’ first mention in Intel’s manuals, Intel has documented that every processor that sets the XSS bit will also set the XSAVEC bit. This is necessary, since the xsaves instruction saves state in the same compacted form as does the xsavec instruction, but the necessity does not go the other way: Intel leaves open that XSAVEC may be set on its own. The kernel, however, is entirely uninterested in using xsavec without xsaves. Given that both are available, and again that no configurable option acts to the contrary, the kernel uses xsaves and xrstors where it would have used xsave (or xsaveopt) and xrstor when switching threads and in some other situations it uses xsavec in preference to xsave.
What’s distinctive about xsaves and xrstors is that they execute only in ring 0. They can save and restore the same user state components as can the other variants, but they also can save and restore supervisor state components. The state-component bitmap that sub-leaf 0 produces in edx:eax (see above) is just of user components. A set XSS bit in eax from sub-leaf 1 indicates that there are also supervisor components. The state-component bitmap of supervisor components is in edx:ecx from sub-leaf 1.
Right from version 6.1, the kernel executes cpuid with 13 in eax and a state component number in ecx once for each state component that is numbered 2 or higher and is in the state-component bitmap obtained from sub-leaf 0. Starting with version 10.0, the kernel does this also for each of the supervisor state components that are learnt from sub-leaf 1. (The kernel just assumes that the legacy components are not supervisor components.)
The kernel has three interests in what a state-component sub-leaf tells about the corresponding state component:
For any non-legacy component, what the corresponding sub-leaf produces in eax is the size in bytes of the component’s part of the XSAVE area. Where this part goes in the XSAVE area depends on whether the area is saved in standard or compacted form. What a state-component sub-leaf produces in ebx is the offset in bytes from the beginning of the XSAVE area to the component’s part of the XSAVE area, if the XSAVE area has the standard form. It is irrelevant to an XSAVE area in compacted form. The xsaves instruction, which is needed for supervisor components, saves in compacted form and so the kernel ignores ebx when enumerating supervisor components.
When enumerating user state components, the kernel records both the offsets and sizes for the components’ parts in standard form for later computation of the total size of an XSAVE area for which components are eventually enabled. This total size is at least the 0x0200 bytes that are required for legacy components and another 0x40 bytes for the XSAVE header. The total size must reach from there to the end of whichever component’s part ends furthest from the start of the XSAVE area. Note that the XSAVE area’s size is not simply a sum of the enabled parts’ sizes. An XSAVE area in standard form may have holes for state components that are not enabled.
The compaction for the compacted form is that the processor removes such holes by downwardly relocating the parts for higher-numbered enabled compoonents. By default, each enabled component’s part in a compacted XSAVE area follows immediately from that of the next lower-numbered enabled component (else from the XSAVE header). If a state component needs that its part in the XSAVE area has 64-byte alignment, even after compaction, its sub-leaf sets the 0x00000002 bit in ecx. Its part in a compacted XSAVE area is then at the first 64-byte boundary at or after that of the next lower-numbered enabled component.
The first Instruction Set Reference that documents this alignment indicator in ecx for state-component sub-leaves of cpuid leaf 13 is Order Number 253666-055, dated June 2015, the month before Windows 10 was released.