KPRCB (i386)

The name KPRCB stands for (Kernel) Processor Control Block. The kernel keeps one KPRCB (formally _KPRCB) for each logical processor as the PrcbData member of the same processor’s KPCR. The KPRCB holds most of what the kernel needs ready access to while managing a processor and while managing resources that are themselves managed more simply (and quickly) per processor. Neither of these structures is formally documented. Both are highly specific to the processor architecture. This page concerns itself only with the KPRCB in 32-bit Windows for the processor architecture that’s variously named i386 or x86. The x64 KPRCB (amd64) is presented separately.

Access

Kernel-mode code can easily find the KPRCB for whichever processor it’s executing on, by finding the current KPCR first. The latter is well-known to be accessible through the fs register. Its Prcb member points to the KPRCB without depending on it to be embedded in the KPCR. Given a C-language definition of the KPCR, getting the current processor’s KPRCB can be conveniently wrapped into one inline function:

FORCEINLINE
KPRCB *KeGetCurrentPrcb (VOID)
{
    return (KPRCB *) __readfsdword (FIELD_OFFSET (KPCR, Prcb));
}

which, less some dressing, is mostly how Microsoft’s own programmers have been doing it, as confirmed by the NTOSP.H that Microsoft published (possibly by oversight) in early editions of the Windows Driver Kit (WDK) for Windows 10. Go back far enough, to version 5.0 and earlier, and the kernel has this as a self-standing routine that is apparently written in assembly language. The .DBG symbol files for Windows NT 4.0 not only name i386pcr.asm as the source file but even tell us that the routine’s two instructions are at lines 61 and 64.

The part of the KPCR that’s ahead of the embedded KPRCB is highly stable. Of particular importance is that the offset of the Prcb member is reliable over all Windows versions. See dword ptr fs:[20h] in kernel-mode code for any 32-bit Windows version and you know that what’s sought is the currently executing processor’s KPRCB.

The KPRCB that Prcb points to is the PrcbData member. Its offset within the KPCR also is stable over all versions. Members of the KPRCB are sometimes accessed through offsets from the KPCR. For some members, this is even the most usual access. Notably, the KeGetCurrentThread function is implemented very like

KTHREAD *KeGetCurrentThread (VOID)
{
    return (KTHREAD *) __readfsdword (FIELD_OFFSET (KPCR, PrcbData.CurrentThread));
}

both as exported and when inlined throughout the kernel. This access to the KPRCB members as nested in the KPCR is formalised in Microsoft’s assembly-language header KS386.INC through such definitions as PcCurrentThread for the preceding field offset. This offset too is stable: see dword ptr fs:[0124h] in kernel-mode code for any 32-bit Windows version and you know that what’s sought is the current thread.

Whether such access to KPRCB members from fs is written in assembly language or C, one case is known of it going wrong, such that the offset applied to fs is relative to the KPRCB instead of the KPCR. Look far below (to offset 0x1A18) for the DpcInterruptRequested member in Windows Vista SP1, specifically. It would not surprise if there have been others. Microsoft understandably does not say and I have neither time nor taste for tracking them down.

Processor Switching

Problems of incorrect offset computation aside, acessing a KPRCB member through the fs register and an offset from the containing KPCR generally is better. This is because the address that some such routine as KeGetCurrentPrcb obtains is merely the address of the KPRCB for the processor that the current thread was being run on at the time. It remains the address of the current KPRCB only while the thread can ensure it is not switched to another processor.

To sense the danger, look again at KeGetCurrentThread and imagine it as coded in two steps:

KTHREAD *KeGetCurrentThread_BAD (VOID)
{
    return KeGetCurrentPrcb () -> CurrentThread;
}

This is unsafe for general use. If you don’t already see why, then kernel-mode programming is not yet an accomplishment—and if you don’t see why by the end of this paragraph, then please leave kernel-mode programming alone for a while. Suppose this bad KeGetCurrentThread is called from thread X. A first instruction executes on processor A and gets the address of A’s KPRCB. A second gets the address of the KTHREAD for the thread that this KPRCB says is executing on processor A. If a thread switch can occur between these two instructions, then although the second instruction also executes for thread X, processor A can by then be executing some other thread Y (and thread X can instead be executing on some other processor B). The routine will find the KTHREAD for Y, not X.

Often, the circumstances are such that the thread can’t be switched. But often is not always, even for the kernel’s own use. How much trouble has been caused by unsynchronised access to a KPRCB for some processor that the thread is no longer running on may be impossible to assess even roughly but I doubt it’s negligible.

Certainly it has not always been attended to closely even by Microsoft’s kernel programmers. As perhaps the simplest possible cases (though also perhaps the most inconsequential), consider the per-processor performance counters CcFastReadNoWait, CcFastReadWait and CcFastReadNotPossible. These are important enough to have been defined from the start—see below at offset 0x0240 for version 3.10. Put aside that it wasn’t until version 5.1 that the writers of third-party file system (filter) drivers were given such functions as FsRtlIncrementCcFastReadNoWait so that the count can include work that they do independently of FsRtlCopyRead, etc. Look instead at the implementations. All that’s needed is one inc instruction relative to fs. Even without a lock prefix, as long as the counter in each KPRCB is never incremented any other way, each counter can only be incremented by the one intended processor. Each truly keeps a per-processor count. But Microsoft did not deliver this simplicity until version 6.1.

Up to and including version 5.2, for both the exported functions and the kernel for its own purposes, the counters are incremented in two steps: first, to get a pointer to the KPRCB; then an (unlocked) inc instruction, using the counter’s offset relative to the KPRCB. This leaves some very slight chance that thread X runs on processor A for the first step but is switched to processor B for the second step while thread Y gets run on processor A and also reaches the second step. The two threads are now running on different processors, X on B and Y on A, but both have pointers to the KPRCB for processor A and seek to increment the same counter concurrently. Without at least a lock prefix, the two processors’ reads and writes for their increments can be interleaved and one of the increments may be lost. Whether someone at Microsoft deduced this as having happened in a real-world case or merely contemplated it may never be known, but version 6.0 changes from inc to a lock xadd. For the limited purpose of these counters, this is good enough: given that thread X in the preceding scenario does run on both processors A and B in and around whatever event is being counted, who’s to say the count is wrong to go to one processor rather than the other?

This case—here just to lose an increment for statistical use only—is arguably of no great consequence. This may be why it went unattended for a decade or so. But this case is also as simple as problems with multi-processing can be, and yet it is hardly trivial. So, let that be a warning for accessing KPRCB members in real-world code!

Documentation Status

The KPRCB is not documented. Though C-language definitions of the KPRCB are in NTDDK.H from as long ago as the Device Driver Kit (DDK) for Windows NT 3.51, they are for other processors than the x86, do not continue beyond Windows XP, and are anyway incomplete. Microsoft’s first known disclosure of a C-language definition of the KPRCB for the x86 is in the NTOSP.H from early editions of the WDK for Windows 10. Publication of this header was possibly an oversight—Microsoft did not repeat it for the 1607 edition—and its definition too is incomplete. Comments explain that the published definitions are just of an “architecturally defined section” that “may be directly addressed by vendor/platform specific HAL code and will not change from version to version of NT.”

The practical equivalent of a C-language definition of the whole structure is available as type information in public symbol files for the kernel, starting with Windows 2000 SP3. Starting with Windows 8, these also tell that the type information came from compiling a header named i386_x.h. Some other symbol files, e.g., those for the HAL in Windows 7 and higher, have type information for only the architecturally defined section at the structure’s start. From these it is known that the incomplete definition is not only in the twice-published NTOSP.H but also in the never-published NTHAL.H.

Layout

The KPRCB is highly variable. The layout changes not just from one version to another but even between builds. To save space and tedium, this article’s presentation of the variations refers to early and late builds of some versions, as defined in the following table of the structure’s varying size:

These sizes, and the names, types and offsets below, are from Microsoft’s public symbols for the kernel, starting from Windows 2000 SP3. Similarly definitive sources are scarce for earlier versions. A statically linked library named craShlib.Lib (sic) from sample code in a Software Development Kit (SDK) for Windows NT 4.0 has an early form of type information for the KPRCB. Members are also named by the !strct command of the KDEX2X86 debugger extension for Windows NT 4.0 and Windows 2000. For most versions before Windows 2000 SP3, members are not known with certainty. Much use in the earliest versions can be more or less easily matched to use in versions for which names and types are known from symbol files. But where such correspondence isn’t known, Microsoft’s names are lost to history and types can only be guessed. It seems I never knew of any use for some large tracts of the KPRCB in the earliest versions. There’s only so much that’s practical to do about that now.

Please bear in mind that the KPRCB is among the largest of kernel-mode structures, not just in terms of size but of member count. There have been frequent rearrangements such that finding a good presentation of this structure’s development even through the versions that are known from symbol files can only ever be a work in progress.

Architecturally Defined Section

Relative to the overall variability of the KPRCB, even just of the architecturally defined section, the very beginning of the structure is remarkably stable through many versions. The first change at all is that Windows Vista found space for a NestingLevel in what had been Reserved and refined the definition of the CpuStep (to clarify that the high and low bytes are model and stepping, respectively). The first change that breaks the stability of this region is when Windows 7 removes SetMember and thus shifts almost all subsequent members of the architecturally defined section.

USHORT MinorVersion;

USHORT MajorVersion;

KTHREAD *CurrentThread;

KTHREAD *NextThread;

KTHREAD *IdleThread;

CHAR Number;

UCHAR Number;

UCHAR LegacyNumber;

CHAR Reserved;

UCHAR NestingLevel;

USHORT BuildType;

KAFFINITY SetMember;

CHAR CpuType;

CHAR CpuID;

USHORT CpuStep;

union {
    USHORT CpuStep;
    struct {
        UCHAR CpuStepping;
        UCHAR CpuModel;
    };
};

KPROCESSOR_STATE ProcessorState;

By no stretch can even the architecturally defined section be thought to have lived up to Microsoft’s comment that it “will not change from version to version of NT”, but the MinorVersion and MajorVersion never have changed. All known versions of the x86 kernel set both to 1. From NTDDK.H for other processors in versions 3.51 to 5.1 inclusive, it can be inferred that Microsoft has the symbolic names PRCB_MAJOR_VERSION and PRCB_MINOR_VERSION for these version numbers, which NTOSP.H confirms for the x86.

Checked builds of the kernel set the 0x0001 bit in the BuildType. Single-processor builds set the 0x0002 bit. Whether a kernel is checked or single-processor is up to the kernel. There can be surprises. For instance, a checked build of NTOSKRNL.EXE for Windows NT 3.51 has the 0x0001 bit set but the 0x0002 bit clear because although it has the name of a single-processor build it has the code of a multi-processor build. Again, Microsoft’s names PRCB_BUILD_DEBUG and PRCB_BUILD_UNIPROCESSOR for these bits can be inferred from definitions for other processors in early versions and are confirmed for the x86 by the NTOSP.H that is published for Windows 10.

The CpuType is what the processor manuals refer to as the family. In eax from cpuid leaf 1, bits 8 to 11 inclusive make a 4-bit family. If all four bits are set, then an 8-bit family to keep as the CpuType is formed by adding the 4-bit family, i.e., 15, to the 8-bit extended family from bits 20 to 27. Or so things go now, both in Intel’s manuals and in the Windows kernel. Beware, though, that the kernel has not always computed it this way exactly. The extended family is ignored before version 5.1. The earliest versions, up to and including the version 4.0 from Windows NT 4.0 SP5, read only a 3-bit family from bits 8 to 10. Indeed, this departure of Microsoft’s from Intel’s specification of a 4-bit family may be the reason that Intel had to introduce the extended family.

The CpuID is redundant now that Windows assumes all processors have the cpuid instruction. Up to and including version 6.2, it is set to 1 if the processor has an acceptable cpuid instruction. Starting with version 3.50, acceptable means that the instruction supports at least leaf 1. Before the version 4.0 from Windows NT 4.0 SP6, it means also that the instruction does not support any leaf higher than 3. In version 6.3 and higher, CpuID is necessarily 1 (except briefly while the KPRCB exists but the processor’s family, model and stepping aren’t yet known).

The CpuStepping and CpuModel are named straightforwardly from the manuals. The CpuStepping is bits 0 to 3 inclusive of eax from cpuid leaf 1. The CpuModel is bits 4 to 7 inclusive, except that versions 5.1 and higher allow for two cases in which an extended model in bits 16 to 19 supply four high bits to make an 8-bit CpuModel. This use of the extended model is indicated if either: the 4-bit family is 15; or it is 6 and the vendor as known from cpuid leaf 0 is either GenuineIntel or, in version 6.2 and higher, CentaurHauls.

Within a processor family, the CpuModel and CpuStepping are much like major and minor version numbers. They usually are taken together, especially to compare against some cut-off, as when some feature flag is set but the indicated feature is thought to have been faulty before some model and stepping—or, the other way round, when the feature is known to have been present without the formal indication. Versions 3.10 and 3.50 have a case of taking the CpuID and CpuType together as a word, which may be a coding oversight but at least has some merit as comparing for either a minimum family or cpuid support. Versions before 6.3 have a curious case of taking CpuModel, CpuStepping, CpuID and CpuType together as a dword. This isn’t credibly anything other than a coding oversight. It will have had a possibly harmless side-effect for versions up to and including 3.51 when running on an 80386: presence of an 80387 will have caused the kernel to set the cr0 bit NE (5) which the 80386 doesn’t have.

Whatever Microsoft may have started out meaning by “architecturally defined”, it did not mean even as early as Windows 2000 that architecturally defined members do not move. The KPROCESSOR_STATE contains a CONTEXT. The latter has long been documented, with C-language definitions in WINNT.H and NTDDK.H, but because its size changed for Windows 2000, due to the addition of ExtendedRegisters, even the ancient KPRCB members that are defined beyond this point all change their position at least once.

The ProcessorState itself moved for version 6.1. This shift gave it 8-byte alignment, which may have been seen as a happy side-effect when version 6.2 added a 64-bit register to the KSPECIAL_REGISTERS that’s nested in the KPROCESSOR_STATE.

Kernel And HAL Reservations

Two specifically reserved areas, one each for the kernel and the HAL, are also supposed to be architectural. The kernel’s reserved area starts to get fleshed out with Windows 8.1, though only then to define two members at the start.

KNODE *ParentNode;

CHAR *PriorityState;

ULONG KernelReserved [0x10];

ULONG KernelReserved [0x0E];

ULONG HalReserved [0x10];

The HAL’s reserved area certainly does get used by at least some HALs—even from long ago, e.g., HALMPS.DLL from Windows NT 3.50—but the kernel knows nothing of it.

Architectural Padding

The preceding reservations for the kernel and HAL look like they originally ended the architecturally defined section: the next few members in versions 3.10 to 4.0 survive to later versions whose symbol files place them firmly beyond the reach of the NTOSP.H definition. Version 5.0 inserted an array of per-processor spin lock queues as the architecturally defined section’s new end: see LockQueue at offset 0x03BC in version 5.0. Version 5.1 pushed this array further into the structure by inserting 0x5C bytes explicitly as padding: see PrcbPad0, below.

There seem to have been two intentions. One is that the LockQueue, initially at offset 0x03BC, should have some cache alignment. For reasons not yet understood, the cache alignment is not of the array from its beginning but from its second element. This is clearly by design, for the same outcome is achieved differently in the x64 KPRCB, and it is confirmed by a comment in the NTOSP.H from the WDK for Windows 10:

// N.B. The following padding is such that the first lock entry falls in the
//      last eight bytes of a cache line.

A plausible second intention was to set aside not the bare minimum for cache alignment but enough space that future additions to the architecturally defined section should not again shift the lock queues (which are presumably at the end of the architecturally defined section so that they too can grow without shifting anything else that’s architectural).

These future additions arrived with Windows Vista, which claims four bytes at the start. Its first Service Pack brings members from (much) further into the structure, presumably for better-defined access to them from outside the kernel—and the HAL certainly does access many of them. But note two things. First, these members were not moved here to remain at their new locations forever. In particular, the CpuVendor was moved into this padding for version 6.1 but then version 6.3 reordered it relative to what remained of the padding. Second, no matter what is added, or even moved after being added, the padding is always adjusted—and even split—so that whatever was at offset 0x0418 after this insertion of padding for version 5.1 is there forever.

ULONG CFlushSize;

UCHAR CoresPerPhysicalProcessor;

UCHAR LogicalProcessorsPerCore;

UCHAR CpuVendor;

UCHAR PrcbPad0 [0x5C];

UCHAR PrcbPad0 [0x58];

UCHAR PrcbPad0 [2];

UCHAR PrcbPad0 [1];

ULONG MHz;

UCHAR CpuVendor;

UCHAR GroupIndex;

USHORT Group;

UCHAR Group;

UCHAR PrcbPad05 [2];

KAFFINITY GroupSetMember;

ULONG Number;

BOOLEAN ClockOwner;

UCHAR PendingTick;

union {
    UCHAR PendingTickFlags;
    struct {
        UCHAR PendingTick : 1;          // 0x01
        UCHAR PendingBackupTick : 1;    // 0x02
    };
};

UCHAR PrcbPad1 [0x50];

UCHAR PrcbPad1 [0x48];

UCHAR PrcbPad1 [0x46];

UCHAR PrcbPad10 [0x46];

The CFlushSize is the size of the cache line in bytes as used for the clflush and clflushopt instructions. The kernel’s first use of clflush is for the exported function KeInvalidateRangeAllCaches in version 6.0. Other uses have been found since, and this first use is replaced in version 10.0 and higher by clflushopt if it’s available. In all versions, the cache line’s size is computed as eight times the second lowest byte, i.e., bits 8 to 15 inclusive, of what cpuid leaf 1 returns in ebx.

The number of CoresPerPhysicalProcessor defaults to 1, of course. That it can be anything else isn’t noticed before version 6.0. Generally, it is one more than what cpuid leaf 4 returns in the high 6 bits of eax. For AMD processors, the kernel instead interprets what cpuid leaf 0x80000008 returns in ecx. Either way, versions 6.1 and higher round up to a power of two.

The CpuVendor is a convenient interpretation of the CPU vendor string that is produced by cpuid leaf 0. It takes its values from the (x86-specific) CPU_VENDORS enumeration.

Spin Lock Queues

That the kernel has a “per processor lock queue” is well known from comments in NTDDK.H from as long ago as the DDK for Windows XP (moved to WDM.H in the WDK for Windows Vista). The comments come a little before the definition of the KSPIN_LOCK_QUEUE structure, which appears to be provided only so that callers of such newly introduced HAL functions as KeAcquireInStackQueuedSpinLock can create the necessary KLOCK_QUEUE_HANDLE. Less well known is that this structure, and queued spin locks as a feature, date from Windows 2000. This first existence was not for general use but for dedicated purposes, with queueing supported only through the following array for each processor:

KSPIN_LOCK_QUEUE LockQueue [0x10];

KSPIN_LOCK_QUEUE LockQueue [LockQueueMaximumLock];

UCHAR PrcbPad1 [8];

As noted above, trouble is taken to keep the LockQueue reliably at offset 0x0418 ever since Windows XP. Comparison with the 64-bit KPRCB suggests that what’s wanted is perhaps not a specific offset but that the second entry is cache-aligned or that the first is in a separate cache line from all the others. Such alignment to 0x40 bytes becomes ever more important through successive versions. Remember always that what counts for alignment is not the offset within the KPRCB but within the containing KPCR (which is necessarily not just cache-aligned but page-aligned). Programmers who deal much with the KPCR and KPRCB in the debugger—and reverse engineers in their disassemblies—will be familiar with adding and subtracting 0x0120.

The LockQueue array is indexed by members of the enumeration KSPIN_LOCK_QUEUE_NUMBER, which is defined in WDM.H and is used in NTIFS.H for the declarations of the exported functions such as KeAcquireQueuedSpinLock that operate on the applicable spin locks through these per-processor lock queues. These declarations would have it that the functions require at least Windows XP. The functions are documented, but only to say they’re reserved.

The array initially allows for 0x10 entries even though use was not defined for all 0x10 of them until version 5.2 The versions that trouble over cache-alignment for the second entry also have either that the array ends at a cache-line boundary or is padded to the next boundary. Since version 5.1, then, the non-architectural section is cache-aligned.

Non-Architectural

The NTOSP.H from the WDK for Windows 10 confirms LockQueue as the last member of the architecturally defined section that is all that NTOSP.H defines of the KPRCB. Though trouble has been taken through successive revisions to keep the LockQueue array at a reliable offset, the array’s highly variable size through versions 5.1 to 6.1 means that most members in the non-architectural section shift wildly in these versions even when not reordered.

KTHREAD *NpxThread;

ULONG InterruptCount;

LARGE_INTEGER KernelTime;

ULONG KernelTime;

LARGE_INTEGER UserTime;

ULONG UserTime;

LARGE_INTEGER DpcTime;

ULONG DpcTime;

ULONG DebugDpcTime;

ULONG DpcTimeCount;

LARGE_INTEGER InterruptTime;

ULONG InterruptTime;

The InterruptCount, KernelTime, UserTime, DpcTime and InterruptTime are all retrievable through the SystemProcessorPerformanceInformation (0x08) case of NtQuerySystemInformation in all known versions. See that version 3.10 keeps the raw 64-bit times. Later versions scale by the so-called maximum increment.

The exported KeUpdateRunTime function that updates those times (and DpcTimeCount in version 6.0 and higher) also reduces the current thread’s quantum. If this leaves the thread with no more quantum, version 3.10 schedules a DPC that will find another thread to run. This was improved upon as early as version 3.50 so that the 0x20 bytes of this KDPC whose name may never be known seem to have became spare. This is here supposed as the origin of the Spare2 that is known from symbol files in later versions. The space recovered from this unknown KDPC started to get used in version 3.51:

ULONG ApcBypassCount;

ULONG DpcBypassCount;

ULONG AdjustDpcThreshold;

ULONG DebugDpcTime;

ULONG Spare2 [8];

ULONG Spare2 [5];

ULONG Spare2 [4];

The ApcBypassCount and DpcBypassCount are retrievable through the SystemInterruptInformation (0x17) case of NtQuerySystemInformation in the applicable versions. Since their discontinuation in version 5.1, the corresponding members in the retrieved information are both zero.

No use of DebugDpcTime is known in any version, either here (where type information from public symbol files for the kernel places it definitively, at least for the later builds of version 5.0) or at its next position until version 6.0 reuses it (or relabels it) as DpcTimeCount. Type information in CRASHLIB.LIB from Dr. Watson sample code in the SDK for Windows NT 4.0 has Spare2 follow immediately from AdjustDpcThreshold, confirming that DebugDpcTime was not yet defined.

ULONG AdjustDpcThreshold;

ULONG PageColor;

ULONG SkipTick;

UCHAR SkipTick;

UCHAR MultiThreadSetBusy;

UCHAR DebuggerSavedIRQL;

UCHAR NodeColor;

UCHAR DeepSleep;

UCHAR TbFlushListActive;

UCHAR Spare2 [3];

UCHAR Spare1 [6];

UCHAR Spare1;

UCHAR PollSlot;

UCHAR PrcbPad20 [2];

UCHAR PrcbPad20 [6];

UCHAR PrcbPad20 [5];

UCHAR PrcbPad20;

PVOID volatile CachedStack;

ULONG NodeShiftedColor;

KNODE *ParentNode;

ULONG MultiThreadProcessorSet;

KPRCB *MultiThreadSetMaster;

No use of ThreadStartCount, below, is known in any version before symbol files show its reuse for late builds of version 5.2. That it is defined as early as version 3.51 is uncertain. It is confirmed for version 4.0 from type information in the statically linked library CRASHLIB.LIB from contemporaneous Dr. Watson sample code.

ULONG ThreadStartCount [2];

ULONG SecondaryColorMask;

LONG Sleeping;

ULONG DpcTimeLimit;

Inter-Processor Interrupts

The very earliest versions deal here with inter-processor interrupts. The implementation soon increased in sophistication and the supporting members were relocated much further into the KPRCB (and reordered).

BOOLEAN volatile IpiCommands [4];

UCHAR volatile IpiFrozen;

ULONG volatile ReverseStall;

ULONG volatile IpiLastPacket;

The IpiCommands array records which of the four possible types of packet-less request are being sent to this processor. They are set by the kernel’s KiIpiSend function (which is exported in these versions) on the way to asking the HAL’s HalRequestIpi function to interrupt this processor. They get cleared at this processor by the kernel’s exported KiIpiServiceRoutine. The name IpiCommands is inferred with reasonable certainty from the I386KD debugger for version 3.10 and the KDEXTX86 debugger extension for version 3.51. Their descriptions of the four types of request are: APC, DPC, Clock and Freeze. These commands are essentially signals to this processor to start some pre-determined activity such as executing Deferred Procedure Calls. Each signal is either set or not. Each can have been set by multiple sending processors concurrently before this processor gets interrupted.

The names IpiFrozen and IpiLastPacket are also known from the debugger (and the latter from the extension).

The IpiLastPacket is the 32-bit sequence number of the packet that the processor last serviced (or most recently started servicing). In these versions, a processor can send a packet-based request to arbitrarily many processors but multiple processors cannot send requests concurrently. The packet’s three parameters (or four, in version 3.50) and the address of the worker routine that is to execute for the packet on each target processor are kept in the kernel’s own data. So too is an array of boolean flags, one for each of the possible 32 processors, which the worker routine must signal. The sending processor holds a spin lock and waits for the signal from each target processor. Among the inefficiencies is that while the sending processor waits, any target processor can receive not just the one inter-processor interrupt to service the packet-based request but others for packet-less commands. The sequence number is the defence against re-servicing a packet. Whether this sequence number is signed or unsigned is not known: type information never becomes available because it does not survive to version 4.0, which provides for concurrent senders.

Per-Processor File Locking

Two pointers that are explicitly spare in version 5.0 according to symbol files are known to be used in the earliest Windows versions. They each point to a chain of freed fixed-sized allocations for managing file locks. The first pointer is for shared locks, the second for exclusive. The allocations are containers for a copy of the FILE_LOCK_INFO structure that is given when locking bytes in a file. Some number (10 in 3.51 but 8 in 4.0) of allocations are obtained in advance from the non-paged pool and are pre-freed to these caches. In essence, these are rough-and-ready implementations of what version 5.0 would formalise as per-processor look-aside lists (see PPLookasideList, below, at offset 0x0500). Late builds of version 4.0 do indeed change to lookaside lists for file locking, but they use ordinary look-aside lists in the kernel’s data, and per-processor caching for file locking never is returned to.

SINGLE_LIST_ENTRY FsRtlFreeSharedLockList;

PVOID SpareHotData [2];

SINGLE_LIST_ENTRY FsRtlFreeExclusiveLockList;

Performance Counters

Though performance counters evidently were never intended as part of an “architecturally defined section”, the KPRCB has at least some from the very beginning, as with the following which help assess the performance of Fast I/O by the file system (including third-party file system drivers). That the kernel keeps a “per processor control block of cache manager system counters” is disclosed by Microsoft in documentation of such functions as FsRtlIncrementCcFastReadNoWait.

The point to counting separately for each processor is typically not to learn how the count for any one processor differs from that for any other. The total count is what matters, but keeping it is faster if done in parts. Given that the counter for each processor is incremented only by code that is running on that processor, the increments can be done safely and quickly without concern for synchronisation. They don’t even need to be interlocked. To evaluate the counter is to tally the per-processor instances, but since this will be wanted only infrequently its overhead is nothing relative to the gain from the likely numerous increments.

Sets of counters that are likely to be touched together would better be in the same cache line. In versions 5.1 to 6.0, the first six counters do indeed start at a 64-byte address boundary, relative to the containing KPCR. Although this cache alignment is achieved without explicit padding, it seems unlikely to be accidental, especially given that version 5.1 is when Windows starts paying attention to cache lines. Later versions work to keep it, first by padding explicitly, then by bringing the space into use:

ULONG PrcbPad21 [2];

ULONG PrcbPad21 [3];

ULONG PrcbPad21 [2];

PVOID MmFlushList;

PVOID MmInternal;

KPRCBFLAG PrcbFlags;

PVOID SchedulerAssist;

Totals for the first six of the per-processor performance counters have always been retrievable through the SystemPerformanceInformation (0x02) case of NtQuerySystemInformation:

ULONG CcFastReadNoWait;

ULONG CcFastReadWait;

ULONG CcFastReadNotPossible;

ULONG CcCopyReadNoWait;

ULONG CcCopyReadWait;

ULONG CcCopyReadNoWaitMiss;

Windows Vista adds substantially, including some that Windows Server 2003 SP1 had added further on. All except MmSpinLockOrdering are retrievable through the SystemPerformanceInformation case of NtQuerySystemInformation (as are the preceding originals). They always have been, but as simple counters in kernel data. It just took a while before Windows counted them in per-processor parts.

LONG volatile MmSpinLockOrdering;

LONG volatile IoReadOperationCount;

LONG volatile IoWriteOperationCount;

LONG volatile IoOtherOperationCount;

LARGE_INTEGER IoReadTransferCount;

LARGE_INTEGER IoWriteTransferCount;

LARGE_INTEGER IoOtherTransferCount;

ULONG CcFastMdlReadNoWait;

ULONG CcFastMdlReadWait;

ULONG CcFastMdlReadNotPossible;

ULONG CcMapDataNoWait;

ULONG CcMapDataWait;

ULONG CcPinMappedDataCount;

ULONG CcPinReadNoWait;

ULONG CcPinReadWait;

ULONG CcMdlReadNoWait;

ULONG CcMdlReadWait;

ULONG CcLazyWriteHotSpots;

ULONG CcLazyWriteIos;

ULONG CcLazyWritePages;

ULONG CcDataFlushes;

ULONG CcDataPages;

ULONG CcLostDelayedWrites;

ULONG CcFastReadResourceMiss;

ULONG CcCopyReadWaitMiss;

ULONG CcFastMdlReadResourceMiss;

ULONG CcMapDataNoWaitMiss;

ULONG CcMapDataWaitMiss;

ULONG CcPinReadNoWaitMiss;

ULONG CcPinReadWaitMiss;

ULONG CcMdlReadNoWaitMiss;

ULONG CcMdlReadWaitMiss;

ULONG CcReadAheadIos;

The preceding selection of performance counters for the Cache Manager expanded as soon as version 3.50 with some miscellany for the Kernel Core. Several are retrievable through NtQuerySystemInformation:

• SystemPerformanceInformation (again) for KeContextSwitches, KeFirstLevelTbFills, KeSecondLevelTbFills and KeSystemCalls;
• SystemExceptionInformation for KeAlignmentFixupCount, KeExceptionDispatchCount and KeFloatingEmulationCount.

No use is known of KeDcacheFlushCount or KeIcacheFlushCount in any version.

ULONG KeAlignmentFixupCount;

ULONG KeContextSwitches;

ULONG SpareCounter0;

ULONG KeDcacheFlushCount;

ULONG KeExceptionDispatchCount;

ULONG KeFirstLevelTbFills;

ULONG KeFloatingEmulationCount;

ULONG KeIcacheFlushCount;

ULONG KeSecondLevelTbFills;

ULONG KeSystemCalls;

ULONG AvailableTime;

When Windows Server 2003 SP1 added counters for I/O operations, it appended to the existing counters. But they didn’t stay for long: Windows Vista moved them forward.

LONG volatile IoReadOperationCount;

LONG volatile IoWriteOperationCount;

LONG volatile IoOtherOperationCount;

LARGE_INTEGER IoReadTransferCount;

LARGE_INTEGER IoWriteTransferCount;

LARGE_INTEGER IoOtherTransferCount;

The early versions follow their performance counters with space for which no use is known other than two pointers that added to the original optimisation of file locking. It is mere supposition that this unaccounted space in the earliest versions is just a larger allowance for the same reservation that is known for Windows NT 4.0 and Windows 2000 from type information.

SINGLE_LIST_ENTRY FsRtlFreeWaitingLockList;

SINGLE_LIST_ENTRY FsRtlFreeLockTreeNodeList;

ULONG ReservedCounter [0x10];

ULONG ReservedCounter [0x20];

ULONG ReservedCounter [6];

ULONG ReservedCounter [8];

ULONG SpareCounter0 [1];

ULONG SpareCounter1;

ULONG SpareCounter1 [8];

ULONG PrcbPad1 [3];

ULONG PrcbPad2 [3];

ULONG PrcbPad22 [2];

That space had been set aside for counters was remembered in the names until a change for version 6.0. But whatever the name, every version from 4.0 onwards allows for expansion but takes care to adjust the padding to end exactly at the next 64-byte alignment boundary relative to the containing KPCR.

Per-Processor Lookaside Lists

Symbol files for Windows 2000 SP3 and SP4 and even the CRASHLIB.LIB for Windows NT 4.0 define the following pointers, apparently to chains of freed structures as a cache to speed their reuse. However, no use is known of them. They perhaps remain from development of the per-processor lookaside lists for special purposes (taken up next). After all, the names suggest the same purposes in the same order.

PVOID SmallIrpFreeEntry;

PVOID LargeIrpFreeEntry;

PVOID MdlFreeEntry;

PVOID CreateInfoFreeEntry;

PVOID NameBufferFreeEntry;

PVOID SharedCacheMapFreeEntry;

ULONG CachePad0 [2];

Note that this first name that explicitly suggests padding for cache alignment does not actually cache-align what follows. It does within the KPRCB, but the KPRCB itself is not cache-aligned within the KPCR.

System Lookaside Lists

Though lookaside lists were introduced for version 4.0, it’s not until version 5.0 that the kernel sets some up for per-processor use. Broadly speaking, there are two separate types of per-processor lookaside lists. First come the system lookaside lists.

PP_LOOKASIDE_LIST PPLookasideList [0x10];

The PPLookasideList array is indexed by the undocumented enumeration PP_NPAGED_LOOKASIDE_NUMBER. Its different values represent lookaside lists that cache very different fixed-size structures that each have a very specific purpose.

The undocumented PP_LOOKASIDE_LIST structure is a pair of pointers, P and L, to the actual lookaside lists. Ideally, they point to separate lists: the first just for the processor; the second shared. Allocations are sought first from the per-processor list, for speed, else from the shared. Allocations are freed to the per-processor list for easy re-allocation, except that if that list has reached its capacity the allocation is instead freed to the shared list.

To support the SystemLookasideInformation case of the NtQuerySystemInformation function the kernel keeps a double-linked list of all the lookaside lists that are pointed to from the PPLookasideList arrays for all processors. Although additions to the list, when building the KPRCB for a newly added processor, can be made by only one thread at a time, no synchronisation is known with this function’s enumeration of the list.

Note that in version 5.1 and higher, the PPLookasideList array starts on a 64-byte boundary, relative to the containing KPCR. Indeed, all the array that is known to be used before Windows 7 fits in one 64-byte cache line. The allowance for 0x10 lists when only 9 are yet defined is presumably intended so that the pool lookaside lists that follow are also cache-aligned.

Pool Lookaside Lists

GENERAL_LOOKASIDE_POOL PPNxPagedLookasideList [0x20];

PP_LOOKASIDE_LIST PPNPagedLookasideList [8];

PP_LOOKASIDE_LIST PPNPagedLookasideList [0x20];

GENERAL_LOOKASIDE_POOL PPNPagedLookasideList [0x20];

PP_LOOKASIDE_LIST PPPagedLookasideList [8];

PP_LOOKASIDE_LIST PPPagedLookasideList [0x20];

GENERAL_LOOKASIDE_POOL PPPagedLookasideList [0x20];

The pool lookaside lists help with the efficiency of small allocations from various types of pool (NonPagedPool, PagedPool and, in version 6.2 and higher, NonPagedPoolNx). Successive lookaside lists in each array are for successively larger sizes of allocation from that pool type. When first introduced, for Windows 2000, the caching was relatively coarse. The first list was for all allocations up to and including 0x20 bytes, which was also the increment from one list to the next. Eight lists thus supported per-processor caching of freed pool allocations up to and including 0x0100 bytes. Windows XP and higher have the same coverage but in increments of 0x08 bytes.

To support the SystemPerformanceInformation and SystemLookasideInformation cases of the NtQuerySystemInformation function the kernel keeps a double-linked list of all the pool lookaside lists for all processors. Although additions to the list, when building the KPRCB for a newly added processor, can be made by only one thread at a time, no synchronisation is known with this function’s enumeration of the list.

ULONG ReservedPad [0x80];

ULONG ReservedPad [0x20];

Inter-Processor Interrupts

As an earlier implementation of inter-processor interrupts grew in sophistication it moved here for version 3.51, and has mostly stayed, albeit with reordering. Insertions for versions 4.0 and 5.1 put the supporting members into three sets, each in its own cache line.

ULONG volatile PacketBarrier;

ULONG volatile ReverseStall;

PVOID IpiFrame;

UCHAR PrcbPad2 [0x34];

UCHAR PrcbPad3 [0x34];

The name ReverseStall takes its name from the stall that is typically entered by the sender to wait for its request to be serviced by all the targets. The reverse is that the target stalls too. Having signalled that it is done with the packet, it stalls until someone changes whatever was in the sender’s ReverseStall.

PVOID volatile CurrentPacket [3];

KAFFINITY volatile TargetSet;

PKIPI_WORKER volatile WorkerRoutine;

ULONG volatile IpiFrozen;

ULONG CachePad1 [2];

UCHAR PrcbPad3 [0x28];

UCHAR PrcbPad4 [0x28];

These additions for version 4.0 are the essence of what may at the time have been a big advance in sending packet-based requests between processors. Before version 4.0, a processor could send a packet to multiple targets but two processors cannot send packets concurrently, not even to separate targets. The new CurrentPacket, TargetSet and WorkerRoutine are in essence the packet: three arbitrary parameters; a bitmap of targets; a worker routine that’s to execute on each target (and be given the three parameters). The bottleneck in earlier versions is that these are kept in the kernel’s own data. Version 4.0 keeps them in the sender’s KPRCB. Each target knows who sent the packet because the sender records the address of its own KPRCB as the SignalDone member (in the next cache line) in each target’s KPRCB.

The type PKIPI_WORKER is is defined in one or more of the NTDDK.H, WDM.H and NTIFS.H from the Device Driver Kit (DDK) for Windows NT 4.0 up to and including the WDK for Windows, and then only ever again in the NTOSP.H from those early editions of the WDK for Windows 10. The type has a subtle change between versions 4.0 and 5.0. In all versions it is

typedef 
VOID 
(*PKIPI_WORKER) (
    PKIPI_CONTEXT PacketContext, 
    PVOID Parameter1, 
    PVOID Parameter2, 
    PVOID Parameter3);

but the PKIPI_CONTEXT is a pointer to a ULONG in version 4.0 and then becomes a pointer to void. In version 3.51, this first argument is a pointer to a 20-byte structure which perhaps was named KIPI_CONTEXT. Microsoft’s names for its members are not known, though it would not surprise to find that some of them survive as (otherwise quirky) names of other new KPRCB members.

BOOLEAN volatile IpiCommands [4];

ULONG volatile RequestSummary;

ULONG volatile IpiLastPacket;

KPRCB volatile *SignalDone;

LONG volatile TargetCount;

ULONG volatile ReverseStall;

UCHAR volatile IpiFrozen;

PVOID IpiFrame;

The RequestSummary starts with the old implementation as an array of bytes that record which of the four possible types of packet-less request are being sent to the processor. Version 4.0 changed to bit flags, perhaps anticipating new types of requests, one of which was added for version 5.0. Microsoft’s assembly-language names for these bit flags have long been public: see, for instance, IPI_APC, in the KS386.INC from the DDK for Windows Server 2003 SP1.

ULONGLONG LastNonHrTimerExpiration;

ULONG PrcbPad94 [2];

ULONGLONG TrappedSecurityDomain;

union {
    USHORT BpbState;
    struct {
        USHORT BpbIbrsPresent : 1;
        USHORT BpbStibpPresent : 1;
        USHORT BpbSmepPresent : 1;
        USHORT BpbSimulateSpecCtrl : 1;
        USHORT BpbSimulateIbpb : 1;
        USHORT BpbIbpbPresent : 1;
        USHORT BpbCpuIdle : 1;
        USHORT BpbClearSpecCtrlOnIdle : 1;
        USHORT BpbHTDisabled : 1;
        USHORT BpbUserToUserOnly : 1;
        USHORT BpbReserved : 6;
    };
};

union {
    UCHAR BpbState;
    struct {
        UCHAR BpbCpuIdle : 1;
        UCHAR BpbFlushRsbOnTrap : 1;
        UCHAR BpbIbpbOnReturn : 1;
        UCHAR BpbIbpbOnTrap : 1;
        UCHAR BpbReserved : 4;
    };
};

union {
    UCHAR BpbFeatures;
    struct {
        UCHAR BpbClearOnIdle : 1;
        UCHAR BpbEnabled : 1;
        UCHAR BpbSmep : 1;
        UCHAR BpbFeaturesReserved : 5;
    };
};

UCHAR BpbSpecCtrlValue;

UCHAR BpbCurrentSpecCtrl;

UCHAR BpbCtxSwapSetValue;

UCHAR BpbKernelSpecCtrl;

UCHAR BpbNmiSpecCtrl;

UCHAR BpbUserSpecCtrl;

UCHAR PrcbPad49 [2];

ULONG ProcessorSignature;

ULONG ProcessorFlags;

ULONG CachePad2 [4];

UCHAR PrcbPad4 [0x38];

UCHAR PrcbPad5 [0x38];

UCHAR PrcbPad50 [0x38];

UCHAR PrcbPad50 [0x30];

UCHAR PrcbPad50 [0x28];

UCHAR PrcbPad50 [0x20];

UCHAR PrcbPad50 [0x14];

UCHAR PrcbPad50 [8];

Despite being named CachePad2 initially, the preceding padding does not actually cache-align what follows in versions 4.0 and 5.0. It doesn’t in version 6.2 and higher, but deliberately. These versions instead squeeze two or four members into the end of the cache line:

ULONG InterruptLastCount;

ULONG InterruptRate;

ULONG DeviceInterrupts;

PVOID IsrDpcStats;

What IsrDpcStats points to, when it does hold a pointer, is an ISRDPCSTATS structure.

Deferred Procedure Calls

A region of members concerned with DPC management was rearranged so much for version 5.2, with its introduction of Threaded DPCs, that it seems better to separate the layouts. Even before then, there were substantial introductions for version 3.51 and then rearrangements for version 5.1.

ULONG volatile DpcInterruptRequested;

PVOID ChainedInterruptList;

ULONG CachePad3 [3];

ULONG CachePad3 [2];

ULONG MaximumDpcQueueDepth;

ULONG MinimumDpcRate;

ULONG CachePad4 [2];

There is some suggestion that the members from DpcListHead to DpcLock, below, were once modelled as a sub-structure. For instance, code for KiDispatchInterrupt in version 3.50 accesses the lock as an offset from the list head, and this seems unlikely to be just an optimisation by the contemporaneous compiler. The space between these members is here thought to have been the original KernelReserved2, as if planned for development as the DPC functionality got more sophisticated.

LIST_ENTRY DpcListHead;

PVOID DpcStack;

ULONG DpcCount;

ULONG DpcQueueDepth;

ULONG volatile DpcQueueDepth;

ULONG DpcRoutineActive;

ULONG volatile DpcRoutineActive;

ULONG volatile DpcInterruptRequested;

ULONG DpcCount;

ULONG DpcLastCount;

ULONG DpcRequestRate;

ULONG MaximumDpcQueueDepth;

ULONG MinimumDpcRate;

PVOID DpcStack;

ULONG QuantumEnd;

ULONG KernelReserved2 [0x0F];

ULONG KernelReserved2 [0x0B];

ULONG KernelReserved2 [0x0A];

UCHAR PrcbPad5 [0x10];

KSPIN_LOCK DpcLock;

UCHAR PrcbPad6 [0x3C];

UCHAR PrcbPad6 [0x1C];

When DpcRoutineActive was moved here from the KPCR it didn’t become a ULONG just to formalise that the kernel sometimes accessed it as 32-bit boolean. The implementation changed such that DpcRoutineActive holds a pointer into the stack of the kernel routine that holds the DpcLock and is retiring the DPCs that it removes ffrom the DpcListHead. That the exported function KeIsExecutingDpc continued to return all 32 bits may mean it was never intended to return a BOOLEAN specifically. There’s no documention even now or a contemporaneous declaration to go by. Yet when DpcRoutineActive was reimplemented as a BOOLEAN for version 5.2, the code was changed to return a zero-extended byte.

DPC Management in Windows Server 2003 and Higher

Version 5.2 introduced Threaded DPCs. These reproduce some of the control data that support normal DPCs, and thus DpcListHead, DpcLock, DpcQueueDepth and DpcCount were gathered into a structure. Each processor has two of these KDPC_DATA structures: the first for normal DPCs; the second for Threaded DPCs.

KDPC_DATA DpcData [2];

PVOID DpcStack;

ULONG MaximumDpcQueueDepth;

LONG MaximumDpcQueueDepth;

ULONG DpcRequestRate;

ULONG MinimumDpcRate;

BOOLEAN volatile DpcInterruptRequested;

BOOLEAN volatile DpcThreadRequested;

BOOLEAN volatile DpcRoutineActive;

BOOLEAN volatile DpcThreadActive;

The DpcInterruptRequested member is notable for some confusion in Windows Vista SP1. This build added one instruction at very nearly the start of the KiDispatchInterrupt function just to clear this member. This function addresses KPRCB members via a pointer to the current processor’s KPCR, relying on the KPRCB to be embedded in the KPCR as the PrcbData member (at offset 0x0120). For Windows Vista SP1, however, the function addresses DpcInterruptRequested relative to the KPRCB not the KPCR. Instead of clearing the intended byte, it clears a byte 0x0120 bytes lower in memory. Put aside any ill effect from not clearing the intended byte. Look instead at what byte gets cleared by mistake. Fortunately, this byte is in the Tag in the PPPagedLookasideList for paged pool allocations of 0x0100 bytes. This means the corruption is harmless but it also gives the curiosity of being observable from user mode (in the output from the SystemLookasideInformation case of NtQuerySystemInformation). The error in addressing was corrected as early as Windows Vista SP2 and it was soon made irrelevant by a reworking of DPC management for Windows 7.

As noted earlier, version 5.2 made DpcRoutineActive into a BOOLEAN and recoded the undocumented KeIsExecutingDpc function to return just the zero-extended BOOLEAN. Version 6.0 computes its answer as TRUE or FALSE not from DpcRoutineActive alone but from it and DpcThreadActive taken together as one 16-bit word. Its answer for whether its caller is in turn called from someone’s handling of a DPC is therefore yes if either byte is non-zero.

ULONG PrcbLock;

ULONG DpcLastCount;

ULONG PrcbLock;

KGATE DpcGate;

ULONG volatile TimerHand;

ULONG volatile TimerRequest;

PVOID PrcbPad41;

PVOID DpcThread;

KEVENT DpcEvent;

BOOLEAN ThreadDpcEnable;

UCHAR IdleState;

BOOLEAN volatile QuantumEnd;

UCHAR PrcbPad50;

BOOLEAN volatile DpcRoutineActive;

BOOLEAN volatile IdleSchedule;

LONG DpcSetEventRequest;

union {
    LONG volatile DpcRequestSummary;
    SHORT DpcRequestSlot [2];
    /*  changing members, follow link  */
};

Timing

LONG Sleeping;

ULONG volatile TimerHand;

ULONG LastTimerHand;

ULONG LastTick;

LONG MasterOffset;

ULONG PrcbPad41 [2];

ULONG PeriodicCount;

ULONG PeriodicBias;

UCHAR PrcbPad5 [0x16];

UCHAR PrcbPad5 [0x12];

UCHAR PrcbPad5 [6];

UCHAR PrcbPad51 [6];

LONG TickOffset;

ULONGLONG TickOffset;

ULONG ClockInterrupts;

ULONG ReadyScanTick;

UCHAR BalanceState;

BOOLEAN GroupSchedulingOverQuota;

UCHAR ThreadDpcEnable;

UCHAR PrcbPad41 [10];

UCHAR PrcbPad41 [3];

UCHAR PrcbPad41 [2];

UCHAR PrcbPad41 [6];

TO BE DONE

KTIMER_TABLE TimerTable;

ULONG PrcbPad92 [12];

KDPC CallDpc;

LONG ClockKeepAlive;

UCHAR ClockCheckSlot;

UCHAR ClockPollCycle;

UCHAR PrcbPad6 [2];

UCHAR PrcbPad6 [4];

LONG DpcWatchdogPeriod;

LONG DpcWatchdogCount;

LONG ThreadWatchdogPeriod;

LONG ThreadWatchdogCount;

LONG volatile KeSpinLockOrdering;

ULONG DpcWatchdogProfileCumulativeDpcThreshold;

ULONG PrcbPad7 [8];

ULONG PrcbPad70 [2];

ULONG PrcbPad70 [1];

TO BE DONE

ULONG QueueIndex;

SINGLE_LIST_ENTRY DeferredReadyListHead;

LIST_ENTRY WaitListHead;

ULONG ReadySummary;

LONG AffinitizedSelectionMask;

KSPIN_LOCK WaitLock;

ULONG ReadySummary;

LIST_ENTRY WaitListHead;

ULONG SelectNextLast;

ULONG QueueIndex;

ULONG ReadyQueueWeight;

LIST_ENTRY DispatcherReadyListHead [0x20];

SINGLE_LIST_ENTRY DeferredReadyListHead;

KPRCB *BuddyPrcb;

ULONG ScbOffset;

ULONG ReadyThreadCount;

ULONGLONG StartCycles;

ULONGLONG TaggedCyclesStart;

ULONGLONG TaggedCycles [2];

ULONGLONG TaggedCycles [3];

ULONGLONG GenerationTarget;

ULONGLONG CycleTime;

ULONGLONG volatile CycleTime;

ULONG volatile HighCycleTime;

ULONG ScbOffset;

ULONGLONG AffinitizedCycles;

ULONGLONG ImportantCycles;

ULONGLONG UnimportantCycles;

ULONGLONG ReadyQueueExpectedRunTime;

ULONGLONG AffinitizedCycles;

ULONG volatile HighCycleTime;

ULONGLONG Cycles [4][2];

ULONG PrcbPad72 [11];

ULONGLONG PrcbPad71 [3];

ULONG PrcbPad71;
ULONGLONG PrcbPad72 [2];

ULONG PrcbPad71;

ULONG PrcbPad71 [10];

ULONG PrcbPad71 [2];

ULONG PrcbPad71;

ULONG DpcWatchdogSequenceNumber;

TO BE DONE

LIST_ENTRY DispatcherReadyListHead [0x20];

PVOID ChainedInterruptList;

LONG LookasideIrpFloat;

RTL_RB_TREE ScbQueue;

LIST_ENTRY ScbList;

ULONG SpareFields0 [6];

ULONG SpareFields0 [4];

LONG volatile MmPageFaultCount;

LONG volatile MmCopyOnWriteCount;

LONG volatile MmTransitionCount;

LONG volatile MmCacheTransitionCount;

LONG volatile MmDemandZeroCount;

LONG volatile MmPageReadCount;

LONG volatile MmPageReadIoCount;

LONG volatile MmCacheReadCount;

LONG volatile MmCacheIoCount;

LONG volatile MmDirtyPagesWriteCount;

LONG volatile MmDirtyWriteIoCount;

LONG volatile MmMappedPagesWriteCount;

LONG volatile MmMappedWriteIoCount;

ULONG volatile CachedCommit;

ULONG volatile CachedResidentAvailable;

PVOID HyperPte;

ULONG SpareFields0 [1];

The Old End

BOOLEAN SkipTick;

UCHAR CpuVendor;

UCHAR PrcbPad9 [3];

UCHAR PrcbPad8 [3];

UCHAR PrcbPad8 [4];

UCHAR VendorString [0x0D];

UCHAR InitialApicId;

UCHAR CoresPerPhysicalProcessor;

UCHAR LogicalProcessorsPerPhysicalProcessor;

UCHAR PrcbPad9 [5];

ULONG MHz;

ULONG FeatureBits;

ULONGLONG FeatureBits;

LARGE_INTEGER UpdateSignature;

ULONG QuantumEnd;

The InitialApicId is the high byte of what’s returned in ebx for cpuid leaf 1.

Given a GenuineIntel processor whose CpuType, i.e., family, is at least 6, the UpdateSignature is all 64 bits that are read from the Model Specific Register 0x8B (IA32_BIOS_SIGN_ID), having first written zero to that register and then executed the cpuid instruction’s leaf 1. Starting with version 6.2, Windows also gets the UpdateSignature—by a straightforward read of the MSR—if the vendor is AuthenticAMD and the family is at least 15.

What QuantumEnd holds is an indicator of which thread was most recently seen to have ended its quantum while running on this processor. The indicator is an address of apparently no signficance from the thread’s stack at the time. Later versions eventually turn it into a BOOLEAN. The kernel’s KiDispatchInterrupt function checks for this indicator after retiring DPCs so that action to take at the end of the quantum is in effect a lowest-priority DPC. Version 3.10 actually did schedule this action as a DPC, with no control over the ordering.

Appended for Windows 2000

The significant development of power management for Windows 2000 brought to the KPRCB a large structure for which Microsoft provides a C-language definition in NTPOAPI.H. The PROCESSOR_POWER_STATE has since changed beyond recognition, yet the definition for Windows 2000 remains even in the NTPOAPI.H from the WDK for Windows 10. Throughout, a comment describes it as the “Power structure in each processor’s PRCB”.

The PowerState and NpxSaveArea anyway got swapped for version 5.1. The latter needs 16-byte alignment. This left version 5.2 with 8 bytes to use for the IsrTime. That it remains seems to have given later versions some work.

PROCESSOR_POWER_STATE PowerState;

ULONGLONG volatile IsrTime;

ULONGLONG SpareField1;

ULONGLONG RuntimeAccumulation;

ULONG Stride;

ULONG PrcbPad90;

ULONG PrcbPad90 [2];

ULONGLONG GenerationTarget;

FX_SAVE_AREA NpxSaveArea;

Appended for Windows XP

PROCESSOR_POWER_STATE PowerState;

The intention behind the following padding is not known.

KDPC ForceIdleDpc;

ULONGLONG MsrIa32TsxCtrl;

ULONG PrcbPad91 [1];

ULONG PrcbPad91 [11];

ULONG PrcbPad91 [8];

ULONG PrcbPad91 [14];

ULONG PrcbPad91 [6];

Appended For Windows Vista

RTL_HASH_TABLE *DpcRuntimeHistoryHashTable;

KDPC *DpcRuntimeHistoryHashTableCleanupDpc;

ULONGLONG CurrentDpcRuntimeHistoryCached;

ULONGLONG CurrentDpcStartTime;

KDEFERRED_ROUTINE *CurrentDpcRoutine;

ULONG DpcWatchdogProfileSingleDpcThreshold;

KDPC DpcWatchdogDpc;

KTIMER DpcWatchdogTimer;

PVOID WheaInfo;

PVOID EtwSupport;

SLIST_HEADER InterruptObjectPool;

The WHEA in the naming of WheaInfo is the Windows Hardware Error Architecture. What the WheaInfo points to is a WHEAP_INFO_BLOCK.

Windows 8 moves the WheaInfo, EtwSupport and (8-byte aligned) InterruptObjectPool further into the structure, keeping them together. Whether they’re intended as some sort of set is not known, but they are also together in the x64 KPRCB.

According to what the WMITRACE.DLL debugger extension’s undocumented !systrace StackTraceCounters command would do if given the expected kernel symbols, what the EtwSupport points to is an ETW_TRACING_BLOCK. The kernel’s public symbols do not show this structure, nor do any symbol files in the downloadable packages of public symbols. Whether any at Microsoft’s public symbol server do, I don’t know, but it’s safe to say that the ETW_TRACING_BLOCK is as obscure as can be for a kernel structure whose name is known with the certainty of its being looked for by a debugger extension. Searching for it through Google today, 8th February 2023, both with and without a leading underscore, gets zero matches.

LARGE_INTEGER HypercallPagePhysical;

SLIST_HEADER HypercallPageList;

PVOID HypercallPageVirtual;

PVOID HypercallCachedPages;

PVOID VirtualApicAssist;

ULONGLONG *StatisticsPage;

PVOID RateControl;

CACHE_DESCRIPTOR Cache [5];

ULONG CacheCount;

ULONG CacheProcessorMask [5];

UCHAR LogicalProcessorsPerCore;

UCHAR PrcbPad8 [3];

KAFFINITY PackageProcessorSet;

KAFFINITY_EX PackageProcessorSet;

That the KPRCB has a KAFFINITY_EX among its members has a curious side-effect in the symbol file WNTDLL.PDB for the WOW64 variant of NTDLL.DLL. Its type information for the KPRCB is wrong because the KAFFINITY_EX is compiled with the greater allowance that 64-bit Windows has for processor groups (20 instead 1). For this symbol file, which the 32-bit debugger uses for the 32-bit NTDLL.DLL when debugging a 32-bit application, everything after PackageProcessorSet is said to be further into the KPRCB than it truly is.

The wonder, of course, is that a symbol file for any user-mode DLL has type information for the kernel-mode KPRCB. Even the symbol file for NTDLL surely is not intended to. It picks up KPRCB from a #include of I386_X.H. That this header gets included may itself be unintended. But included it is, and since it not only defines the KPRCB but uses it in an inline routine even the public symbols get type information for it. The practical consequence for debugging is nothing, the KPRCB having no meaning in the circumstances, but the significance for reverse engineering is two-fold. First, type information can turn up in symbol files for modules that make no run-time use of the type. This also has significance for programmers, and perhaps their lawyers, who are concerned for what’s revealed in symbol files. Second, type information is not as certainly correct as reverse engineers tend to think (or blindly hope).

ULONG CacheProcessorMask [5];

ULONG SharedReadyQueueMask;

ULONG PrcbPad91 [1];

ULONG ScanSiblingMask;

KSHARED_READY_QUEUE *SharedReadyQueue;

ULONG SharedQueueScanOwner;

ULONG CoreProcessorSet;

Inserted For Windows 8

ULONG ScanSiblingMask;

ULONG LLCMask;

ULONG CacheProcessorMask [5];

ULONG ScanSiblingIndex;

ULONG LLCLevel;

PVOID WheaInfo;

PVOID EtwSupport;

SLIST_HEADER InterruptObjectPool;

ULONG SharedReadyQueueOffset;

PVOID *DpcWatchdogProfile;

PVOID *DpcWatchdogProfileCurrentEmptyCapture;

ULONG PrcbPad92 [8];

ULONG PrcbPad92 [2];

ULONG PrcbPad92 [3];

ULONG PrcbPad92 [1];

ULONG PackageId;

ULONG PteBitCache;

ULONG PteBitOffset;

ULONG PrcbPad93;

PROCESSOR_PROFILE_CONTROL_AREA *ProcessorProfileControlArea;

PVOID ProfileEventIndexAddress;

Appended For Windows 7

KDPC TimerExpirationDpc;

Windows 7 appends numerous performance counters for synchronisation, but they didn’t stay as separate members for long.

ULONG SpinLockAcquireCount;

ULONG SpinLockContentionCount;

ULONG SpinLockSpinCount;

ULONG IpiSendRequestBroadcastCount;

ULONG IpiSendRequestRoutineCount;

ULONG IpiSendSoftwareInterruptCount;

ULONG ExInitializeResourceCount;

ULONG ExReInitializeResourceCount;

ULONG ExDeleteResourceCount;

ULONG ExecutiveResourceAcquiresCount;

ULONG ExecutiveResourceContentionsCount;

ULONG ExecutiveResourceReleaseExclusiveCount;

ULONG ExecutiveResourceReleaseSharedCount;

ULONG ExecutiveResourceConvertsCount;

ULONG ExAcqResExclusiveAttempts;

ULONG ExAcqResExclusiveAcquiresExclusive;

ULONG ExAcqResExclusiveAcquiresExclusiveRecursive;

ULONG ExAcqResExclusiveWaits;

ULONG ExAcqResExclusiveNotAcquires;

ULONG ExAcqResSharedAttempts;

ULONG ExAcqResSharedAcquiresExclusive;

ULONG ExAcqResSharedAcquiresShared;

ULONG ExAcqResSharedAcquiresSharedRecursive;

ULONG ExAcqResSharedWaits;

ULONG ExAcqResSharedNotAcquires;

ULONG ExAcqResSharedStarveExclusiveAttempts;

ULONG ExAcqResSharedStarveExclusiveAcquiresExclusive;

ULONG ExAcqResSharedStarveExclusiveAcquiresShared;

ULONG ExAcqResSharedStarveExclusiveAcquiresSharedRecursive;

ULONG ExAcqResSharedStarveExclusiveWaits;

ULONG ExAcqResSharedStarveExclusiveNotAcquires;

ULONG ExAcqResSharedWaitForExclusiveAttempts;

ULONG ExAcqResSharedWaitForExclusiveAcquiresExclusive;

ULONG ExAcqResSharedWaitForExclusiveAcquiresShared;

ULONG ExAcqResSharedWaitForExclusiveAcquiresSharedRecursive;

ULONG ExAcqResSharedWaitForExclusiveWaits;

ULONG ExAceResSharedWaitForExclusiveNotAcquires;

ULONG ExSetResOwnerPointerExclusive;

ULONG ExSetResOwnerPointerSharedNew;

ULONG ExSetResOwnerPointerSharedOld;

ULONG ExTryToAcqExclusiveAttempts;

ULONG ExTryToAcqExclusiveAcquires;

ULONG ExBoostExclusiveOwner;

ULONG ExBoostSharedOwners;

ULONG ExEtwSynchTrackingNotificationsCount;

ULONG ExEtwSynchTrackingNotificationsAccountedCount;

Windows 8 formalises the preceding as one structure and inserts another for file and disk I/O.

SYNCH_COUNTERS SynchCounters;

FILESYSTEM_DISK_COUNTERS FsCounters;

TO BE DONE

CONTEXT *Context;

ULONG ContextFlags;

ULONG ContextFlagsInit;

XSAVE_AREA *ExtendedState;

Appended For Windows 8

KENTROPY_TIMING_STATE EntropyTimingState;

Appended For Windows 8.1

PVOID IsrStack;

KINTERRUPT *VectorToInterruptObject [0xD0];

SINGLE_LIST_ENTRY AbSelfIoBoostsList;

SINGLE_LIST_ENTRY AbPropagateBoostsList;

KDPC AbDpc;

The VectorToInterruptObject array is for interrupt vectors 0x30 to 0xFF inclusive. This is the range that all x86 versions allow for hardware interrupts.

Appended For Windows 10

The additions for Windows 10 pad three times, apparently for cache alignment in each case.

IOP_IRP_STACK_PROFILER IoIrpStackProfilerCurrent;

IOP_IRP_STACK_PROFILER IoIrpStackProfilerPrevious;

KTIMER_EXPIRATION_TRACE TimerExpirationTrace [0x10];

ULONG TimerExpirationTraceCount;

PVOID ExSaPageArray;

XSAVE_AREA_HEADER *ExtendedSupervisorState;

ULONG PrcbPad100 [10];

ULONG PrcbPad100 [9];

TO BE DONE

KSHARED_READY_QUEUE LocalSharedReadyQueue;

UCHAR PrcbPad95 [12];

One might wonder, though, at having a cache line for just one pointer:

REQUEST_MAILBOX *Mailbox;

UCHAR PrcbPad [0x3C];

UCHAR PrcbPad [0x05FC];

The preceding padding is greatly expanded for Version 1803 so that additions are not just cache-aligned but page-aligned:

ULONG KernelDirectoryTableBase;

ULONG EspBaseShadow;

ULONG UserEspShadow;

ULONG ShadowFlags;

ULONG UserDS;

ULONG UserES;

ULONG UserFS;

PVOID EspIretd;

ULONG RestoreSegOption;

ULONG SavedEsi;

USHORT VerwSelector;

USHORT PrcbShadowPad;

ULONG TaskSwitchCount;

ULONG DbgLogs [0x0200];

ULONG DbgCount;

ULONG PrcbPadRemainingPage [0x01F5];

ULONG PrcbPadRemainingPage [0x01F3];

The RequestMailbox array is at the very end so that the kernel can easily provide as many mailboxes as there can ever be processors.

REQUEST_MAILBOX RequestMailbox [ANYSIZE_ARRAY];