Geoff Chappell - Software Analyst
Starting with Windows 8, an I/O Request Packet (IRP) can have an extension which can carry multiple new parameters to and from drivers that are in the know. It seems to be important to Microsoft that the delivery should by-pass drivers along the way, for although these other drivers see all the I/O requests go up and down the device-object stacks, they don’t know of the IRP extension that holds the extra parameters. Microsoft keeps IRP extensions secret.
Let’s be clear that Microsoft has gone to unusual trouble to keep IRP Extensions undocumented. More is going on here than just Microsoft keeping some structures and functions as internal, not documenting them and perhaps not even declaring them in the headers that Microsoft publishes in the Windows Driver Kit (WDK). More is going on than just Microsoft defining some structure as semi-opaque with members defined in plain sight but as being reserved for the system’s use. IRP extensions are more reserved than that. They count among the relatively few features that have Microsoft define a structure differently in public and private.
To support IRP extensions, Microsoft has two definitions of the age-old IRP structure: one for programmers in general; the other for Microsoft’s own programmers (and perhaps for favoured programmers outside Microsoft). Symbol files for the kernel in Windows 8 and higher show clearly that the kernel is now built with an IRP whose Tail.Overlay ends with a pointer named IrpExtension. The kernel, though, does not get the IRP from the same headers that Microsoft publishes in the WDK. Symbol files show that the kernel gets the IRP structure from a header file named “io_x.h”. For all we’re to know without access to all that’s involved in building Microsoft’s source-code tree, even this header is not the original source of the definition. It could be built by extracting from yet another file, not even necessarily a C-language header. Somewhere, however, there seems likely to be a master definition. Whatever may be this master definition, and however it passes into “io_x.h” for the kernel, it differs from what ends up in WDM.H for programmers in general. What shows at the end of the Tail.Overlay in WDM.H from the WDK is the same OriginalFileObject that has been at the end since the definition in NTDDK.H from the Device Driver Kit (DDK) for Windows NT 3.1. The new IrpExtension isn’t blocked out of the IRP definition in WDM.H by conditional compilation: it just isn’t there.
Some years after Windows 8, Microsoft briefly published another header that contains an IRP definition. This header is named NTOSP.H and can be found in the WDK for Windows 10, both originally and for the 1511 release, but not in later editions. Since it is anyway in a subdirectory of a directory named “um”, as if for user-mode programming in contrast to “km”, its publication doesn’t seem intended for writing device drivers or file system drivers or anything else that ever sees an IRP. Yet its IRP is either the kernel’s or is much closer. Notably, it has the otherwise secret IrpExtension—along with comments that make clear that IrpExtension is intentionally edited out of WDM.H.
Of course, Microsoft does not play these games for fun or mischief. There is real-world use for the IrpExtension. Especially as devices become ever more capable, there’s obvious merit to expanding what can be sent with an I/O request and what sort of detailed error information (such as a SCSI sense key and additional sense codes) can be returned. The example that brought IRP extensions to my attention is of hybrid disks that have non-volatile caches that can be accessed without the penalty of waiting for heads to seek or disks to spin. These devices can take a hint to use one form of storage rather than another. That Windows 8.1 can send such hints when writing to a paging file is surely a great help, but the hint needs to go with the data—not in a separate control request, but as control data in the same request that supplies the data that’s to be stored. The hint goes with the write request as an IRP extension. It’s set by the kernel’s Memory Manager when forming the IRP for an asynchronous page write. It’s then fished out when the request reaches such drivers as CLASSPNP.SYS, RDYBOOST.SYS and STORPORT.SYS. Perhaps only drivers from Microsoft could use the hint even if others knew of it.
The IrpExtension is declared as pointing to void. What it actually points to, if anything, is an undocumented structure named IOP_IRP_EXTENSION that can hold multiple types of extension data. In Windows 8, just the one type is defined, for associating the IRP with an Activity ID to help with performance monitoring of I/O requests. But the definition of that structure, as known from public symbol files, makes clear that multiple types were planned from the start.
One type of extension that was added for Windows 8.1 is small enough to fit in the IRP in place of the IrpExtension pointer if it’s the only type of extension that is yet set. New bits in the AllocationFlags of the IRP distinguish the cases. (Microsoft’s names for these new bits are known from NTOSP.H.) When IRP_EXTENSION_GENERIC_ONLY (0x80) is set, the IRP has only a generic IRP extension whose four bytes overlay the IrpExtension pointer. When IrpExtension is not NULL, it points to an IOP_IRP_EXTENSION. If this is a separate allocation (from non-paged no-execute pool), the IRP_EXTENSION_ALLOCATED bit (0x40) is set in the AllocationFlags. The IOP_IRP_EXTENSION can instead be in the same memory block as the IRP, as an expansion of space allowed for the I/O stack locations.
An IRP that has an IOP_IRP_EXTENSION can have multiple types of extension data (though some types are mutually exclusive, if only for now). To each type there corresponds one bit in the TypesAllocated member. The bit is set if the extension has the type. Microsoft’s names for these bits are not known.
Of course, even Microsoft’s device driver programmers who know of IRP extensions do not access IrpExtension directly and depend on these implementation details, which Microsoft may change readily, and has already. Instead, they call functions, which Microsoft may change but rarely does. There are rather many for a feature seems so slight and obscure:
None are documented (though an earlier set of functions such as IoGetActivityIdIrp, which use IRP extensions internally, are). Functions whose names are shaded orange above are declared in one or other of NTDDK.H and NTIFS.H from the ordinary WDK, even for the version that first exports the functions. The others are highlighted orange to draw attention to them. They too are declared but obscurely—in one or both of NTOSP.H and NTOSIFS.H from the “minwin” subdirectory of headers in some early editions of the WDK for Windows 10.