Geoff Chappell - Software Analyst
Research and writing continues, though it might better not!
The attempt over the last few months to document the ZwQuerySystemInformation function and its companions and all their numerous cases and related structures will keep me tidying up for a long while yet. That was partly the point to starting it: once the overall organisation of the documentation is settled, the many cases might each be picked off in otherwise idle time between items of paid work.
One of the more complex cases of this system information pertains to Event Tracing for Windows (ETW). Regular readers will know that I have long been interested in ETW and was among the first critical examiners of its significantly expanded use for Windows Vista. See, for instance, Event Trace Security or The Service Control Manager Eventlog Provider. Now I find myself fascinated not just by the diagnostic information that’s exposed through the NT Kernel Logger but also by the mechanism.
So much about ETW and especially about the NT Kernel Logger is undocumented, not just the implementation details but even the means of using it through interfaces independently of the tools that Microsoft provides in, for instance, the Windows Driver Kit and the Windows Performance Toolkit. That this is very deliberate by Microsoft is less troubling to me than is the programming community’s evident interest in the feature but apparent lack of interest in getting the feature understood in any systematic way—not even for what I’d have thought is the very practical matter of developing alternatives to Microsoft’s tools.
I do sympathise, sort of. More than almost anyone else, I know that studying a large, complex operating system like Windows to a useful depth and scale would be hard work even with source code to follow. But while the world does so little high-quality research into Windows, what opportunities for programming Windows don’t so much go begging as get left undreamt? It’s so much easier to grumble that Windows doesn’t come with source code or that Microsoft is stingy with the documentation. Do something about it!
See that I’ve tacked on, albeit as drafts, attempts at documenting some of the native API functions that support profiling. These don’t send events to loggers but instead produce counters of how often a profile interrupt detects execution at particular addresses. That can be invaluable diagnostic detail, whether the purpose is constructive or mischievous. It really hasn’t been remarked on nearly enough that for much of the history of Windows these undocumented interfaces didn’t require any privilege at all even to profile kernel-mode execution from user mode, and still don’t require much. Even now, these functions give an unprivileged user-mode caller the means to get large amounts of data mapped into kernel-mode address space and to keep changing the content and to control how long it stays there—which may not be an exploitable vulnerability but is at least incautious.
But that’s a side-track. I add the profiling functions not for any amount of noteworthiness (though I do believe they warrant more than a little), nor even because they and ETW are both for diagnostics, but because they and ETW are closely related in the implementation. That’s the nature of studying a complex system: one thing leads to another, which leads to another, and before you know it you have a year of research laid out. How is this not someone’s paid work?
On that last point, please indulge me. Google, both as a search engine reporting on the world at large and as a company with its own software development, shows no shortage of interest in profiling. The feature is certainly not obscure or neglected, but the mechanism plainly is. If I ask Google specifically for pages that mention NtCreateProfile I see nothing that’s even near to being as detailed as my drafts. Yet these functions are now 20 years old. How is this left to happen?
Of course, it’s not enough to start piecing together a kernel-mode knowledge of ETW. Anyone who wants to put it into practice will want its user-mode interfaces. It just doesn’t stop! I can’t commit to much more as free work, especially because I know that highly regarded people on good salaries don’t just fumble around with this feature in the hope of finding the magic incantations but actually claim to make progress. If nothing else, that’s a tidy reminder of how absurd it is that I do the research and writing for free. Still, let’s start with two relatively new functions that are vital for getting advanced use from ETW but for which Microsoft’s documentation is remarkably poor.