Geoff Chappell, Software Analyst
As with many who start programming but are soon drawn to study the operating system that they’re programming for, my interests have long lain with what happens at the lower levels, not so much down with the hardware but certainly where the operating system is the platform that’s depended on by all the software that does what users see as their real work that the computer exists to help them with.
It was this way for me immediately that I first started looking at computer programming as more than a tool that a mathematician may occasionally want for numerical analysis. Before I fully appreciated that my diversion from mathematics might be the start of what would instead have to pass as my life’s achievement, I was picking apart the IO.SYS and MSDOS.SYS from MS-DOS 3.30, all as a theoretical exercise in deduction, working from debug output that I printed to continuous-feed paper and marked up with ruler and pencil. As DOS gave way to protected-mode Windows and the Windows that runs on DOS gave way to the Windows that we have today, I continued to feel that the most rewarding study is of what these operating systems do with memory, files and storage devices, and how they support the generality of code written by unknown programmers of highly variable quality who may each write as if their code more or less owns the machine.
For the Windows that we have today, the management of memory, files and processes and such is ultimately the business of a multi-megabyte kernel, highly modularised internally but distributed as one executable, supported by a handful of other essential executables, most notably the Hardware Abstraction Layer (HAL), which all together support an extensible scheme of drivers to complete what Windows applications have to take for granted as the system.
Indeed, to Windows applications and to most of their programmers, the system that I talk about here is more a system in a system. When a Windows programmer writes a call to CreateFile as the way to have a Windows application open a file, it makes perfect sense to think that this is a call to the operating system. The other side of the call, however, is still in the processor’s outermost privilege ring (numbered 3, but commonly referred to as user mode). What the Windows application calls is actually just a subsystem that does substantial work of its own but depends in turn on the true system that executes in the processor’s innermost privilege ring (numbered 0, but commonly referred to as kernel mode). This section of this website is concerned solely with this kernel-mode execution, and thus with
There is inevitably some overlap with the user-mode subsystem. The user-mode NTDLL.DLL is arguably less the lowest-level component of any subsystem than the kernel’s user-mode footprint in every process that has user-mode code. Still, for anything that executes in user mode, even if named KERNEL-something, please see the separate Win32 study (not that it has an introduction yet).
Although the present Windows has from its beginning as Windows NT had vastly better system-level documentation than did the earlier Windows that ran on DOS, the difference was more in depth and reliability than in coverage. As with the Windows that ran on DOS, there has always been much about low-level Windows NT that Microsoft leaves undocumented or, worse, under-documented—or, in some ways worst, unreliably documented. These problems are specially true of kernel-mode Windows.
When I speak of this level of Windows as the most rewarding to study, I should like to mean intellectually rewarding, but I cannot ignore that the wider research presented at this website was only ever possible because kernel-mode programming is, or at least has been, financially rewarding. As will be known all too well to any software manufacturer that has ever needed any sort of kernel-mode driver, kernel-mode programming is expensive. Why is that?
There is of course some expense to expect of any highly specialised work. Make no mistake: kernel-mode programming is about as specialised as programming gets. It has its own interfaces and terminology, not just as a repackaging of anything from user-mode programming but because it has to deal with numerous considerations that have no equivalent in or even relevance to user-mode programming. Much of what the kernel-mode programmer must master is not immediately helpful or even transferrable to user-mode programming. Conversely, many of the crutches that many user-mode programmers take for granted or even as fundamental—notably, the C Run-Time and the Standard Template Library—are largely unavailable in kernel-mode programming or are at best used only with difficulty and care. Indeed, it’s not that long ago that C++ itself—yes, the whole language—was widely thought more trouble to use safely in kernel mode than it could be worth. As with any other specialisation, nobody commits to kernel-mode programming except by regarding it as a substantial investment—from which, naturally, they seek a substantial return.
Another reason, at least in the private thoughts of anyone who calls himself a kernel-mode programmer, is that kernel-mode programming actually does require more skill than does most programming. Even the most complicated or advanced issues in user-mode programming are eased by large measures of control and cooperation. Splitting a program’s work over multiple threads can get intricate enough to tax even the most competent of programmers—and if those threads are created in DLLs you’ll soon be sorting your programming wheat from the chaff—but at least the threads are your creatures that exist to do your bidding. Even if you have to synchronise with a thread in someone else’s process, it will at least be a process that you are cooperating with more or less explicitly. In user-mode programming you have the luxury of looking in or down to an operating system that is a reliable provider not just of functionality but of isolation from other people’s code. Except for the very particular matter of driving devices, the perspective for most kernel-mode programming is more or less the opposite: looking out or up to a tower of other people’s code in all its generality of purpose. All that code competes for your attention and expects that you let it proceed as if it’s alone, even while you in the kernel almost always have at least the potential of being called from multiple threads of other people’s code to do much the same work at much the same time. Getting all this right for everyone is not for the faint-hearted.
But although additional expense is only to be expected for specialised work by programmers who will anyway tend to be among the best and brightest, I have long wondered if there is more to it. So much that programmers need is undocumented. Much of what’s documented now used to be undocumented—for years, even—and is now documented inaccurately, incompletely or just so sparingly that it was arguably better left undocumented. There is a substantial element of “black art” to kernel-mode programming.
The several hundred pages that exist in this section of the website have been slow in coming but do by now amount to a substantial resource. They’re a long way from banishing the “black art” label but they may, I hope, expose the subject to a little light.