Geoff Chappell, Software Analyst
This function acquires a queued spin lock at high IRQL.
VOID
FASTCALL
KeAcquireInStackQueuedSpinLockAtDpcLevel (
KSPIN_LOCK *SpinLock,
KLOCK_QUEUE_HANDLE *LockHandle);
The SpinLock argument provides the address of the spin lock that is to be acquired.
The LockHandle argument provides the address of an opaque context structure. This address is re-presented when releasing the lock. The context must have no other use until the lock has been released.
The KeAcquireInStackQueuedSpinLockAtDpcLevel function assumes that the IRQL is at least DISPATCH_LEVEL. The spin lock and the context structure must be in non-paged memory when this function begins and must remain so until the lock is released.
The KeAcquireInStackQueuedSpinLockAtDpcLevel function is exported by name from the kernel in version 5.1 and higher.
The KeAcquireInStackQueuedSpinLockAtDpcLevel function is documented.
Not until the Windows Driver Kit (WDK) for Windows 7 does the documentation explicitly allow the function’s use at IRQL above DISPATCH_LEVEL.
The KeAcquireInStackQueuedSpinLockAtDpcLevel function begins by initialising the KLOCK_QUEUE_HANDLE. Since the function does not change the IRQL, it makes no use of the OldIrql member. Its interest is only in the LockQueue. This it initialises by clearing the Next member to NULL and pointing the Lock member to the spin lock.
The remainder of the function is an inlining of an internal routine
VOID FASTCALL KeAcquireQueuedSpinLockAtDpcLevel (KSPIN_LOCK_QUEUE *);
This might pass unmentioned as an implementation detail except that this internal routine is the common code for acquiring a queued spin lock at high IRQL through other exported functions, notably KeAcquireQueuedSpinLock, the notes for which refer here for the details.
If the spin lock is not already owned, it will hold zero either from its initialisation by KeInitializeSpinLock or from being reset when its last release left it with no owner. While the spin lock is owned, it has a queue of one owner and zero or more waiters. Each provides its own KSPIN_LOCK_QUEUE. The owner’s is the head of the queue. Its Next member points to the KSPIN_LOCK_QUEUE for the first waiter. Successive Next members link successive waiters until the last has NULL for Next. The queue is never followed from head to tail. Releasing removes from the head. This function appends to the tail. The head is known, when it needs to be, because the owner re-presents its KSPIN_LOCK_QUEUE when releasing the lock. The tail is known because the lock always points to it.
Appending the caller to the lock’s queue may need multiple steps. The first is to update the lock. The function uses an xchg instruction (and its implied lock) to point the lock to the caller’s KSPIN_LOCK_QUEUE as the queue’s new tail while discovering the previous tail. If this exchange had the lock previously containing NULL, then there was no queue to append to: the caller is the new owner and the function is done.
Otherwise, the caller must expect to wait its turn. The lock has a queue but the caller is not yet linked into it. The moment that it is, the caller can become the new owner. It must before then have a signal that its wait can end. The function sets the LOCK_QUEUE_WAIT bit (0x01) in the Lock member of the caller’s KSPIN_LOCK_QUEUE, expecting that it will get cleared on some subsequent release that would transfer ownership to the caller. The function then links the previous tail’s Next to the caller’s KSPIN_LOCK_QUEUE. Now the caller waits in the spin loop until the LOCK_QUEUE_WAIT bit in the caller’s KSPIN_LOCK_QUEUE gets cleared. When that eventually happens, the caller is the new owner and the function is done.
See that when multiple processors contend for the same queued spin lock, each waits for a clear LOCK_QUEUE_WAIT bit in a different KSPIN_LOCK_QUEUE. That each processor’s spin loop polls a different address, which will ideally be close to the processor, may not be the greatest advantage that queued spin locks have over the classic kind, but it may be the most understated.
The 32-bit implementation is in assembly language before version 6.2. It keeps two bits in the Lock member: LOCK_QUEUE_WAIT as above but also LOCK_QUEUE_OWNER (0x02). When this function sees that the caller becomes the new owner immediately, this function sets the LOCK_QUEUE_OWNER bit before returning. When the caller instead goes into a spin loop, the LOCK_QUEUE_OWNER bit is set concurrently with the clearing of the LOCK_QUEUE_WAIT bit by whichever function releases the lock to make this function’s caller the new owner.
Between its tests for whether LOCK_QUEUE_WAIT is yet clear, the spin loop is originally just a pause instruction. The version 6.0 from Windows Vista SP1 adds hypervisor notification. How many spins pass between notifications depends on a LongSpinWaitCount parameter that is learnt from the cpuid instruction’s 0x40000004 leaf during the kernel’s initialisation. (See HV_X64_ENLIGHTENMENT_INFORMATION). To the assembly-language implementation, such notification is extra to the pause. To the C-language implementation, it’s an alternative to the pause.
Version 6.1 introduces performance counting. Each KPRCB has three counters whose repeated sampling may give some sense of overall demand for spin locks. Attempting to acquire any type of spin lock increments a SpinLockAcquireCount. Each acquisition that is not satisfied immediately is a contention and increments a SpinLockContentionCount. However many times the processor then spins in its loop before acquiring the lock gets added to a SpinLockSpinCount. Note that each count is only 32 bits wide. In version 6.2 and higher, these counts are maintained only if the PERF_SPINLOCK_CNTRS group mask is enabled for at least one system logger session.
More sophisticated event tracing for system logger sessions also dates from version 6.1 but only in the C-language coding: the x86 builds don’t get it until version 6.2. This event tracing requires that the PERF_SPINLOCK group mask be enabled. Its effect on this function is to record into the EtwSupport area of the processor’s KPRCB a little information about the function’s entry and exit. When the caller eventually releases the lock, information from the acquisition is retrieved. If sampling conditions are satisfied, information about the acquisition and release are put into a WMI_SPINLOCK and written as a PERFINFO_LOG_TYPE_SPINLOCK event.