Geoff Chappell - Software Analyst
This page is being substantially reworked to account for its subject’s early history. Some content is temporarily missing. Temporarily may turn into indefinitely. New content may be more than usually defective. The page is published only as a revision in progress. Use with caution.
An event provider delivers an event to Event Tracing for Windows (ETW) by specifying some general description of the event in combination with arbitrary data that is specific to that occurrence of the event. ETW marshals the description and the data, along with system properties such as times and the current process and thread, into one data block in a trace buffer. Each such buffer is a fixed-size WMI_BUFFER_HEADER and a sequence of these variable-size data blocks, each with (nowadays) 8-byte alignment. If the event tracing session is configured to flush the trace buffers to an Event Trace Log (ETL) file, the buffers and thus also the events in this raw form become more than a transitory implementation detail and instead persist for easy inspection.
Broadly speaking, each data block that describes some event’s occurrence is in two parts:
The fixed-size header might ideally be the same structure for all events, but the implementation is not nearly so neat. ETW has accreted through the decades by fitting together several schemes that appear to have been developed not quite independently but also not in continuous progression. This can be seen, for instance, in the documentation’s separate talk of classic and manifest-based event providers, each with their own sets of API functions. One way it shows within the implementation is that several types of header are defined for events as held in trace buffers. They fall naturally into two sets according to whether they begin with a size. This difference also aligns roughly with whether their use is external or internal, and with some sense of their being old or new. For now, however, it suffices just to list the known types of trace header alphabetically, which has the advantage of starting with the most prosaically named:
This article aims only for an overview. It goes no further than to differentiate these types of trace header according to what they have in common near their start. For details of how each trace header continues and is used, follow the links.
The EVENT_INSTANCE_HEADER and EVENT_TRACE_HEADER are documented, and have been from the beginning. The corresponding C-language definitions are in EVNTRACE.H. What counts as the beginning is Windows 2000 as far as released versions go, but these structures and their supporting interfaces had been in development for years: as supplied with the Device Driver Kit (DDK) for Windows 2000, EVENTRACE.H has a comment that dates its creation to “15-Sep-1997”. For no known reason, this comment disappeared as soon as the DDK for Windows XP.
The EVENT_HEADER also is documented, and has been since its introduction for Windows Vista. Its C-language definition is in EVNTCONS.H. A comment in this header dates its creation to “26-Aug-2004”. The substantial reworking of ETW for Windows Vista also must be reckoned in years.
None of the other trace headers are documented, but they all have C-language definitions in a header file named NTWMI.H which Microsoft published, apparently by oversight, in the original and Version 1511 editions of the Windows Driver Kit (WDK) for Windows 10. A comment near the top of this header dates a “Public/Private header split” to “23-Jan-2003”. The file is surely older and whatever was split does not seem to have affected which trace headers are public or private.
The practical equivalent of C-language definitions (for having almost all the content that’s meaningful to the compiler, though none of the comments that might help a human reader but often don’t) is available for many more versions in the form of type information in symbol files for a very small selection of Windows binaries and, even more rarely, in statically linked libraries.
That any of the trace headers are documented is plainly not so that programmers or other interested computer users can read the events directly in ETL files. There have always been documented API functions both for providing events and for consuming them. New and preferred API functions abstract the events at both ends, but in the relatively simple architecture that preceded Windows Vista, what’s kept in the trace buffers and gets saved in ETL files is very much closer to what the API functions feed into the ETW machinery or get from it.
The EVENT_TRACE_HEADER and EVENT_INSTANCE_HEADER are involved right from the start for some types of event. They are the forms in which a so-called classic event provider describes an event to ETW through the documented user-mode API functions TraceEvent and TraceInstanceEvent and through the documented kernel export IoWMIWriteEvent. Note that although these functions are not formally deprecated, they all have some sense of legacy to them.
The EVENT_TRACE_HEADER and EVENT_HEADER also have essential roles at the end of an event’s lifetime. No matter what type of header an event had when presented to ETW or had while in the trace buffers or still has in an ETL file, what an event consumer is told of the event is a translation that starts with either an EVENT_TRACE_HEADER or an EVENT_HEADER. To retrieve events, an event consumer supplies the documented OpenTrace function with an EVENT_TRACE_LOGFILE structure that specifies a callback function, and then calls the documented ProcessTrace function to get the callback called back, recurrently, once for each available event. The original form of callback presents each event as an EVENT_TRACE_HEADER as the beginning of an EVENT_TRACE. A new style gets an EVENT_HEADER as the beginning of an EVENT_RECORD.
Several of the trace headers show signs of having developed from (or in common with) the WNODE_HEADER structure from the Windows Management Infrastructure (WMI). See especially that the EVENT_TRACE_HEADER and the WNODE_HEADER have the same size and duplicate several members at the same offsets.
Historically, anything that might be a WNODE_HEADER is reinterpreted as an EVENT_TRACE_HEADER if it has WNODE_FLAG_TRACED_GUID set among its Flags at offset 0x2C. The reverse—given what might be a trace header, what sort is it, or is it instead a WNODE_HEADER—is complicated by the proliferation of trace headers, most types of which do not share the Flags.
What counts for differentiating the trace headers among themselves is the first four bytes. The WNODE_HEADER has these as a 32-bit BufferSize. That the highest bit would ever be set is all but unimaginable, and so a set high bit in this first dword is a convenient marker that what might be a WNODE_HEADER is instead some type of trace header. The rest of the dword is then available for reinterpretation.
Most types of trace header keep the size at offset 0x00 but reduced to 16 bits. The high 16 bits are used for differentiating the types. The scheme for differentiation looks like it accommodated originally separate considerations and then settled. What remains is a mixture of bit flags for the byte at offset 0x03 and an enumeration for the byte at offset 0x02, but with few enough combinations in actual use that they are alternatively defined as values for the word at offset 0x02 or as masks from which to form the whole dword, e.g., by combining with the size.
Though the high byte’s history as bit fields is plain enough from the high bit’s use for differentiating trace headers from a WNODE_HEADER, the neater presentation is to start with the values that most types of trace headers have for the byte at offset 0x02. The neatness is that values are defined even for types of trace header that don’t keep the value at offset 0x02 (or anywhere). No formal enumeration is known, just macro definitions from NTWMI.H:
| Value | Name | Versions | Header |
|---|---|---|---|
| 0x01 | TRACE_HEADER_TYPE_SYSTEM32 | 5.0 and higher | SYSTEM_TRACE_HEADER followed by 32-bit event data |
| 0x02 | TRACE_HEADER_TYPE_SYSTEM64 | 5.0 and higher | SYSTEM_TRACE_HEADER followed by 64-bit event data |
| 0x03 | TRACE_HEADER_TYPE_COMPACT32 | 6.2 and higher | compact SYSTEM_TRACE_HEADER followed by 32-bit event data |
| 0x04 | TRACE_HEADER_TYPE_COMPACT64 | 6.2 and higher | compact SYSTEM_TRACE_HEADER followed by 64-bit event data |
| 0x0A | TRACE_HEADER_TYPE_FULL_HEADER32 | 5.0 and higher | EVENT_TRACE_HEADER followed by 32-bit event data |
| 0x0B | TRACE_HEADER_TYPE_INSTANCE32 | 5.0 to 5.1 | EVENT_INSTANCE_HEADER followed by 32-bit event data |
| 5.2 and higher | EVENT_INSTANCE_GUID_HEADER followed by 32-bit event data |
||
| 0x0C | TRACE_HEADER_TYPE_TIMED | 5.0 and higher | |
| 0x0D | TRACE_HEADER_TYPE_ERROR | 5.0 and higher | |
| 0x0E | TRACE_HEADER_TYPE_WNODE_HEADER | 5.0 and higher | |
| 0x0F | TRACE_HEADER_TYPE_MESSAGE | 5.1 and higher | MESSAGE_TRACE_HEADER followed by message arguments |
| 0x10 | TRACE_HEADER_TYPE_PERFINFO32 | 5.1 and higher | PERFINFO_TRACE_HEADER with 32-bit event data as Data array |
| 0x11 | TRACE_HEADER_TYPE_PERFINFO64 | 5.1 and higher | PERFINFO_TRACE_HEADER with 64-bit event data as Data array |
| 0x12 | TRACE_HEADER_TYPE_EVENT_HEADER32 | 6.0 and higher | EVENT_HEADER followed by 32-bit event data |
| 0x13 | TRACE_HEADER_TYPE_EVENT_HEADER64 | 6.0 and higher | EVENT_HEADER followed by 64-bit event data |
| 0x14 | TRACE_HEADER_TYPE_FULL_HEADER64 | 6.0 and higher | EVENT_TRACE_HEADER followed by 64-bit event data |
| 0x15 | TRACE_HEADER_TYPE_INSTANCE64 | 6.0 and higher | EVENT_INSTANCE_GUID_HEADER followed by 64-bit event data |
See that even in the late 1990s, the definition of these types anticipated that the event-specific data may have different layouts when written by 32-bit and 64-bit event providers. These differences need to be distinguished for correct interpretation by 32-bit and 64-bit event consumers.
Though types 0x0C to 0x0E were still defined in 2015 for Microsoft’s disclosure of NTWMI.H, all three have long been out of use in one sense or another.
WORK IN PROGRESS