A couple of weeks ago I wrote a script called BSODCheck.vbs. This script will remotely check to see if a machine has created a new memory.dmp file. The existence of a new file indicates that a machine has blue screened and crashed.
This script is intended to be run periodically and the results of the script are compared against existing results to let you know if a new dump has been created since the last time the script has run. Well after using this script for a period of weeks I noticed that some of our servers were crashing for no apparent reason.
The first step to resolving this issue was to take a look at the dump files themselves. To do that I needed to copy the dump files to a local directory and examine them with a utility called WinDbg. These debugging tools are available free of cost from Microsoft.
After downloading and installing the debugging tools we need to configure WinDbg to use the correct symbols files. This is done by clicking “File -> Symbol File Path … ->” then entering the following text in the “Symbol path:” box.
|
1 |
SRV*c:\symbols*http://msdl.microsoft.com/download/symbols |
Now that this is configured I can open the memory.dmp files that I copied off of the machines earlier. That is done by clicking “File -> Open Crash Dump …” and browsing to the location of the memory.dmp file.
After the file has been opened I can analyze the crash with the “!analyze –v” command. That command revealed the following crash dump analysis information:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 |
******************************************************************************* * * * Bugcheck Analysis * * * ******************************************************************************* RDR_FILE_SYSTEM (27) If you see RxExceptionFilter on the stack then the 2nd and 3rd parameters are the exception record and context record. Do a .cxr on the 3rd parameter and then kb to obtain a more informative stack trace. The high 16 bits of the first parameter is the RDBSS bugcheck code, which is defined as follows: RDBSS_BUG_CHECK_CACHESUP = 0xca550000, RDBSS_BUG_CHECK_CLEANUP = 0xc1ee0000, RDBSS_BUG_CHECK_CLOSE = 0xc10e0000, RDBSS_BUG_CHECK_NTEXCEPT = 0xbaad0000, Arguments: Arg1: baad0080 Arg2: f78de840 Arg3: f78de53c Arg4: b9ebf79d Debugging Details: ------------------ EXCEPTION_RECORD: f78de840 -- (.exr fffffffff78de840) ExceptionAddress: b9ebf79d (SYMEVENT!SYMEvent_GetVMDataPtr+0x00004dbd) ExceptionCode: c0000005 (Access violation) ExceptionFlags: 00000000 NumberParameters: 2 Parameter[0]: 00000000 Parameter[1]: 00000028 Attempt to read from address 00000028 CONTEXT: f78de53c -- (.cxr fffffffff78de53c) eax=88ff58e8 ebx=00000000 ecx=00000004 edx=00000000 esi=89f67cf0 edi=b6f4cfa9 eip=b9ebf79d esp=f78de908 ebp=f78de974 iopl=0 nv up ei pl nz na po nc cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010202 SYMEVENT!SYMEvent_GetVMDataPtr+0x4dbd: b9ebf79d 8a5124 mov dl,byte ptr [ecx+24h] ds:0023:00000028=?? Resetting default scope PROCESS_NAME: System CURRENT_IRQL: 0 ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The memory could not be "%s". READ_ADDRESS: 00000028 BUGCHECK_STR: 0x27 DEFAULT_BUCKET_ID: NULL_CLASS_PTR_DEREFERENCE LAST_CONTROL_TRANSFER: from 8080be67 to b9ebf79d STACK_TEXT: WARNING: Stack unwind information not available. Following frames may be wrong. f78de974 8080be67 88ff58e8 89f67cf0 8a113c00 SYMEVENT!SYMEvent_GetVMDataPtr+0x4dbd f78de9c0 8080db05 8a0d5aa8 00000000 89f67c00 nt!IopCompleteUnloadOrDelete+0xce f78de9d8 b9ebd268 89f67c00 8a113c80 b9ec9c28 nt!IoDeleteDevice+0x81 f78de9e4 b9ec9c28 00000000 b9ec56d8 00000001 SYMEVENT!SYMEvent_GetVMDataPtr+0x2888 f78de9ec b9ec56d8 00000001 8a113c80 89f67cf0 SYMEVENT!EventObjectCreate+0xb98 00000000 00000000 00000000 00000000 00000000 SYMEVENT!SYMEvent_GetSubTask+0x18c8 FOLLOWUP_IP: SYMEVENT!SYMEvent_GetVMDataPtr+4dbd b9ebf79d 8a5124 mov dl,byte ptr [ecx+24h] SYMBOL_STACK_INDEX: 0 FOLLOWUP_NAME: MachineOwner MODULE_NAME: SYMEVENT IMAGE_NAME: SYMEVENT.SYS DEBUG_FLR_IMAGE_TIMESTAMP: 42ddb287 SYMBOL_NAME: SYMEVENT!SYMEvent_GetVMDataPtr+4dbd STACK_COMMAND: .cxr 0xfffffffff78de53c ; kb FAILURE_BUCKET_ID: 0x27_SYMEVENT!SYMEvent_GetVMDataPtr+4dbd BUCKET_ID: 0x27_SYMEVENT!SYMEvent_GetVMDataPtr+4dbd Followup: MachineOwner |
When we look at crash dumps we can keep some things in mind. Most of the time, even though Microsoft has a bad reputation for being unstable, the cause of the crash is not related to Microsoft code at all. So 99% of the time we can exclude code created by Microsoft as the likely culprit of the problem.
In this particular dump analysis we can see that the module that faulted was SYMEVENT.SYS. This file is a driver created by Symantec that is used to scan files for viruses.
|
1 2 3 4 5 6 7 8 9 10 |
1: kd> lmvm SYMEVENT start end module name b9afa000 b9b16cc0 SYMEVENT (export symbols) SYMEVENT.SYS Loaded symbol image file: SYMEVENT.SYS Image path: \??\C:\Program Files\Symantec\SYMEVENT.SYS Image name: SYMEVENT.SYS Timestamp: Tue Jul 19 21:10:15 2005 (42DDB287) CheckSum: 00024467 ImageSize: 0001CCC0 Translations: 0000.04b0 0000.04e0 0409.04b0 0409.04e0 |
I also checked the event log of this machine and it revealed the following stop error:
After Googling for this particular combination of stop error and the faulting module, I found the following link explaining the issue in some detail.
From the article:
Situation:
You install Symantec AntiVirus on a computer that runs Windows 2003/XP/2000/NT. After the installation, the computer unexpectedly restarts or encounters a blue screen with a STOP message similar to the following:STOP 0x0000007f (0x00000008, 0x00000000, 0x00000000, 0x00000000)
UNEXPECTED_KERNEL_MODE_TRAPYou may see the following message in the Event Log: “Event ID: 1005. Source: SAVRT: Symantec AntiVirus Auto-Protect could not scan file <path><filename> for viruses due to low kernel stack.”
A common configuration for this situation is a Windows 2000 Server with Terminal Services in Remote Administration Mode with a combination of any of the following applications: Symantec AntiVirus Corporate Edition, St. Bernard Open File Manager, Quota Manager, Legato RepliStor, or other “filter drivers” that register with the Kernel Stack.
Solution:
This problem occurs because there is a limited amount of kernel space available for kernel drivers. If the operating system runs out of kernel space, then the computer displays a blue screen error message.
To fix this problem, do all of the procedures and all of the steps within each procedure. Do the procedures and steps in the order in which they appear.