EDAC - Error Detection And Correction
-Written by Doug Thompson <norsk5@xmission.com>
+Written by Doug Thompson <dougthompson@xmission.com>
7 Dec 2005
+17 Jul 2007 Updated
-EDAC was written by:
- Thayne Harbaugh,
- modified by Dave Peterson, Doug Thompson, et al,
- from the bluesmoke.sourceforge.net project.
+EDAC is maintained and written by:
+ Doug Thompson, Dave Jiang, Dave Peterson et al,
+ original author: Thayne Harbaugh,
+
+Contact:
+ website: bluesmoke.sourceforge.net
+ mailing list: bluesmoke-devel@lists.sourceforge.net
+
+"bluesmoke" was the name for this device driver when it was "out-of-tree"
+and maintained at sourceforge.net. When it was pushed into 2.6.16 for the
+first time, it was renamed to 'EDAC'.
+
+The bluesmoke project at sourceforge.net is now utilized as a 'staging area'
+for EDAC development, before it is sent upstream to kernel.org
+
+At the bluesmoke/EDAC project site, is a series of quilt patches against
+recent kernels, stored in a SVN respository. For easier downloading, there
+is also a tarball snapshot available.
============================================================================
EDAC PURPOSE
The 'edac' kernel module goal is to detect and report errors that occur
-within the computer system. In the initial release, memory Correctable Errors
-(CE) and Uncorrectable Errors (UE) are the primary errors being harvested.
+within the computer system running under linux.
+
+MEMORY
+
+In the initial release, memory Correctable Errors (CE) and Uncorrectable
+Errors (UE) are the primary errors being harvested. These types of errors
+are harvested by the 'edac_mc' class of device.
Detecting CE events, then harvesting those events and reporting them,
CAN be a predictor of future UE events. With CE events, the system can
-continue to operate, but with less safety. Preventive maintainence and
+continue to operate, but with less safety. Preventive maintenance and
proactive part replacement of memory DIMMs exhibiting CEs can reduce
the likelihood of the dreaded UE events and system 'panics'.
+NON-MEMORY
+
+A new feature for EDAC, the edac_device class of device, was added in
+the 2.6.23 version of the kernel.
+
+This new device type allows for non-memory type of ECC hardware detectors
+to have their states harvested and presented to userspace via the sysfs
+interface.
+
+Some architectures have ECC detectors for L1, L2 and L3 caches, along with DMA
+engines, fabric switches, main data path switches, interconnections,
+and various other hardware data paths. If the hardware reports it, then
+a edac_device device probably can be constructed to harvest and present
+that to userspace.
+
+
+PCI BUS SCANNING
In addition, PCI Bus Parity and SERR Errors are scanned for on PCI devices
in order to determine if errors are occurring on data transfers.
+
The presence of PCI Parity errors must be examined with a grain of salt.
-There are several addin adapters that do NOT follow the PCI specification
+There are several add-in adapters that do NOT follow the PCI specification
with regards to Parity generation and reporting. The specification says
the vendor should tie the parity status bits to 0 if they do not intend
to generate parity. Some vendors do not do this, and thus the parity bit
can "float" giving false positives.
-The PCI Parity EDAC device has the ability to "skip" known flakey
-cards during the parity scan. These are set by the parity "blacklist"
-interface in the sysfs for PCI Parity. (See the PCI section in the sysfs
-section below.) There is also a parity "whitelist" which is used as
-an explicit list of devices to scan, while the blacklist is a list
-of devices to skip.
+In the kernel there is a pci device attribute located in sysfs that is
+checked by the EDAC PCI scanning code. If that attribute is set,
+PCI parity/error scannining is skipped for that device. The attribute
+is:
+
+ broken_parity_status
-EDAC will have future error detectors that will be added or integrated
-into EDAC in the following list:
+as is located in /sys/devices/pci<XXX>/0000:XX:YY.Z directorys for
+PCI devices.
+
+FUTURE HARDWARE SCANNING
+
+EDAC will have future error detectors that will be integrated with
+EDAC or added to it, in the following list:
MCE Machine Check Exception
MCA Machine Check Architecture
============================================================================
EDAC VERSIONING
-EDAC is composed of a "core" module (edac_mc.ko) and several Memory
+EDAC is composed of a "core" module (edac_core.ko) and several Memory
Controller (MC) driver modules. On a given system, the CORE
is loaded and one MC driver will be loaded. Both the CORE and
-the MC driver have individual versions that reflect current release
-level of their respective modules. Thus, to "report" on what version
-a system is running, one must report both the CORE's and the
-MC driver's versions.
+the MC driver (or edac_device driver) have individual versions that reflect
+current release level of their respective modules.
+
+Thus, to "report" on what version a system is running, one must report both
+the CORE's and the MC driver's versions.
LOADING
EDAC presents a 'sysfs' interface for control, reporting and attribute
reporting purposes.
-EDAC lives in the /sys/devices/system/edac directory. Within this directory
-there currently reside 2 'edac' components:
+EDAC lives in the /sys/devices/system/edac directory.
+
+Within this directory there currently reside 2 'edac' components:
mc memory controller(s) system
- pci PCI status system
+ pci PCI control and status system
============================================================================
Memory Controller (mc) Model
First a background on the memory controller's model abstracted in EDAC.
-Each mc device controls a set of DIMM memory modules. These modules are
-layed out in a Chip-Select Row (csrowX) and Channel table (chX). There can
-be multiple csrows and two channels.
+Each 'mc' device controls a set of DIMM memory modules. These modules are
+laid out in a Chip-Select Row (csrowX) and Channel table (chX). There can
+be multiple csrows and multiple channels.
Memory controllers allow for several csrows, with 8 csrows being a typical value.
Yet, the actual number of csrows depends on the electrical "loading"
of a given motherboard, memory controller and DIMM characteristics.
Dual channels allows for 128 bit data transfers to the CPU from memory.
+Some newer chipsets allow for more than 2 channels, like Fully Buffered DIMMs
+(FB-DIMMs). The following example will assume 2 channels:
Channel 0 Channel 1
DIMM_B1
Labels for these slots are usually silk screened on the motherboard. Slots
-labeled 'A' are channel 0 in this example. Slots labled 'B'
+labeled 'A' are channel 0 in this example. Slots labeled 'B'
are channel 1. Notice that there are two csrows possible on a
physical DIMM. These csrows are allocated their csrow assignment
based on the slot into which the memory DIMM is placed. Thus, when 1 DIMM
Memory DIMMs come single or dual "ranked". A rank is a populated csrow.
Thus, 2 single ranked DIMMs, placed in slots DIMM_A0 and DIMM_B0 above
will have 1 csrow, csrow0. csrow1 will be empty. On the other hand,
-when 2 dual ranked DIMMs are similiaryly placed, then both csrow0 and
+when 2 dual ranked DIMMs are similarly placed, then both csrow0 and
csrow1 will be populated. The pattern repeats itself for csrow2 and
csrow3.
Panic on UE control file:
- 'panic_on_ue'
+ 'edac_mc_panic_on_ue'
An uncorrectable error will cause a machine panic. This is usually
desirable. It is a bad idea to continue when an uncorrectable error
LOAD TIME: module/kernel parameter: panic_on_ue=[0|1]
- RUN TIME: echo "1" >/sys/devices/system/edac/mc/panic_on_ue
+ RUN TIME: echo "1" >/sys/devices/system/edac/mc/edac_mc_panic_on_ue
Log UE control file:
- 'log_ue'
+ 'edac_mc_log_ue'
Generate kernel messages describing uncorrectable errors. These errors
are reported through the system message log system. UE statistics
LOAD TIME: module/kernel parameter: log_ue=[0|1]
- RUN TIME: echo "1" >/sys/devices/system/edac/mc/log_ue
+ RUN TIME: echo "1" >/sys/devices/system/edac/mc/edac_mc_log_ue
Log CE control file:
- 'log_ce'
+ 'edac_mc_log_ce'
Generate kernel messages describing correctable errors. These
errors are reported through the system message log system.
LOAD TIME: module/kernel parameter: log_ce=[0|1]
- RUN TIME: echo "1" >/sys/devices/system/edac/mc/log_ce
+ RUN TIME: echo "1" >/sys/devices/system/edac/mc/edac_mc_log_ce
Polling period control file:
- 'poll_msec'
+ 'edac_mc_poll_msec'
The time period, in milliseconds, for polling for error information.
Too small a value wastes resources. Too large a value might delay
necessary handling of errors and might loose valuable information for
- locating the error. 1000 milliseconds (once each second) is about
- right for most uses.
+ locating the error. 1000 milliseconds (once each second) is the current
+ default. Systems which require all the bandwidth they can get, may
+ increase this.
LOAD TIME: module/kernel parameter: poll_msec=[0|1]
- RUN TIME: echo "1000" >/sys/devices/system/edac/mc/poll_msec
-
-
-Module Version read-only attribute file:
-
- 'mc_version'
-
- The EDAC CORE modules's version and compile date are shown here to
- indicate what EDAC is running.
-
+ RUN TIME: echo "1000" >/sys/devices/system/edac/mc/edac_mc_poll_msec
============================================================================
-DIMM capability attribute file:
-
- 'edac_capability'
-
- The EDAC (Error Detection and Correction) capabilities/modes of
- the memory controller hardware.
-
-
-DIMM Current Capability attribute file:
-
- 'edac_current_capability'
-
- The EDAC capabilities available with the hardware
- configuration. This may not be the same as "EDAC capability"
- if the correct memory is not used. If a memory controller is
- capable of EDAC, but DIMMs without check bits are in use, then
- Parity, SECDED, S4ECD4ED capabilities will not be available
- even though the memory controller might be capable of those
- modes with the proper memory loaded.
-
-
-Memory Type supported on this controller attribute file:
-
- 'supported_mem_type'
-
- This attribute file displays the memory type, usually
- buffered and unbuffered DIMMs.
-
-
Memory Controller name attribute file:
'mc_name'
that is being utilized.
-Memory Controller Module name attribute file:
-
- 'module_name'
-
- This attribute file displays the memory controller module name,
- version and date built. The name of the memory controller
- hardware - some drivers work with multiple controllers and
- this field shows which hardware is present.
-
-
Total memory managed by this memory controller attribute file:
'size_mb'
'device'
- Symlink to the memory controller device
+ Symlink to the memory controller device.
+
+Sdram memory scrubbing rate:
+
+ 'sdram_scrub_rate'
+
+ Read/Write attribute file that controls memory scrubbing. The scrubbing
+ rate is set by writing a minimum bandwith in bytes/sec to the attribute
+ file. The rate will be translated to an internal value that gives at
+ least the specified rate.
+
+ Reading the file will return the actual scrubbing rate employed.
+
+ If configuration fails or memory scrubbing is not implemented, the value
+ of the attribute file will be -1.
'size_mb'
This attribute file displays, in count of megabytes, of memory
- that this csrow contatins.
+ that this csrow contains.
Memory Type attribute file:
This attribute file will display what type of memory is currently
on this csrow. Normally, either buffered or unbuffered memory.
+ Examples:
+ Registered-DDR
+ Unbuffered-DDR
EDAC Mode of operation attribute file:
'dev_type'
- This attribute file will display what type of DIMM device is
- being utilized. Example: x4
+ This attribute file will display what type of DRAM device is
+ being utilized on this DIMM.
+ Examples:
+ x1
+ x2
+ x4
+ x8
Channel 0 CE Count attribute file:
If logging for UEs and CEs are enabled then system logs will have
error notices indicating errors that have been detected:
-MC0: CE page 0x283, offset 0xce0, grain 8, syndrome 0x6ec3, row 0,
+EDAC MC0: CE page 0x283, offset 0xce0, grain 8, syndrome 0x6ec3, row 0,
channel 1 "DIMM_B1": amd76x_edac
-MC0: CE page 0x1e5, offset 0xfb0, grain 8, syndrome 0xb741, row 0,
+EDAC MC0: CE page 0x1e5, offset 0xfb0, grain 8, syndrome 0xb741, row 0,
channel 1 "DIMM_B1": amd76x_edac
for any parity error regardless of whether Parity is enabled on the
device. (The spec indicates parity is generated in some cases).
On Header Type 01 bridges, the secondary status register is also
-looked at to see if parity ocurred on the bus on the other side of
+looked at to see if parity occurred on the bus on the other side of
the bridge.
'panic_on_pci_parity'
- This control files enables or disables panic'ing when a parity
+ This control files enables or disables panicking when a parity
error has been detected.
-PCI Device Whitelist:
+=======================================================================
- 'pci_parity_whitelist'
- This control file allows for an explicit list of PCI devices to be
- scanned for parity errors. Only devices found on this list will
- be examined. The list is a line of hexadecimel VENDOR and DEVICE
- ID tuples:
+EDAC_DEVICE type of device
- 1022:7450,1434:16a6
+In the header file, edac_core.h, there is a series of edac_device structures
+and APIs for the EDAC_DEVICE.
- One or more can be inserted, seperated by a comma.
+User space access to an edac_device is through the sysfs interface.
- To write the above list doing the following as one command line:
+At the location /sys/devices/system/edac (sysfs) new edac_device devices will
+appear.
- echo "1022:7450,1434:16a6"
- > /sys/devices/system/edac/pci/pci_parity_whitelist
+There is a three level tree beneath the above 'edac' directory. For example,
+the 'test_device_edac' device (found at the bluesmoke.sourceforget.net website)
+installs itself as:
+ /sys/devices/systm/edac/test-instance
+in this directory are various controls, a symlink and one or more 'instance'
+directorys.
- To display what the whitelist is, simply 'cat' the same file.
+The standard default controls are:
+ log_ce boolean to log CE events
+ log_ue boolean to log UE events
+ panic_on_ue boolean to 'panic' the system if an UE is encountered
+ (default off, can be set true via startup script)
+ poll_msec time period between POLL cycles for events
-PCI Device Blacklist:
+The test_device_edac device adds at least one of its own custom control:
- 'pci_parity_blacklist'
+ test_bits which in the current test driver does nothing but
+ show how it is installed. A ported driver can
+ add one or more such controls and/or attributes
+ for specific uses.
+ One out-of-tree driver uses controls here to allow
+ for ERROR INJECTION operations to hardware
+ injection registers
- This control file allows for a list of PCI devices to be
- skipped for scanning.
- The list is a line of hexadecimel VENDOR and DEVICE ID tuples:
+The symlink points to the 'struct dev' that is registered for this edac_device.
- 1022:7450,1434:16a6
+INSTANCES
- One or more can be inserted, seperated by a comma.
+One or more instance directories are present. For the 'test_device_edac' case:
- To write the above list doing the following as one command line:
+ test-instance0
- echo "1022:7450,1434:16a6"
- > /sys/devices/system/edac/pci/pci_parity_blacklist
+In this directory there are two default counter attributes, which are totals of
+counter in deeper subdirectories.
- To display what the whitelist current contatins,
- simply 'cat' the same file.
+ ce_count total of CE events of subdirectories
+ ue_count total of UE events of subdirectories
-=======================================================================
+BLOCKS
+
+At the lowest directory level is the 'block' directory. There can be 0, 1
+or more blocks specified in each instance.
+
+ test-block0
+
+
+In this directory the default attributes are:
+
+ ce_count which is counter of CE events for this 'block'
+ of hardware being monitored
+ ue_count which is counter of UE events for this 'block'
+ of hardware being monitored
+
+
+The 'test_device_edac' device adds 4 attributes and 1 control:
-PCI Vendor and Devices IDs can be obtained with the lspci command. Using
-the -n option lspci will display the vendor and device IDs. The system
-adminstrator will have to determine which devices should be scanned or
-skipped.
+ test-block-bits-0 for every POLL cycle this counter
+ is incremented
+ test-block-bits-1 every 10 cycles, this counter is bumped once,
+ and test-block-bits-0 is set to 0
+ test-block-bits-2 every 100 cycles, this counter is bumped once,
+ and test-block-bits-1 is set to 0
+ test-block-bits-3 every 1000 cycles, this counter is bumped once,
+ and test-block-bits-2 is set to 0
+ reset-counters writing ANY thing to this control will
+ reset all the above counters.
-The two lists (white and black) are prioritized. blacklist is the lower
-priority and will NOT be utilized when a whitelist has been set.
-Turn OFF a whitelist by an empty echo command:
- echo > /sys/devices/system/edac/pci/pci_parity_whitelist
+Use of the 'test_device_edac' driver should any others to create their own
+unique drivers for their hardware systems.
-and any previous blacklist will be utililzed.
+The 'test_device_edac' sample driver is located at the
+bluesmoke.sourceforge.net project site for EDAC.
git.openpandora.org / pandora-kernel.git / blobdiff