3 EDAC - Error Detection And Correction
5 Written by Doug Thompson <dougthompson@xmission.com>
9 (c) Mauro Carvalho Chehab <mchehab@redhat.com>
10 05 Aug 2009 Nehalem interface
12 EDAC is maintained and written by:
14 Doug Thompson, Dave Jiang, Dave Peterson et al,
15 original author: Thayne Harbaugh,
18 website: bluesmoke.sourceforge.net
19 mailing list: bluesmoke-devel@lists.sourceforge.net
21 "bluesmoke" was the name for this device driver when it was "out-of-tree"
22 and maintained at sourceforge.net. When it was pushed into 2.6.16 for the
23 first time, it was renamed to 'EDAC'.
25 The bluesmoke project at sourceforge.net is now utilized as a 'staging area'
26 for EDAC development, before it is sent upstream to kernel.org
28 At the bluesmoke/EDAC project site is a series of quilt patches against
29 recent kernels, stored in a SVN repository. For easier downloading, there
30 is also a tarball snapshot available.
32 ============================================================================
35 The 'edac' kernel module goal is to detect and report errors that occur
36 within the computer system running under linux.
40 In the initial release, memory Correctable Errors (CE) and Uncorrectable
41 Errors (UE) are the primary errors being harvested. These types of errors
42 are harvested by the 'edac_mc' class of device.
44 Detecting CE events, then harvesting those events and reporting them,
45 CAN be a predictor of future UE events. With CE events, the system can
46 continue to operate, but with less safety. Preventive maintenance and
47 proactive part replacement of memory DIMMs exhibiting CEs can reduce
48 the likelihood of the dreaded UE events and system 'panics'.
52 A new feature for EDAC, the edac_device class of device, was added in
53 the 2.6.23 version of the kernel.
55 This new device type allows for non-memory type of ECC hardware detectors
56 to have their states harvested and presented to userspace via the sysfs
59 Some architectures have ECC detectors for L1, L2 and L3 caches, along with DMA
60 engines, fabric switches, main data path switches, interconnections,
61 and various other hardware data paths. If the hardware reports it, then
62 a edac_device device probably can be constructed to harvest and present
68 In addition, PCI Bus Parity and SERR Errors are scanned for on PCI devices
69 in order to determine if errors are occurring on data transfers.
71 The presence of PCI Parity errors must be examined with a grain of salt.
72 There are several add-in adapters that do NOT follow the PCI specification
73 with regards to Parity generation and reporting. The specification says
74 the vendor should tie the parity status bits to 0 if they do not intend
75 to generate parity. Some vendors do not do this, and thus the parity bit
76 can "float" giving false positives.
78 In the kernel there is a PCI device attribute located in sysfs that is
79 checked by the EDAC PCI scanning code. If that attribute is set,
80 PCI parity/error scanning is skipped for that device. The attribute
85 as is located in /sys/devices/pci<XXX>/0000:XX:YY.Z directories for
88 FUTURE HARDWARE SCANNING
90 EDAC will have future error detectors that will be integrated with
91 EDAC or added to it, in the following list:
93 MCE Machine Check Exception
94 MCA Machine Check Architecture
95 NMI NMI notification of ECC errors
96 MSRs Machine Specific Register error cases
99 These errors are usually bus errors, ECC errors, thermal throttling
103 ============================================================================
106 EDAC is composed of a "core" module (edac_core.ko) and several Memory
107 Controller (MC) driver modules. On a given system, the CORE
108 is loaded and one MC driver will be loaded. Both the CORE and
109 the MC driver (or edac_device driver) have individual versions that reflect
110 current release level of their respective modules.
112 Thus, to "report" on what version a system is running, one must report both
113 the CORE's and the MC driver's versions.
118 If 'edac' was statically linked with the kernel then no loading is
119 necessary. If 'edac' was built as modules then simply modprobe the
120 'edac' pieces that you need. You should be able to modprobe
121 hardware-specific modules and have the dependencies load the necessary core
126 $> modprobe amd76x_edac
128 loads both the amd76x_edac.ko memory controller module and the edac_mc.ko
132 ============================================================================
135 EDAC presents a 'sysfs' interface for control, reporting and attribute
138 EDAC lives in the /sys/devices/system/edac directory.
140 Within this directory there currently reside 2 'edac' components:
142 mc memory controller(s) system
143 pci PCI control and status system
146 ============================================================================
147 Memory Controller (mc) Model
149 First a background on the memory controller's model abstracted in EDAC.
150 Each 'mc' device controls a set of DIMM memory modules. These modules are
151 laid out in a Chip-Select Row (csrowX) and Channel table (chX). There can
152 be multiple csrows and multiple channels.
154 Memory controllers allow for several csrows, with 8 csrows being a typical value.
155 Yet, the actual number of csrows depends on the electrical "loading"
156 of a given motherboard, memory controller and DIMM characteristics.
158 Dual channels allows for 128 bit data transfers to the CPU from memory.
159 Some newer chipsets allow for more than 2 channels, like Fully Buffered DIMMs
160 (FB-DIMMs). The following example will assume 2 channels:
164 ===================================
165 csrow0 | DIMM_A0 | DIMM_B0 |
166 csrow1 | DIMM_A0 | DIMM_B0 |
167 ===================================
169 ===================================
170 csrow2 | DIMM_A1 | DIMM_B1 |
171 csrow3 | DIMM_A1 | DIMM_B1 |
172 ===================================
174 In the above example table there are 4 physical slots on the motherboard
182 Labels for these slots are usually silk screened on the motherboard. Slots
183 labeled 'A' are channel 0 in this example. Slots labeled 'B'
184 are channel 1. Notice that there are two csrows possible on a
185 physical DIMM. These csrows are allocated their csrow assignment
186 based on the slot into which the memory DIMM is placed. Thus, when 1 DIMM
187 is placed in each Channel, the csrows cross both DIMMs.
189 Memory DIMMs come single or dual "ranked". A rank is a populated csrow.
190 Thus, 2 single ranked DIMMs, placed in slots DIMM_A0 and DIMM_B0 above
191 will have 1 csrow, csrow0. csrow1 will be empty. On the other hand,
192 when 2 dual ranked DIMMs are similarly placed, then both csrow0 and
193 csrow1 will be populated. The pattern repeats itself for csrow2 and
196 The representation of the above is reflected in the directory tree
197 in EDAC's sysfs interface. Starting in directory
198 /sys/devices/system/edac/mc each memory controller will be represented
199 by its own 'mcX' directory, where 'X' is the index of the MC.
209 Under each 'mcX' directory each 'csrowX' is again represented by a
210 'csrowX', where 'X' is the csrow index:
220 Notice that there is no csrow1, which indicates that csrow0 is
221 composed of a single ranked DIMMs. This should also apply in both
222 Channels, in order to have dual-channel mode be operational. Since
223 both csrow2 and csrow3 are populated, this indicates a dual ranked
224 set of DIMMs for channels 0 and 1.
227 Within each of the 'mcX' and 'csrowX' directories are several
228 EDAC control and attribute files.
230 ============================================================================
234 In 'mcX' directories are EDAC control and attribute files for
235 this 'X' instance of the memory controllers:
238 Counter reset control file:
242 This write-only control file will zero all the statistical counters
243 for UE and CE errors. Zeroing the counters will also reset the timer
244 indicating how long since the last counter zero. This is useful
245 for computing errors/time. Since the counters are always reset at
246 driver initialization time, no module/kernel parameter is available.
248 RUN TIME: echo "anything" >/sys/devices/system/edac/mc/mc0/counter_reset
250 This resets the counters on memory controller 0
253 Seconds since last counter reset control file:
255 'seconds_since_reset'
257 This attribute file displays how many seconds have elapsed since the
258 last counter reset. This can be used with the error counters to
263 Memory Controller name attribute file:
267 This attribute file displays the type of memory controller
268 that is being utilized.
271 Total memory managed by this memory controller attribute file:
275 This attribute file displays, in count of megabytes, of memory
276 that this instance of memory controller manages.
279 Total Uncorrectable Errors count attribute file:
283 This attribute file displays the total count of uncorrectable
284 errors that have occurred on this memory controller. If panic_on_ue
285 is set this counter will not have a chance to increment,
286 since EDAC will panic the system.
289 Total UE count that had no information attribute fileY:
293 This attribute file displays the number of UEs that have occurred
294 with no information as to which DIMM slot is having errors.
297 Total Correctable Errors count attribute file:
301 This attribute file displays the total count of correctable
302 errors that have occurred on this memory controller. This
303 count is very important to examine. CEs provide early
304 indications that a DIMM is beginning to fail. This count
305 field should be monitored for non-zero values and report
306 such information to the system administrator.
309 Total Correctable Errors count attribute file:
313 This attribute file displays the number of CEs that
314 have occurred wherewith no informations as to which DIMM slot
315 is having errors. Memory is handicapped, but operational,
316 yet no information is available to indicate which slot
317 the failing memory is in. This count field should be also
318 be monitored for non-zero values.
324 Symlink to the memory controller device.
326 Sdram memory scrubbing rate:
330 Read/Write attribute file that controls memory scrubbing. The scrubbing
331 rate is set by writing a minimum bandwidth in bytes/sec to the attribute
332 file. The rate will be translated to an internal value that gives at
333 least the specified rate.
335 Reading the file will return the actual scrubbing rate employed.
337 If configuration fails or memory scrubbing is not implemented, the value
338 of the attribute file will be -1.
342 ============================================================================
345 In the 'csrowX' directories are EDAC control and attribute files for
346 this 'X' instance of csrow:
349 Total Uncorrectable Errors count attribute file:
353 This attribute file displays the total count of uncorrectable
354 errors that have occurred on this csrow. If panic_on_ue is set
355 this counter will not have a chance to increment, since EDAC
356 will panic the system.
359 Total Correctable Errors count attribute file:
363 This attribute file displays the total count of correctable
364 errors that have occurred on this csrow. This
365 count is very important to examine. CEs provide early
366 indications that a DIMM is beginning to fail. This count
367 field should be monitored for non-zero values and report
368 such information to the system administrator.
371 Total memory managed by this csrow attribute file:
375 This attribute file displays, in count of megabytes, of memory
376 that this csrow contains.
379 Memory Type attribute file:
383 This attribute file will display what type of memory is currently
384 on this csrow. Normally, either buffered or unbuffered memory.
390 EDAC Mode of operation attribute file:
394 This attribute file will display what type of Error detection
395 and correction is being utilized.
398 Device type attribute file:
402 This attribute file will display what type of DRAM device is
403 being utilized on this DIMM.
411 Channel 0 CE Count attribute file:
415 This attribute file will display the count of CEs on this
416 DIMM located in channel 0.
419 Channel 0 UE Count attribute file:
423 This attribute file will display the count of UEs on this
424 DIMM located in channel 0.
427 Channel 0 DIMM Label control file:
431 This control file allows this DIMM to have a label assigned
432 to it. With this label in the module, when errors occur
433 the output can provide the DIMM label in the system log.
434 This becomes vital for panic events to isolate the
435 cause of the UE event.
437 DIMM Labels must be assigned after booting, with information
438 that correctly identifies the physical slot with its
439 silk screen label. This information is currently very
440 motherboard specific and determination of this information
441 must occur in userland at this time.
444 Channel 1 CE Count attribute file:
448 This attribute file will display the count of CEs on this
449 DIMM located in channel 1.
452 Channel 1 UE Count attribute file:
456 This attribute file will display the count of UEs on this
457 DIMM located in channel 0.
460 Channel 1 DIMM Label control file:
464 This control file allows this DIMM to have a label assigned
465 to it. With this label in the module, when errors occur
466 the output can provide the DIMM label in the system log.
467 This becomes vital for panic events to isolate the
468 cause of the UE event.
470 DIMM Labels must be assigned after booting, with information
471 that correctly identifies the physical slot with its
472 silk screen label. This information is currently very
473 motherboard specific and determination of this information
474 must occur in userland at this time.
476 ============================================================================
479 If logging for UEs and CEs are enabled then system logs will have
480 error notices indicating errors that have been detected:
482 EDAC MC0: CE page 0x283, offset 0xce0, grain 8, syndrome 0x6ec3, row 0,
483 channel 1 "DIMM_B1": amd76x_edac
485 EDAC MC0: CE page 0x1e5, offset 0xfb0, grain 8, syndrome 0xb741, row 0,
486 channel 1 "DIMM_B1": amd76x_edac
489 The structure of the message is:
490 the memory controller (MC0)
493 offset in the page (0xce0)
494 the byte granularity (grain 8)
495 or resolution of the error
496 the error syndrome (0xb741)
498 memory channel (channel 1)
499 DIMM label, if set prior (DIMM B1
500 and then an optional, driver-specific message that may
501 have additional information.
503 Both UEs and CEs with no info will lack all but memory controller,
504 error type, a notice of "no info" and then an optional,
505 driver-specific error message.
508 ============================================================================
509 PCI Bus Parity Detection
512 On Header Type 00 devices the primary status is looked at
513 for any parity error regardless of whether Parity is enabled on the
514 device. (The spec indicates parity is generated in some cases).
515 On Header Type 01 bridges, the secondary status register is also
516 looked at to see if parity occurred on the bus on the other side of
522 Under /sys/devices/system/edac/pci are control and attribute files as follows:
525 Enable/Disable PCI Parity checking control file:
530 This control file enables or disables the PCI Bus Parity scanning
531 operation. Writing a 1 to this file enables the scanning. Writing
532 a 0 to this file disables the scanning.
535 echo "1" >/sys/devices/system/edac/pci/check_pci_parity
538 echo "0" >/sys/devices/system/edac/pci/check_pci_parity
545 This attribute file will display the number of parity errors that
549 ============================================================================
552 Panic on UE control file:
554 'edac_mc_panic_on_ue'
556 An uncorrectable error will cause a machine panic. This is usually
557 desirable. It is a bad idea to continue when an uncorrectable error
558 occurs - it is indeterminate what was uncorrected and the operating
559 system context might be so mangled that continuing will lead to further
560 corruption. If the kernel has MCE configured, then EDAC will never
563 LOAD TIME: module/kernel parameter: edac_mc_panic_on_ue=[0|1]
565 RUN TIME: echo "1" > /sys/module/edac_core/parameters/edac_mc_panic_on_ue
572 Generate kernel messages describing uncorrectable errors. These errors
573 are reported through the system message log system. UE statistics
574 will be accumulated even when UE logging is disabled.
576 LOAD TIME: module/kernel parameter: edac_mc_log_ue=[0|1]
578 RUN TIME: echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ue
585 Generate kernel messages describing correctable errors. These
586 errors are reported through the system message log system.
587 CE statistics will be accumulated even when CE logging is disabled.
589 LOAD TIME: module/kernel parameter: edac_mc_log_ce=[0|1]
591 RUN TIME: echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ce
594 Polling period control file:
598 The time period, in milliseconds, for polling for error information.
599 Too small a value wastes resources. Too large a value might delay
600 necessary handling of errors and might loose valuable information for
601 locating the error. 1000 milliseconds (once each second) is the current
602 default. Systems which require all the bandwidth they can get, may
605 LOAD TIME: module/kernel parameter: edac_mc_poll_msec=[0|1]
607 RUN TIME: echo "1000" > /sys/module/edac_core/parameters/edac_mc_poll_msec
610 Panic on PCI PARITY Error:
612 'panic_on_pci_parity'
615 This control files enables or disables panicking when a parity
616 error has been detected.
619 module/kernel parameter: edac_panic_on_pci_pe=[0|1]
622 echo "1" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe
625 echo "0" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe
629 =======================================================================
632 EDAC_DEVICE type of device
634 In the header file, edac_core.h, there is a series of edac_device structures
635 and APIs for the EDAC_DEVICE.
637 User space access to an edac_device is through the sysfs interface.
639 At the location /sys/devices/system/edac (sysfs) new edac_device devices will
642 There is a three level tree beneath the above 'edac' directory. For example,
643 the 'test_device_edac' device (found at the bluesmoke.sourceforget.net website)
646 /sys/devices/systm/edac/test-instance
648 in this directory are various controls, a symlink and one or more 'instance'
651 The standard default controls are:
653 log_ce boolean to log CE events
654 log_ue boolean to log UE events
655 panic_on_ue boolean to 'panic' the system if an UE is encountered
656 (default off, can be set true via startup script)
657 poll_msec time period between POLL cycles for events
659 The test_device_edac device adds at least one of its own custom control:
661 test_bits which in the current test driver does nothing but
662 show how it is installed. A ported driver can
663 add one or more such controls and/or attributes
665 One out-of-tree driver uses controls here to allow
666 for ERROR INJECTION operations to hardware
669 The symlink points to the 'struct dev' that is registered for this edac_device.
673 One or more instance directories are present. For the 'test_device_edac' case:
678 In this directory there are two default counter attributes, which are totals of
679 counter in deeper subdirectories.
681 ce_count total of CE events of subdirectories
682 ue_count total of UE events of subdirectories
686 At the lowest directory level is the 'block' directory. There can be 0, 1
687 or more blocks specified in each instance.
692 In this directory the default attributes are:
694 ce_count which is counter of CE events for this 'block'
695 of hardware being monitored
696 ue_count which is counter of UE events for this 'block'
697 of hardware being monitored
700 The 'test_device_edac' device adds 4 attributes and 1 control:
702 test-block-bits-0 for every POLL cycle this counter
704 test-block-bits-1 every 10 cycles, this counter is bumped once,
705 and test-block-bits-0 is set to 0
706 test-block-bits-2 every 100 cycles, this counter is bumped once,
707 and test-block-bits-1 is set to 0
708 test-block-bits-3 every 1000 cycles, this counter is bumped once,
709 and test-block-bits-2 is set to 0
712 reset-counters writing ANY thing to this control will
713 reset all the above counters.
716 Use of the 'test_device_edac' driver should any others to create their own
717 unique drivers for their hardware systems.
719 The 'test_device_edac' sample driver is located at the
720 bluesmoke.sourceforge.net project site for EDAC.
722 =======================================================================
723 NEHALEM USAGE OF EDAC APIs
725 This chapter documents some EXPERIMENTAL mappings for EDAC API to handle
726 Nehalem EDAC driver. They will likely be changed on future versions
729 Due to the way Nehalem exports Memory Controller data, some adjustments
730 were done at i7core_edac driver. This chapter will cover those differences
732 1) On Nehalem, there are one Memory Controller per Quick Patch Interconnect
733 (QPI). At the driver, the term "socket" means one QPI. This is
734 associated with a physical CPU socket.
736 Each MC have 3 physical read channels, 3 physical write channels and
737 3 logic channels. The driver currenty sees it as just 3 channels.
738 Each channel can have up to 3 DIMMs.
740 The minimum known unity is DIMMs. There are no information about csrows.
741 As EDAC API maps the minimum unity is csrows, the driver sequencially
742 maps channel/dimm into different csrows.
744 For example, suposing the following layout:
745 Ch0 phy rd0, wr0 (0x063f4031): 2 ranks, UDIMMs
746 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
747 dimm 1 1024 Mb offset: 4, bank: 8, rank: 1, row: 0x4000, col: 0x400
748 Ch1 phy rd1, wr1 (0x063f4031): 2 ranks, UDIMMs
749 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
750 Ch2 phy rd3, wr3 (0x063f4031): 2 ranks, UDIMMs
751 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
752 The driver will map it as:
753 csrow0: channel 0, dimm0
754 csrow1: channel 0, dimm1
755 csrow2: channel 1, dimm0
756 csrow3: channel 2, dimm0
761 Each QPI is exported as a different memory controller.
763 2) Nehalem MC has the hability to generate errors. The driver implements this
764 functionality via some error injection nodes:
766 For injecting a memory error, there are some sysfs nodes, under
767 /sys/devices/system/edac/mc/mc?/:
770 Controls the error injection mask register. It is possible to specify
771 several characteristics of the address to match an error code:
772 dimm = the affected dimm. Numbers are relative to a channel;
773 rank = the memory rank;
774 channel = the channel that will generate an error;
775 bank = the affected bank;
776 page = the page address;
777 column (or col) = the address column.
778 each of the above values can be set to "any" to match any valid value.
780 At driver init, all values are set to any.
782 For example, to generate an error at rank 1 of dimm 2, for any channel,
783 any bank, any page, any column:
784 echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm
785 echo 1 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank
787 To return to the default behaviour of matching any, you can do:
788 echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm
789 echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank
792 specifies what bits will have troubles,
795 specifies what ECC cache section will get the error:
801 specifies the type of error, being a combination of the following bits:
806 inject_enable starts the error generation when something different
809 All inject vars can be read. root permission is needed for write.
811 Datasheet states that the error will only be generated after a write on an
812 address that matches inject_addrmatch. It seems, however, that reading will
813 also produce an error.
815 For example, the following code will generate an error for any write access
816 at socket 0, on any DIMM/address on channel 2:
818 echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/channel
819 echo 2 >/sys/devices/system/edac/mc/mc0/inject_type
820 echo 64 >/sys/devices/system/edac/mc/mc0/inject_eccmask
821 echo 3 >/sys/devices/system/edac/mc/mc0/inject_section
822 echo 1 >/sys/devices/system/edac/mc/mc0/inject_enable
823 dd if=/dev/mem of=/dev/null seek=16k bs=4k count=1 >& /dev/null
825 For socket 1, it is needed to replace "mc0" by "mc1" at the above
828 The generated error message will look like:
830 EDAC MC0: UE row 0, channel-a= 0 channel-b= 0 labels "-": NON_FATAL (addr = 0x0075b980, socket=0, Dimm=0, Channel=2, syndrome=0x00000040, count=1, Err=8c0000400001009f:4000080482 (read error: read ECC error))
832 3) Nehalem specific Corrected Error memory counters
834 Nehalem have some registers to count memory errors. The driver uses those
835 registers to report Corrected Errors on devices with Registered Dimms.
837 However, those counters don't work with Unregistered Dimms. As the chipset
838 offers some counters that also work with UDIMMS (but with a worse level of
839 granularity than the default ones), the driver exposes those registers for
842 They can be read by looking at the contents of all_channel_counts/
844 $ for i in /sys/devices/system/edac/mc/mc0/all_channel_counts/*; do echo $i; cat $i; done
845 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm0
847 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm1
849 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm2
852 What happens here is that errors on different csrows, but at the same
853 dimm number will increment the same counter.
854 So, in this memory mapping:
855 csrow0: channel 0, dimm0
856 csrow1: channel 0, dimm1
857 csrow2: channel 1, dimm0
858 csrow3: channel 2, dimm0
859 The hardware will increment udimm0 for an error at the first dimm at either
860 csrow0, csrow2 or csrow3;
861 The hardware will increment udimm1 for an error at the second dimm at either
862 csrow0, csrow2 or csrow3;
863 The hardware will increment udimm2 for an error at the third dimm at either
864 csrow0, csrow2 or csrow3;
866 4) Standard error counters
868 The standard error counters are generated when an mcelog error is received
869 by the driver. Since, with udimm, this is counted by software, it is
870 possible that some errors could be lost. With rdimm's, they displays the
871 contents of the registers