SMART Check and Attribute 190

cPanel appears to have recently corrected the auto-detection for SATA drives. When this correction went out, we got a flood of support requests asking about SMART emails being received. Almost all of the requests were regarding errors that occurred in the past.

AndyReed posted details about old errors in a post here:

http://forums.cpanel.net/showthread.php?t=53613#4

But we're also noticing that reports are being generated due to attributes 190 having an "In_the_past" value for WHEN_FAILED. This attribute is the Airflow_Temperature_Cel and indicates when the drive's temperature exceeded a threshold. This is certainly good information to have if your drive is running hot, but after you install a fan and and get it running cool again, the WHEN_FAILED value will continue to trip SMART email alerts.

I've already contact Seagate and have confirmed that it is impossible to reset the attributes of the drive.

The smartd SMART daemon makes use of /etc/smartd.conf to control which which options and drives should be monitored with SMART checks. Within this file, there as a -I option which allows you to ignore certain attributes in error reports. cPanel's smartcheck script does not use this file so we can't omit attributes from the error alerts. Is there a sanctioned method of ignoring the "In_the_past" value for a given attribute in /scripts/smartcheck?

If not, we'll probably be disabling /scripts/smartcheck across all of our servers and continuing to use the smartd service to monitor drives instead. It might be a prudent move to have cPanel simply setup /etc/smartd.conf and allow the SMART daemon to do the actual monitoring. You could even add smartd to tailwatchd for good measure.

 

Thanks, cpanelkenneth. We already went digging through the smartctl flags to see if there was a way to suppress certain errors and just add them to /scripts/smartcheck's binary call directly. The smartctl binary itself doesn't use a config file, but when you setup the smartd service, you can use /etc/smartd.conf to control which drives are automatically checked and the config file itself has a set of it's own flags which you can use to prune the alerts.

Relevant Documentation Snippet from /etc/smartd.conf:

Code:

# HERE IS A LIST OF DIRECTIVES FOR THIS CONFIGURATION FILE
#   -d TYPE Set the device type to one of: ata, scsi
#   -T TYPE set the tolerance to one of: normal, permissive
#   -o VAL  Enable/disable automatic offline tests (on/off)
#   -S VAL  Enable/disable attribute autosave (on/off)
#   -H      Monitor SMART Health Status, report if failed
#   -l TYPE Monitor SMART log.  Type is one of: error, selftest
#   -f      Monitor for failure of any 'Usage' Attributes
#   -m ADD  Send warning email to ADD for -H, -l error, -l selftest, and -f
#   -M TYPE Modify email warning behavior (see man page)
#   -p      Report changes in 'Prefailure' Normalized Attributes
#   -u      Report changes in 'Usage' Normalized Attributes
#   -t      Equivalent to -p and -u Directives
#   -r ID   Also report Raw values of Attribute ID with -p, -u or -t
#   -R ID   Track changes in Attribute ID Raw value with -p, -u or -t
#   -i ID   Ignore Attribute ID for -f Directive
#   -I ID   Ignore Attribute ID for -p, -u or -t Directive
#   -v N,ST Modifies labeling of Attribute N (see man page)
#   -a      Default: equivalent to -H -f -t -l error -l selftest
#   -F TYPE Use firmware bug workaround. Type is one of: none, samsung
#   -P TYPE Drive-specific presets: use, ignore, show, showall
#    #      Comment: text after a hash sign is ignored
#    \      Line continuation character
# Attribute ID is a decimal integer 1 <= ID <= 255
# All but -d, -m and -M Directives are only implemented for ATA devices

Would be nice to have these options available directly within cPanel. Thanks for checking with the devs. Look forward to hearing details.

 

Hello cpdan,

1) Yup:

Code:

[root@colorado ~]# smartctl -q errorsonly -H -l selftest -l error -d ata /dev/sda
Please note the following marginal Attributes:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
190 Airflow_Temperature_Cel 0x0022   059   044   045    Old_age   Always   In_the_past 41 (0 3 46 33)

2) Nope:

Code:

=======> UNRECOGNIZED OPTION: I

Use smartctl -h to get a usage summary

The options I posted in my last post are for the smartd.conf file directives. The -i flag is recognized by smartctl, but it doesn't have the same meaning. Within the smartd.conf -i means to ignore failures of any attributes, passing it to the binary means to print info about the device.

Thanks for the response. Let me know if there's anything else I can do to help!

 

Yup, the error's present in both. Yeah, we'll probably go back to using smartd until cPanel's script is able to include some configurable options. Thanks for the help.

 

Within the smartd.conf file, you can set them as such (SATA drive):

Code:

/dev/sda -d sat -S on -o on -a -I 194 -I 190 -s (S/../.././(01|05|09|13|17|21)|L/../../5/08) -m mycontact@mydomain.com

This sets up the drive to be checked with an offline short test every 4 hours and a long test once each Friday at 8. The -I says to ignore attribute 194 and 190. -m is the contact email to send to.

Full Manpage:

http://smartmontools.sourceforge.net/man/smartd.conf.5.html

 

Sorry, just reread your question. smartctl wouldn't receive the flags directly. I believe the smartd service does some post-test parsing of the results using the flags you set in smartd.conf to decide what is/isn't worth notifying the contact about.

 

Yup, that's all the output:

Code:

[root@##### ~]# smartctl -q errorsonly -H -l selftest -l error -d ata /dev/sda
Please note the following marginal Attributes:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
190 Airflow_Temperature_Cel 0x0022   059   044   045    Old_age   Always   In_the_past 41 (0 3 46 33)

The error is not present when I omit the -H flag and now that I'm thoroughly testing smartd, I don't think the -I flag is actually omitting error reporting on the WHEN_FAILED value. According to the smartd manpage, the -H option for smartd reports only when the prefailure attributes are less than or equal to their THRESH values:

So I think using smartd, period, is enough to suppress reports generated by the "WHEN_FAILED" value being In_the_past. I interpret the smartd manpage as saying smartd doesn't report on the WHEN_FAILED value at all, it checks the current Attribute values for failures. The -I 190 -I 194 flags further reduce the frivolous emails by ignoring small changes in ambient temperature which would normally be reported using the -t flag. Reports that would be generated by these values exceeding their THRESH would still be sent since -I does not ignore that attribute for -f failures.