verdon

Well-Known Member
Nov 1, 2003
919
12
168
Northern Ontario, Canada
cPanel Access Level
Root Administrator
Hi

For the last 3 days, I've been getting slocate errors from cron.daily. The exact error is

/etc/cron.daily/slocate.cron: line 3: 17746 Segmentation fault /usr/bin/updatedb -f "nfs,smbfs,ncpfs,proc,devpts" -e "/tmp,/var/tmp,/usr/tmp,/afs,/net"

I tried renaming my /var/lib/slocate dir and creating a new one, then running /etc/cron.daily/slocate.cron manually, to see what would happen. I still get the error however.

When I check out the file var/lib/slocate/slocate.db.tmp that is created during this process, it has 534 lines and the last one is ...

... hmmm the forum here won't accept the example I am trying to include, even wrapped in code tags

Any ideas how I might proceed from here?

Thanks,
 
Last edited:

chirpy

Well-Known Member
Verifed Vendor
Jun 15, 2002
13,437
31
473
Go on, have a guess
A few ideas:

1. Make sure that /var isn't out of space ;)

2. Empty out /var/lib/slocate/* again and then check the perms/owner of /var/lig/slocate it should be:

drwxr-x--- 2 root slocate 4096 Oct 8 04:32 /var/lib/slocate/

3. Run updatedb with the -v qualifier so that you can see what files are being processed and see if it keels over at a particular point:

/usr/bin/updatedb -fv "nfs,smbfs,ncpfs,proc,devpts" -e "/tmp,/var/tmp,/usr/tmp,/afs,/net"

4. If it doesn't get as far as listing files before the seg fault from the verbose tag, try running it through strace, though you're likely to get a lot of output:

strace /usr/bin/updatedb -f "nfs,smbfs,ncpfs,proc,devpts" -e "/tmp,/var/tmp,/usr/tmp,/afs,/net"

5. Last try, remove the slocate rpm (package, depends what OS you're using) and try reinstalling it.
 

verdon

Well-Known Member
Nov 1, 2003
919
12
168
Northern Ontario, Canada
cPanel Access Level
Root Administrator
chirpy said:
A few ideas:
1. Make sure that /var isn't out of space ;)
Yup, that's fine

2. Empty out /var/lib/slocate/* again and then check the perms/owner of /var/lig/slocate it should be:

drwxr-x--- 2 root slocate 4096 Oct 8 04:32 /var/lib/slocate/
That's OK too

3. Run updatedb with the -v qualifier so that you can see what files are being processed and see if it keels over at a particular point:

/usr/bin/updatedb -fv "nfs,smbfs,ncpfs,proc,devpts" -e "/tmp,/var/tmp,/usr/tmp,/afs,/net"
No output

4. If it doesn't get as far as listing files before the seg fault from the verbose tag, try running it through strace, though you're likely to get a lot of output:

strace /usr/bin/updatedb -f "nfs,smbfs,ncpfs,proc,devpts" -e "/tmp,/var/tmp,/usr/tmp,/afs,/net"
Tons of output here. The final few lines before it dies are...
Code:
fchdir(5)                               = 0
getdents64(5, /* 24 entries */, 4096)   = 672
lstat64(".registry", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64(".lock", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
lstat64(".filemap", {st_mode=S_IFREG|0644, st_size=2312, ...}) = 0
lstat64("Archive", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("doc", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("Console", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("data", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("pearcmd.php",  <unfinished ...>
+++ killed by SIGSEGV +++
5. Last try, remove the slocate rpm (package, depends what OS you're using) and try reinstalling it.
I'll leave this as a last ditch effort. I am using CentOS 3.5. When I use WHM -> Install a RPM to have a look, I don't see anything for slocate though?

Finally, one other thing I've noticed in the last day or 2 is that when running top, I can see a zombie process that won't clear and I don't seem to be able to kill.
5 root 9 0 0 0 0 Z 0.0 0.0 195:06 0 kswapd <defunct>
I don't know if this is related or not, but it's something I've not seen before in the 294 days of uptime that I've had this server.

Chirpy, thanks for your advice on this issue :)
 

chirpy

Well-Known Member
Verifed Vendor
Jun 15, 2002
13,437
31
473
Go on, have a guess
kswapd is a system process and shouldn't be killed. If the server has been up that long you should probably try a report first as you could be suffering from memory fragmentation. I would also assume that if it's been up that long you're due a kernel upgrade which might be an opportune time to do that if you're rebooting.

Lastly, you could try and find out where that pearcmd.php is on disk and then look within the same directory and see if there's anything that looks like file system corruption. Either way, it would certainly be work forcing an FSCK of your disks when you reboot to check for any corruption just incase the file system is on it's way out.
 

gemininetcom

Active Member
Nov 29, 2003
36
0
156
Last night received same problem with run-parts /etc/cron.daily and segmentation fault:

*** start error log

/etc/cron.daily/slocate.cron:

/etc/cron.daily/slocate.cron: line 3: 31164 Segmentation fault /usr/bin/updatedb -f "nfs,smbfs,ncpfs,proc,devpts" -e "/tmp,/var/tmp,/usr/tmp,/afs,/net"

*** end error log

Ran you suggestions 1 up/included 4 and last lines result of (4)
strace /usr/bin/updatedb -f "nfs,smbfs,ncpfs,proc,devpts" -e "/tmp,/var/tmp,/usr/tmp,/afs,/net"
is below


*** start of strace

...
....
fchdir(5) = 0
close(5) = 0
open("browse", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 5
fstat64(5, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl64(5, F_SETFD, FD_CLOEXEC) = 0
fstat64(5, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fchdir(5) = 0
getdents64(5, /* 13 entries */, 4096) = 448
lstat64("column_footers.inc", {st_mode=S_IFREG|0644, st_size=645, ...}) = 0
lstat64("actions.inc", {st_mode=S_IFREG|0644, st_size=3273, ...}) = 0
lstat64("column_headers.inc", {st_mode=S_IFREG|0644, st_size=2075, ...}) = 0
lstat64("contactrow.inc", {st_mode=S_IFREG|0644, st_size=2689, ...}) = 0
lstat64("footer.inc", {st_mode=S_IFREG|0644, st_size=274, ...}) = 0
lstat64("header.inc", {st_mode=S_IFREG|0644, st_size=65, ...}) = 0
lstat64("javascript.inc", {st_mode=S_IFREG|0644, st_size=2504, ...}) = 0
lstat64("search.inc", {st_mode=S_IFREG|0644, st_size=4576, ...}) = 0
lstat64("select.inc", {st_mode=S_IFREG|0644, st_size=659, ...}) = 0
lstat64("footerAlpha.inc", {st_mode=S_IFREG|0644, st_size=796, ...}) = 0
lstat64("search_criteria.inc", {st_mode=S_IFREG|0644, st_size=325, ...}) = 0
getdents64(5, /* 0 entries */, 4096) = 0
close(5 <unfinished ...>
+++ killed by SIGSEGV +++
[email protected] [~]#

*** end of strace

Seems that strace at last "close 5(" got kicked out.

System changes of last 7 days:
- had last Sunday (16th) a full memory replacement
- on Monday 17th no problem: upcp and cpbackup OK and no errors
- on Tuesday 18th suddenly no cpbackup performed -- upcp was OK
- today Wednesday 19th upcp and cpbackup OK but segmentation problem.

Any ideas Jonathan ?
 
Last edited:

chirpy

Well-Known Member
Verifed Vendor
Jun 15, 2002
13,437
31
473
Go on, have a guess
Have you tried looking for the previous file search_criteria.inc and see if anything in that directory looks troublesome?

My only other idea apart from those mentioned would be to reboot and run and FSCK to check for/correct and file system corruption.

I have seen segfaults with slocate before and tracked it down to a corrupt file on the file system and simply put in an exclusion into the /etc/cron.daily/slocate.cron file.
 

gemininetcom

Active Member
Nov 29, 2003
36
0
156
Located "search_criteria.inc" in [/usr/local/cpanel/base/horde/turba/templates/browse] and tried a list, but system came back immediately with "Segmentation fault". So can't even see what's in that directory.

Tonight a reboot with FSCK is scheduled. My NOC suggested a restore, but find this the extreme :D
 
Last edited:

gemininetcom

Active Member
Nov 29, 2003
36
0
156
You wouldn't believe it, but last Sunday the Segmentation fault mysteriously dispappeared :eek:

Still using 10.8.1-RELEASE_4 both on last Saturday and Sunday according to upcp.

Tested
(1)
/usr/bin/updatedb -fv "nfs,smbfs,ncpfs,proc,devpts" -e "/tmp,/var/tmp,/usr/tmp,/afs,/net" without errors
(2)
Directory /usr/local/cpanel/base/horde/turba/templates/browse with search_criteria.inc is accessable and contents can be listed

... thus my problem solved but it beats me what did the trick :D