Weird issue with 2 servers

[TSD500]

Ok we have two servers. Now for some reason our DNS clustering has stoped working and they wont evne connect to each other. When I try to add the second server to server 1 I get this message:
There was an error while processing your request: Cpanel::Accounting returned [HTTP/1.0 900 NET OR SSL ERROR ./cgi/trustclustermaster.cgi 5139: open_tcp_connection: failed `server2ip', 2087 (No route to host) ]

when I try to add server 1 to server 2 I get a timeout after a minute of wiating. I can't even get a ping responce from them. This is getting very confusing for me becuase I can't seem to find whats wrong. Thanks for your help!

[chirpy]

Do have have a firewall running on/for either server? If so, make sure that port 2087 is open IN and OUT bound.

[TSD500]

Nope the ports are all open, its not just cpanel, I cant even have the boxes ping each other, from box 1 to box 2 I get "destination host unreachable" and from box 2 to box 1 I get "timed out".

Edit: We have APF on each server but we dont restrict outgoing traffic and the port is in the allow list for incoming ports.

[Keston.e]

I am recieving the same problem.

[chirpy]

Sounds like an IP routing problem at the NOC then, if you cannot even ping each server from the other.

[TSD500]

Heres the traceroute if it helps any:
root@server1 [/etc/apf]# traceroute 66.197.173.141
traceroute to 66.197.173.141 (66.197.173.141), 30 hops max, 38 byte packets
1 66.197.173.197 (66.197.173.197) 2998.551 ms !H 2994.306 ms !H 2999.938 ms
!H

root@server2 [/etc]# traceroute 65.75.137.80
traceroute to 65.75.137.80 (65.75.137.80), 30 hops max, 38 byte packets
1 64.191.111.1 (64.191.111.1) 0.724 ms 0.550 ms 1.440 ms
2 66.197.191.81 (66.197.191.81) 1.431 ms 1.884 ms 0.522 ms
3 66.197.191.46 (66.197.191.46) 5.519 ms 5.554 ms 5.293 ms
4 ge3-0.2.cr01.phl02.pccwbtn.net (63.218.31.5) 6.500 ms 6.172 ms 5.294 ms
5 pos5-0.cr02.ash01.pccwbtn.net (63.216.0.18) 17.514 ms 18.654 ms 18.189 ms
6 206.111.0.25 (206.111.0.25) 14.177 ms 13.788 ms 13.270 ms
7 p5-0-0.RAR1.Washington-DC.us.xo.net (65.106.3.133) 19.854 ms 20.819 ms 52.056 ms
8 p1-0-0.RAR1.SanJose-CA.us.xo.net (65.106.0.38) 97.461 ms 147.702 ms 96.558 ms
9 p0-0-0.MAR1.Fremont-CA.us.xo.net (65.106.5.134) 98.462 ms 96.834 ms 97.023 ms
10 p0-0.CHR1.Fremont-CA.us.xo.net (207.88.80.178) 98.999 ms 98.382 ms 98.734 ms
11 67.104.60.222.ptr.us.xo.net (67.104.60.222) 91.833 ms 93.738 ms 92.455 ms
12 66.79.175.2 (66.79.175.2) 92.111 ms 91.887 ms 92.215 ms
13 * * *
after 13 it keeps on going until it gets to 30 and times out.

[TSD500]

O and its only between servers that ping fails, I can ping both fine from everywhere else.

[chirpy]

You mentioned that you do use APF - is it possible that one or the other has blacklisted the other. You can check on both with:

iptables -L -n | grep other-server-ip-address