Yes i am able to 95% of the times except the other 5% i get the same error as wordpress. The same happends to wordpress. Also this happens both for smtp and pop3.I would expect either the server IP or localhost to work well for that connection type.
As a test, are you able to access email through a client using the authentication details configured in the WordPress plugin?