I have tried to view my /var/log/secure file and discovered that someone was trying my username and password repeatedly. The atttempt was from one single IP and it tried my sshd on many port number.
What can I do to prevent this kind of attempt?
Depending on your OS, you could look at installing APF and BFD on your server, BFD is the tool for detecting such attacks. You can find them at http://www.rfxnetworks.com and there are tutorials around on how best to configure them on the forums and on web sites.