How do I check if my VPS has been hacked? [duplicate]

My service provider for my VPS has informed me that I have high traffic on my server on port 25.

How can I make sure my.server has not been hacked.

I have fail2ban installed, but I have clearly not configured it correctly.

Your question is too broad to really answer in our Q&A format.

Note your VPS being abused is not necessarily the same as the being the victim of a hack.

One way that servers can be abused is due to an insecure configuration created by the server admin. Although the result seems the same that is usually somehwat different from a server that was sucessfully "hacked". The subtle difference with such abuse is that the (private) data stored on your server isn't exposed and compromised and the attacker didn't modify the server/configuration nor did they install their own (malicious) software on the server. The only thing being abused is your bandwidth.

For mail such an insecure configuration is called an "open relay" and for web servers the result is typically an "open proxy".