Thanks – I can see the Limit Login Attempts module is working well and the login-failure notices are interesting, but I don’t know if it’s you I must thank for testing it out, or miniOrange for creating it so I can block the perpetrators who are.

Take Note: I’m logging all attempts, and you have been warned.

Brute Force Attack – The Defence

The attack on my server got quite intense while I was testing alternate defenses and the controller(s) were obviously aware someone was reacting. The perpetrators were using hundreds of connections from almost every country in the world including but not limited to (CN, ID, QA, MY, PH, ES, NL, HK, RU, IT, CO, GB, BG, PL, FR, DE, and TW) focused on a block of 16 IP addresses where each connection would attempt to log in through ssh under root (mostly), or a random name, using random passwords (well, not random but from a Most Common Passwords list, then random).

I was immediately notified when the attack began, and after a brief evaluation blocked all countries except Canada and all IPs except one. This brought the attack down to a reasonable level where I could experiment with various alternatives to come up with the best defense not only to stop the current attack but to prevent it from happening again. If you are a cPanel System Administrator you might find it interesting what I tried, my results, and my final solution.

Your Password is the Weak Link

If you don’t have a strong password, go no further until you fix it or you could be broken into while trying to prevent a break-in. Never use a common name, familiar term, or even an English word or common phrase for your password. After going through the list of common passwords, perpetrators begin a dictionary attack – so if your password is not (a common password, in the dictionary, or a common phrase), there’s very little chance they will guess it.

  • Use a random combination of:
    • upper and lower case A-Z
    • digits 0-9
    • special chars !@#$%^&*()_+-={}[]:";',?/.
  • Some experts suggest at least 8 characters but 9 is almost 100x the protection so I try to use at least 9 so it takes up to 572,994,802,228,616,704 attempts to break. If they could try 1000/sec it would take 6631884285 years. No matter how much computing power the hackers have, the attack is still limited by server bandwidth so there’s no way, it’s futile.

I can’t stress enough, the importance of a strong password. Nothing gives you a bad day more than a server break-in with information theft, destruction, or both. That’s one mistake you don’t want to learn the hard way.

If you don’t have strong passwords, read no further; CHANGE YOUR PASSWORDS NOW

Preventing Lock-Out

Since we are going to be messing with firewalls automatically blocking IPs here, you’ll want to take every precaution that your IP doesn’t get accidentally blocked or your day could get even worse. Once we are sure that can’t happen, we can go ahead with further experimentation.

We’ll put up three safeguards as we work:

  1. Login to the server via SSH as either the root, or a user that has su privileges to become root and leave this connection open. To test a login, use a separate instance of the SSH Client, do not log out of this connection or you may not be able to log back in
  2. Login to WHM through a web browser so we can control (as much as possible) the settings and operations from WHM before having to resort to the command line.
  3. Set up Public Key Access so no matter what walls we put up; we can always connect to the server.

1) Connection to Server via SSH

Using your favorite SSH Client, connect to the server as root or another user that can su root so that we can edit the necessary settings, and restart services as we make changes. You can be sure that you are the root user by looking at the prompt (~ #) or with the whoami command.

# whoami
root

2) Login to WHM

Leave the SSH terminal open and with your web browser, log into WHM. We’ll need both WHM and the command line to make the necessary changes.

3) Setup Public Key Access

This gets a little complex, but its a necessary step and I know you’ll agree by the end of this document, so follow along.

Public Key Authentication works like https used for Online banking, shopping, and wherever secure communications is used. It uses two keys (sequences of binary numbers) for the purpose of uniquely identifying one person or entity (Private Key), with one or more remote persons or services (Public Key).

How Public Key Authentication Works

A single communications exchange (session) begins with a handshake as follows:

  • First, a Key Pair is generated, one Private (for the unique person to be identified), and one Public (shared with everyone for securely communicating with the unique identity). Only the Private key can decrypt messages encrypted with the Public Key and only the Public keys can decrypt messages encoded with the Private key. So, everyone with the public key can decrypt any messages sent with the Private key and only the single Private key can decrypt messages sent from any of the Public Keys.
  • Sessions start with a handshake to verify the authenticity of the unique identity (the workstation), to a remote service (the server) that goes something like this:
    • The workstation sends a request to the server encoded using the private key
    • The server uses the public key to decrypt the request, and sends back a unique identifier encrypted using the public key
    • The workstation decodes the response from the server, re-encodes it using the private key, and sends it back to the server
    • The server decodes the response, verifies it to be what was sent, and thereby knows that this can only be the unique identity with the private key, and so a secure session begins

Setting Up Public Key Authentication

There are many ways to generate key pairs with most SSH client programs having their own utility and most operating systems (including Windows, OSX, and Linux) having one as well. Rather than discussing all possibilities, problems, and workarounds for each situation, I’ll discuss what we’re doing here (integrating public-key authentication on cPanel). I’ll also assume that you currently have an open SSH connection as root with the server, and WHM open in a tab on your browser.

Generating the Key Pair

For cPanel admins, the easiest and most reliable way to create and manage keys is using WHM’s Key Generator which can be found at WHM > Security Center > Manage Root’s SSH Keys.

  • Click the Manage Root’s SSH Keys menu item to start the service
  • Chose a Key Name representing what the key is for (ie ServernameRoot)
  • Add a passphrase (Password) of reasonable complexity (>65) and confirm it on the second line
  • Set Key Type to RSA, Key Size to 2048, and click [Generate Key] to generate the key
  • After the key is created, click the link to Return to SSH Manager where the keys you just created should be shown in the tables at the top
  • You can authorize the key through the Manage Authorization link in the Public Key table. Authorize the key and the server is ready to go, all we must do is let your SSH client know what the private key is

The Private Key must be downloaded to your computer for your SSH Client to be able to use it. A good place to save it is in your {your_user}/.ssh folder, create the folder if it doesn’t already exist. You should download both the passcode protected Private Key, and the passcode protected PuTTY Key, you’ll need one or the other. You can erase them off the server after saving them on your local computer, they are of no use to the server.

I’m trying to keep this document to a reasonable length, so I won’t be discussing how to add a private key to each SSH client. If you aren’t already familiar with it, a quick web search will give you the answers you need for your SSH client.

With the private key loaded into the SSH client, open a new SSH client window and try a connection. If it works (Hurray!), you are safe. If it doesn’t work its probably how you configured your SSH client, consult the software docs for help. If you are asked for a password its the keyphrase you used when generating the key pair, not a user password on the server.

(DON’T GO ANY FURTHER UNTIL YOU CAN SUCESSFULLY CONNECT TO THE SERVER USING THE KEYS).

cPHulk

cPHulk is a utility within the WHM control panel (WHM > Security Center > cPHulk Brute Force Protection) providing protection against Brute Force attacks on the (cPanel, WHM, Mail, FTP, and SSH) logins. It blocks connections with the only indication to the perpetrator being a message saying, “The login is invalid“.

Note: cPHulk does not affect public key authentication. If you generated and tested a public key as described earlier, you should be immune to failure as even ‘Locked’ accounts can always authenticate using public keys.

Log in to WHM, and visit cPHulk to verify it is enabled and the green light is on, then follow along while we go through the settings.

Configure Settings

  • Username-based Protection [ON]
    • Period: 20 – Number of minutes over which cPHulk keeps track of login attempts for the same user (1-1440)
    • Max Failures: 5 – Will lockout user for same number of minutes as attempts (ie 5min after 5 tries)
    • Select – Apply protection to local addresses only
    • [CHECK] – Allow username protection to lock the “root (we have an open SSH session and the public key as backup)

    No notifications are sent for these actions.

  • IP Address-based Protection
    • Period: 15 min – Number of minutes over which cPHulk keeps track of login attempts from same IP (1-1440)
    • Max Failures: 9
    • The Command to Run is mainly for logging, you can leave it empty
    • [CHECK] – Block IP addresses at the firewall level if they trigger brute force protection
  • One-day Blocks
    • Max Failures: 8 – Max attempts before cPHulk blocks IP for one-day
    • The Command to Run is mainly for logging or notification, you can leave it empty
    • [CHECK] – Block IP addresses at the firewall level if they trigger this
  • Login History

    360 (minutes)

  • Notifications
    • [CHECK] – Send a notification upon successful root login when the IP address is not on the whitelist
    • [UNCHECK] – Send a notification upon successful root login when the IP address is not on the whitelist, but from a known netblock
    • [CHECK] – Send a notification when the system detects a brute force user

Click [Save] to save the settings.

Whitelist Management

Always check your Internet IP while sitting at the computer you wish to connect to the server from. Using a web service such as whatsmyip.org will show you your external web IP which is probably different than your computer’s IP (it does NOT begin with 192.168..).

Enter that IP in the New Whitelist Records box with a possible comment indicating where the IP is and click the [Add] button. You should see your IP listed in the right-hand column. This is added insurance against accidental blocking but limited in lifespan since your IP is probably dynamic and changes occasionally.

Blacklist Management

This is where you can manually add IPs to the blacklist if you know an IP you don’t want to connect. For now, you can leave this tab alone.

Countries Management

This is how I disabled all foreign countries at the start. You should disable all countries except your own unless you are hosting websites for foreigners who need login access from another country. All countries who need access must have ‘ Not Specified’ selected, all other countries should have ‘ Blacklisted’ selected. Don’t select ‘ Whitelisted’ or no other checks will be done on that entire country.

History Reports

This provides a list of all timed lockouts currently pending. It does not show permanent bans which have been passed to the firewall.

You can return to the first tab and click [Save] again just to be sure all is saved.

Restart cpHulk

The changes should have taken effect immediately but since we’re currently logged in to the server lets restart csf from the command line so we can check the status.

# /usr/local/cpanel/scripts/restartsrv_cphulkd
Waiting for “cphulkd” to restart gracefully ……waiting for “cphulkd” to initialize ………finished.

Service Status
           cphulkd (cPhulkd - processor
                           ) is running as root with PID 22475 (systemd+/proc check method).

Startup Log Aug 25 17:48:35 svrname systemd[1]: Starting cPanel brute force detector services... Aug 25 17:48:35 svrname systemd[1]: Started cPanel brute force detector services. cphulkd restarted successfully.

cPHulk can also be controlled at the command line cPanel – cPHulk Management on the Command Line.

Full cPHulk docs available at cPHulk Brute Force Protection.

At this point, any attack against your server should have been greatly reduced but we’re still not done, we still have the silver bullet.

Editing sshd_config

The file at /etc/ssh/sshd_config is the master SSH configuration file. Other than cPHulk settings (above), this is where you control the various options for SSH security.
I’ll be discussing these options here and why most of them won’t work (at least for long).
There is one silver bullet among them, and I’ll describe why that one works, and why it leaves no chance for a Brute Force Attack to bother you again.

You may want to open the file at /etc/ssh/sshd_config for editing in your favorite editor as this is the last stop on our road to success.

Change SSH Port

The most common response to a Brute Force attack is to change the SSH port number. This is done using the line containing #Port 22, erasing the # at the start and 22 to another port number not already in use.

This is only a temporary fix as perpetrators can easily do a port scan and continue the attack. They may just stop trying and go away, but don’t count on it, its not fail-safe.

If you do decide to try it out, the hard part is choosing an alternative port number. The netstat command can show you all the ports already in use on your server.

netstat -an | grep 'LISTEN'

But there are other ports you shouldn’t use either. A complete list of them is available at MIT – Red Hat Enterprise Linux 4: Security Guide – Common Ports.

So, choosing an alternate port is both difficult, and futile.. don’t count on it.

Allow Login from Only a Single IP Address

This is a very good precaution for any server handling multiple IP addresses. Its not necessary for every website on the server with a static IP granted SSH access to log in to their account using that same static IP, any IP address attached to the server can be used with equal effectiveness. The account name they log in under determines the login account, not the IP.

If only one IP address is used for all SSH traffic into the server, all accounts except that one will be impenetrable to brute force attacks and won’t even respond to SSH connection requests. Odds of being hit are reduced in direct proportion to the number of IPs locked out (50% for 2 IPs, 25% for 4 IPs, 10% for 10 IPs).

To do this, change the #ListenAddress 0.0.0.0 line by eliminating the # at the start and replacing 0.0.0.0 with the only IP address you wish to allow connections on. This alone will not block all Brute Force attacks, but it should greatly reduce them.

Create a wheel user and Disable Root Login

A wheel user is a user given the privilege of becoming root by using the ‘su root‘ command. After creating a wheel user, you can disable the root user from being able to login and always have root capabilities by logging in as the wheel user and ‘su root‘ing to gain root control.

The problem is that it doesn’t stop the attacks, hackers see they can’t log in under root and so know you are using a wheel user, and continue using random names or by selecting usernames listed in your blog as authors.. and the attack continues.

To try it, from the command line you would: create a new user, give them a password, and assign them to the wheel group.

# adduser {wheel_user_name}
# passwd {wheel_user_name}
# usermod -aG wheel {wheel_user_name}

You can check it out (without disabling root login) and see it works but again ‘it won’t stop the attacks’.

Disable Password Authentication

This is the silver bullet, and would it surprise you to know you have already set yourself up for it?
You did it way back at the start when you setup Private Key Authentication.
I could have told you back then to come straight here but you would have always wondered “what about the other options?”.

If you disable password authentication a Brute Force attack is useless, they can’t log in with ANY username or password even if it was correct.
It’s the absolute best solution and if you have been following along you have already tried it and know it works.

Change the line in sshd_config containing #PasswordAuthentication yes by eliminating the # character at the start and changing the value from yes to no

That’s it, you are done, your server can no longer be a victim of Brute Force and you learned a lot about security in the process.

Save the changes to the sshd_config file and restart the SSH daemon. You could do it from WHM WHM > SSH Server (OpenSSH), or do it from the command line

# system restart ssh

You will be able to login as you’ve done previously using the key, only from the IP address you set, and not be able to get in using any usernames or passwords.


That’s it, we’re done, hope you learned something, and drop by again.

– Kent

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *