Hardening Linux Server Setup

01 Feb 2020 in IT on Linux, Hardening 2021-04-20

The system administrator is responsible for security of the Linux servers. In this post, I will provide Linux server hardening tips for default installation of Linux system.
Hacker Noon^©

For more information : See the document, written by ANSSI, the French National Information Security Agency, presents the “Configuration recommendations of a GNU/LINUX system”

Disk Partitions
Disable IPv6 Networking - Turn Off IPv6
Use Secure Shell (SSH)
Avoid Using FTP, Telnet, and Rlogin / Rsh services on Linux
Keep system updated
1. Applying Debian security updates only
2. Updating all packages (OS and other installed packages)
No accounts with empty Passwords
Make sure no non-root accounts have UID set to 0
Disable root Login
IPtable, fail2ban, IPset and GeoIP filtered access

Disk Partitions

It’s important to have different partitions to obtain higher data security in case if any disaster happens. By creating different partitions, data can be separated and grouped. When an unexpected accident occurs, only data of that partition will be damaged, while the data on other partitions survived. Make sure you must have following separate partitions and sure that third party applications should be installed on separate file systems under /opt.

    /
    /boot
    /usr
    /var
    /home
    /tmp
    /opt

Disable IPv6 Networking - Turn Off IPv6

echo 'net.ipv6.conf.all.disable_ipv6 = 1' > /etc/sysctl.d/90-disable-ipv6.conf
sysctl -p -f /etc/sysctl.d/90-disable-ipv6.conf

Note: The first one created a file /etc/sysctl.d/90-disable-ipv6.conf which contains the configuration setting to disable IPv6 and the second one applies the change.

Use Secure Shell (SSH)

Telnet and rlogin protocols uses plain text, not encrypted format which is the security breaches. SSH is a secure protocol that use encryption technology during communication with server. Never login directly as root unless necessary. Use sudo to execute commands. sudo are specified in /etc/sudoers file.

See Hardening SSH Server Setup post for more information.

Avoid Using `FTP`, `Telnet`, and `Rlogin` / `Rsh` services on Linux

Under most network configurations, user names, passwords, FTP / telnet / rsh commands and transferred files can be captured by anyone on the same network using a packet sniffer.

sudo apt-get --purge remove xinetd nis yp-tools tftpd atftpd tftpd-hpa telnetd rsh-server rsh-redone-server

Keep system updated

Applying Debian security updates only

sudo grep security /etc/apt/sources.list | tee /etc/apt/security.sources.list
sudo apt update
sudo apt upgrade -o Dir::Etc::SourceList=/etc/apt/security.sources.list

Press “Y” and “Enter”. Security updates will be installed and your system will be up to date.

To check and apply updates at a later stage do this:

sudo apt update
sudo apt upgrade -o Dir::Etc::SourceList=/etc/apt/security.sources.list

Updating all packages (OS and other installed packages)

sudo apt update && sudo apt upgrade

No accounts with empty Passwords

Type the following command to identify account whith empty Passwords

sudo awk -F: '($2 == "") {print}' /etc/shadow

and lock all empty password accounts

sudo passwd -l <accountName>

Make sure no non-root accounts have UID set to 0

Only root account have UID 0 with full permissions to access the system. Type the following command to display all accounts with UID set to 0:

awk -F: '($3 == "0") {print}' /etc/passwd

You should only see one line as follows:

root:x:0:0:root:/root:/bin/bash

If you see other lines, delete them or make sure other accounts are authorized by you to use UID 0.

Never ever login as root user. You should use sudo to execute root level commands as and when required. sudo does greatly enhances the security of the system without sharing root password with other users and admins. sudo provides simple auditing and tracking features too.

To disable, you can remove the password of the account or lock it down, or even do both of them:

Remove the root password:
```
sudo passwd -d root
```
Lock the account:
```
sudo passwd -l root
```
Change root User’s Shell

sudo vi /etc/passwd

Change the line:

root:x:0:0:root:/root:/bin/bash

root:x:0:0:root:/root:/sbin/nologin

From now on, when root user logs in, he/she will get the message This account is currently not available. This is the default message, but, you can change it and set a custom message in the the file /etc/nologin.txt. This method is only effective with programs that require a shell for user login, otherwise, sudo, ftp and email clients can access the root account.

IPtable, fail2ban, IPset and GeoIP filtered access

IPtable

iptables is an utility program that allows a system administrator to configure the tables provided by the Linux kernel firewall (implemented as different Netfilter modules) and the chains and rules it stores. Different kernel modules and programs are currently used for different protocols; iptables applies to IPv4, ip6tables to IPv6, arptables to ARP, and ebtables to Ethernet frames.

iptables requires elevated privileges to operate and must be executed by user root, otherwise it fails to function. On most Linux systems, iptables is installed as /usr/sbin/iptables.

iptables let’s you configure default policies for chains in the filter table, where INPUT, FORWARD and OUTPUT, are the main ones (or at least the most used). Users can even define new chains.

These aforementioned chains, are better explained in this graph that comes from Wikipedia.

SSH, CryptCheck, screenshot Packet flow in Netfilter and General Networking

Viewing current configuration

sudo iptables -L

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

You may prefer to use iptables -L -vn to get more information, and to see ports as numbers instead of its names.

Delete rules

Delete all rules

sudo iptables -F

Delete spécific rules

sudo iptables -L --line-numbers

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    ACCEPT     all  --  192.168.0.4          anywhere
2    ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
3    ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
4    ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh

Notes: For deleted the record number 3 type the command below

sudo iptables -D INPUT 3

Save after reboot

sudo /sbin/iptables-save

Or if you want deleted all rule and save

sudo iptables -F
sudo /sbin/iptables-save

Basic commands

Block from an IP

sudo iptables -A INPUT -s 11.22.33.44 -j DROP

If you want to block only on an specific NIC

sudo iptables -A INPUT -s 11.22.33.44 -i eth0 -j DROP

Or an specific port

sudo iptables -A INPUT -s 11.22.33.44 -p tcp -dport 22 -j DROP

Using a Network and not only one IP

sudo iptables -A INPUT -s 11.22.33.0/24 -j DROP

Block traffic from a specific MAC address Suppose you want to bloc traffic some a MAC address instead of an IP address. This is handy if a DHCP server is changing the IP of the maching you want to protect from.
```
sudo iptables -A INPUT -m mac --mac-source 00:11:2f:8f:f8:f8 -j DROP
```
Block incoming traffic to a port Suppose we need to block port 21 for incoming traffic
```
sudo iptables -A INPUT -p tcp --destination-port 21 -j DROP
```
Block outgoing traffic to a port If you want to forbid outgoing traffic to port 25, this is useful, in the case you are running a Linux firewall for your office, and you want to stop virus from sending emails.
```
sudo iptables -A FORWARD -p tcp --dport 25 -j DROP
```

Persist at reboot

In /etc/network/interfaces add below face eth0 inet dhcp this lines :

pre-up iptables-restore < /etc/iptables.up.rules

So at eatch boot le system load the file /etc/iptables.up.rules

IPset

IP sets are a framework inside the Linux kernel, which can be administered by the ipset utility. Depending on the type, an IP set may store IP addresses, networks, (TCP/UDP) port numbers, MAC addresses, interface names or combinations of them in a way, which ensures lightning speed when matching an entry against a set. If you want to :

store multiple IP addresses or port numbers and match against the collection by iptables at one swoop;
dynamically update iptables rules against IP addresses or ports without performance penalty;
express complex IP address and ports based rulesets with one single iptables rule and benefit from the speed of IP sets

then ipset may be the proper tool for you.

IPset install

sudo apt update
sudo apt install ipset

Listing / adding / removing IP addresses from our blacklist

To list current IP addresses in our list, do:

sudo ipset list

To add an IP address, simply do:

sudo ipset add blacklist <IP-ADDRESS_OR_CIDR-IF-NETHASH>

To remove an IP address, do:

sudo ipset del blacklist <IP-ADDRESS_OR_CIDR-IF-NETHASH>

To remove ALL IP addresses in a list, use the flush command

sudo ipset flush blacklist

Querying to see if IP address is in list

You can use the test command to check if an IP address is in a specific list:

sudo ipset test blacklist <IP-ADDRESS>

Deleting a list You might want to delete a list. To do so you can use the ‘destroy’ command:

sudo ipset destroy blacklist

Persist at reboot

In /etc/network/interfaces add below face eth0 inet dhcp this lines :

pre-up ipset restore -! < /etc/ipset.up.rules

So at eatch boot le system load the file /etc/ipset.up.rules

GeoIP

Install and configure GeoIP

Create an free account on GeoIP maxmind to access at the GeoIP database
Install geoipupdate_4.0.6-2_arm64.deb debian package
```
sudo apt install geoipupdate
```
Note: The last version of geoipupdate doesn’t exist in debian buster version, but the bullseye version of geoipupdate package is compatible to buster. Get the geoipupdate package in bullseye debian version here for your CPU arch.
```
cd /tmp/
wget http://ftp.ch.debian.org/debian/pool/contrib/g/geoipupdate/geoipupdate_4.0.6-2_arm64.deb
sudo apt install /tmp/geoipupdate_4.0.6-2_arm64.deb
```
Create a licence key on https://www.maxmind.com/en/accounts/<user_id>/license-key

Create a config file GeoIP.conf in /etc/

sudo touch /etc/GeoIP.conf

And add this line :

 # file: "GeoIP.conf"
 # GeoIP.conf file for `geoipupdate` program, for versions >= 3.1.1.
 # Used to update GeoIP databases from https://www.maxmind.com.
 # For more information about this config file, visit the docs at
 # https://dev.maxmind.com/geoip/geoipupdate/.

 # "AccountID" is from your MaxMind account.
 AccountID <user_id>

 # "LicenseKey" is from your MaxMind account
 LicenseKey <LicenseKey>

 # "EditionIDs" is from your MaxMind account.
 EditionIDs GeoLite2-ASN GeoLite2-City GeoLite2-Country

Automatic Updates for GeoIP2 MaxMind provides the GeoIP Update program, which performs automatic updates for both GeoIP2 and GeoIP Legacy binary databases. Please follow the instructions below.

Step 1 – Install GeoIP Update The latest release may be downloaded from GitHub Releases. See here for installation instructions.
Note: If you are using an older version of GeoIP Update, you may need to upgrade to GeoIP Update 4.x or later version. The 4.x and later versions meet our requirement for using TLS 1.2 or greater for all requests to our servers to keep your data secure.

sudo apt update
sudo apt install geoipupdate

Step2 – Run GeoIP

sudo geoipupdate -v

Using config file /etc/GeoIP.conf
Using database directory /var/lib/GeoIP
Acquired lock file lock (/var/lib/GeoIP/.geoipupdate.lock)
Performing get filename request to https://updates.maxmind.com/app/update_getfilename?product_id=GeoLite2-ASN
Calculated MD5 sum for /var/lib/GeoIP/GeoLite2-ASN.mmdb: 8b19690ec6c8091c9f7ae0bc3b739aa1
Performing update request to https://updates.maxmind.com/geoip/databases/GeoLite2-ASN/update?db_md5=8b19690ec6c8091c9f7ae0bc3b739aa1
No new updates available for GeoLite2-ASN
Performing get filename request to https://updates.maxmind.com/app/update_getfilename?product_id=GeoLite2-City
Calculated MD5 sum for /var/lib/GeoIP/GeoLite2-City.mmdb: 4a355b3f54f07257e37e96ac1d51daf8
Performing update request to https://updates.maxmind.com/geoip/databases/GeoLite2-City/update?db_md5=4a355b3f54f07257e37e96ac1d51daf8
No new updates available for GeoLite2-City
Performing get filename request to https://updates.maxmind.com/app/update_getfilename?product_id=GeoLite2-Country
Calculated MD5 sum for /var/lib/GeoIP/GeoLite2-Country.mmdb: 8d03dd66154470d3e1318f4a5dc74694
Performing update request to https://updates.maxmind.com/geoip/databases/GeoLite2-Country/update?db_md5=8d03dd66154470d3e1318f4a5dc74694
No new updates available for GeoLite2-Country

ls -la /var/lib/GeoIP

total 70896
drwxr-xr-x  2 root root     4096 Jan 29 06:48 .
drwxr-xr-x 38 root root     4096 Jan 31 18:32 ..
-rw-------  1 root root        0 Jan 14 05:44 .geoipupdate.lock
-rw-r--r--  1 root root  6570183 Jan 29 06:47 GeoLite2-ASN.mmdb
-rw-r--r--  1 root root 61914533 Jan 29 06:48 GeoLite2-City.mmdb
-rw-r--r--  1 root root  4099378 Jan 29 06:48 GeoLite2-Country.mmdb

Step 3 – Run GeoIP Update Run geoipupdate. To fully automate this process on Linux or Unix, use a crontab file like:

###############
##  GeoIP    ##
###############
0 0 * * 7 		sudo -u root /usr/bin/geoipupdate

Notes: This cron expression would run every week at 00:00 on Sunday.

Direct download GeoIP `CVS` format database

You can direct download GeoIP CVS format database whith command line below :

cd /tmp
curl https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-Country-CSV&license_key=<license_key>&suffix=zip
curl https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-ASN-CSV&license_key=<license_key>&suffix=zip
curl https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City-CSV&license_key=<license_key>&suffix=zip

Note: <LicenseKey> is your maxmind.com licence key created in bullet 3.

Secured acces now with `iptable`, `ipset` and `GeoIP`

Now we know iptable. And ipset & GeoIP are installed. So we can secure outcoming and incoming network traffic from our server.

iptables to allow traffic from private network and one country only

create a hashmap with ipset to allow private IP to access your host
create a hashmap with ipset to allow only french IP to access your host
loading this rules in iptable
And persist rules for each system reboot
Add a cron entry for the regular update of the hash table

The scripts below (os_ipset_update.sh ) do it automatically

#file: "os_ipset_update.sh"
#!/bin/bash
#
# Update the ipset that iptables references for allowing based on country.
# Takes 2 parameters: ipset name (no spaces), country name e.g. 'Australia', 'france' ...
#
# iptables should have an existing '--match-set' rule e.g
#  $ iptables -I INPUT -p tcp --dport 22 -m set --match-set australia4 src -j ACCEPT
#  $ ip6tables -I INPUT -p tcp --dport 22 -m set --match-set australia6 src -j ACCEPT

CHANGELOG="/var/log/ipset_changes.log"
IPSET="/usr/sbin/ipset"
GEOIP_URL="https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-Country-CSV&license_key=<LicenseKey>&suffix=zip"

IPSET_UP_RULES="/etc/ipset.up.rules"
IPTABLES_UP_RULES="/etc/iptables.up.rules"

if [ $# -ne 2 ]; then
    echo "Usage $0 <ipset name> <country>"
    exit
fi

SETNAME=$1
COUNTRY=$2
TMPDIR=/tmp/$$_tmp
GEOIP_FILE=${TMPDIR}/$$_geoip.zip
CIDR_FILE=${TMPDIR}/$$_${SETNAME}_CIDR.zone
mkdir -p $TMPDIR

function fail {
    echo "FAILED: "$1
        cleanup
    exit 1
}

function cleanup {
	rm -rf $TMPDIR
}

cd $TMPDIR

[ $? -ne 0 ] && fail "Problem using tmp directory $TMPDIR"
curlout=`/bin/curl $GEOIP_URL -o $GEOIP_FILE 2>&1`
[ $? -ne 0 ] && fail "Problem downloading GeoIP database: $curlout"
unzip -q $GEOIP_FILE
cd GeoLite2-Country-CSV*

GEOID=`egrep -i "${COUNTRY}" GeoLite2-Country-Locations-en.csv | cut -f1 -d,`
[ $? -ne 0 ] && fail "$COUNTRY not found in GeoIP file"
for IPvX in 4 6; do
	SET=${SETNAME}${IPvX}
	grep $GEOID GeoLite2-Country-Blocks-IPv${IPvX}.csv | cut -f1 -d, > $CIDR_FILE
    if [ "$IPvX" -eq "4" ]; then
        # Add all locahost Address
        echo 127.0.0.0/8 >> $CIDR_FILE
        # Add all IPv4 Private Address
        echo 10.0.0.0/8 >> $CIDR_FILE
        echo 172.16.0.0/12 >> $CIDR_FILE
        echo 192.168.0.0/16 >> $CIDR_FILE

        $IPSET -exist create ${SET} hash:net family inet
        $IPSET create ${SET}-tmp hash:net family inet
    else
        $IPSET -exist create ${SET} hash:net family inet6
        $IPSET create ${SET}-tmp hash:net family inet6
    fi
    [ $? -ne 0 ] && fail "Problem creating tmp ipset"
    while read line; do
    $IPSET add ${SET}-tmp $line;
            [ $? -ne 0 ] && fail "Problem updating tmp ipset"
    done < $CIDR_FILE

    # Check for changes
    if [ -n $CHANGELOG ]; then
        $IPSET save ${SET} > ${TMPDIR}/$$_${SET}.ipset
        $IPSET save ${SET}-tmp > ${TMPDIR}/$$_${SET}-tmp.ipset
        echo `date`": Processing $COUNTRY..." >> $CHANGELOG
        echo `wc -l $CIDR_FILE | cut -f1 -d' '`" blocks loaded" >> $CHANGELOG
                diffout=`diff ${TMPDIR}/$$_${SET}.ipset ${TMPDIR}/$$_${SET}-tmp.ipset`
        if [ $? -ne 0 ]; then
            echo $diffout >> $CHANGELOG
        else
            echo "No changes" >> $CHANGELOG
        fi
        echo >> $CHANGELOG
    fi

    # Flush ipset whitelist hashmap
    $IPSET flush $COUNTRY  > /dev/null
    $IPSET flush $SETNAME  > /dev/null

    $IPSET swap ${SET}-tmp $SET
        [ $? -ne 0 ] && fail "Problem updating live ipset"
    $IPSET destroy ${SET}-tmp
        [ $? -ne 0 ] && fail "Problem destroying tmp ipset"
done

cleanup

# Update rules and persit at reboot
cp ${IPSET_UP_RULES} ${IPSET_UP_RULES}.save
$IPSET save > ${IPSET_UP_RULES}

cp ${IPTABLES_UP_RULES} ${IPTABLES_UP_RULES}.save
cat <<EOF >${IPTABLES_UP_RULES}
# Generated by $0 on $(date)
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:LOGGING - [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -m set ! --match-set ${1}4 src -j DROP
COMMIT
# Completed on $(date)
EOF

exit 0

Note: In script os_ipset_update.sh
<LicenseKey> is your GeoIP maxmind.com licence key
Only French public IP are allow
All private IP are allow
And localhost are allow

To fully automate this process on Linux or Unix, use a crontab file like.

###############
##  GeoIP    ##
###############
0 23 * * 6		sudo -u root /<path>/os_ipset_update.sh <hashmap_name> <contry_name>

Notes: run every week at 23:00 on Saturday for example.

fail2ban

See part on fail2ban in Protect against DDOS and brute force attacks with fail2ban post.

Disk Partitions

Disable IPv6 Networking - Turn Off IPv6

Use Secure Shell (SSH)

Avoid Using FTP, Telnet, and Rlogin / Rsh services on Linux

Keep system updated

Applying Debian security updates only

Updating all packages (OS and other installed packages)

No accounts with empty Passwords

Make sure no non-root accounts have UID set to 0

Disable root Login

IPtable, fail2ban, IPset and GeoIP filtered access

IPtable

Viewing current configuration

Delete rules

Save after reboot

Basic commands

Persist at reboot

IPset

IPset install

Listing / adding / removing IP addresses from our blacklist

Persist at reboot

GeoIP

Install and configure GeoIP

Direct download GeoIP CVS format database

Secured acces now with iptable, ipset and GeoIP

iptables to allow traffic from private network and one country only

fail2ban

Templates (for web app):

Error

Avoid Using `FTP`, `Telnet`, and `Rlogin` / `Rsh` services on Linux

Direct download GeoIP `CVS` format database

Secured acces now with `iptable`, `ipset` and `GeoIP`