Configuring IPsec for Greenplum Database

This topic describes how to set up and configure Internet Protocol Security (IPsec) for a Greenplum Database cluster.

IPsec Overview

Internet Protocol Security (IPsec) is a suite of protocols that authenticate and encrypt communications at the IP network level (OSI layer 3). Using IPsec can help to prevent network attacks, such as packet sniffing, altering network packets, identity spoofing, and man-in-the-middle attacks in the Greenplum Database system.

When IPsec is enabled for Greenplum Database, a virtual private network (VPN), or tunnel, is established between every pair of hosts in the cluster and every packet exchanged between them is encrypted and sent through the tunnel. If you have n hosts in the cluster, n(n-1)/2 VPNs are needed to connect each host to every other host. You may choose to add other hosts on the network, for example ETL servers, to the IPsec configuration.

Encrypting IP traffic has a cost in network performance. To ensure suitable network bandwidth is available after IPsec configuration, use the Greenplum Database gpcheckperf utility. See the Greenplum Database Utility Guide for help with gpcheckperf. If network bandwidth is insufficient for performance and database workloads you may need to tune the configuration or use a higher bandwidth network medium.

This section describes how to set up and configure IPsec for a Greenplum cluster on Red Hat or CentOS hosts using Openswan, a popular IPsec implementation for Linux. Openswan provides user tools to enable IPsec on Linux. It uses the Internet Key Exchange (IKE) protocol and X.509 certificates to handshake and exchange session keys, and uses the netlink API to interface with the IPsec support built into the Linux kernel.

The IKE protocol allows two peers to negotiate the authentication and encryption algorithms the tunnels will use. The negotiation occurs in two phases. During phase 1, the peers perform a Diffie-Hellman key exchange to establish a secure, encrypted channel. Phase 1 must successfully complete before phase 2 begins.

During phase 2, the peers negotiate the authentication and encryption algorithms to be used with the IPsec tunnels. The result of the phase 2 negotiation is a security association (SA). It contains the source, the destination, and an instruction. The Linux kernel uses the SA to set up the connection.

The peers can authenticate each other using one of the following methods:
  • RSA public key encryption. Each host has a public and private key. Public keys are distributed to all hosts in the cluster so that any host can authenticate any other host.
  • Pre-shared keys. This method is easiest to configure, but is not as secure as using RSA public key encryption.
  • X.509 certificates. A certificate is issued for each host by a certificate authority (CA). The host is authenticated based on trust conferred by the CA. This is most often used when many hosts connect to a central gateway.

RSA public key encryption is the preferred method.

There are two connection modes for a connection: tunnel or transport. In tunnel mode, the entire IP packet is encrypted, including the IP headers. In transport mode, the headers are exposed. The tunnel mode is preferred for greater security.

The following resources are recommended for additional information about IPsec and Openswan:
  • Internet Key Exchange (IKE) - Wikipedia article that describes the IKE protocol used to set up a security association.
  • Security Association - Wikipedia article that describes the attributes and purpose of a security association.
  • AES instruction set - Wikipedia article that provides an overview of the Intel Advanced Encryption Standard (AES) instruction set and lists the CPU families that support it.
  • ipsec.conf(5) - man page for the ipsec.conf configuration file.
  • setkey(8) - man page for the setkey utility used to manage the Security Association Database (SAD) and Security Policy Database (SPD) in the Linux kernel.
  • Openswan - Red Hat Openswan package overview; applies to CentOS also.
  • Host-to-Host VPN Using Openswan - Red Hat guide for creating a host-to-host VPN using Openswan; can also be used with CentOS.

Installing Openswan

Openswan may be installed using the package manager on your system, by downloading an installable package from the Openswan Web site, or by downloading and compiling source code.

Pivotal recommends that you use Openswan version 2.6.43 or later. If your package manager has an earlier version, you can download an RPM of the latest Openswan release from the Openswan Web site. You can also download Openswan source code from the Openswan Web site.

The following instructions assume you are installing Openswan on hosts running 64-bit Red Hat 6.x or CentOS 6.x.

First, determine if Openswan is already installed and if so, which version:
$ sudo yum info installed openswan

If the recommended version is already installed, continue with Configuring and Verifying the Openswan Installation.

If an older version is installed, uninstall it before continuing:
$ sudo yum remove openswan

Installing Openswan with an RPM

Enter the following command to see which version of Openswan is available in the package repository:
$ sudo yum list available openswan
If the recommended version is available, install it on each host in the cluster:
$ sudo yum install -y openswan

If the recommended version is not in the repository, you can download it from the Openswan Web site at https://download.openswan.org. Browse to the /rhel6/x86_64 directory to find the RPM.

Install the downloaded RPM with a command like the following:
$ sudo rpm -i openswanX-version.x86_64.rpm

Installing Openswan from Source

If you cannot install Openswan with an RPM you can download the source, compile it, and install it.

  1. Download the Openswan source from the Openswan Web site at Openswan Web site.
  2. Extract the archive and review the README file to ensure that the prerequisite packages are installed on your build machine. For example:
    sudo yum install gmp gmp-devel gawk flex bison \
        iproute2 iptables sed awk bash cut python
  3. Build the Openswan tools by following the instructions in the README file. For example:
    $ make programs
    $ sudo make install

Configuring and Verifying the Openswan Installation

Follow the steps in this section to configure each host and verify the Openswan installation.

  1. Edit /etc/sysctl.conf and modify or add the following variables:
    net.ipv4.conf.default.accept_redirects = 0
    net.ipv4.conf.all.accept_redirects = 0
    net.ipv4.conf.default.send_redirects = 0
    net.ipv4.conf.all.send_redirects = 0
    net.ipv4.ip_forward = 1

    Execute sysctl -p to reload the file.

  2. Restore the default SELinux security contexts for the IPsec directory by running the following command:
    # restorecon -Rv /etc/ipsec.d
  3. If a firewall is enabled, modify it to allow IPsec packets:
    • UDP port 500 for the Internet Key Exchange (IKE) protocol
    • UDP port 4500 for IKE NAT-Traversal
    • Protocol 50 for Encapsulated Security Payload (ESP) IPsec packets
    • (not recommended) Protocol 51 for Authenticated Header (AH) IPsec packets
    Here is an example of IPsec rules for iptables:
    iptables -A INPUT -p udp --sport 500 --dport 500 -j ACCEPT
    iptables -A OUTPUT -p udp --sport 500 --dport 500 -j ACCEPT
    iptables -A INPUT -p udp --sport 4500 --dport 4500 -j ACCEPT
    iptables -A OUTPUT -p udp --sport 4500 --dport 4500 -j ACCEPT
    iptables -A INPUT -p 50 -j ACCEPT
    iptables -A OUTPUT -p 50 -j ACCEPT
  4. Edit the /etc/ipsec.conf file and make the following changes:
    • Change protostack from auto to netkey:
      protostack=netkey
    • Uncomment or add the following line:
      include /etc/ipsec.d/*.conf
      This allows you to create and install a separate configuration file for each host-to-host tunnel.
  5. Start IPsec with the service command.
    # service start ipsec
  6. Run ipsec verify to check the IPsec installation. Python must be installed to run this command.
    $ sudo ipsec verify
    The output looks like the following:
    Checking if IPsec got installed and started correctly:
    
    Version check and ipsec on-path                          [OK]
    Openswan U2.6.43/K2.6.32-504.16.2.el6.x86_64 (netkey)
    See `ipsec --copyright' for copyright information.
    Checking for IPsec support in kernel                     [OK]
     NETKEY: Testing XFRM related proc values
             ICMP default/send_redirects                     [OK]
             ICMP default/accept_redirects                   [OK]
             XFRM larval drop                                [OK]
    Hardware random device check                      	     [N/A]
    Two or more interfaces found, checking IP forwarding     [OK]
    Checking rp_filter                                	     [ENABLED]
     /proc/sys/net/ipv4/conf/all/rp_filter                   [ENABLED]
     /proc/sys/net/ipv4/conf/lo/rp_filter                    [ENABLED]
     /proc/sys/net/ipv4/conf/eth0/rp_filter                  [ENABLED]
     /proc/sys/net/ipv4/conf/pan0/rp_filter                  [ENABLED]
    Checking that pluto is running                           [OK]
     Pluto listening for IKE on udp 500                      [OK]
     Pluto listening for IKE on tcp 500                      [NOT IMPLEMENTED]
     Pluto listening for IKE/NAT-T on udp 4500               [OK]
     Pluto listening for IKE/NAT-T on tcp 4500               [NOT IMPLEMENTED]
     Pluto listening for IKE on tcp 10000 (cisco)            [NOT IMPLEMENTED]
    Checking NAT and MASQUERADEing                           [TEST INCOMPLETE]
    Checking 'ip' command                                    [OK]
    Checking 'iptables' command                              [OK]
    
    Note: The result for Checking 'ip' command may be [IP XFRM BROKEN], depending on the version of iproute on your system. This can be a misdiagnosis caused by a change in IP XFRM message output between iproute versions.
  7. Enable starting IPsec on boot with the following command:
    # chkconfig ipsec on

Configuring Openswan Connections

Set up an IPsec tunnel between each pair of hosts on the cluster by creating a connection. On each connection, one host is designated the "left" host and the other the "right" host. For example, if you have a master (mdw), a standby master (smdw), and three segment hosts (sdw1, sdw2, sdw3), you will need ten connections, as shown in the following table.
Table 1. IPsec connections for a five-host cluster
Connection Number Left host Right host
1 mdw smdw
2 mdw sdw1
3 mdw sdw2
4 mdw sdw3
5 smdw sdw1
6 smdw sdw2
7 smdw sdw3
8 sdw1 sdw2
9 sdw1 sdw3
10 sdw2 sdw3

Create Host Keys

To enable RSA public key authentication for the tunnels, each host must have an RSA key pair. As the root user on each host, enter the following command to generate an authentication key.
# ipsec newhostkey --output /etc/ipsec.d/ipsec.secrets --bits 4096

The key is saved in the /etc/ipsec.d/ipsec.secrets file and its attributes are set to allow access to the root user only.

To view a host's public key, use the ipsec showhostkey command with the --left or --right option. The command outputs the public key in a format suitable for pasting into the connection configuration file. In this example, the keys are shortened for readability:
# ipsec showhostkey --left
	# rsakey AQOW+RwpL
	leftrsasigkey=0sAQOW+RwpLg7CGoyywCnv+vnasGJI7...
# ipsec showhostkey --right
	# rsakey AQOW+RwpL
	rightrsasigkey=0sAQOW+RwpLg7CGoyywCnv+vnasGJI7... 

You will need to use this command as you configure the tunnel between each host pair.

Create a Connection Configuration File

IPsec tunnels are configured by creating a conn section in the /etc/ipsec.conf file. Because we added the include /etc/ipsec.d/*.conf directive to /etc/ipsec.conf, we can create a .conf file for each connection.

Follow these steps to configure a connection for each pair of hosts.
  1. Log in to the host that will be on the "left" side of the tunnel.
  2. Create a new configuration file in the /etc/ipsec.d directory. Choose a name that includes both host names and has a .conf extension. The following configuration file, named mdw-sdw1.conf, configures the connection between the hosts mdw and sdw1:
    conn mdw-sdw1
        leftid=mdw 
        left=192.1.2.214
        leftrsasigkey=0sAQOW+RwpLg7CGoyywCnv+vnasGJI7... # shortened for readability
        rightid=sdw1 
        right=192.1.2.215
        rightrsasigkey=0sAQNfdDCoDte5bGaGLGkHTKa5GMRl... # shortened for readability
        type=tunnel
        authby=rsasig   
        ike=aes192-sha2;dh20
        phase2alg=aes_gcm_c-160-null    
        auto=start
    

    See the ipsec.conf man page for the complete list of available parameters and their default values.

    The connection name in the example is mdw-sdw1.

    For the leftrsasigkey use the output from running ipsec showhostkey --left on the "left" host. For rightrsasigkey use the output from running ipsec showhostkey --right on the "right" host.

    Following are recommendations for configuring parameters for Greenplum Database IPsec connections to obtain the best security and performance:
    type
    Set to tunnel, the default mode.
    authby
    Set to rsasig. This is more secure than using pre-shared keys (psk).
    auto
    Set to start so that the tunnel is brought up when IPsec starts up.
    ike
    The ike parameter is used during phase 1 to authenticate the peers and negotiate secure session keys for phase2. The parameter value is an entry in the format:
    cipher-hash;modpgroup, cipher-hash;modpgroup, .... 
    • cipher is an encryption algorithm. AES is more secure than 3DES, which is more secure than DES. AES has length of 128, 192, or 256 bits. The more bits, the stronger the encryption, but more time is required for computation.
    • hash is the hash algorithm. SHA2 is stronger than SHA1, which is stronger than MD5. SHA2 is recommended, but if SHA2 is not supported on the device, use SHA1. SHA2 is a family of hash functions—SHA-224, SHA-256, SHA-384, SHA-512, SHA-512/224, SHA-512/256—not all of which are supported by Openswan. To find out which algorithms Openswan supports, run the following command after starting the ipsec service:
      # ipsec auto -status
      000 algorithm IKE hash: id=2, name=OAKLEY_SHA1, hashsize=20
      000 algorithm IKE hash: id=4, name=OAKLEY_SHA2_256, hashsize=32
      000 algorithm IKE hash: id=6, name=OAKLEY_SHA2_512, hashsize=64
      000 algorithm ESP encrypt: id=2, name=ESP_DES, ivlen=8, keysizemin=64, keysizemax=64
      000 algorithm ESP encrypt: id=3, name=ESP_3DES, ivlen=8, keysizemin=192, keysizemax=192
      000 algorithm ESP encrypt: id=18, name=ESP_AES_GCM_A, ivlen=8, keysizemin=160,
            keysizemax=288000 algorithm ESP encrypt: id=19, name=ESP_AES_GCM_B, ivlen=12, keysizemin=160,
            keysizemax=288
      000 algorithm ESP encrypt: id=20, name=ESP_AES_GCM_C, ivlen=16, keysizemin=160,
            keysizemax=288
      See http://en.wikipedia.org/wiki/SHA-2 for information about SHA2.
    • modpgroup is the Diffie-Hellman group. The peers negotiate a shared secret using the Diffie-Hellman protocol. The Diffie-Hellman group is a set of standardized parameters the peers agree to use as the basis for their calculations. The groups are numbered, and higher numbered groups are more secure (and more compute-intensive) than lower numbered groups. Avoid the lowest numbered groups: 1 (modp768), 3 (modp1024), and 5 (modp1576), which are not considered secure. Choose a higher level group, such as dh14, dh15, dh19, dh20, dh21, or dh24.
    phase2
    Set to esp, the default, to encrypt data. The ah setting creates a connection that authenticates, but does not encrypt IP packets.
    phase2alg
    The phase2alg parameter specifies algorithms to use for encrypting and authenticating data. The format and defaults are the same as for the ike parameter.
    The AES cipher and SHA hash algorithm are more secure. For effective use of emerging 10-gigabit and 40-gigabit network devices, and to enable high speed communication channels, the AES_GCM algorithm is currently the recommended best option. To use AES_GCM, verify that the CPU supports the AES_NI instruction set. See AES instruction set for a list of CPUs that support AES_NI.
    To see if the CPU supports AES-NI, see if the aes flag is set in /proc/cpuinfo:
    grep aes /proc/cpuinfo
    To see if AES-N1 has been enabled, search /proc/crypto for the module:
    grep module /proc/crypto | sort –u
    To see if the aesni_intel kernel module is loaded:
    /sbin/modinfo aesni_intel
    To specify the AES_GCM algorithm, use the following syntax:
    phase2alg=aes_gcm_c-160-null
    Openswan requires adding the salt size (32 bits) to the key size (128, 192, or 256 bits). In the example above, "160" is calculated by adding a 128-bit key size to the 32 bit salt size. The other valid values are 224 and 288.
  3. Use scp to copy the configuration file to the "right" host. For example:
    # scp /etc/ipsec.d/mdw-sdw1.conf sdw1:/etc/ipsec.d/
  4. Ensure that IPsec is started by executing the following command on both the "left" and "right" hosts:
    # ipsec service start
  5. Load the tunnel on both left and right hosts with the following command:
    # ipsec auto --add mdw-sdw
  6. Bring up the tunnel on both left and right hosts with the following command:
    # ipsec auto --up mdw-sdw

Test the IPsec Connection

To verify IPsec packets are flowing through a network interface, run the following tcdump command on one host and then ping that host from another host.
tcdump -n -i interface_name host hostname
For example, run the tcpdump command on sdw1 and then, on mdw, ping sdw2:
# tcpdump -n -i eth0 host mdw
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
08:22:10.186743 IP 192.168.1.214 > 192.168.1.215: ESP(spi=0xe56f19ea,seq=0x1), length 132
08:22:10.186808 IP 192.168.1.214 > 192.168.1.215: ICMP echo request, id 30987, seq 1, length 64
08:22:10.186863 IP 192.168.1.215 > 192.168.1.214: ESP(spi=0x4e55824c,seq=0x1), length 132
08:22:11.189663 IP 192.168.1.214 > 192.168.1.215: ESP(spi=0xe56f19ea,seq=0x2), length 132
08:22:11.189707 IP 192.168.1.214 > 192.168.1.215: ICMP echo request, id 30987, seq 2, length 64

The ESP packets verify that the IP packets are encrypted and encapsulated.

When you have connections set up between all of the hosts in the cluster and Greenplum Database is running, you can run the tcpdump command on segment hosts to observe database activity in the IPsec tunnels.