What Is IPsec and What Does It Do for a VPN?
The Internet Engineering Task Force (IETF) developed IPsec in response to the security vulnerabilities of the Internet Protocol (IPv4). It was not a challenging task for anyone to intercept data on the internet delivered over IPv4. Therefore, IPsec came into play. IPsec addresses the security deficiencies of IPv4 by using a set of security protocols to ensure communications over the internet are secure.
Operating at the network layer of the Open Systems Interconnection (OSI) model, you can use IPsec to communicate securely between networks and hosts. As a result, IPsec implementations occur in host-to-host, network-to-host, and network-to-network communications.
IPsec is popular in any industry that requires end-to-end security between two endpoints, especially the VPN industry. Among other protocols, VPNs use IPsec to secure data in transit as it provides data integrity, confidentiality, and authentication. This piece dwells on understanding IPsec and why VPNs depend on it to protect the internet traffic of their consumers.
What Is IPsec?
Internet Protocol Security (IPsec) is a protocol suite or a group of protocols that work together to set up a secure connection between computers over the internet. IPsec provides encrypted connections, preventing third parties from seeing the contents of the communication between the computers involved. Aside from encryption, IPsec also provides a mechanism for authenticating data packets to be sure they come from the right source. Let’s look at the standard security features IPsec provides below.
1. Authentication: By authenticating data at both ends of the connection, IPsec ensures that the endpoints are trustworthy before data transmission begins.
2. Confidentiality: By providing end-to-end data encryption, IPsec ensures malicious entities cannot infiltrate the network, steal data, or eavesdrop on the network.
3. Integrity: IPsec uses hashing to check data integrity, ensuring that it intercepts altered data.
4. Anti-replay: Using sequence numbers, IPsec ensures that bad actors cannot replicate data packets to infiltrate the network.
You can use IPsec between two hosts (host-to-host), between two networks/gateways (network-to-network), or between a host and a network (network-to-host). In addition, you will find the implementation of IPsec in security domains such as Virtual Private Networks (VPNs), routing security, and application-level security. IPsec alone might not be effective enough for high-level security, which is why it is implemented together with other protocols in some cases.
IPSec Protocols
There are three protocols that are part of the IPsec suite. These protocols work together to provide data authentication, integrity, confidentiality, and anti-replay. Let’s look at them below.
1. Authentication Header (AH)
The Authentication Header’s main job is to authenticate IP data packets. Using hash functions and a secret key, it validates every packet from each end of the VPN tunnel. This prevents malicious entities from modifying data and passing it off as genuine. The AH provides authentication, integrity, and anti-replay protection. It does not offer any form of encryption.
2. Encapsulating Security Payload (ESP)
The Encapsulating Security Payload’s main job is to encrypt data packets. Depending on the IPsec mode, it encrypts only the payload or the payload and the IP header. Transport mode only allows ESP to encrypt the transport layer segment and the payload, leaving the IP header without encryption. On the other hand, ESP in tunnel mode encrypts the IP header with the transport layer segment and the payload. The ESP provides authentication, integrity, confidentiality, and anti-replay protection.
3. Security Association (SA)
IPSec uses security associations to establish security protocols that the endpoints use to negotiate encryption algorithms and keys. In simple terms, a security association is just a way for the two endpoints to agree on parameters that will establish and keep the IPsec tunnel secure.
Security Associations of IPsec require three things: a Security Parameter Index (SPI), the destination IP address, and the IPsec protocol (AH or ESP). SAs operate in a single direction; therefore, you need two SAs (outbound and inbound) for each endpoint. IPsec uses Internet Security Association and Key Management Protocol (ISAKMP) to establish SAs.
How Does IPsec Work?
Below, we look at how IPsec establishes a secure connection between two endpoints and protects data from its origin to destination.
1. Key Exchange
The Internet Key Exchange Protocol (IKE) handles the negotiation of keys and algorithms that both endpoints will use. As a result, it is a vital process in using IPsec to establish a secure and reliable connection. IKEv1 has two phases: IKE phase 1 and IKE phase 2.
The purpose of IKE phase 1 is for the endpoints to exchange proposals on how to authenticate and secure the connection between them. They exchange proposals for security parameters such as encryption algorithms, authentication algorithms, Diffie-Hellman (DH) group, and pre-shared keys or RSA/DSA certificates. As a result, by the end of this phase, at least a single bi-directional ISAKMP SA should be established between peers. You can use phase 1 in either Main Mode (total of six messages between endpoints) or Aggressive Mode (total of three messages between endpoints).
IKE phase 2 begins after the endpoints or peers have established a secure connection. In this phase, the endpoints negotiate SAs that will keep data that passes through the IPsec tunnel safe. A security protocol (ESP or AH), encryption algorithms, and authentication algorithms are negotiated at this phase. In addition, a Diffie-Hellman (DH) group and Perfect Forward Secrecy can also be part of the proposal.
IKEv2, the updated version of IKEv1, does not have a phase 1 or phase 2. However, the endpoints exchange four messages to negotiate security parameters for the IPsec tunnel. VPNs prefer to use IKEv2 because it is easier to configure and more secure.
2. Packet Headers and Trailers
After the endpoints have agreed on security parameters, they can now send data to each other. First, IP packets need to be encapsulated. Depending on whether you’re using tunnel or transport mode, IPsec adds the necessary header(s) and trailer(s) to each packet that needs to be transported.
3. Authentication and Encryption
IPsec applies authentication and encryption using AH and ESP. AH does not provide encryption but authenticates data packets. On the other hand, ESP offers encryption and authentication.
4. Transmission
IP packets can now move through the IPsec connection to the endpoints with the help of a transport protocol (preferably UDP).
5. Decryption
Decryption occurs at the receiving end of the connection. Once decrypted, data moves to the application that needs it.
IPsec Transport Mode Vs. Tunnel Mode
IPsec has two modes of operation: Tunnel and Transport. Let’s look at what makes each one different from the other below.
Transport Mode
An IPsec circuit operating in transport mode is usually a direct connection between two hosts. The endpoints don’t encrypt or authenticate the entire IP packet in transport mode, only the payload. Since the IP header is not modified or encapsulated, third-party devices outside the two endpoints can see the destination and origin of packets.
Tunnel Mode
An IPsec circuit operating in tunnel mode is usually between two gateways (network-to-network communications): two routers or a router and a firewall. However, you can use it for host-to-host and host-to-network communications. In this mode, it is not just the payload that is encrypted; the whole IP packet is encrypted and authenticated. VPNs use IPsec tunnel mode because it secures the entire network using end-to-end encryption and not just the data that passes through it.
IPsec in a VPN
Since a VPN provides a way you can securely communicate over the internet, it is only reasonable that IPsec is one of the protocols that VPNs use. When VPNs use IPsec, they usually use ESP in tunnel mode because of the end-to-end encryption that it provides. In some cases, you will see either IKEv2/IPsec or IKEv2 as VPN protocols you can use. They both mean the same thing as IKEv2 uses the IPsec tunnel mode to establish a safe connection.
In addition, VPNs combine less secure protocols like L2TP with IPsec to ensure secure connections. It is why you’ll see L2TP/IPsec as an option in some VPN applications. Now, let’s look at the fundamental way IPsec works when you click the connect button on your VPN application.
1. Once you click the “Connect” button, IPsec begins to establish a secure connection using ESP and Tunnel Mode.
2. The endpoints of the tunnel agree on security parameters using Security Associations (SAs).
3. Data can now move from one end to another securely as encryption and decryption occur at both endpoints. One endpoint receives IP packets, encrypts them, and transfers them to the other endpoint. At the receiving end, the other endpoint decrypts IP packets and sends them to the necessary application.
The above is not a thorough explanation, but it should give you an idea of how IPsec works in a VPN.
Conclusion
IPsec is important if you need data confidentiality, integrity, and authentication between two endpoints. Most VPNs include IPsec (IKEv2/IPsec or L2TP/IPsec) as part of the secure VPN protocols that they offer. We hope this article helps you understand IPsec and how it is an important protocol suite VPNs use to secure communications on the internet.