IPsec (Internet Protocol Security) VPN

IPSec (Internet Protocol Security) is the most common internet security protocol. When used as part of a VPN, the protocol creates encrypted and anonymous connections across a virtual network.

As data moves from origin to destination on the VPN, it flows through encrypted connections known as "tunnels." Protocols like IPSec define how these tunnels are constructed and play a major role in determining the security and speed of VPN services.

IPSec is an extremely popular way to route data securely, and you will probably encounter the protocol when you fine-tune your information security setup. But what is an IPSec VPN and how does it work? What separates IPSec from other protocols? Let's find out more detail and explore the protocol's strengths and weaknesses.

How does IPsec work?

The first thing to understand about IPSec is that it is not a single protocol. IPSec is formed from various parts. Together they deliver a set of core security functions:

• Host recognition
• Negotiation
• Authentication
• Encryption
• Transmission
• Decryption
• Termination

As the name suggests, IPSec functions at the IP layer of the OSI hierarchy (level 3). This is useful because applying encryption at level 3 allows companies to add VPN coverage across an entire network, and not just separate devices.

Users generally access IPSec VPN protection via a VPN client. This acts as a secure gateway, and forms the first link in a chain reaching to the payload destination.

IPSec VPNs work in five separate stages. Each stage adds another element to IPSec VPN protection, enabling users to send sensitive data securely anywhere in the world.

Stage 1: Host recognition

Before data packets enter an IPSec VPN's secure gateway, they must be recognized and analyzed to prepare for transmission. Host recognition determines which data packets require encryption. Relevant data segments are authenticated by the host and then prepared for encryption according to local IPSec policies.

This process adds important information to data packets. Payloads are assigned a fresh IP header containing information agreed by the two hosts.

Encryption and authentication headers are also added. The encryption header allows recipients to unwrap payloads when they arrive. The authentication header demonstrates that each data packet is what it claims to be and that the source is legitimate.

Stage 2: Negotiation

The next step in the IPSec VPN process is creating a secure route from origin to destination. Two hosts must negotiate the terms of the transmission before data can depart. Both hosts must agree on the policy settings regulating the transfer. This creates the basic structure for the tunneling protocol to apply IKEv2 encryption.

Negotiation can be either aggressive or main. Main style negotiation is consensual, and the two hosts take time to agree on terms. Aggressive negotiation involves the origin host forcing terms on the destination. Main mode tends to be more secure as the policy in use is agreed upon and clear. But aggressive mode will usually be faster.

Stage 3: Creating the IPSec circuit

The next stage in an IPSec VPN transmission is the creation of an encrypted tunnel which applies the terms of the negotiation. Once the basic terms are agreed upon, the two hosts determine which encryption algorithm to use.

At this point, hosts exchange encryption keys. Keys are strings of characters that are unique to each transmission. Both the origin host and the destination device must know the key in order to complete the IPSec circuit and decode data.

Cryptographic nonces are also exchanged at this stage. These unique number codes authenticate the transfer, ensuring it proceeds smoothly and securely.

When the relevant keys have been exchanged, the VPN tunnel is ready for use. Hosts are aware of the data being transferred, the payload format, and the encryption in use. Data should be well protected from outsiders at every point in its journey.

Stage 4: Transmission

Transmission can now take place. The IPSec VPN will send data through the encrypted tunnel. Encryption policies agreed upon earlier make it possible to encrypt and decrypt the data as needed.

Most data transfer methods use the Transmission Control Protocol (TCP) to send payloads. IPSec uses the User Datagram Protocol (UDP) transmission protocol to send data instead. This enables IPSec payloads to pass through firewalls without impediment.

Stage 5: Terminating the session

IPSec VPN sessions must be terminated once the transmission is complete. Negotiations agree on a data or time limit before transmission and the session ends once this limit is reached. At that stage, any encryption keys are destroyed immediately to minimize the risk of spoofing attacks.

What are the IPsec VPN ports?

VPN protocols use different ports to make transfers. In the case of IPSec, the default ports are UDP/500 and UDP/4500. IPSec will usually employ UDP/500 to create IKEv2 encrypted tunnels. UDP/4500 will be used if the target host is behind the Network Address Translation (NAT).

What is IPsec used for?

The IPSec protocol originated in the 1990s as a general data transfer tool. Companies needed a way to transmit data safely across the internet. IPSec offered a solution. Since then, IPSec has remained an open protocol, allowing constant refinements and updates. Thanks to its open nature, IPSec continues to be a common feature of Virtual Private Network services.

VPNs create virtual networks overlaying public networks. These virtual networks route encrypted traffic and conceal data from outsiders via IP shielding. And they are a critical aspect of modern cybersecurity management.

As a VPN protocol IPSec has plenty of practical uses, including working around geo-blockers and avoiding surveillance. But the VPN's most well-known use case is creating secure connections between remote users and central offices.

Companies that rely on work-from-home or on-site staff can create a remote network with strong encryption. With an IPSec VPN, remote users can send and receive data wherever they are, and the risks of data loss or interception are low. Companies can also integrate an IPSec VPN into firewall setups, extending cybersecurity protection to network devices in bedrooms, airport lounges, or coffee shops.

Key IPsec VPN protocols

Protocols are sets of rules or standards that allow devices to talk to each other. IPSec itself is often described as a VPN tunneling protocol. However, the technology actually features a group of protocols. For instance, important IPSec VPN protocols include the following elements:

Authentication Header (AH)

The Authentication Header protocol authenticates data, ensures data integrity, and guards against repeated transmissions via replay protection.

AH generates a message authentication code called MD5. This code or checksum is created for every data packet passing over the IPSec VPN. It detects any changes to data packets during transmission, ensuring no modifications occur along the way.

The IPSec algorithm also includes shared keys that AH uses to authenticate each packet. This guarantees that data belongs on the IPSec chain and can be safely transmitted.

Finally, AH adds a sequence number field to the packet header. This creates a one-time transfer identity linked to a specific moment in time. Replay or spoofing attacks can be detected if packets are later transmitted out of sequence.

The outcome of these three components is payload protection from end to end. Data should remain in its original condition, and external tampering should be impossible or at least extremely difficult.

AH is generally used in the IPSec VPN transport mode. In this mode, AH authenticates IP datagrams, and data inside the packet remains encrypted. However, external attackers can read information about the packet when in transit.

Encapsulating Security Payload (ESP)

ESP performs a similar function to AH, but with one key difference. ESP applies encryption to payload and IP headers, while also delivering replay protection, authentication, and data integrity.

When users transfer data via ESP, decryption follows authentication at the packet destination. The destination host determines that data packets are authentic and remain unmodified, and uses shared keys to decrypt the data.

ESP is commonly used in the IPSec tunnel mode, which is more secure but data-intensive. This IPSec mode wraps data packets in an additional IP header. Information on the header does not relate to the IP information of the data packet inside. Instead, the IP data of the packet is concealed alongside the data itself. Critical information about the data's origins and nature is invisible to outsiders.

Internet Security Association and Key Management Protocol (ISAKMP)

ISAKMP is an encryption protocol that manages security keys and establishes connections between those sharing keys. It does not apply encryption to data. Instead, ISAKMP creates the conditions for protocols like IKE (see below) to work.

The ISAKMP configuration defines the permitted routes of exchange and the payloads operating across IPSec tunnels. This relationship between hosts is known as the Security Association (SA).

As discussed earlier, the SA is formed in two negotiating stages. Hosts determine how to set up their connection and the policies in use to secure data. When that is achieved, ISAKMP brings in ESP to manage data encryption or AH to apply IP authentication.

IKE

Internet Key Exchange (IKE) works with ISAKMP during the IPSec VPN process. IKE establishes a secure key exchange between hosts, enabling data to travel securely from end to end of the Virtual Private Network tunnel. To achieve this, IKE must generate secure keys. These keys are used to encrypt data at the source and decrypt data at the end of the journey.

IPsec advantages & disadvantages

IPSec VPN technology is extremely popular with companies and private individuals seeking digital privacy. However, the protocol was created in the 1990s and has been challenged by competitors since then. This makes it essential to understand its pros and cons.

Advantages of applying IPSec VPN protection

• Strong security levels. IPSec has a good security reputation provided users implement it properly. In-built authentication systems combine with IKE encryption to protect data effectively. However, users can choose which encryption standard to apply. Not all encryption systems are watertight, and poorly configured IPSec VPNs can be vulnerable.
• Transparency for apps. IPSec functions as a drop-in security solution at the Network Layer. This enables it to work with virtually any application. Encrypting data channels via alternatives like SSL can lead to compatibility issues and complex solutions. IPSec is much easier to implement and delivers similar cybersecurity benefits.
• Easy to implement, low error rate. IPSec covers all network traffic automatically. There is no need to make modifications to applications as can be the case with SSL-based VPNs. Users just need to make simple operating system changes and VPN protection slots in without any issues.

These advantages make a strong case for using an IPSec VPN. But they also need to be weighed against the protocol's potential drawbacks.

Disadvantages of IPSec

• Data usage. IPSec encrypts all traffic and applies strict authentication processes. Both operations consume network bandwidth and raise data usage. This makes IPSec a less attractive option for networks handling large numbers of small data packets. In those situations, SSL-based VPNs may be superior.
• Can be deceptive. When configured correctly with strong encryption of IP packets, IPSec delivers excellent security. But this can be a problem in some cases. For example, IPSec may be configured without ESP, consciously or accidentally. Without the encryption provided by ESP, managers may not be aware that data is at risk.
• Known security concerns. Although IPSec offers solid encryption and authentication, it does have a couple of well-known security weaknesses. Firstly, users must have tight control over key exchanges. Encryption keys must remain confidential and protected from external observers. Secondly, IPSec has a wide access range. This can allow users excessive freedom to access network resources, potentially leading to cyber attacks.
• Complexity. IPSec is simple to apply but can be complex to use. The protocol has several moving parts that deliver different features. The process of creating an encrypted tunnel also has various stages, with multiple transfers to authenticate, encrypt, and monitor data. The existence of tunneling and transport modes adds extra complexity, which can make finding a secure configuration difficult.

IPsec modes: tunnel & transport

Before deploying IPSec VPN technology, it is essential to know the difference between the two IPSec modes. Each mode has a specific purpose, and each mode has strengths and weaknesses.

Tunnel mode

Tunnel mode is the main IPSec setting. In this mode, ESP creates an IP trailer around the encrypted data payload. The entire packet is wrapped in an additional IP header and proceeds along the VPN tunnel to its destination.

In tunnel mode, both ends of the process must have a gateway configured to unwrap and decrypt IPSec packets. This gateway could be a specialist VPN router. However, it could also be a VPN client connected to the remote network.

Why is tunnel mode so useful? When data travels via an IPSec VPN in tunnel mode, the entire packet is encrypted. The encryption of IP network packets conceals the internal IP header from external observers. This makes data far less vulnerable to cyber attacks.

The application of an extra IP header also makes it possible to change routing information when needed. Adding an extra header makes it easier for VPN services to route data through their servers and anonymize IP information. It also helps when traversing firewalls, an area where transport mode struggles.

Transport mode

Transport mode is a faster IPSec VPN mode that is generally used to create end-to-end encrypted tunnels. Data payloads are protected via LKE encryption, as with IPSec's tunnel mode. AH protocols authenticate packets as normal and encrypt data. However, in this case, there is no additional IP header.

But because there is no need for an extra IP header, the process is less data intensive and faster. This is beneficial for establishing connections between remote networks and central servers. But it comes with a security cost.

Without the external IP header, IP information remains exposed. Transport mode may also lead to compatibility issues with secure gateways and traversal NATs. However, it is a good solution for creating direct encrypted links between specific devices.

Does NordLayer support IPsec?

IPSec/IKEv2 is one of the primary tunneling protocols used by NordLayer. It's used for two main functions:

• As a VPN tunneling protocol for Shared Gateways connections.
• As a technology securing site-to-site tunnels between separate branches.

Existing NordLayer clients can also create site-to-site connections by requesting them through the Control Panel. It can be a quick and simple solution when linking internal resources for remote employees.

That said, NordLayer has a wide variety of tunneling protocols for diverse use cases. IPSec is just one example of its full suite.

IPSec — vital part of cybersecurity worldwide

In summary: IPSec is a core component of VPN services, especially when paired with tunneling protocols like L2TP. It includes a series of sub-protocols that encrypt data packets. These sub-protocols also add authentication data to packets and encrypt the entire packet with an IP header. This creates a data transfer format that suits VPN routing and can anonymize data efficiently.

FAQ

How does IPsec impact MSS and MTU?

VPNs always result in changes to the maximum transmission unit (MTU). That's because the MTU determines how large packets can be when they pass across the network infrastructure.

IPSec comes with an encapsulation overhead which must be factored into the packet size. If packet sizes exceed the MTU, they will be fragmented for transmission. This results in slowdown and poor network performance. It may also lead to data loss.

Most forms of ESP encryption add 73 bytes, while an extra IP layer adds 20 bytes. Assuming an average ethernet frame size of 1,500 bytes, that leaves an MTU of 1,407.

IPSec VPN protection also affects the maximum segment size (MSS). MSS refers to payload sizes when headers are not included. For IPSec, most networks use an MSS of 1,400 with a TCP-MSS adjustment of 1,360 bytes. Any packets above that size will not be transmitted and will probably be dropped.

What is the difference between IPsec tunnel and IPsec transport modes?

IPSec has two modes: tunnel and transport.

Tunnel mode is the default setting. In this mode, packets are encrypted and authenticated by either ESP or AH. IPSec adds an extra IP header over the packet, hiding the contents from external observers. This extra header allows routers to move packets easily and makes data suitable for VPN transmission.

Transport is the second IPSec mode. In transport mode, packets are encrypted and authenticated by the IPSec protocol. But there is no additional layer of IP information. This is the fastest transmission mode and is commonly used in direct device-to-device communication.

What Is an IPSec Tunnel?

An IPSec tunnel is a secure connection between two users or devices. The tunnel is created by a series of IPSec components, which ensure data integrity, authenticate transmissions, anonymize IP information, and encrypt data contents.

When IP anonymization and encryption combine, this form of tunneling can be used in highly secure and fast IPSec VPN services. These VPN services can be used to secure intranets, and extranets with partners, or to ensure safe remote access for employees.

IPSec is not the only encryption and authentication protocol. OpenVPN, Wireguard, and SSL are potential alternatives. But it remains a relevant cybersecurity tool, and widely used around the world.