IPsec

Updated: April 19, 2023

Internet security protocol

Synopsis:

#include <sys/types.h>
#include <netinet/in.h>
#include <netinet6/ipsec.h>

int socket( PF_KEY, 
            SOCK_RAW, 
            PF_KEY_V2 );

Description:

IPsec is a security protocol for the Internet Protocol layer. It consists of these sub-protocols:

AH (Authentication Header)
Guarantees the integrity of the IP packet and protects it from intermediate alteration or impersonation by attaching a cryptographic checksum computed by one-way hash functions.
ESP (Encapsulated Security Payload)
Protects the IP payload from wire-tapping by encrypting it using secret-key cryptography algorithms.

IPsec has these modes of operation:

Kernel interface

The IPsec protocol behavior is controlled by these engines:

These engines are located in the socket manager. The socket manager implements the PF_KEY interface and allows you to define IPsec policy similar to per-packet filters. Note that the socket manager code doesn't implement the dynamic encryption key exchange protocol IKE (Internet Key Exchange)—that implementation should be done at the application level (usually as daemons), using the previously described APIs.

Policy management

The socket manager implements experimental policy management. You can manage the IPsec policy in these ways:

If the socket manager finds no matching policy, the system-wide default value is applied.

For a list of net.inet6.ipsec6.* variables, see the sysctl utility in the Utilities Reference.

Miscellaneous sysctl variables

The following variables are accessible via the sysctl utility for tweaking socket manager IPsec behavior:

Name Type Changeable?
net.inet.ipsec.ah_cleartos Integer Yes
net.inet.ipsec.ah_offsetmask Integer Yes
net.inet.ipsec.dfbit Integer Yes
net.inet.ipsec.ecn Integer Yes
net.inet.ipsec.debug Integer Yes
net.inet6.ipsec6.ecn Integer Yes
net.inet6.ipsec6.debug Integer Yes

The variables are interpreted as follows:

ipsec.ah_cleartos
When computing AH authentication data, the socket manager clears the type-of-service field in the IPv4 header if the value is set to a nonzero value. The variable tweaks AH behavior to interoperate with devices that implement RFC 1826 AH. Set this to a nonzero value (clear the type-of-service field) if you want to conform to RFC 2402.
ipsec.ah_offsetmask
When computing AH authentication data, the socket manager includes the 16-bit fragment offset field (including flag bits) in the IPv4 header, after computing a logical “AND” with the variable. This variable tweaks the AH behavior to interoperate with devices that implement RFC 1826 AH. Set this value to zero (clear the fragment offset field during computation) if you want to conform to RFC 2402.
ipsec.dfbit
Configures the socket manager behavior for IPv4 IPsec tunnel encapsulation. The variable is supplied to conform to RFC 2403 Chapter 6.1.
If the value is set to: Then:
0 The DF bit on the outer IPv4 header is cleared.
1 The outer DF bit on the header is set from the inner DF bit.
2 The DF bit is copied from the inner header to the outer.
ipsec.ecn
If set to nonzero, the IPv4 IPsec tunnel encapsulation/decapsulation behavior supports ECN (Explicit Congestion Notification), as documented in the IETF draft draft-ietf-ipsec-ecn-02.txt.
ipsec.debug
If set to nonzero, debug messages are generated to the syslog.

Variables under the net.inet6.ipsec6 tree have meaning similar to their net.inet.ipsec counterparts.

Protocols

Because the IPsec protocol works like a plugin to the INET and INET6 protocols, IPsec supports most of the protocols defined upon those IP-layer protocols. Some of the protocols, like ICMP or ICMP6, may behave differently with IPsec. This is because IPsec can prevent ICMP or ICMP6 routines from looking into the IP payload.

Setting the policy

You can set the policy manually by calling setkey, or set it permanently in /etc/inetd.conf. Valid policy settings include:

for setkey:
-P direction discard
-P direction ipsec request ...
-P direction none
for /etc/inetd.conf:
direction bypass
direction entrust
direction ipsec request ...

where:

direction
The direction in which the policy is applied. It's either in or out.
bypass
(/etc/inetd.conf only) Bypass the IPsec processing and transmit the packet in clear text. This option is for privileged sockets.
discard
(setkey only) Discard the packet matching indexes.
entrust
(/etc/inetd.conf only) Consult the Security Policy Database (SPD) in the stack. The SPD is set by setkey (see the Utilities Reference).
ipsec request ...
Put the IPsec operation into the packet. You can specify one or more request strings using the following format:
protocol/mode/src-dst[/level]

For detailed descriptions of the arguments in the request string, see below.

none
(setkey only) Don't put the IPsec operation into the packet.

Arguments for request

protocol
One of:
  • ah — Authentication Header. Guarantees the integrity of the IP packets and protects them from intermediate alteration or impersonation, by attaching cryptographic checksums computed by one-way hash functions.
  • esp — Encapsulated Security Payload. Protects the IP payload from wire-tapping by encrypting it with secret key cryptography algorithms.
  • ipcomp — IP Payload Compression Protocol.
mode
Security protocol to be used, which is one of:
  • transport — Protects peer-to-peer communication between end nodes.
  • tunnel — Includes IP-in-IP encapsulation operations and is designed for security gateways, like VPN configurations.
dst, src
The “receiving node” (dst) and “sending node” (src) endpoint addresses of the Security Association (SA). When the direction specified is in, dst would represent this node and src the other node (peer).

If transport is specified as the mode, you can omit these values.

level
One of:
  • default — The stack should consult the system default policy that's set by the sysctl utility.
  • require — An SA is required whenever the kernel deals with the packet.
  • use — Use an SA if it's available; otherwise, keep the normal operation.
  • unique — (setkey only) Similar to require, but adds the restriction that the SA for outbound traffic is used only for this policy.

    You may need the identifier in order to relate the policy and the SA when you define the SA by manual keying. You can put the decimal number as the identifier after unique, such as: unique: number

    The value of number must be between 1 and 32767. If the request string is kept unambiguous, the level and slash prior to level can be omitted. However, you should specify them explicitly to avoid unintended behaviors.

Note: If the level isn't specified in the setkey command, unique is used by default.

Based on:

RFC 2367, RFC 1826, RFC 2402, RFC 2403

Detailed documentation about the IP security protocol may be found at the IPsec FAQ website at http://www.netbsd.org/Documentation/network/ipsec/.

Caveats:

The IPsec support is subject to change as the IPsec protocols develop.

There's no single standard for policy engine API, so the policy engine API described herein is just for KAME implementation.

The AH tunnel may not work as you might expect. If you configure the require policy against AH tunnel for inbound, tunneled packets will be rejected. This is because AH authenticates the encapsulating (outer) packet, not the encapsulated (inner) packet.

Under certain conditions, a truncated result may be returned from the socket manager from SADB_DUMP and SADB_SPDDUMP operations on a PF_KEY socket. This occurs if there are too many database entries in the socket manager and the socket buffer for the PF_KEY socket is too small. If you manipulate many IPsec key/policy database entries, increase the size of socket buffer.