Network Security Protocols: IPsec vs. TLS/SSL vs. SSH – Part II

In Part I, I provided some background information on the OSI layer, some of the protocols and on security. In Part II, I’ll be discussing the different network security protocols: IPsec, TLS/SSL and SSH.


I’ve included SSH as it is a network security protocol though my exposure to SSH isn’t as in depth as the other two.

I consider SSH the least similar between the three.  It does provide the same level of protection as SSL/TLS and IPsec but I consider it specific to a type of service (access to remote shells).

Secure shell (SSH) is the replacement for various shell based protocols such as telnet, and other remote logins as well as file transfer protocols such as FTP and remote file copy protocols such as RCP. The deficiency with these protocols is that when you authenticate yourself to a server, it is always sent in the clear. An eavesdropper can easily intercept these requests and masquerade as a potential user. Also, all subsequent data that is transmitted is also in plaintext. If telnet is being used to access a corporate machine, it could potentially store sensitive material that could fall into the hands of an intruder. Corporations must disable telnet, ftp, and remote login services and replace them with the secured versions SSH, SFTP and SCP.  This is usually performed as part of the OS hardening guidelines before product deployment.


  • An application layer solution. It is a connection-oriented service and thus uses TCP ONLY. SSH is primarily used for shell based solutions and ideally won’t be used to protect web browsing sessions and other application services (though it can via port forwarding)
  • Uses public key cryptography to prove the authenticity of the remote user. SSH can generate an RSA key pair (I believe Diffie-Hellman can also be used). It uses what’s known as a fingerprint which is a snapshot of an individual host’s actual public key (for instance the RSA public portion).  The fingerprints are usually 128 bits in length. It is what the user can use to verify that a public key is that of an individual or host.  SSH will maintain a list of trusted hosts. The actual data communication is secured using symmetric cryptography such as AES or 3DES, IDEA
  • The public portion of the key is transmitted to the remote endpoint/server via an out-of-band mechanism. Also a lack of key management
  • SSH does provide some extensible features.  Two of these are port forwarding and secure tunneling.  With port forwarding, you can tell the SSH daemon to listen to data communication on a particular port and forward this communication to the encrypted SSH session. This allows you to protect other services as well.  In Linux, X11 forwarding takes advantage of this. X11 is used for graphical display of a remote system on your localhost.  With port forwarding, SSH can be used to provide an encrypted session for X11

When to use

For shell access and file transfer, SSH is the simplest, most widely deployed and preferred secure solution.

TLS and IPsec

These two security protocols provide very similar functionality but at different layers at the IP stack layer. There are a lot of differences between the two protocols and some key reasons why certain solutions prefer one over the other.  For instance, in 3GPP (3G Wireless),  IMS defines both of these protocols for the secure signaling between User Equipments (end devices) and edge device proxies.


Transport Layer Security (TLS – RFC 2246) is based on SSLv3. It is a Layer 4 protocol as it runs directly on top of TCP ONLY.  It uses PKI to provide user authentication as well as symmetric keying for confidentiality protection. It is designed to prevent eavesdropping, tampering, and message forgery. TLS can be used in two methods:

  • Mutual Authentication:  User Equipment and server both provide certificates when establishing a session. The server must validate the UE certificate, and the UE must validate the server certificate. Mutual authentication provides a high degree of security however is computationally extremely expensive due to the fact that it is public key cryptography. In a secure web browsing session, this would be equivalent to both the user and the server authenticating themselves (which doesn’t happen today)
  • Server-Side Authentication:  Only the server provides a certificate when establishing a session. This is the common solution we see today with using secure web sessions between a user and a web server (HTTPS request).  This approach avoids the extra computational overhead of a PKI operation on the User Equipment. It provides a medium level of security, with lower CPU requirements on the UE. Also, the User is not mandated to own a valid certificate


  • TLS is the recommended security mechanism specified in RFC 3261 by the IETF. This is the Session Initiation Protocol (SIP) used for the call establishment of Voice over IP
  • SSL is used to secure http sessions (HTTPS)
  • There is a general shift now towards the use of TCP to better handle longer messages
  • TLS supports NAT traversal at the protocol layer while IPsec doesn’t
  • TLS is implemented at the application level instead of the kernel level, which provides some advantages such as easier support in multiple environments
  • May be used to secure HTTP Digest sessions in SIP environments.
  • In VoIP, More SIP devices such as phones and soft clients support TLS and not IPsec
  • Provides privacy (private user identity)
  • Provides user authentication instead of data-origin authentication (higher degree of authentication)
  • Preferred in e-commerce solutions such as online banking
  • A push towards using SSL based VPNs as they are easier to deploy since it’s deployed via a browser and doesn’t require specialized client software. They also provide more granular controls as applications can restrict particular users based on a service
  • Becoming the de facto standard for WPA2 wireless security authentication (EAP-TLS)


  • Both of the TLS models require the server and client to support PKI features, such as certificate validation and certificate management. Not all clients and solutions support PKI. PKI is typically used in complex environments
  • PKI is computationally expensive since it uses public key cryptography
  • TCP and TLS pose significant memory consumption and scaling issues when you have tens of thousands of TCP connections. UDP and IPsec are easier to scale. TCP is not well liked by service providers since the overheads associated with its mass use are significant compared to UDP.
  • Runs on top of TCP only (connection-oriented). There is a subset version of TLS that is supported for use with UDP called DTLS (RFC 4347)
  • Provides only hop-by-hop security. What this means is that every intermittent hop would need to be secured with TLS. With this, it doesn’t provide true end-2-end security
  • TLS cannot be used to secure VoIP RTP media streams
  • In Server-Side Authentication, only one end is authenticated
  • TLS does not handle dead office recovery scenarios efficiently.  As mentioned, PKI is CPU intensive therefore when you need to handle re-authentications for all endpoints, this is going to significantly slow down your system

When to Use

Every web browser now has the SSL protocol built into it.  There is no configuration required to utilize it. Therefore to use a secure HTTP session or an SSL VPN requires no additional client software to install.  Ease of use with little maintenance required is what makes TLS/SSL extremely popular. As I mentioned above, one of the benefits of an SSL VPN is that it allows for granular controls. If I wanted to provide a VPN for a particular application and not the entire network, I can achieve this using an SSL VPN.  This is why SSL VPNs are becoming more and more popular.   You will see TLS used more frequently when its use is tied to an individual instead of a device

The real bottleneck with TLS/SSL is the requirement for PKI.  In environments where we can tolerate a slower session, require user authentication and non-repudiation (such as online banking), TLS is the choice.

I am still skeptical about the deployment in VoIP and Wireless networks.  Would you as a user, pay extra money for your monthly service and tolerate slower registration times when you power on your phone so you can have a secure authentication? I highly doubt it.  People still tend to talk freely and are oblivious to those around them. The majority of Social Engineering attacks occur because eavesdroppers are standing next to people and can interpret the conversation. Users always want the cheapest solution and service providers know this.  Even if identity theft or fraud occurs, the cost of the risk and exposure doesn’t outweigh the cost to implement the appropriate safeguards. It’s much cheaper to simply accept the risk.


IPsec (RFCs 2401, 2406, 2409, 2411); is a protocol suite that runs at the networking layer (L3).  It provides confidentiality, integrity protection, data origin authentication and replay protection of each message by encrypting and signing every message.  IPsec is combination of many RFCs and defines two main protocols to use: Authentication Header (AH) and Encapsulating Security Payload (ESP). ESP is the preferred choice as it provides both authentication and confidentiality while AH doesn’t provide confidentiality protection.  In order to establish an IPsec Security Association (SA) between two endpoints, the SAs need to be dynamically established via a key management protocol. This is normally done via IKEv1/IKEv2 in the internet world however for 3GPP access security, they mandate an alternative key management: IMS-AKA(outside the scope of this post)


  • Doesn’t mandate PKI (pre-shared key can be used instead)
  • Since it runs at L3, will protect any transport or application on top of L3. This would allow us to secure both any transport protocol or any application (i.e. it’s application independent)
  • Transparent to applications as it’s integrated into the kernel
  • Mandated by 3GPP for 3G Wireless networks to secure the access side connection and for network domain security (NDS). NDS is used in both inter and intra domain networks
  • Supports multiple modes (transport or tunnel).  Transport mode is used for secure sessions between end devices where tunnel mode is used between security gateways. Tunnel mode adds an extra layer of security as both the header and payload are encrypted (in transport mode, only the payload is encrypted)
  • In 3GPP network IPsec presents no problem because there are no NAPT devices between the phone and the server
  • Since keying information is preloaded onto ISIM in 3GPP, setup time for IPsec SA’s is not computationally expensive
  • Transparent enough that it can be used with many key management protocols (manual keying, IKEv1, IKEv2, IMS-AKA)
  • Most VPNs use IPsec
  • Can be used with Kerberos


  • In IETF,  TLS is preferred to IPsec
  • IPsec is not NA(P)T friendly. When a NAPT device encounters an IPsec ESP packet it no longer has access to the L4 ports since it’s encrypted as part of the L3 payload and will usually revert to NAT-only operation (translate only the IP header and not the ports). This has been solved by using NAT-T and UDP Encapsulation as defined in RFCs 3948 and 3947. The ESP packet is encapsulated within a UDP header (usually communicates on UDP port 4500)
  • UDP NA(P)T bindings will timeout in one minute or so (no standard timeout). This means that the clients need to send a datagram typically every 30th – 40th second to refresh the NA(P)T binding, which of course increases the amount of traffic significantly
  • UDP cannot handle fragmentation, so it is unsuitable for big datagrams
  • No dead peer detection support. No way of knowing if you’ve lost connectivity to your peer since UDP is stateless (see RFC 3706. Periodic keepalive heart beats are sent between peers)
  • Not widely used in SIP Security
  • Since it’s embedded within the IP stack, it would require kernel level changes
  • At best, provides data-origin authentication. This is network to network authentication. You aren’t authenticating a particular user but anyone that could be using that device
  • With an IPsec VPN, it doesn’t allow for application filtering. You have access to the full network (i.e. you are part  of the corporate network)
  • Both devices must be configured to talk to each other and require the use of third party hardware and software
  • Configuring IPsec and IKE is complex and cumbersome

When to use

IPsec VPNs are still the most widely used, especially for larger corporations.  The computer connecting via an IPsec VPN is now part of the corporate network and has the exact same access as if he was at his office. Whenever I needed to connect remotely to work from either home or traveling, I would use an IPsec VPN to connect.

You will see IPsec used more for secure communication between devices (such as proxy servers, security gateways, edge routers) than any other network protocol.  Since we don’t require user authentication in these instances, data-origin authentication is sufficient.

In 3GPP networks, IPsec was chosen over TLS for several reasons, primarily though that it doesn’t require PKI to implement and thus doesn’t require the computation overhead.  IPsec is also more versatile since it can secure TCP, UDP and SCTP


The real answer is that there is no better protocol.  They all have their benefits.  In order to decide which one to use, you really need to understand what you are trying to secure. Once you understand that question, the choice of what network security protocol to use will be easy enough!

Add to DeliciousAdd to DiggAdd to FaceBookAdd to Twitter

About the author

jojo.maalouf -

4 Responses to "Network Security Protocols: IPsec vs. TLS/SSL vs. SSH – Part II"

  1. Very interesting info, i am waiting for more ! Keep updating your site and you will have a lot o readers

  2. You post great posts, i have bookmarked for future referrence !

Do you have something to say?

Your email is never published nor shared.
Required fields are marked *

Anti-Spam Protection by WP-SpamFree