Enhancing Internet connectivity with MultiPath TCP

A wide variety of access network technologies have been proposed, implemented and standardised to enable a growing number of users to access the Internet. The early dial-up networks have been replaced by a range of Digital Subscriber Line (DSL) technologies. The coaxial cable that was initially deployed to only distribute television has been upgraded to carry Internet services. Satellite services and most mobile networks are now used to carry more and more Internet services. Finally, various forms of fiber have started to be deployed.

Despite the variety of access networks, most users still depend on a single access technology, typically a wired solution at home and a wireless network on their smartphones. This is mainly because the Internet protocols were designed with the assumption that each computer is attached to a network through a single link. This assumption was valid in the early 1970s when few computers had a network interface.

Fig. 1 – Illustration of a TCP limitation

Today, this is not true since any smartphone or laptop is equipped with two or more network interfaces. Although these devices only use one access network at a given time, this has opened up new possibilities.

About 10 years ago a group of researchers¹ started to work on a new protocol to support devices which are equipped with several network interfaces: Multipath TCP. Multipath TCP is an extension to the Transmission Control Protocol (TCP). TCP and IP are the core Internet protocols. Thanks to IP, any host can send packets to any destination.TCP ensures the reliable delivery of the data through the Internet and automatically adjusts its bandwidth in response to network load. It is used by most Internet applications. Multipath TCP extends TCP by enabling it to be used on devices that are equipped with two or more network interfaces. Thanks to Multipath TCP, it is possible to simultaneously use different network interfaces or seamlessly switch from one to another without interruption. Multipath TCP has been discussed over the last four years within the Internet Engineering Task Force (IETF). In January 2013, this standardisation body published the complete specification for Multipath TCP in RFC6824. The publication of  RFC6824 was an important milestone since it indicated that the protocol was stable and that the industry could start to leverage it to deploy new services. In September 2013, Apple deployed Multipath TCP on all its iPhones to support the Siri voice recognition application. The main benefit of Multipath TCP for this application is that it allows switching seamlessly from WiFi and cellular as we demonstrated earlier.

Another very important benefit of Multipath TCP is that it allows us to aggregate the bandwidth of different links to provide a faster aggregated pipe. In early 2013, we demonstrated that Multipath TCP could reach more than 50 Gbps by aggregating six 10 Gbps Ethernet interfaces on a high-end server using our Linux implementation. This confirmed the performance and maturity of this new protocol.

However, the benefits of Multipath TCP can only be realised by installing this new protocol on all Internet connected devices. Apple was able to perform this upgrade on all iPhones, but no public Internet server supported Multipath TCP at that time.

In December 2013, we proposed an efficient end-to-end solution running on home gateways and generic servers to solve this deployment problem. This is illustrated in the figure below that appeared in the original paper. We introduced an efficient Multipath TCP proxy (depicted as M in the figure below) that enables clients (C in the figure) to use Multipath TCP in the access networks while using TCP to interact with the legacy server. We modified Multipath TCP to include the server address in the connection establishment packet that is sent to the Multipath TCP proxy. We also proposed an efficient implementation for such Multipath TCP proxies that leverages the Linux kernel.

Fig. 2 – Multipath TCP deployment with a proxy

We explored various use cases for Multipath TCP and in 2014 our discussion with network operators indicated that some of them were investigating the possibility of combining their xDSL and LTE networks to provide higher bandwidth services to users that are connected with a long DSL line. We could quickly port our Multipath TCP proxy to existing DSL routers and demonstrated MPTCP benefits . This led to the commercialisation of our Hybrid access solution.

MPTCP deployments

Description of the SOCKS protocol

SOCKS was initially designed to let computers in an enterprise network connect to Internet servers through an explicit proxy. Its operation is illustrated in the figure 3 below assuming the Korea Telecom deployment where SOCKS is used on smartphones.

Fig. 3 – Establishment of a TCP connection through a SOCKS proxy

In the above figure, the client is attached to two network interfaces. The SOCKS proxy has one address (@p).

To establish a connection to the server, the smartphone first needs to create a connection to the SOCKS proxy. This enables it to use Multipath TCP on the WiFi and LTE networks, but using SOCKS delays the establishment of all connections to a remote server because the smartphone must first initiate a Multipath TCP connection to the proxy, then negotiate the version of SOCKS that will be used and finally inform the proxy of the address of the remote server. As shown above, SOCKS is a ‘chatty’ protocol and its usage increases the connection establishment time by two round-trip-times (RTT) between the smartphone and the proxy. In today’s Internet where latency is a key concern, adding several tens of millisecond to each TCP connection establishment attempt cannot be seen as an improvement, especially as many web applications require frequent but short data transfers.

Another problem with using SOCKS above the open-source implementation of Multipath TCP is that, by default, this implementation tries to use all the available network interfaces and prefers the one with the lowest delay. Most network operators prefer to use the DSL network first and only start to use the LTE network once DSL is fully utilized. We solved this problem with a flexible path manager that received an Applied Networking Research Prize from the IRTF.

A third operational issue with SOCKS is that all the connections that reach the Server appear to have been initiated by the Proxy. This implies that the SOCKS proxy operates like a carrier grade NAT. in many countries, law enforcement agencies are concerned that NATs may make it more difficult to detect criminal activities. However, it is still possible for operators to detect criminal activities in this case but that requires more IP addresses (in the case of IPv4).

Examples of implementations of SOCKS protocol

In July 2015, Korea Telecom, announced that they were using Multipath TCP on smartphones to combine their WiFi and LTE networks to reach bandwidths of up to 1 Gbps.

In September 2015, OVH announced the Overthebox service that combines several low speed lines such as DSL together. In both cases, Multipath TCP provides more bandwidth and more redundancy than using a single network.

In November 2016, Swisscom launched their hybrid broadband package called Internet Booster that delivers download speeds up to 40 Mbps to customers suffering from slow Internet services.

Reducing latency with the 0-RTT Convert protocol

Given the known limitations of SOCKS, Tessares did not consider it a viable approach when designing Hybrid Access Networks. Instead, Tessares partnered with network operators, starting with Proximus Belgium, to develop new solutions that can be efficiently implemented and meet the operator and law enforcement requirements. Our first solution was demonstrated at the Broadband World Forum in late 2015 where it received an award.

We use the  terminology that the Broadband Forum adopted in TR-348 to specify the architecture for Hybrid Access Networks. The CPE that embeds our software is called a Hybrid CPE (HCPE). We assume that a single HCPE having both DSL and LTE interfaces is used, although several deployments use two different CPEs that are connected through a high speed network interface such as fast or gigabit Ethernet. This HCPE includes a transparent Multipath TCP proxy that converts the TCP connections established by the attached clients into Multipath TCP connections. The second element of the architecture defined in TR-348 is the Hybrid Access Gateway (HAG). This network function must terminate the Multipath TCP connections initiated by the HCPE over the DSL and LTE networks.

Fig. 4 – Architecture of Hybrid Access Broadband Network Architecture

Our first innovation was to place the HAG on the path followed by the packets of the DSL network. This requires some routing configuration on the DSL network, but significantly simplifies both the architecture and the operation of hybrid networks. From an architectural viewpoint, the HAG is simply a transparent proxy that converts Multipath TCP connections into TCP connections. It is equivalent to the software used on the HCPE, except that it must operate at a larger scale since it handles the connections for thousands of HCPEs. From an operational viewpoint, this approach has two key benefits compared to the SOCKS-based approach. First, there is no additional delay for the establishment of TCP connections, this is very important for interactive web-based applications. Second, the HAG does not perform any network translation.

This deployment model has been presented at the IETF. It has later been adopted as “implicit mode” in the WT-378 specification that will be published soon by the Broadband Forum. See figure 5. It has already been commercially deployed by network operators in four different countries and two new customers will be unveiled in the first half of 2019.

The client attached to the HCPE initiates a TCP connection by sending a SYN packet. The HCPE converts this TCP connection establishment attempt into a Multipath TCP connection attempt and forwards it over the DSL network. The HAG intercepts the connection attempt and transparently terminates the Multipath TCP connection. It then initiates a connection towards the final destination server. The Multipath TCP connection to the HCPE is only confirmed once the server has accepted the connection attempt. The same applies to the connection between the Client and the HCPE. Within a single RTT, the client has established a connection with the remote server. This connection is transparently proxied on the HCPE and the HAG and there is no address translation on the HAG. Shortly after the establishment of the Multipath TCP connection, the HAG advertises its address on the LTE network (@h2) to the HCPE which can later decide, e.g. based on the load of the DSL link, to create a subflow over the LTE interface to also use it.

Fig. 5 – Establishment of a TCP connection through the implicit mode

In parallel with this first deployment model, together with Orange and other network operators we explored the possibility of standardizing Multipath TCP proxies within the Internet Engineering Task Force. The starting point for this work was the idea of placing the destination address of the server in the SYN packet as we already proposed 2013. After many discussions, the IETF did not agree to to adopt this plain-mode approach.

We then went back to the drawing board and studied in detail the comments raised by the IETF and proposed the 0-RTT Convert protocol, our second innovation. This new solution leverages the recently standardised TCP Fast Open and encodes the destination address as well as additional information in the payload of the initial SYN packet. It has been accepted as a working group document within the TCPM working group. Its scope is broader than Multipath TCP proxies, but we focus here on this use case which appears to be the most important one today. From a deployment viewpoint, the main benefit of this approach is that the HAG does not need to reside on the DSL path. This simplifies the routing configuration of the DSL network and eases the scaling of the HAGs. Furthermore, it is possible to deploy such a HAG without performing any network address translation on the HAGs. This new deployment model is the second model that is recommended by the Broadband Forum as the explicit mode of operation in the forthcoming WT-378 specification.

Its operation is illustrated in the figure below. The client initiates a TCP connection by sending a SYN packet. This connection attempt is proxied by the HCPE that initiates a Multipath TCP connection towards the HAG (address @h1). This connection attempt includes two TCP options: MP_CAPABLE (Multipath TCP) and TFO  (TCP Fast Open) as well as a 0-rtt Convert message that provides the address of the destination server. The main benefit of this deployment model is that since the SYN is directly addressed to the HAG, the HAG can be located anywhere inside the network. The HAG extracts the server address from the payload of the SYN packet and then initiates the connection towards the final server. As in the implicit deployment model, the client can establish a connection with a server within only one RTT, three RTT less than SOCKS protocol.

Fig. 6 – Establishment of a TCP connection via the 0-RTT Convert protocol

Our R&D team performed tests to measure the performance related to latency for the time to exchange SOCKSv5 commands between the client and SOCKS Proxy. It is noteworthy that the last 15% of connections had RTTs higher than 50ms, thus increasing latency by a minimum of 150ms due to the SOCKSv5 protocol. More details available in this blog post.

References :