by Chandrasekar Balasubramanian

Network overlays is a method which creates additional layer of network abstraction providing new application or benefits. VxLAN is a Layer-2 overlay scheme. It runs over the existing network infrastructure and provides a means to extend a layer-2 network.

Let us see why we need VxLAN in the first place. In a typical networking environment, IP phones, hosts and servers are connected to the layer-2 devices like switch in a network. The switch in turn connects to other switches or to layer-3 devices like routers. Each host or server has a unique Media Access Control (MAC) address which is used in Layer-2 communication between the switch and the connected devices.  The switches maintain a MAC address table which helps a host, or a server connected to the switch to communicate with the other devices connected to the switch. This is achieved through the functionality of switching in the switch.  A switch is a layer-2 device having multiple ports, which receives data, processes it and forwards to the destination device based on the MAC address.

VLANs provide a logical segmentation by department, team or application.  E.g. a VLAN for HR, another VLAN for finance, another VLAN for engineering.  Hosts, servers and other devices connected to the switch become members of the VLAN by assignment of the ports (to which the hosts/servers are connected) to a VLAN.  Traffic originating in one host or server connected to a VLAN is only forwarded to the devices connected in the VLAN because it’s a single broadcast domain within the VLAN. And another VLAN has a different broadcast domain.  If you need traffic to be forwarded to another device which is connected to another VLAN, then either a layer-3 switch or a router needs to be used.

Challenges due to Server virtualization and Containers:

Due to server virtualization, multiple Virtual Machines (VM) residing on the server are assigned unique Media Access Control (MAC) addresses by the Virtualized server. By default, the server has its own MAC address. Likewise, a Docker also assigns unique MAC addresses to multiple containers. As you know the MAC address is a unique Layer-2 address assigned to each Ethernet device. In this case all the VMs are treated as Ethernet devices in addition to the virtualized server.  Likewise, all containers are treated as Ethernet devices in addition to the docker.  To recollect, the switch is the Layer-2 networking device which connects the servers and hosts to the network.  Switch maintains a MAC address table and a VLAN table.  We use Virtual trunk protocol (VTP) and trunk links to extend VLAN on one switch to all other switches. Dot1Q encapsulation is needed for the ethernet frames to be identified with VLANs.  I.e., VLAN information is stored in the header of the frame using Dot1Q encapsulation.  The encapsulation will add 4 bytes of tag to the frame. Within this 16-bit VLAN Id is stored in the frame. This will help for the receiving switch to identify for which VLANs the frame belongs to.

We require larger MAC address table due to hundreds of thousands of VMs.    We also require larger MAC address table due to hundreds of thousands of containers as well.  VMs in a data center are grouped according to the VLANs to which they belong to. Likewise, containers are grouped according to the VLANs to which they belong to.  One might need thousands of VLANs to take care to segment the traffic belonging to thousands of VMs into multiple VLANs.   Similarly, we need thousands of VLANs to segment the traffic belonging to thousands of containers into multiple VLANs. But there is a VLAN limit of 4094.  Clearly a better solution is needed more than what VLAN provides in this scaled environment. This is where VxLAN comes in.

Challenges due to Multi-tenant environment in Cloud:

As you know multi-tenancy is an architecture in which a single instance of software runs on a server and serves multiple customers. Cloud service providers provide on demand elastic provisioning of resources to multiple tenants by using the same physical infrastructure maintained by Cloud service provider. Isolation of traffic for a tenant can either be done via layer-2 or layer-3 networks. In the case of Layer-2 network, we can have each tenant to have its own VLANs.  Since many tenants are serviced by a cloud service provider, the number of VLANs required will be more than maximum number of VLANs supported which is 4094. One thing that aggravates the VLAN limitation is that each tenant will require multiple VLANs.  This is again where VxLAN comes in.

Limitations of Spanning Tree Protocol (STP):

Layer-2 switches use spanning tree protocols to avoid loops due to duplicate paths. STP blocks the ports so that loops are avoided.   Some of the ports end up being unused though the cost for the ports is paid. Also, there is no way to build resiliency using STP. Multipathing is not there in STP. An important requirement for a virtualized environment using a layer-2 network is to scale the layer-2 network is to scale across the data center or between data centers.  Using STP in such cases will lead to large number of disabled links due to loop detection. Newer mechanisms like Transparent interconnection of lot of links (TRILL) used in VxLAN helps in alleviating this problem.

Top of the Rack (TOR) switch limitation: The top of the rack switches which connect to the servers need to also learn and maintain an address table about the MAC addresses of Virtual machines within the servers and containers within the servers.  In some large environments where are there are several thousands of VMs and containers, it can lead to overflow of the MAC address table maintained by the Top of the rack switches leading to issues like stoppage of learning and flooding of unknown destination frames.

Virtual eXtensible Local Area Network (VxLAN):

As you know, Network overlays is a method which creates additional layer of network abstraction providing new application or benefits. VxLAN is a Layer-2 overlay scheme. It runs over the existing network infrastructure and provides a means to extend a layer-2 network. It’s a layer-2 overlay scheme on a layer-3 network.  Each overlay is a VxLAN segment. Only VM or a network device or a container connected to the same VxLAN segment can communicate with each other. Each VxLAN segment is identified with a 24-bit segment ID.  Here, in the following diagram we can see that the Layer-2 Ethernet frame is sent as part of a Layer-3 packet. Outer MAC address is the MAC address of the packet carrying the Overlayed Layer-2 packet. Outer IP header and Outer UDP header are the IP and UDP header of the packet carrying the OVerlayed Layer-2 packet. VxLAN Header is the header which facilities to identify and segment based on VxLAN information.  Then we have the overlayed Ethernet frame followed by checksum.

VxLAN Header:

Each VxLAN segment is identified by a 24-bit segment ID. This is called VxLAN Network Identifier (VNI). This allows up to 16 Milliion VxLAN segments to co-exist within the same administrative domain. VxLAN hence overcoems the limitation of 4094 VLANs.

VxLAN Tunnel Endpoint (VTEP):

VTEPs does all the work related to VxLAN in terms of encapsulation and de-encapsulation which makes the VxLAN overlay of Layer-2 over Layer-3 work.

VTEP is the endpoint of the tunnel which is located within the hypervisor on the VM server which hosts the VM. Likewise in containerized environment the VTEP endpoint is located within the docker machine.

VxLAN Deployment:

VxLAN is deployed in either data centers or cloud environments where containers and/or virtual machines are used.

VxLAN Security: Layer-2 Over Layer-3 overlay mechanism used in VxLAN increases the attack surface. The Layer-3 attack on the tunnelled traffic can be secured by using IPSec which authenticates and also encrypts VxLAN Traffic. Layer-2 attacks can be mitigated using 802.1x authentication.

VxLAN Monitoring:

VxLAN monitoring and visibility is an open area where there are not much of tools and existing tools also needs to mature.  We at GAVS Technologies, believe that VxLAN is an important Technology that will be used more in future and we need to support it fully with a Monitoring tool. We also  can support Managd services of Network infrastructures involving VxLAN environments

Conclusion:. VxLAN is a great technology which helps to overcome the limitations of VLAN in a scaled environment involving Virtual machines and Containers.