Quickstart Cisco BGP EVPN with Terraform

BGP EVPN VXLAN can be deployed in campus networks with Cisco Catalyst 9000 Series Switches running Cisco IOS XE software. This solution is a result of proposed IETF standards and Internet drafts submitted by the BGP Enabled ServiceS (bess) workgroup. It is designed to provide a unified overlay network solution and also address the challenges and drawbacks of existing technologies while providing Layer 2 and Layer 3 network services.

Network Configuration State with Terraform

Terraform allows you to keep the state of the configuration that you configured for your specific resources. The state is stored in terraform.tfstate file locally, but it can also be stored remotely. This helps when working in a team environment and managing the same resources.
Here’s how you install Terraform CLI.

Example of a Terraform resource with Cisco BGP EVPN Provider

resource "ciscoevpn_bgp_neighbor" "leafs" {
  depends_on = [
    ciscoevpn_bgp_system.ibgp,
    ciscoevpn_loopback.leaf1_100,
    ciscoevpn_loopback.leaf2_100,
  ]
  roles  = ["leafs"]
  bgp_id = ciscoevpn_bgp_system.ibgp.bgp_id
  neighbors = [
    "${ciscoevpn_loopback.spine1_100.ipv4_address}",
    "${ciscoevpn_loopback.spine2_100.ipv4_address}",
  ]
  update_source  = ciscoevpn_loopback.leaf1_100.interface_name
  remote_as      = ciscoevpn_bgp_system.ibgp.bgp_id
  send_community = "both"
  activate       = true
  l2vpn_evpn     = true
}

Example of a Terraform state

{
 "version": 4,
 "terraform_version": "1.1.9",
 "serial": 2,
 "lineage": "1bedee12-cff9-4d21-58a0-457e8e8336c0",
 "outputs": {},
 "resources": [{
  "mode": "managed",
  "type": "ciscoevpn_bgp_neighbor",
  "name": "leafs_borders",
  "provider": "provider[\"registry.terraform.io/robertcsapo/ciscoevpn\"]",
  "instances": [{
   "schema_version": 0,
   "attributes": {
    "activate": true,
    "bgp_id": 65534,
    "id": "bgp_id_65534",
    "l2vpn_evpn": true,
    "neighbors": [
     "100.119.11.1",
     "100.119.11.2"
    ],
    "remote_as": 65534,
    "response": null,
    "roles": [
     "leafs",
     "borders"
    ],
    "route_reflector_client": false,
    "send_community": "both",
    "update_source": "Loopback100"
   },
   "sensitive_attributes": [],
   "private": "bnVsbA==",
   "dependencies": [
    "ciscoevpn_bgp_system.ibgp",
    "ciscoevpn_loopback.border1_100",
    "ciscoevpn_loopback.border2_100",
    "ciscoevpn_loopback.leaf1_100",
    "ciscoevpn_loopback.leaf2_100",
    "ciscoevpn_loopback.spine1_100",
    "ciscoevpn_loopback.spine2_100"
   ]
  }]
 }]
}

The benefit is that we gather values from different Terraform resources, without needing to manually enter the same value in several places.

bgp_id = ciscoevpn_bgp_system.ibgp.bgp_id

<RESOURCE TYPE>.<NAME> represents a managed resource of the given type and name. In this case, ciscoevpn is the Terraform Provider,bgp_system is the resource and ibgp is the instance we want to create.

resource "ciscoevpn_bgp_system" "ibgp" {
  depends_on = [
    ciscoevpn_loopback.spine1_100,
    ciscoevpn_loopback.spine2_100,
    ciscoevpn_loopback.leaf1_100,
    ciscoevpn_loopback.leaf2_100,
  ]
  roles                = ["spines", "leafs"]
  router_id            = ciscoevpn_loopback.spine1_100.interface_name
  bgp_id               = 65534
  log_neighbor_changes = true
  default_ipv4_unicast = false
}

ciscoevpn_bgp_system.ibgp is the resource we will create here and we can enter the BGP id (ASN) once and re-use that value across our configuration.

Cisco EVPN Terraform Provider with Catalyst 9000 Switching

The terraform-provider-ciscoevpn allows Terraform to leverage YANG models when deploying EVPN with BGP.

Prerequisites:

• Catalyst 9000 Switching Platforms
• Cisco IOS-XE 17.8+
• Cisco Network Advantage / DNA Advantage
• Underlay network (connectivity between all the nodes)
• RESTCONF enabled
• Terraform CLI

Getting started with a template config

I’ve created a GitHub repository for simplicity that includes the basic configuration needed to deploy BGP EVPN.

git clone https://github.com/robertcsapo/cisco-iac-c9k.git
cd cisco-iac-c9k/evpn/terraform/tutorial/

There will be a folder called tutorial under evpn/terraform in this repository that we will follow.

The base structure of the configuration

In this folder, you will several files ending with .tf suffix.

bgp.tf  evpn.tf  loopback.tf  main.tf  nve.tf  svi.tf  variables.tf  vlan.tf  vrf.tf

If you look at the main.tf file you’ll see the base configuration in Terraform for the ciscoevpn provider.

terraform {
  required_providers {
    ciscoevpn = {
      source  = "robertcsapo/ciscoevpn"
      version = "1.0.0"
    }
  }
}
# ciscoevpn provider settings
provider "ciscoevpn" {
  username = var.username
  password = var.password
  insecure = var.insecure
  timeout  = var.timeout
  debug    = false
  roles {
    spines {
      iosxe = var.iosxe_spines
    }
    borders {
      iosxe = var.iosxe_borders
    }
    leafs {
      iosxe = var.iosxe_leafs
    }
  }
}

required_providers will make Terraform download the ciscoevpn provider from Terraform Public registry. Best practice is to pin the provider to the preferred version, else it will use the latest version.

Roles are required for the provider to know what kind of configuration it should push and configure onto the devices.
These roles are required, spines , leafs . But borders role is optional.

The network devices are configured in the .auto.tfvars that assign ip addresses to the selected network devices in the preferred role.

iosxe_spines = [
    "100.119.1.1",
    "100.119.1.2",
]
iosxe_leafs = [
    "100.119.1.3",
    "100.119.1.3",
]

username ,password , insecure , timeout can be configured through environments or in a Terraform file. In this case, we will use variables.tf, and these settings are common for all the network devices.

variable "username" {
  type        = string
  default     = "developer"
  description = "IOS-XE Username"
}
variable "password" {
  type        = string
  default     = "C1sco12345"
  description = "IOS-XE Passwordd"
}
variable "insecure" {
  type        = bool
  default     = true
  description = "IOS-XE HTTPS insecure"
}
variable "timeout" {
  type        = number
  default     = 120
  description = "IOS-XE Client timeout"
}
variable "iosxe_spines" {
  type        = list(string)
  default     = []
  description = "IOS-XE Devices as Spines"
}
variable "iosxe_leafs" {
  type        = list(string)
  default     = []
  description = "IOS-XE Devices as Leafs"
}
variable "iosxe_borders" {
  type        = list(string)
  default     = []
  description = "IOS-XE Devices as Borders"
}

Download the ciscoevpn Terraform Provider

Execute the following command to verify that Terraform can execute our configuration.

terraform init

This will result in either downloading the plugin (ciscoevpn provider) or verifying that existing downloaded plugins match the shasum for the selected provider against Terraform Registry.

Configuring BGP EVPN using ciscoevpn

There are several files that build the configuration for this BGP EVPN example that we have in this repository.

This example will construct Layer 3 VNI and Layer 2 VNI.

All the configuration is done in these files.

bgp.tf
evpn.tf
loopback.tf
nve.tf
svi.tf
vlan.tf
vrf.tf

We can use the terraform plan command to get the overall configuration before it’s deployed.

Based on this, you can adjust the .tf files to match the environment that you have.

Deploy the network configuration for BGP EVPN

Run the terraform apply command to prepare to deploy the configuration to the network devices.

Similar to terraform plan, we will get the overall configuration for the environment. But in this case, proceed with this change by approving and typing yes.

The benefit of our configuration is that we have applied depends_on in our Terraform resources (configuration). This adds the logic for Terraform to understand in which order it should create those resources and allows us to wait until specific resources are created.

We have now successfully deployed our BGP EVPN on Cisco Catalyst 9000 Switches and we have Layer 2 VNI and Layer 3 VNI with VXLAN configured on our Leafs and Spines.

Apply complete! Resources: 27 added, 0 changed, 0 destroyed.

You can verify the EVPN operational status in one of the leafs.

show bgp l2vpn evpn summary

show nve peers

show l2fib bridge-domain detail

Updating existing network configuration state

If we look at our terraform.tfstate file in the local folder, we can see the current assumed state we have configured on our network devices.

We can modify the state by making changes or removing resources that we have configured, by modifying the Terraform configuration (.tf files)

On the Layer 2 VNI, we want to change the IPv4 Network in svi.tf

resource "ciscoevpn_svi" "svi_102" {
  depends_on   = [ciscoevpn_vrf.blue]
  roles        = ["leafs"]
  svi_id       = 102
  autostate    = false
  vrf          = ciscoevpn_vrf.blue.name
  ipv4_address = "100.119.202.1"
  ipv4_mask    = "255.255.255.0"
}

Once we run terraform apply for our configuration, Terraform will flag the changes before we approve them.

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

The configuration of VLAN 102 is now updated.

#show running-config interface vlan 102
Building configuration...
Current configuration : 146 bytes
!
interface Vlan102
 description Managed by Terraform (ciscoevpn)
 vrf forwarding blue
 ip address 100.119.202.1 255.255.255.0
 no autostate
end

Lastly, we want to remove the Layer 3 VNI routing for VRF blue, hence we’ll remove this configuration from svi.tf by deleting the lines in the file.

resource "ciscoevpn_svi" "svi_104" {
  depends_on = [ciscoevpn_vrf.blue]
  roles      = ["leafs"]
  svi_id     = 104
  autostate  = false
  vrf        = ciscoevpn_vrf.blue.name
  unnumbered = ciscoevpn_loopback.leaf1_200.interface_name
}

Using terraform apply again we can see that Terraform is going to destroy (remove) our Layer 3 VNI resource called svi_104. But it will keep the remaining configuration of the EVPN Fabric.

Apply complete! Resources: 0 added, 0 changed, 1 destroyed.

Removing all network configurations for BGP EVPN

It’s not more complicated than asking Terraform to delete all the resources by using terraform destroy. This will remove all managed resources in the Terraform state file.

Confirm that you want to proceed by removing these resources (network configuration), by typing yes

It will use the dependency logic again mapped out in the terraform.tfstate file and remove the resources in the reversed order.

Destroy complete! Resources: 26 destroyed.

You’ve now successfully deleted all your BGP EVPN network configuration that was deployed with Terraform on your Catalyst 9000 Switches running IOS-XE.

For more information

• Why Transition to BGP EVPN VXLAN in Enterprise Campus?
• Next-Generation Enterprise Campus Network Virtualization with BGP EVPN VXLAN
• VXLAN EVPN — De-Facto Standard, Open and Interoperable
• BGP EVPN VXLAN Configuration Guide, Cisco IOS XE Cupertino 17.8.x
• Cisco BGP EVPN Terraform Provider on GitHub
• Cisco Catalyst 9000 EVPN Terraform Example on GitHub
• Cisco BGP EVPN with Ansible on GitHub

Disclaimer: I work for Cisco as a Product Manager
These articles are my own and not Cisco’s