1Virtual Routing and Forwarding (VRF)
2====================================
3The VRF device combined with ip rules provides the ability to create virtual
4routing and forwarding domains (aka VRFs, VRF-lite to be specific) in the
5Linux network stack. One use case is the multi-tenancy problem where each
6tenant has their own unique routing tables and in the very least need
7different default gateways.
8
9Processes can be "VRF aware" by binding a socket to the VRF device. Packets
10through the socket then use the routing table associated with the VRF
11device. An important feature of the VRF device implementation is that it
12impacts only Layer 3 and above so L2 tools (e.g., LLDP) are not affected
13(ie., they do not need to be run in each VRF). The design also allows
14the use of higher priority ip rules (Policy Based Routing, PBR) to take
15precedence over the VRF device rules directing specific traffic as desired.
16
17In addition, VRF devices allow VRFs to be nested within namespaces. For
18example network namespaces provide separation of network interfaces at the
19device layer, VLANs on the interfaces within a namespace provide L2 separation
20and then VRF devices provide L3 separation.
21
22Design
23------
24A VRF device is created with an associated route table. Network interfaces
25are then enslaved to a VRF device:
26
27         +-----------------------------+
28         |           vrf-blue          |  ===> route table 10
29         +-----------------------------+
30            |        |            |
31         +------+ +------+     +-------------+
32         | eth1 | | eth2 | ... |    bond1    |
33         +------+ +------+     +-------------+
34                                  |       |
35                              +------+ +------+
36                              | eth8 | | eth9 |
37                              +------+ +------+
38
39Packets received on an enslaved device and are switched to the VRF device
40in the IPv4 and IPv6 processing stacks giving the impression that packets
41flow through the VRF device. Similarly on egress routing rules are used to
42send packets to the VRF device driver before getting sent out the actual
43interface. This allows tcpdump on a VRF device to capture all packets into
44and out of the VRF as a whole.[1] Similarly, netfilter[2] and tc rules can be
45applied using the VRF device to specify rules that apply to the VRF domain
46as a whole.
47
48[1] Packets in the forwarded state do not flow through the device, so those
49    packets are not seen by tcpdump. Will revisit this limitation in a
50    future release.
51
52[2] Iptables on ingress supports PREROUTING with skb->dev set to the real
53    ingress device and both INPUT and PREROUTING rules with skb->dev set to
54    the VRF device. For egress POSTROUTING and OUTPUT rules can be written
55    using either the VRF device or real egress device.
56
57Setup
58-----
591. VRF device is created with an association to a FIB table.
60   e.g, ip link add vrf-blue type vrf table 10
61        ip link set dev vrf-blue up
62
632. An l3mdev FIB rule directs lookups to the table associated with the device.
64   A single l3mdev rule is sufficient for all VRFs. The VRF device adds the
65   l3mdev rule for IPv4 and IPv6 when the first device is created with a
66   default preference of 1000. Users may delete the rule if desired and add
67   with a different priority or install per-VRF rules.
68
69   Prior to the v4.8 kernel iif and oif rules are needed for each VRF device:
70       ip ru add oif vrf-blue table 10
71       ip ru add iif vrf-blue table 10
72
733. Set the default route for the table (and hence default route for the VRF).
74       ip route add table 10 unreachable default metric 4278198272
75
76   This high metric value ensures that the default unreachable route can
77   be overridden by a routing protocol suite.  FRRouting interprets
78   kernel metrics as a combined admin distance (upper byte) and priority
79   (lower 3 bytes).  Thus the above metric translates to [255/8192].
80
814. Enslave L3 interfaces to a VRF device.
82       ip link set dev eth1 master vrf-blue
83
84   Local and connected routes for enslaved devices are automatically moved to
85   the table associated with VRF device. Any additional routes depending on
86   the enslaved device are dropped and will need to be reinserted to the VRF
87   FIB table following the enslavement.
88
89   The IPv6 sysctl option keep_addr_on_down can be enabled to keep IPv6 global
90   addresses as VRF enslavement changes.
91       sysctl -w net.ipv6.conf.all.keep_addr_on_down=1
92
935. Additional VRF routes are added to associated table.
94       ip route add table 10 ...
95
96
97Applications
98------------
99Applications that are to work within a VRF need to bind their socket to the
100VRF device:
101
102    setsockopt(sd, SOL_SOCKET, SO_BINDTODEVICE, dev, strlen(dev)+1);
103
104or to specify the output device using cmsg and IP_PKTINFO.
105
106By default the scope of the port bindings for unbound sockets is
107limited to the default VRF. That is, it will not be matched by packets
108arriving on interfaces enslaved to an l3mdev and processes may bind to
109the same port if they bind to an l3mdev.
110
111TCP & UDP services running in the default VRF context (ie., not bound
112to any VRF device) can work across all VRF domains by enabling the
113tcp_l3mdev_accept and udp_l3mdev_accept sysctl options:
114
115    sysctl -w net.ipv4.tcp_l3mdev_accept=1
116    sysctl -w net.ipv4.udp_l3mdev_accept=1
117
118These options are disabled by default so that a socket in a VRF is only
119selected for packets in that VRF. There is a similar option for RAW
120sockets, which is enabled by default for reasons of backwards compatibility.
121This is so as to specify the output device with cmsg and IP_PKTINFO, but
122using a socket not bound to the corresponding VRF. This allows e.g. older ping
123implementations to be run with specifying the device but without executing it
124in the VRF. This option can be disabled so that packets received in a VRF
125context are only handled by a raw socket bound to the VRF, and packets in the
126default VRF are only handled by a socket not bound to any VRF:
127
128    sysctl -w net.ipv4.raw_l3mdev_accept=0
129
130netfilter rules on the VRF device can be used to limit access to services
131running in the default VRF context as well.
132
133################################################################################
134
135Using iproute2 for VRFs
136=======================
137iproute2 supports the vrf keyword as of v4.7. For backwards compatibility this
138section lists both commands where appropriate -- with the vrf keyword and the
139older form without it.
140
1411. Create a VRF
142
143   To instantiate a VRF device and associate it with a table:
144       $ ip link add dev NAME type vrf table ID
145
146   As of v4.8 the kernel supports the l3mdev FIB rule where a single rule
147   covers all VRFs. The l3mdev rule is created for IPv4 and IPv6 on first
148   device create.
149
1502. List VRFs
151
152   To list VRFs that have been created:
153       $ ip [-d] link show type vrf
154         NOTE: The -d option is needed to show the table id
155
156   For example:
157   $ ip -d link show type vrf
158   11: mgmt: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
159       link/ether 72:b3:ba:91:e2:24 brd ff:ff:ff:ff:ff:ff promiscuity 0
160       vrf table 1 addrgenmode eui64
161   12: red: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
162       link/ether b6:6f:6e:f6:da:73 brd ff:ff:ff:ff:ff:ff promiscuity 0
163       vrf table 10 addrgenmode eui64
164   13: blue: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
165       link/ether 36:62:e8:7d:bb:8c brd ff:ff:ff:ff:ff:ff promiscuity 0
166       vrf table 66 addrgenmode eui64
167   14: green: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
168       link/ether e6:28:b8:63:70:bb brd ff:ff:ff:ff:ff:ff promiscuity 0
169       vrf table 81 addrgenmode eui64
170
171
172   Or in brief output:
173
174   $ ip -br link show type vrf
175   mgmt         UP             72:b3:ba:91:e2:24 <NOARP,MASTER,UP,LOWER_UP>
176   red          UP             b6:6f:6e:f6:da:73 <NOARP,MASTER,UP,LOWER_UP>
177   blue         UP             36:62:e8:7d:bb:8c <NOARP,MASTER,UP,LOWER_UP>
178   green        UP             e6:28:b8:63:70:bb <NOARP,MASTER,UP,LOWER_UP>
179
180
1813. Assign a Network Interface to a VRF
182
183   Network interfaces are assigned to a VRF by enslaving the netdevice to a
184   VRF device:
185       $ ip link set dev NAME master NAME
186
187   On enslavement connected and local routes are automatically moved to the
188   table associated with the VRF device.
189
190   For example:
191   $ ip link set dev eth0 master mgmt
192
193
1944. Show Devices Assigned to a VRF
195
196   To show devices that have been assigned to a specific VRF add the master
197   option to the ip command:
198       $ ip link show vrf NAME
199       $ ip link show master NAME
200
201   For example:
202   $ ip link show vrf red
203   3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000
204       link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
205   4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000
206       link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
207   7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN mode DEFAULT group default qlen 1000
208       link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff
209
210
211   Or using the brief output:
212   $ ip -br link show vrf red
213   eth1             UP             02:00:00:00:02:02 <BROADCAST,MULTICAST,UP,LOWER_UP>
214   eth2             UP             02:00:00:00:02:03 <BROADCAST,MULTICAST,UP,LOWER_UP>
215   eth5             DOWN           02:00:00:00:02:06 <BROADCAST,MULTICAST>
216
217
2185. Show Neighbor Entries for a VRF
219
220   To list neighbor entries associated with devices enslaved to a VRF device
221   add the master option to the ip command:
222       $ ip [-6] neigh show vrf NAME
223       $ ip [-6] neigh show master NAME
224
225   For example:
226   $  ip neigh show vrf red
227   10.2.1.254 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE
228   10.2.2.254 dev eth2 lladdr 5e:54:01:6a:ee:80 REACHABLE
229
230   $ ip -6 neigh show vrf red
231   2002:1::64 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE
232
233
2346. Show Addresses for a VRF
235
236   To show addresses for interfaces associated with a VRF add the master
237   option to the ip command:
238       $ ip addr show vrf NAME
239       $ ip addr show master NAME
240
241   For example:
242   $ ip addr show vrf red
243   3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000
244       link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
245       inet 10.2.1.2/24 brd 10.2.1.255 scope global eth1
246          valid_lft forever preferred_lft forever
247       inet6 2002:1::2/120 scope global
248          valid_lft forever preferred_lft forever
249       inet6 fe80::ff:fe00:202/64 scope link
250          valid_lft forever preferred_lft forever
251   4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000
252       link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
253       inet 10.2.2.2/24 brd 10.2.2.255 scope global eth2
254          valid_lft forever preferred_lft forever
255       inet6 2002:2::2/120 scope global
256          valid_lft forever preferred_lft forever
257       inet6 fe80::ff:fe00:203/64 scope link
258          valid_lft forever preferred_lft forever
259   7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN group default qlen 1000
260       link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff
261
262   Or in brief format:
263   $ ip -br addr show vrf red
264   eth1             UP             10.2.1.2/24 2002:1::2/120 fe80::ff:fe00:202/64
265   eth2             UP             10.2.2.2/24 2002:2::2/120 fe80::ff:fe00:203/64
266   eth5             DOWN
267
268
2697. Show Routes for a VRF
270
271   To show routes for a VRF use the ip command to display the table associated
272   with the VRF device:
273       $ ip [-6] route show vrf NAME
274       $ ip [-6] route show table ID
275
276   For example:
277   $ ip route show vrf red
278   unreachable default  metric 4278198272
279   broadcast 10.2.1.0 dev eth1  proto kernel  scope link  src 10.2.1.2
280   10.2.1.0/24 dev eth1  proto kernel  scope link  src 10.2.1.2
281   local 10.2.1.2 dev eth1  proto kernel  scope host  src 10.2.1.2
282   broadcast 10.2.1.255 dev eth1  proto kernel  scope link  src 10.2.1.2
283   broadcast 10.2.2.0 dev eth2  proto kernel  scope link  src 10.2.2.2
284   10.2.2.0/24 dev eth2  proto kernel  scope link  src 10.2.2.2
285   local 10.2.2.2 dev eth2  proto kernel  scope host  src 10.2.2.2
286   broadcast 10.2.2.255 dev eth2  proto kernel  scope link  src 10.2.2.2
287
288   $ ip -6 route show vrf red
289   local 2002:1:: dev lo  proto none  metric 0  pref medium
290   local 2002:1::2 dev lo  proto none  metric 0  pref medium
291   2002:1::/120 dev eth1  proto kernel  metric 256  pref medium
292   local 2002:2:: dev lo  proto none  metric 0  pref medium
293   local 2002:2::2 dev lo  proto none  metric 0  pref medium
294   2002:2::/120 dev eth2  proto kernel  metric 256  pref medium
295   local fe80:: dev lo  proto none  metric 0  pref medium
296   local fe80:: dev lo  proto none  metric 0  pref medium
297   local fe80::ff:fe00:202 dev lo  proto none  metric 0  pref medium
298   local fe80::ff:fe00:203 dev lo  proto none  metric 0  pref medium
299   fe80::/64 dev eth1  proto kernel  metric 256  pref medium
300   fe80::/64 dev eth2  proto kernel  metric 256  pref medium
301   ff00::/8 dev red  metric 256  pref medium
302   ff00::/8 dev eth1  metric 256  pref medium
303   ff00::/8 dev eth2  metric 256  pref medium
304   unreachable default dev lo  metric 4278198272  error -101 pref medium
305
3068. Route Lookup for a VRF
307
308   A test route lookup can be done for a VRF:
309       $ ip [-6] route get vrf NAME ADDRESS
310       $ ip [-6] route get oif NAME ADDRESS
311
312   For example:
313   $ ip route get 10.2.1.40 vrf red
314   10.2.1.40 dev eth1  table red  src 10.2.1.2
315       cache
316
317   $ ip -6 route get 2002:1::32 vrf red
318   2002:1::32 from :: dev eth1  table red  proto kernel  src 2002:1::2  metric 256  pref medium
319
320
3219. Removing Network Interface from a VRF
322
323   Network interfaces are removed from a VRF by breaking the enslavement to
324   the VRF device:
325       $ ip link set dev NAME nomaster
326
327   Connected routes are moved back to the default table and local entries are
328   moved to the local table.
329
330   For example:
331   $ ip link set dev eth0 nomaster
332
333--------------------------------------------------------------------------------
334
335Commands used in this example:
336
337cat >> /etc/iproute2/rt_tables.d/vrf.conf <<EOF
3381  mgmt
33910 red
34066 blue
34181 green
342EOF
343
344function vrf_create
345{
346    VRF=$1
347    TBID=$2
348
349    # create VRF device
350    ip link add ${VRF} type vrf table ${TBID}
351
352    if [ "${VRF}" != "mgmt" ]; then
353        ip route add table ${TBID} unreachable default metric 4278198272
354    fi
355    ip link set dev ${VRF} up
356}
357
358vrf_create mgmt 1
359ip link set dev eth0 master mgmt
360
361vrf_create red 10
362ip link set dev eth1 master red
363ip link set dev eth2 master red
364ip link set dev eth5 master red
365
366vrf_create blue 66
367ip link set dev eth3 master blue
368
369vrf_create green 81
370ip link set dev eth4 master green
371
372
373Interface addresses from /etc/network/interfaces:
374auto eth0
375iface eth0 inet static
376      address 10.0.0.2
377      netmask 255.255.255.0
378      gateway 10.0.0.254
379
380iface eth0 inet6 static
381      address 2000:1::2
382      netmask 120
383
384auto eth1
385iface eth1 inet static
386      address 10.2.1.2
387      netmask 255.255.255.0
388
389iface eth1 inet6 static
390      address 2002:1::2
391      netmask 120
392
393auto eth2
394iface eth2 inet static
395      address 10.2.2.2
396      netmask 255.255.255.0
397
398iface eth2 inet6 static
399      address 2002:2::2
400      netmask 120
401
402auto eth3
403iface eth3 inet static
404      address 10.2.3.2
405      netmask 255.255.255.0
406
407iface eth3 inet6 static
408      address 2002:3::2
409      netmask 120
410
411auto eth4
412iface eth4 inet static
413      address 10.2.4.2
414      netmask 255.255.255.0
415
416iface eth4 inet6 static
417      address 2002:4::2
418      netmask 120
419