1
2
3                  HOWTO for the linux packet generator
4                  ------------------------------------
5
6Enable CONFIG_NET_PKTGEN to compile and build pktgen either in-kernel
7or as a module.  A module is preferred; modprobe pktgen if needed.  Once
8running, pktgen creates a thread for each CPU with affinity to that CPU.
9Monitoring and controlling is done via /proc.  It is easiest to select a
10suitable sample script and configure that.
11
12On a dual CPU:
13
14ps aux | grep pkt
15root       129  0.3  0.0     0    0 ?        SW    2003 523:20 [kpktgend_0]
16root       130  0.3  0.0     0    0 ?        SW    2003 509:50 [kpktgend_1]
17
18
19For monitoring and control pktgen creates:
20	/proc/net/pktgen/pgctrl
21	/proc/net/pktgen/kpktgend_X
22        /proc/net/pktgen/ethX
23
24
25Tuning NIC for max performance
26==============================
27
28The default NIC settings are (likely) not tuned for pktgen's artificial
29overload type of benchmarking, as this could hurt the normal use-case.
30
31Specifically increasing the TX ring buffer in the NIC:
32 # ethtool -G ethX tx 1024
33
34A larger TX ring can improve pktgen's performance, while it can hurt
35in the general case, 1) because the TX ring buffer might get larger
36than the CPU's L1/L2 cache, 2) because it allows more queueing in the
37NIC HW layer (which is bad for bufferbloat).
38
39One should hesitate to conclude that packets/descriptors in the HW
40TX ring cause delay.  Drivers usually delay cleaning up the
41ring-buffers for various performance reasons, and packets stalling
42the TX ring might just be waiting for cleanup.
43
44This cleanup issue is specifically the case for the driver ixgbe
45(Intel 82599 chip).  This driver (ixgbe) combines TX+RX ring cleanups,
46and the cleanup interval is affected by the ethtool --coalesce setting
47of parameter "rx-usecs".
48
49For ixgbe use e.g. "30" resulting in approx 33K interrupts/sec (1/30*10^6):
50 # ethtool -C ethX rx-usecs 30
51
52
53Kernel threads
54==============
55Pktgen creates a thread for each CPU with affinity to that CPU.
56Which is controlled through procfile /proc/net/pktgen/kpktgend_X.
57
58Example: /proc/net/pktgen/kpktgend_0
59
60 Running:
61 Stopped: eth4@0
62 Result: OK: add_device=eth4@0
63
64Most important are the devices assigned to the thread.
65
66The two basic thread commands are:
67 * add_device DEVICE@NAME -- adds a single device
68 * rem_device_all         -- remove all associated devices
69
70When adding a device to a thread, a corresponding procfile is created
71which is used for configuring this device. Thus, device names need to
72be unique.
73
74To support adding the same device to multiple threads, which is useful
75with multi queue NICs, the device naming scheme is extended with "@":
76 device@something
77
78The part after "@" can be anything, but it is custom to use the thread
79number.
80
81Viewing devices
82===============
83
84The Params section holds configured information.  The Current section
85holds running statistics.  The Result is printed after a run or after
86interruption.  Example:
87
88/proc/net/pktgen/eth4@0
89
90 Params: count 100000  min_pkt_size: 60  max_pkt_size: 60
91     frags: 0  delay: 0  clone_skb: 64  ifname: eth4@0
92     flows: 0 flowlen: 0
93     queue_map_min: 0  queue_map_max: 0
94     dst_min: 192.168.81.2  dst_max:
95     src_min:   src_max:
96     src_mac: 90:e2:ba:0a:56:b4 dst_mac: 00:1b:21:3c:9d:f8
97     udp_src_min: 9  udp_src_max: 109  udp_dst_min: 9  udp_dst_max: 9
98     src_mac_count: 0  dst_mac_count: 0
99     Flags: UDPSRC_RND  NO_TIMESTAMP  QUEUE_MAP_CPU
100 Current:
101     pkts-sofar: 100000  errors: 0
102     started: 623913381008us  stopped: 623913396439us idle: 25us
103     seq_num: 100001  cur_dst_mac_offset: 0  cur_src_mac_offset: 0
104     cur_saddr: 192.168.8.3  cur_daddr: 192.168.81.2
105     cur_udp_dst: 9  cur_udp_src: 42
106     cur_queue_map: 0
107     flows: 0
108 Result: OK: 15430(c15405+d25) usec, 100000 (60byte,0frags)
109  6480562pps 3110Mb/sec (3110669760bps) errors: 0
110
111
112Configuring devices
113===================
114This is done via the /proc interface, and most easily done via pgset
115as defined in the sample scripts.
116You need to specify PGDEV environment variable to use functions from sample
117scripts, i.e.:
118export PGDEV=/proc/net/pktgen/eth4@0
119source samples/pktgen/functions.sh
120
121Examples:
122
123 pg_ctrl start           starts injection.
124 pg_ctrl stop            aborts injection. Also, ^C aborts generator.
125
126 pgset "clone_skb 1"     sets the number of copies of the same packet
127 pgset "clone_skb 0"     use single SKB for all transmits
128 pgset "burst 8"         uses xmit_more API to queue 8 copies of the same
129                         packet and update HW tx queue tail pointer once.
130                         "burst 1" is the default
131 pgset "pkt_size 9014"   sets packet size to 9014
132 pgset "frags 5"         packet will consist of 5 fragments
133 pgset "count 200000"    sets number of packets to send, set to zero
134                         for continuous sends until explicitly stopped.
135
136 pgset "delay 5000"      adds delay to hard_start_xmit(). nanoseconds
137
138 pgset "dst 10.0.0.1"    sets IP destination address
139                         (BEWARE! This generator is very aggressive!)
140
141 pgset "dst_min 10.0.0.1"            Same as dst
142 pgset "dst_max 10.0.0.254"          Set the maximum destination IP.
143 pgset "src_min 10.0.0.1"            Set the minimum (or only) source IP.
144 pgset "src_max 10.0.0.254"          Set the maximum source IP.
145 pgset "dst6 fec0::1"     IPV6 destination address
146 pgset "src6 fec0::2"     IPV6 source address
147 pgset "dstmac 00:00:00:00:00:00"    sets MAC destination address
148 pgset "srcmac 00:00:00:00:00:00"    sets MAC source address
149
150 pgset "queue_map_min 0" Sets the min value of tx queue interval
151 pgset "queue_map_max 7" Sets the max value of tx queue interval, for multiqueue devices
152                         To select queue 1 of a given device,
153                         use queue_map_min=1 and queue_map_max=1
154
155 pgset "src_mac_count 1" Sets the number of MACs we'll range through.
156                         The 'minimum' MAC is what you set with srcmac.
157
158 pgset "dst_mac_count 1" Sets the number of MACs we'll range through.
159                         The 'minimum' MAC is what you set with dstmac.
160
161 pgset "flag [name]"     Set a flag to determine behaviour.  Current flags
162                         are: IPSRC_RND # IP source is random (between min/max)
163                              IPDST_RND # IP destination is random
164                              UDPSRC_RND, UDPDST_RND,
165                              MACSRC_RND, MACDST_RND
166                              TXSIZE_RND, IPV6,
167                              MPLS_RND, VID_RND, SVID_RND
168                              FLOW_SEQ,
169                              QUEUE_MAP_RND # queue map random
170                              QUEUE_MAP_CPU # queue map mirrors smp_processor_id()
171                              UDPCSUM,
172                              IPSEC # IPsec encapsulation (needs CONFIG_XFRM)
173                              NODE_ALLOC # node specific memory allocation
174                              NO_TIMESTAMP # disable timestamping
175 pgset 'flag ![name]'    Clear a flag to determine behaviour.
176                         Note that you might need to use single quote in
177                         interactive mode, so that your shell wouldn't expand
178                         the specified flag as a history command.
179
180 pgset "spi [SPI_VALUE]" Set specific SA used to transform packet.
181
182 pgset "udp_src_min 9"   set UDP source port min, If < udp_src_max, then
183                         cycle through the port range.
184
185 pgset "udp_src_max 9"   set UDP source port max.
186 pgset "udp_dst_min 9"   set UDP destination port min, If < udp_dst_max, then
187                         cycle through the port range.
188 pgset "udp_dst_max 9"   set UDP destination port max.
189
190 pgset "mpls 0001000a,0002000a,0000000a" set MPLS labels (in this example
191                                         outer label=16,middle label=32,
192					 inner label=0 (IPv4 NULL)) Note that
193					 there must be no spaces between the
194					 arguments. Leading zeros are required.
195					 Do not set the bottom of stack bit,
196					 that's done automatically. If you do
197					 set the bottom of stack bit, that
198					 indicates that you want to randomly
199					 generate that address and the flag
200					 MPLS_RND will be turned on. You
201					 can have any mix of random and fixed
202					 labels in the label stack.
203
204 pgset "mpls 0"		  turn off mpls (or any invalid argument works too!)
205
206 pgset "vlan_id 77"       set VLAN ID 0-4095
207 pgset "vlan_p 3"         set priority bit 0-7 (default 0)
208 pgset "vlan_cfi 0"       set canonical format identifier 0-1 (default 0)
209
210 pgset "svlan_id 22"      set SVLAN ID 0-4095
211 pgset "svlan_p 3"        set priority bit 0-7 (default 0)
212 pgset "svlan_cfi 0"      set canonical format identifier 0-1 (default 0)
213
214 pgset "vlan_id 9999"     > 4095 remove vlan and svlan tags
215 pgset "svlan 9999"       > 4095 remove svlan tag
216
217
218 pgset "tos XX"           set former IPv4 TOS field (e.g. "tos 28" for AF11 no ECN, default 00)
219 pgset "traffic_class XX" set former IPv6 TRAFFIC CLASS (e.g. "traffic_class B8" for EF no ECN, default 00)
220
221 pgset "rate 300M"        set rate to 300 Mb/s
222 pgset "ratep 1000000"    set rate to 1Mpps
223
224 pgset "xmit_mode netif_receive"  RX inject into stack netif_receive_skb()
225				  Works with "burst" but not with "clone_skb".
226				  Default xmit_mode is "start_xmit".
227
228Sample scripts
229==============
230
231A collection of tutorial scripts and helpers for pktgen is in the
232samples/pktgen directory. The helper parameters.sh file support easy
233and consistent parameter parsing across the sample scripts.
234
235Usage example and help:
236 ./pktgen_sample01_simple.sh -i eth4 -m 00:1B:21:3C:9D:F8 -d 192.168.8.2
237
238Usage: ./pktgen_sample01_simple.sh [-vx] -i ethX
239  -i : ($DEV)       output interface/device (required)
240  -s : ($PKT_SIZE)  packet size
241  -d : ($DEST_IP)   destination IP
242  -m : ($DST_MAC)   destination MAC-addr
243  -t : ($THREADS)   threads to start
244  -c : ($SKB_CLONE) SKB clones send before alloc new SKB
245  -b : ($BURST)     HW level bursting of SKBs
246  -v : ($VERBOSE)   verbose
247  -x : ($DEBUG)     debug
248
249The global variables being set are also listed.  E.g. the required
250interface/device parameter "-i" sets variable $DEV.  Copy the
251pktgen_sampleXX scripts and modify them to fit your own needs.
252
253The old scripts:
254
255pktgen.conf-1-2                  # 1 CPU 2 dev
256pktgen.conf-1-1-rdos             # 1 CPU 1 dev w. route DoS
257pktgen.conf-1-1-ip6              # 1 CPU 1 dev ipv6
258pktgen.conf-1-1-ip6-rdos         # 1 CPU 1 dev ipv6  w. route DoS
259pktgen.conf-1-1-flows            # 1 CPU 1 dev multiple flows.
260
261
262Interrupt affinity
263===================
264Note that when adding devices to a specific CPU it is a good idea to
265also assign /proc/irq/XX/smp_affinity so that the TX interrupts are bound
266to the same CPU.  This reduces cache bouncing when freeing skbs.
267
268Plus using the device flag QUEUE_MAP_CPU, which maps the SKBs TX queue
269to the running threads CPU (directly from smp_processor_id()).
270
271Enable IPsec
272============
273Default IPsec transformation with ESP encapsulation plus transport mode
274can be enabled by simply setting:
275
276pgset "flag IPSEC"
277pgset "flows 1"
278
279To avoid breaking existing testbed scripts for using AH type and tunnel mode,
280you can use "pgset spi SPI_VALUE" to specify which transformation mode
281to employ.
282
283
284Current commands and configuration options
285==========================================
286
287** Pgcontrol commands:
288
289start
290stop
291reset
292
293** Thread commands:
294
295add_device
296rem_device_all
297
298
299** Device commands:
300
301count
302clone_skb
303burst
304debug
305
306frags
307delay
308
309src_mac_count
310dst_mac_count
311
312pkt_size
313min_pkt_size
314max_pkt_size
315
316queue_map_min
317queue_map_max
318skb_priority
319
320tos           (ipv4)
321traffic_class (ipv6)
322
323mpls
324
325udp_src_min
326udp_src_max
327
328udp_dst_min
329udp_dst_max
330
331node
332
333flag
334  IPSRC_RND
335  IPDST_RND
336  UDPSRC_RND
337  UDPDST_RND
338  MACSRC_RND
339  MACDST_RND
340  TXSIZE_RND
341  IPV6
342  MPLS_RND
343  VID_RND
344  SVID_RND
345  FLOW_SEQ
346  QUEUE_MAP_RND
347  QUEUE_MAP_CPU
348  UDPCSUM
349  IPSEC
350  NODE_ALLOC
351  NO_TIMESTAMP
352
353spi (ipsec)
354
355dst_min
356dst_max
357
358src_min
359src_max
360
361dst_mac
362src_mac
363
364clear_counters
365
366src6
367dst6
368dst6_max
369dst6_min
370
371flows
372flowlen
373
374rate
375ratep
376
377xmit_mode <start_xmit|netif_receive>
378
379vlan_cfi
380vlan_id
381vlan_p
382
383svlan_cfi
384svlan_id
385svlan_p
386
387
388References:
389ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/
390ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/examples/
391
392Paper from Linux-Kongress in Erlangen 2004.
393ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/pktgen_paper.pdf
394
395Thanks to:
396Grant Grundler for testing on IA-64 and parisc, Harald Welte,  Lennert Buytenhek
397Stephen Hemminger, Andi Kleen, Dave Miller and many others.
398
399
400Good luck with the linux net-development.
401