Build and run Cisco Trex traffic generator

We recently had a requirement to perform some load testing on some new firewalls, time and cost were two main constraints along with some flexibility in using our own traffic profiles. After debating our options for a short time we settled on a new type of traffic generation tool, its open source, flexible, very cost effective, with no vendor / hardware lock-in…

Cisco Trex is an open source traffic generation toolset, comparable to tools such as IXIA, SPIRENT etc… It’s purpose is to overcome some challenges such as cost, scale, standardisation, and flexibility.

Trex Overview

With a flexible and powerfull traffic generation tool, we can;

  • Test actual throughput rates and packet-per-second claims by vendor equipment
  • Discover bottlenecks in production and test environments
  • Perform accurate measurements of latency, jitter, and throughput
  • Perform Penetration testing
  • Perform Security and vulnerability testing, malformed packets, DoS

What differentiates Trex to other toolsets is its portability, cost, capacity and flexibility. The toolset is open source, and comes with the ability to communicate and receive instructions via RPC and Python API’s. The package is self contained and easily installed on various Linux distros. It also supports virtual interfaces, so can be installed and run on AWS, VirtualBox, ESXi, Fusion, or simply your laptop!

Throughput of up to 200Gbps have been tested with the right server hardware, this includes generating custom traffic profiles at L4-7 and the ability to analyse these flows in real-time for latency, packet loss, jitter and throughput.

Components of Trex

Hardware components

In its current state Trex supports x86 servers and Intel DPDK Network Interface Cards, the following Trex page outlines some recommendations and testing: Trex Manual

For detailed support requirements, NIC hardware, drivers and locations for software refer to the main Trex page Trex Home

Software components

OS requirements at this stage are Fedora 20 or higher, Ubuntu 14.04.1 or Ubuntu 16.04.1 or higher. However we have tested on CentOS and given you have the correct drivers this also works well.

Optional - Python for API access, and Scapy for packet creation (all these requirements and dependancies can be downloaded as part of the Trex Github) ‘trex-core’ branch at trex-core

Building and Running Trex with VirtualBox

This guide will setup a quick Trex environment and test on VirtualBox, so the entire test can be performed on a single laptop. The idea is to just provide a sample of what Trex can do and how flexible it really is.

  1. Download and build the Trex application on a VirtualBox VM
  2. Configure Trex for first test and use
  3. Configure a second Ubuntu VM as the DUT (Device Under Test)
  4. Explain the packet creating and YAML profile configuration process
  5. Run a throughout test and have the Ubuntu VM route the traffic (Linux IP forwarding)
  6. Analyse the output
Download and build the app

There are two options for running Trex on a VM;

  1. Download the packaged VirtualBox VM image, which includes Fedora_21 + Trex v1.63
  2. Create a VM running Fedora_21 and download Trex from the below location, then unpack-age The app is currently stored on a public repo @ http://trex-tgn.cisco.com/release/latest
$mkdir trex
$cd trex
$wget --no-cache http://trex-tgn.cisco.com/release/latest
$tar -xzvf latest

This will make a new directory with current version for example /v2.13

The recommendation would be to just download the VM image and then update / install the latest Trex version once the image is running. If you are building your own VM then the following are recommended resource configurations:

Memory CPU Network
4GB 4vCPU Adapter1 = NAT
Adapter2 = trex_intnet
Adapter3 = intnet

Image below shows VB VM configuration:

Trex VM Config

Log onto the virtualmachine with the forwarded port, the VM setup will forward local port 3022 > 22 (ssh) on the VM

ssh -p 3022 trex@127.0.0.1
username: trex
password: trex

Change dir to latest version supplied using: cd /home/trex/v1.62/

Once the VM is built and running, ssh to it and confirm the interface status and bindings with sudo ./dpdk_setup_ports.py -s

[vagrant@localhost ~]$ ls
latest  netdata  trex
[vagrant@localhost ~]$ cd trex
[vagrant@localhost trex]$ ls
v2.08
[vagrant@localhost trex]$ cd v2.08/
[vagrant@localhost v2.08]$ ls
automation       daemon_server        external_libs     run_functional_tests  _t-rex-64-debug    trex-cfg
avl              doc_process.py       find_python.sh    run_regression        t-rex-64-debug     trex_client_v2.08.tar.gz
bp-sim-64        dpdk_nic_bind.py     ko                stl                   _t-rex-64-debug-o  trex-console
bp-sim-64-debug  dpdk_nic_bind.pyc    libzmq.so.3       stl-sim               t-rex-64-debug-o   trex_daemon_server
cap2             dpdk_setup_ports.py  master_daemon.py  _t-rex-64             _t-rex-64-o
cfg              exp                  python-lib        t-rex-64              t-rex-64-o
[vagrant@localhost v2.08]$ sudo ./dpdk_setup_ports.py  -s

Network devices using DPDK-compatible driver
============================================
<none>

Network devices using kernel driver
===================================
0000:00:03.0 '82540EM Gigabit Ethernet Controller' if=eth0 drv=e1000 unused=igb_uio,vfio-pci,uio_pci_generic *Active*
0000:00:08.0 '82545EM Gigabit Ethernet Controller (Copper)' if=eth1 drv=e1000 unused=igb_uio,vfio-pci,uio_pci_generic 
0000:00:09.0 '82545EM Gigabit Ethernet Controller (Copper)' if=eth2 drv=e1000 unused=igb_uio,vfio-pci,uio_pci_generic 

Other network devices
=====================
<none>
[vagrant@localhost v2.08]$ 
Trex minimum configuration

Trex utilises one main files for setup and configuration which are mandatory to reference each run, /etc/trex_cfg.yaml

You can copy a basic configuration file from cfg folder by running:

$cp cfg/simple_cfg.yaml /etc/trex_cfg.yaml

For this example we will have the following in our Trex_cfg.yaml file

- port_limit      : 2         # this option can limit the number of port of the platform
  version         : 2
  interfaces    : ["00:08.0","00:09.0"] #the interfaces using ./dpdk_setup_ports.py -s
  port_info       :  # set eh mac addr
          - dest_mac        :   [0x08,0x00,0x27,0x81,0x6d,0xe0]    # router mac addr should be taken from DUT
            src_mac         :   [0x08,0x00,0x27,0x8e,0xfc,0x42]  # source mac-addr - taken from ifconfig
          - dest_mac        :   [0x08,0x00,0x27,0x81,0xf6,0x63]  # router mac addr  taken from DUT
            src_mac         :   [0x08,0x00,0x27,0x23,0x21,0xdc]  #source mac-addr  taken from ifconfig
Configure the DUT (Ubuntu VM)

The DUT for this example will be a simple Ubuntu VM connected to the Trex VM via VirtualBox networking, the topology will be two interfaces on the DUT each of which are on a different VLAN connecting to the respective Trex VM VLAN. We will use basic Linux IP_Forwarding to have the traffic routed from one interface to the other, with a couple of static ARP and static routes. The topology is as follows…

Trex client-MAC01----trex-intnet-----MAC03-Ubuntu-enp0s8

Trex Server-MAC02-------intnet-------MAC04-Ubuntu-enp0s9

MAC01: [0x08,0x00,0x27,0x8e,0xfc,0x42]
MAC02: [0x08,0x00,0x27,0x23,0x21,0xdc]

MAC03: [0x08,0x00,0x27,0x81,0x6d,0xe0] IP: 192.168.1.100
MAC04: [0x08,0x00,0x27,0x81,0xf6,0x63] IP: 192.168.2.100
  • Add two interfaces to the VirtualBox VM, each with adapter type of ‘server’
  • Place each one in a separate VLAN [trex_intnet / intnet]
  • Configure the interfaces with an IP on separate subnets
  • Add the static routes for relevant pcap tests being run from Trex [eg; 16.0.0.0 and 48.0.0.0]. Route to dummy IP towards Trex
  • Add the static ARP entires for next hop per interface to the dummy IP
Configure the interfaces with an IP on separate subnets
Add the static routes for relevant pcap tests being run from Trex [eg; 16.0.0.0 and 48.0.0.0]. Route to dummy IP towards Trex


source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto enp0s3
iface enp0s3 inet dhcp

auto enp0s8
iface enp0s8 inet static
	address 192.168.1.100
	netmask 255.255.255.0
	network 192.168.1.0
	broadcast 192.168.1.255
	up route add -net 16.0.0.0 netmask 255.0.0.0 gw 192.168.1.101

auto enp0s9
iface enp0s9 inet static
    address 192.168.2.100
    netmask 255.255.255.0
	network 192.168.2.0
    broadcast 192.168.2.255
up route add -net 48.0.0.0 netmask 255.0.0.0 gw 192.168.2.101

Add the static ARP entires for next hop per interface to the dummy IP

aleks@ubuntu:/etc/network$ sudo arp -i enp0s8 -s 192.168.1.101 08:00:27:8e:fc:42
aleks@ubuntu:/etc/network$ sudo arp -i enp0s9 -s 192.168.2.101 08:00:27:23:21:dc

DUT@ubuntu:/etc/network$ arp
Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.2.101            ether   08:00:27:23:21:dc   CM                    enp0s9
192.168.1.101            ether   08:00:27:8e:fc:42   CM                    enp0s8
10.0.2.2                 ether   52:54:00:12:35:02   C                     enp0s3

DUT@ubuntu:/etc/network$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.0.2.2        0.0.0.0         UG    0      0        0 enp0s3
10.0.2.0        *               255.255.255.0   U     0      0        0 enp0s3
16.0.0.0        192.168.1.101   255.0.0.0       UG    0      0        0 enp0s8
48.0.0.0        192.168.2.101   255.0.0.0       UG    0      0        0 enp0s9
192.168.1.0     *               255.255.255.0   U     0      0        0 enp0s8
192.168.2.0     *               255.255.255.0   U     0      0        0 enp0s9

Thats it! We are ready to run our first test! For details on each component visist the trex guide mentioned above.

Running the first load test with Trex

The quickest way to begin a test is to just use one of the pre-defined .pcap files and yaml configuration files, these are all located under /trex/v2.08/cap2

[vagrant@localhost cap2]$ ls
asa_exploit.pcap              imix_fast_1g_100k_flows.yaml  rtsp_short1.yaml
asa_explot1.yaml              imix_fast_1g.yaml             rtsp_short2.yaml
citrix.pcap                   imix.yaml                     rtsp_short3.yaml
delay_10_rtp_250k_short.pcap  ipv4_vlan.yaml                rtsp_short.pcap
dns_no_delay.yaml             ipv6.pcap                     rtsp.yaml
dns_one_server.yaml           ipv6_vlan.yaml                sfr2.yaml
dns.pcap                      ipv6.yaml                     sfr3.yaml
dns_single_server.yaml        jumbo.pcap                    sfr4.yaml
dns_wlen1.yaml                jumbo.yaml                    sfr_agg_tcp14_udp11_http200msec_new_high_new_nir_profile_ipg_mix.yaml
dns_wlen2.yaml                lb_ex1.yaml                   sfr_agg_tcp14_udp11_http200msec_new_high_new_nir_profile.yaml
dns_wlength.yaml              limit_multi_pkt.yaml          sfr.yaml
dns_wlen.yaml                 limit_single_pkt.yaml         short_tcp.yaml
dns.yaml                      mail_pop.pcap                 sip_short1.yaml
dyn_pyld1.yaml                many_client_example.yaml      sip_short2.yaml
exchange.pcap                 nat_test.yaml                 smtp.pcap
http_browsing.pcap            Oracle.pcap                   test_mac.yaml
http_get.pcap                 per_template_gen1.yaml        test_pcap_mode1.yaml
http_plugin.yaml              per_template_gen2.yaml        test_pcap_mode2.yaml
http_post.pcap                per_template_gen3.yaml        tuple_gen.yaml
http_simple.yaml              per_template_gen4.yaml        udp_1518B.pcap
https.pcap                    per_template_gen5.yaml        udp_594B.pcap
http_very_long.yaml           rtp_160k.pcap                 udp_64B.pcap
http.yaml                     rtp_250k_rtp_only_1.pcap      Video_Calls.pcap
imix_1518.yaml                rtp_250k_rtp_only_2.pcap      Voice_calls_rtp_only.pcap
imix_64_100k.yaml             rtp_250k_rtp_only.pcap        wrong_ip.pcap
imix_64.yaml                  rtsp_full1.yaml               wrong_ip.yaml
imix_9k_burst_10.yaml         rtsp_full2.yaml
imix_9k.yaml                  rtsp_short1_slow.yaml

We will use the http_simple for our test which consists of one .pcap files containing a HTTP session, IP range for clients from 16.0.0.1 to 16.0.0.255

- duration : 0.1
  generator :  
          distribution : "seq"
          clients_start : "16.0.0.1"
          clients_end   : "16.0.0.255"
          servers_start : "48.0.0.1"
          servers_end   : "48.0.255.255"
          clients_per_gb : 201
          min_clients    : 101
          dual_port_mask : "1.0.0.0" 
          tcp_aging      : 0
          udp_aging      : 0
  mac        : [0x0,0x0,0x0,0x1,0x0,0x00]
  cap_ipg    : true
  cap_info : 
     - name: avl/delay_10_http_browsing_0.pcap
       cps : 2.776
       ipg : 10000
       rtt : 10000
       w   : 1

To begin the test and monitor this live on the trex server we just run one simple command with a few parameters

sudo ./t-rex-64 -f cap2/http_simple.yaml -d 100 -m 1 --nc

-f is the source yaml file to be used for the test

-d Duration of the test in seconds (minimum is 30)

-m Factor for bandwidth (multiply the CPS of each template in the .yaml file by this number)

--nc If set, will terminate exacly at the end of the duration. This provides a faster, more accurate TRex termination. In default it wait for all the flow to terminate gracefully. In case of a very long flow the termination might be prolong.

-l Run the latency daemon in this Hz rate. Example: -l 1000 runs 1000 pkt/sec from each interface. A value of zero (0) disables the latency check.

-c Number of cores per dual interface. Use 4 for TRex 40Gb/sec. Monitor the CPU% of TRex - it should be ~50%. TRex uses 2 cores for inner needs, the rest of cores can be used divided by number of dual interfaces. For virtual NICs the limit is -c=1.

Once we begin the test we should see the following output

[vagrant@localhost v2.08]$ sudo ./t-rex-64 -f cap2/http_simple.yaml -d 30 -l 1000 -m 40 --nc
Loading kernel drivers for the first time
/usr/bin/python dpdk_nic_bind.py --bind=igb_uio 0000:00:08.0 
/usr/bin/python dpdk_nic_bind.py --bind=igb_uio 0000:00:09.0 

Starting  TRex v2.08 please wait  ... 
found configuration file at /etc/trex_cfg.yaml 
no client generator pool configured, using default pool
no server generator pool configured, using default pool
zmq publisher at: tcp://*:4500
 Number of ports found: 2 
 set driver name rte_em_pmd 
 wait 1 sec .
port : 0 
------------
link         :  link : Link Up - speed 1000 Mbps - full-duplex
promiscuous  : 0 
port : 1 
------------
link         :  link : Link Up - speed 1000 Mbps - full-duplex
promiscuous  : 0 
 -------------------------------
 core, c-port, c-queue, s-port, s-queue, lat-queue
 ------------------------------------------
 1        0      0       1       0      0  
 -------------------------------
 number of ports         : 2 
 max cores for 2 ports   : 1 
 max queue per port      : 1 
no client generator pool configured, using default pool
no server generator pool configured, using default pool
 -- loading cap file avl/delay_10_http_browsing_0.pcap 
 Generating erf file ...  



-Per port stats table 
      ports |               0 |               1 
 -----------------------------------------------------------------------------------------
   opackets |            1082 |            1422 
     obytes |           77182 |         1296676 
   ipackets |            1422 |            1082 
     ibytes |         1296676 |           77182 
    ierrors |               0 |               0 
    oerrors |               0 |               0 
      Tx Bw |       0.00  bps |       0.00  bps 

-Global stats enabled 
 Cpu Utilization : 0.0  %  0.0 Gb/core 
 Platform_factor : 1.0  
 Total-Tx        :       0.00  bps  
 Total-Rx        :       0.00  bps  
 Total-PPS       :       0.00  pps  
 Total-CPS       :       0.00  cps  

 Expected-PPS    :       4.11 Kpps  
 Expected-CPS    :     111.04  cps  
 Expected-BPS    :      30.58 Mbps  

 Active-flows    :       15  Clients :      255   Socket-util : 0.0001 %    
 Open-flows      :       48  Servers :    65535   Socket :       15 Socket/Clients :  0.1 
 drop-rate       :       0.00  bps   
 current time    : 1.7 sec  
 test duration   : 28.3 sec  

-Latency stats enabled 
 Cpu Utilization : 0.0 %  
 if|   tx_ok , rx_ok  , rx check ,error,       latency (usec) ,    Jitter          max window 
   |         ,        ,          ,     ,   average   ,   max  ,    (usec)                     
 ---------------------------------------------------------------------------------------------------------------- 
 0 |      545,     543,         0,    0,       1045  ,    6552,      60      |  0  0  0  0  0  0  0  0  0  0  0  0  0 
 1 |      545,     543,         0,    0,       1002  ,    6064,      53      |  0  0  0  0  0  0  0  0  0  0  0  0  0 

Trex will provide sampled output every seconds which displays throughput, latency, drop rate, connections, window, CPU utilisation

-Per port stats table 
      ports |               0 |               1 
 -----------------------------------------------------------------------------------------
   opackets |           17783 |           24646 
     obytes |         1268505 |        25882924 
   ipackets |           24646 |           17783 
     ibytes |        25882924 |         1268505 
    ierrors |               0 |               0 
    oerrors |               0 |               0 
      Tx Bw |       1.44 Mbps |      29.67 Mbps 

-Global stats enabled 
 Cpu Utilization : 2.2  %  2.8 Gb/core 
 Platform_factor : 1.0  
 Total-Tx        :      31.10 Mbps  
 Total-Rx        :      31.10 Mbps  
 Total-PPS       :       6.02 Kpps  
 Total-CPS       :     109.21  cps  

 Expected-PPS    :       4.11 Kpps  
 Expected-CPS    :     111.04  cps  
 Expected-BPS    :      30.58 Mbps  

 Active-flows    :       14  Clients :      255   Socket-util : 0.0001 %    
 Open-flows      :      773  Servers :    65535   Socket :       14 Socket/Clients :  0.1 
 drop-rate       :       0.00  bps   
 current time    : 8.3 sec  
 test duration   : 21.7 sec  

-Latency stats enabled 
 Cpu Utilization : 0.3 %  
 if|   tx_ok , rx_ok  , rx check ,error,       latency (usec) ,    Jitter          max window 
   |         ,        ,          ,     ,   average   ,   max  ,    (usec)                     
 ---------------------------------------------------------------------------------------------------------------- 
 0 |     7086,    7085,         0,    0,       1012  ,    6552,      14      |  0  6552  1261  2611  1106  1458  1102  1272  1348  1377  1139  1215  1104 
 1 |     7086,    7085,         0,    0,       1024  ,    6064,      13      |  0  6064  1270  2647  1121  1502  1110  1286  1361  1404  1150  1222  1117 

To confirm the test is working we should see RX and TX packets incrementing, drop rate might increment if the DUT cannot handle the load. Latency is measured in (uSec) While the test is running we can confirm the DUT (in our case Ubuntu VM) is receiving and forwarding the traffic with just ‘ifconfig’ and determine the RX / TX is increasing for each run

enp0s3    Link encap:Ethernet  HWaddr 08:00:27:36:e0:85  
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe36:e085/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1269 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1211 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:169359 (169.3 KB)  TX bytes:132289 (132.2 KB)

enp0s8    Link encap:Ethernet  HWaddr 08:00:27:81:6d:e0  
          inet addr:192.168.1.100  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe81:6de0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:69827 errors:0 dropped:0 overruns:0 frame:0
          TX packets:97056 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:4981561 (4.9 MB)  TX bytes:102567940 (102.5 MB)

enp0s9    Link encap:Ethernet  HWaddr 08:00:27:81:f6:63  
          inet addr:192.168.2.100  Bcast:192.168.2.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe81:f663/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:97048 errors:0 dropped:0 overruns:0 frame:0
          TX packets:69835 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:102567292 (102.5 MB)  TX bytes:4982209 (4.9 MB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:260 errors:0 dropped:0 overruns:0 frame:0
          TX packets:260 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:17760 (17.7 KB)  TX bytes:17760 (17.7 KB)

Check again

enp0s3    Link encap:Ethernet  HWaddr 08:00:27:36:e0:85  
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe36:e085/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1277 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1217 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:170007 (170.0 KB)  TX bytes:139621 (139.6 KB)

enp0s8    Link encap:Ethernet  HWaddr 08:00:27:81:6d:e0  
          inet addr:192.168.1.100  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe81:6de0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:73720 errors:0 dropped:0 overruns:0 frame:0
          TX packets:102471 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:5259266 (5.2 MB)  TX bytes:108296022 (108.2 MB)

enp0s9    Link encap:Ethernet  HWaddr 08:00:27:81:f6:63  
          inet addr:192.168.2.100  Bcast:192.168.2.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe81:f663/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:102463 errors:0 dropped:0 overruns:0 frame:0
          TX packets:73728 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:108295374 (108.2 MB)  TX bytes:5259914 (5.2 MB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:260 errors:0 dropped:0 overruns:0 frame:0
          TX packets:260 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:17760 (17.7 KB)  TX bytes:17760 (17.7 KB)

For a basic introduction and test run this is sufficient to get an understanding of what Trex can do, this tutorial is extremely limited and does not articulate the true power and flexibility of Trex. We will provide more updates to this about Scapy, RPC and Python API, interactive client and more soon. In the meantime for more detail on Trex and configuration, guides etc the following resources are a good start:

Trex Home | Trex manual

comments powered by Disqus