Not all Ethernet NICs are Created Equal - Trying to Capture Invalid Ethernet Frames

Published: 2020-03-13
Last Updated: 2020-03-13 01:08:20 UTC
by Rob VandenBrink (Version: 1)
6 comment(s)

This all started with a simple request.  A client had purchased some new, shiny networking gear, and in each failover pair the active unit was sending 1 "Runt" per second.

A "runt" is a frame that is smaller than the legal minimum, in other words, 64 bytes.  I guess it doesn't qualify as a packet, since the frame would encapsulate the packet, and since the frame isn't valid to decapsulate, there is no packet to see. Usually you see runt frames when you have a duplex mismatch, in this case it looks like they were inflicted on us by the driver on this piece of gear.  In addition to the switch port's "runt" counter incrementing, the "input error" counter on the switch port is counting up in lock step.

No problem, go to the vendor and get it fixed with an update is the answer you'd expect.  But no, the vendor doesn't believe us, or at least not enough to do anything, they want us to capture some of these frames.

OK, so I start by setting up a capture session on the switch, with a SPAN / mirror port.  At this point I find out that the switch port drops invalid frames before they are even completely recieved, so this approach only forwards the valid frames.

Next, we tried using a TAP (an inline network device that has a "listener" port).  It turns out that most modern TAPs (1Gbps or better) are active devices, they're essentially small switches and have the same issue as the switch I was starting from.  So we tried using a passive tap, which is essentially an electrical device that has the "Rx" ethernet pins "tee'd" off to the listener port.  This works!  On with the capture we said!

Not so fast!  Your ethernet NIC does the same thing as a switch port, the hardware drops invalid frames before they reach the OS.  Luckily in Linux you can adjust this, you can use the "ethtool" tool to disable this feature:

ethtool -K eth0 rx-fcs on  will enable the receive of frames that fail the fcs (frame checksum) check.
ethtool -K eth0 rx-all on  will enable receiving of other invalid frames (like runts)

These options are not available for any Windows drivers I've been able to find - you'd expect to find them under the "advanced" tab for the driver configuration.

OK, all that said, good to go right?
Nope, these options aren't available on many NICs, especially USB NICs like we're forced to use on so many modern laptops.

ethtool -k eth0 | grep rx  will list all the receive options available on your NIC.

Most USB adapters are either Realtek or ASIX, and they both show the two key flags as "fixed", as in not changeable.  I had an older Broadcom USB NIC (the Apple Thunderbolt-2 Ethernet card), same deal:

Realtek ASIX Broadcom

# dmesg | grep eth1

[  361.904443] r8152 2-5:1.0 eth1:v1.09.9

 

 

# dmesg | grep eth1

[   41.506128] ax88179_178a 2-5:1.0 eth1:
register 'ax88179_178a' at usb-0000
:00:14.0-5, ASIX AX88179 USB 3.0
Gigabit Ethernet, 00:24:9b:1e:a9:94

# dmesg | grep eth1

[   89.179919] tg3 0000:3e:00.0 eth1:
Tigon3 [partno(BCM957762) rev
57766000] (PCI Express)
MAC address 68:fe:f7:08:0e:e7

 

Realtek ASIX Broadcom

# ethtool -k eth1 | grep rx

rx-checksumming: on
rx-vlan-offload: on
rx-vlan-filter: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]

# ethtool -k eth1 | grep rx

rx-checksumming: on
rx-vlan-offload: off [fixed]
rx-vlan-filter: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]

# ethtool -k eth1 | grep rx

rx-checksumming: on
rx-vlan-offload: on [fixed]
rx-vlan-filter: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]


Luckily, my main laptop has an on-board Intel NIC, which allows you to adjust lots of the knobs available (certainly the ones we're looking for)!

# ethtool -k eth0 | grep rx

rx-checksumming: on
rx-vlan-offload: on
rx-vlan-filter: off [fixed]
rx-fcs: off
rx-all: off

rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]

Ok, NOW we're ready to go, right?  We set the whole thing up, with a capture filter of: len < 65 (this is packet length not frame length, so it still sees ARP and other small packets, but at least it filters the majority of the traffic out)

.... And we still don't get our target frames.  We know that we're still receiving runt frames  - we still see them on the switch, and ethtool shows them on the capturing PC - but tcpdump isn't seeing them?

At that point, we go to the wireshark FAQ, and see that even after all this work, libpcap is our last roadbock.  libpcap will not capture invalid frames, so that means tcpdump, wireshark and anything that uses tcpdump won't.
https://www.wireshark.org/faq.html#_how_can_i_capture_entire_frames_including_the_fcs

Looking into various posts on libpcap, we see the same messages echo'd "we'll capture any valid frame" ....

If I remember right, (way way) back in the day, the Network General Sniffer boxes could do this, but that's going back to 10/100mbps ethernet days.

So the question to the community is - has anyone seen a combo of NIC, driver, OS and library that will capture invalid frames?  Please, use our comment form if you've seen anything that works.  Or if you've been in a similar situation of needing to capture traffic but couldn't we're all ears on that too!

===============
Rob VandenBrink
rob@coherentsecurity.com

 

Keywords: ethtool invalid runts
6 comment(s)

Comments

You're nicer than I am; I would've started the FTC complaint already. I've never personally run into that but a little searching and you 'might' be able to have some luck with Intel based cards with specific adapter driver series. This intel KB last checked earlier this year mentions registry key settings. For example the: Intel G174P E1G42ET series might work for Windows/Linux driver based needs and you can still buy them cheap/used on ebay.

I'm assuming your switch might've reported some items on the RMON "etherStatsUndersizePkts" and "etherStatsFragments" counters. Jasper at Packet Foo in a comment states a few possible specialized hardware solutions like the EOL/EOS Riberbed TurboCAP series (which I can't seem to find on ebay or cdw, ironically I still see 'sniffer pro' boxes in the multiple of thousands) and then there's Napatech and FiberBlaze mentioned. In some other posts he Jasper also mentions Endace which they have a special set of capture cards called the DAQ's that vary in price point. It might be worth reaching out to them to see if you even can set such a mode.

Excerpt from the Intel KB:
Adapter Driver Registry Key
e1g, e1e, e1y MonitorModeEnabled
e1c, e1d, e1k, e1q, e1r, ixe, ixn, ixt MonitorMode

Place the new key (dword) at:

HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318}\00nn

Where nn is the physical instance of the network port where you want to capture the VLAN tags. ControlSet001 might need to be Current Control Set or another 00x number.

When creating or changing registry dword MonitorModeEnabled, set the dword value to one of the following:

0—disabled (Do not store bad packets, Do not store CRCs, Strip 802.1Q vlan tags)
1—enabled (Store bad packets. Store CRCs. Do not strip 802.1Q vlan tags)
When creating or modifying registry dword MonitorMode, set the dword value to one of the following options:

0—disabled (Do not store bad packets, Do not store CRCs, Strip 802.1Q vlan tags)
1—enabled (Receive bad/runt/invalid CRC packets. Leave CRCs attached to the packets. Do not strip VLAN tags and ignore packets sent to other VLANs as per normal operation.)
Note You must restart Windows for the registry change to take effect.

In Linux*
By default, the driver in promiscuous mode does not strip VLAN tags.

To strip VLAN tags: Load the kernel supplied 802.1q module. This step automatically enables the Intel Networking hardware offload capabilities to offload VLAN tag stripping and insertion. For support and information on loading the 802.1q module, contact your distribution.

Your capture software is responsible for enabling promiscuous mode in your driver. If the driver is not in promiscuous mode, the packets are dropped or ignored because of the bad type/len field.

Ref:
https://www.intel.com/content/www/us/en/support/articles/000005498/network-and-i-o/ethernet-products.html
https://blog.packet-foo.com/2013/05/capturing-damaged-frames/
https://www.endace.com/endace-high-speed-packet-capture-solutions/oem/dag/
https://ark.intel.com/content/www/us/en/ark/products/50397/intel-gigabit-et-dual-port-server-adapter.html
Thanks very much for the multiple links (especially those reg keys). I'm at the point where I have the Intel NIC and Linux doing everything needed, except that the actual capture library (libpcap) simply won't ingest those invalid frames. With those registry keys in play, I'll give the two main capture options in Windows a shot - if that approach works it's a win, and a surprising one (since there isn't a good answer on "vanilla" Linux so far)

If my next step is to build a purpose-specific box for this, I think that fix exceeds my current need, but I'm sure that some of our readers will follow up in that direction :-)
In Linux, I would try to recompile the NIC driver.

Since your laptop has an Intel NIC, i would bet e1000 (used to be) - and find the right place to put the "printf" for each byte received - before error checking, etc.

Later, compare the output from tcpdump and the printf from NIC.
I've had folks suggest that, also people have suggested me "fixing" libpcap.
There aren't enough hours in the day for me to tackle either of those - maybe after I retire :-)
Hey Rob. I'm not where I can test but could you use Scapy? Maybe change the conf to a different socket like:

>>> conf.L3socket=L3RawSocket
If decide to give it a try for the Intel driver (e1000/e1000e):

I believe the function e1000_clean_rx_irq can be used to spill the RX bytes into debug with pr_err(). Not pratical for high traffic, of course.

https://elixir.bootlin.com/linux/v5.2.20/source/drivers/net/ethernet/intel/e1000/e1000_main.c

/**
* e1000_clean_rx_irq - Send received data up the network stack; legacy
* @adapter: board private structure
* @rx_ring: ring to clean
* @work_done: amount of napi work completed this call
* @work_to_do: max amount of work allowed for this call to do
*/
static bool e1000_clean_rx_irq(struct e1000_adapter *adapter,
struct e1000_rx_ring *rx_ring,
int *work_done, int work_to_do)
[...]
[...]
process_skb:
total_rx_bytes += (length - 4); /* don't count FCS */
total_rx_packets++;
[...]
[...]


u32 length; -> I believe it is the number of bytes
u8 *data; --> The data to be printed

pr_err() -> the function to print

Diary Archives