I have recently been working on a project that involves working with ipfix/netflow, this was a new concept to me, I primarily deal with data ingestion and analysis of network traffic for monitoring and security purposes, we needed to ingest some data into splunk, in a way that would scale, the intention of this post it to provide a short break down of notes that hope give you a “leg up”, if you ever find yourself in the same situation.
Introduction
IPFIX (Internet Protocol Flow Information Export) is a standardized protocol for exporting network flow data from network devices, such as routers, switches, and firewalls. It provides a way to transmit information about network traffic, including information about the source and destination IP addresses, port numbers, traffic volume, and other metadata associated with individual flows.
IPFIX is based on the earlier NetFlow protocol, but provides additional features and capabilities that make it more flexible and scalable. It has become widely adopted in a variety of different network environments for monitoring, analysis, and troubleshooting purposes.
In this blog post, we will provide an overview of the key concepts and features of IPFIX, including IPFIX elements, enterprise IDs, templates, and exporting flow data.
IPFIX Elements
IPFIX elements are individual data fields that are used to describe network flows. Each element represents a specific piece of information about the flow, such as the source IP address, destination IP address, protocol, port numbers, packet and byte counts, and various other metadata.
IPFIX includes a number of standard elements that are defined in the IPFIX Information Elements registry, as well as enterprise-specific elements that are defined by specific vendors or organizations. Each element is identified by a unique element ID, which is used to reference the element in IPFIX templates.
Here are some examples of standard IPFIX elements:
- Source IP Address (element ID 8)
- Destination IP Address (element ID 12)
- Protocol (element ID 4)
- Source Port (element ID 7)
- Destination Port (element ID 11)
- Packet Count (element ID 2)
- Byte Count (element ID 1)
In addition to these standard elements, IPFIX also supports enterprise-specific elements that are defined by specific vendors or organizations. Enterprise-specific elements are identified by an enterprise ID (EID) and a unique element ID, and are used to distinguish their elements from those defined by the standard.
For example, Cisco defines a set of enterprise-specific elements for IPFIX, which are identified by an EID of 9. Some examples of Cisco’s enterprise-specific elements are:
- Cisco Source VLAN ID (EID 9, element ID 150)
- Cisco Destination VLAN ID (EID 9, element ID 151)
- Cisco TCP Connection Flags (EID 9, element ID 111)
Templates
Templates are used to define the structure of IPFIX messages, including the elements that are included and the format in which they are presented. Templates provide a standardized way of representing flow data, which allows different vendors and organizations to exchange flow data in a consistent and interoperable manner.
Templates define the set of elements that are included in a flow record, their data types, and their positions within the record. A template is identified by a unique template ID, which is used to reference the template in IPFIX messages.
Templates can be defined by network devices that generate flow records, and are typically associated with a particular flow record format. They can also be defined by collectors that receive flow records, and are used to specify the format in which the records should be presented.
Here is an example of an IPFIX template:
| Template ID | Field Count | Field Specifier 1 | Field Specifier 2 | … | Field Specifier N |
|---|---|---|---|---|---|
| 256 | 4 | Source IP Address | Destination IP Address | … | TCP Flags |
In this example, the template has an ID of 256 and defines four fields: source IP address, destination IP address, protocol, and TCP flags.
Exporting Flow Data
IPFIX data is typically exported from network devices to a central collector, which can be used to store, analyze, and visualize flow data. Devices that generate flow records (i.e., exporters) use the IPFIX protocol to transmit flow records to one or more collectors, typically over a UDP or TCP connection.
When an exporter transmits IPFIX data to a collector, it first sends a template record that specifies the format of the flow records that will be sent. The collector uses the template record to parse the flow records that it receives and to extract the individual fields of data.
Once the template record has been sent, the exporter sends flow records that contain information about individual network flows. Each flow record contains a set of fields that describe a single network flow, including the source and destination IP addresses, port numbers, protocol, packet and byte counts, and other metadata.
Flow records can be transmitted in either a “streaming” mode or a “file-based” mode. In streaming mode, flow records are transmitted in real-time as they are generated by the exporter. In file-based mode, flow records are written to a file on the exporter and are then transferred to the collector at a later time.
Conclusion
IPFIX is a powerful protocol for exporting network flow data from network devices. It provides a standardized way of transmitting information about network traffic, including details about individual flows, and is widely adopted in a variety of different network environments.
By providing a flexible and scalable way to export and analyze network flow data, IPFIX helps network administrators to gain valuable insights into their networks, identify potential security threats, and optimize network performance. With its support for standard and enterprise-specific elements, templates, and flexible flow record formats, IPFIX is a versatile tool that can be customized to meet the needs of almost any network environment.
IPFIX & LIBFDS
IPFIXcol2 and libfds are open-source software libraries that are used for processing and exporting network flow data in the IPFIX format.
IPFIXcol2 is a modular software framework for collecting, processing, and exporting IPFIX data. It is designed to be highly scalable and configurable, and can be customized to support a wide range of use cases. IPFIXcol2 can collect flow data from various sources, including network devices, network taps, and other software applications, and can export the data to different destinations, such as databases, log files, or other network devices. It supports a wide range of IPFIX-related protocols and features, such as support for different templates, data types, and export protocols.
libfds is a library of data structures and functions for working with IPFIX data. It provides a set of APIs for parsing, validating, and manipulating IPFIX data, and supports a wide range of IPFIX-related protocols and features, such as support for different templates, data types, and export protocols. libfds is designed to be highly efficient and scalable, and can be used in a wide range of applications that require processing of large volumes of IPFIX data.
Both IPFIXcol2 and libfds are open-source software projects that are freely available under the terms of the BSD license. They are actively maintained and supported by a community of developers and users, and are widely used in a variety of applications and use cases
Building out your .xml file for LIBFDS
Assume you have the below format and need to build out your ipfix elements from a vendor supplied .csv file
| ElementID | Name | Datatype |
| 1 | oper-status | string |
| 2 | admin-status | unsigned32 |
For LIBFDS we need a .XML with our elements in the below format, ensure that you use the correct PEN for your vendor.
This file should live in /etc/libfds/user/elements/[name].xml
The below script will build your elements file that will gather template information and translate the ipfix stream.
import csv
import xml.etree.ElementTree as ET
from xml.dom import minidom
with open('elements_file.csv', 'r') as file:
reader = csv.DictReader(file)
headers = reader.fieldnames
data = [row for row in reader]
unique_data = []
seen_names = set()
for row in data:
name = row[headers[1]]
if name not in seen_names:
unique_data.append(row)
seen_names.add(name)
root = ET.Element("elements")
for row in unique_data:
element = ET.SubElement(root, "element")
element_id = ET.SubElement(element, "id")
element_id.text = row[headers[0]]
element_name = ET.SubElement(element, "name")
name = row[headers[1]]
try:
element_name.text = name[name.rindex(":") + 1:]
except ValueError:
element_name.text = name
element_data_type = ET.SubElement(element, "dataType")
element_data_type.text = row[headers[2]]
xml_str = ET.tostring(root, 'utf-8')
reparsed = minidom.parseString(xml_str)
pretty_xml_str = reparsed.toprettyxml(indent=" ")
with open("elements_output.xml", "w") as file:
file.write(pretty_xml_str)
This will build your elements.xml file in the below format. NOTE you will need to add the ipfix element header and footer tags manually
<?xml version="1.0" encoding="utf-8"?>
<ipfix-elements>
<scope>
<pen>3333</pen>
<name>VENDORNAME </name>
<biflow mode="none"></biflow>
</scope>
<element>
<id>142</id>
<name>Name</name>
<dataType>string</dataType>
</element>
</ipfix-elements>
RFC 7011
RFC 7011 is a standard document that describes the Internet Protocol Flow Information Export (IPFIX) protocol. IPFIX is a standard protocol for exporting network flow information from network devices, such as routers, switches, and firewalls. It is used to transmit information about network traffic, including information about the source and destination IP addresses, port numbers, traffic volume, and other metadata associated with individual flows.
RFC 7011 provides a comprehensive specification for IPFIX, including details about the IPFIX architecture, the format of IPFIX messages, and the procedures for exporting flow data. It defines the key concepts and terminology used in IPFIX, such as flow records, templates, and information elements, and provides guidance on how to implement IPFIX in a variety of different environments.
In addition to the core IPFIX protocol, RFC 7011 also describes a number of related protocols and features, such as options for transporting IPFIX messages over different transport protocols, security considerations for IPFIX deployment, and guidelines for interoperability testing.
Overall, RFC 7011 provides a detailed and authoritative specification for the IPFIX protocol, which has become widely adopted in a variety of different network environments for monitoring, analysis, and troubleshooting purposes.
Check back on this post as I plan to add more info as and when I have time.