About Security Information Exchange (SIE)

The Security Information Exchange (SIE), from Farsight Security® Inc., is a scalable and adaptable real-time data streaming and information sharing platform. SIE collects and provides access to more than 200,000 observations per-second of raw data from its global sensor network. Farsight also applies unique and proprietary methods for improving usability of the data, directly sharing the refined intelligence with SIE customers and DNSDB®, one of the world’s largest passive DNS (pDNS) databases.

The diverse set of data available from SIE includes the following and is relevant and useful for practitioners in various technology roles:

Each unique set of data in SIE is known as a channel and the data acquired from a specific channel can be customized to meet the needs of each customer, enabling you to subscribe to and access only the channels needed to solve your problem. A channel in SIE may be the result from analyzing the data or a subset of data from other channels.

The data available from SIE channel subscription packages includes:

The SIE Channel Guide provides an overview of the channels available from SIE. The list of currently available channels can be found at:

Why Passive DNS (pDNS)?

DNS is a critical component of Internet communication and almost all Internet transactions begin with a DNS query and response.

DNS serves as early warning and detection solution for phishing, spam, malicious and suspicious behaviors, and other attacks. DNS intelligence is considered the only source of “ground truth” information for the Internet.

Farsight Security’s mission is to make the Internet a safer place. We provide security solutions that empower customers with meaningful and relevant intelligence. This information provides customers with insights about the network configuration of a threat and the surrounding network on the Internet for improving the value and impact of threat intelligence and research.

The Security Information Exchange (SIE), from Farsight Security Inc., is designed with privacy in mind. The passive DNS (pDNS) sensors do not collect Personally Identifiable Information (PII) from client resolvers (also known as stub) by deliberately collecting between recursive resolvers and authoritative servers.

The data from SIE enables security professionals to accurately identify, map, and protect their networks from cybercriminal activity by providing global visibility. It provides immediate access to a real-time global sensor network without the need to develop or deploy your own data collection infrastructure.

Methods to access and acquire data from SIE channels are available using SIE Direct Connect, SIE Remote Access (SRA), SIE Batch, or AXAMD. These methods are described in the SIE Technical Overview document. Due to the technical limitations of transporting high bitrate SIE channels across the Internet, some access methods are not available for specific SIE channels. These restrictions are noted below.

SIE is designed with privacy in mind. The passive DNS (pDNS) sensors do not collect Personally Identifiable Information (PII) from client resolvers (also known as stub) by deliberately collecting between recursive resolvers and authoritative servers.

Based on your needs, you can subscribe to an individual channel or a bundle of commonly used channels. A Farsight sales representative or Solution Architect (SAs) can help you select the channels that will best meet your needs.

The Passive DNS Processing “Waterfall” Model

Data is processed by Security Information Exchange (SIE) in what is called a waterfall model. The following diagram can help inform and guide you in understanding the data that is available from the various SIE DNS channels. Farsight’s Solution Architect’s (SAs) are happy to discuss criteria for selecting the appropriate SIE Channels with customers.

Look here to learn more about waterfall models.

SIE passive DNS (pDNS) Waterfall Model

Passive DNS (pDNS) begins with raw DNS traffic that is observed and collected by passive DNS sensors and contributed to Farsight’s Security Information Exchange (SIE) by pDNS sensor operators. Once the data is sent to SIE, the data then passes through a series of processing phases, starting with deduplication.

Waterfall Model Processing Phase #1: Deduplication

Farsight’s passive DNS (pDNS) solution observes and collects unique DNS answers based on analysis of the associated RRname, RRtype, RData, and bailiwick. Since the raw DNS data includes many duplicate answers for common DNS questions, that may be observed many times per-second, SIE deduplicates the data in the first phase of the waterfall model.

The deduplication phase performs data reduction and exports the unique DNS records with counts for the number of times each unique DNS answer was observed in the data.

However, some types of DNS data are also filtered at this state, such as DNS messages that have a bad checksum value or data that has been delayed for more than an hour. Some of the data is sent to Channel 206, DNSDB Rejected Records (also known as Chaff), while other records are discarded.

Waterfall Model Processing Phase #2: Verification

Rogue, malicious, or misconfigured name servers may respond with misleading resource record information for a domain or domains. The verification phase ensures that only bailiwick-appropriate DNS data is passed on to Channel 208, DNSDB Verified Data. DNS data that fails bailiwick verification is sent to Channel 206, DNSDB Rejected Records (also known as Chaff).

For more information on bailiwicks and how they are used in DNSDB, see What is a Bailiwick.

Waterfall Model Processing Phase #3: Filtering

The next phase is filtering and the final phase in the waterfall model processing. In this phase, various categories of DNS data are filtered, which may including the following:

Some DNS records are filtered at this phase and sent to Channel 206, DNSDB Rejected Records (also known as Chaff), while others are discarded, depending on factors beyond the scope of this document.

DNS records that are not filtered are sent to Channel 204, Processed DNS Data, which is the channel also used by DNSDB.

SIE Data Formats

To acquire, prepare, and transport SIE data, Farsight created an adaptable container wire and file format for storing and transmitting blobs of data called Network Message (NMSG). As its core, NMSG leverages Google Protocol Buffers Version 2 for binary encoding using pre-defined schemas, or in a native packetized format like PCAP.

Other data formats, like JSON or XML, can also be encapsulated in NMSG for consistent transport across Farsight’s Security Information Exchange (SIE) infrastructure and acquired and analyzed by receiving systems. This document addresses what is needed to acquire and process NMSG message payloads.

Farsight uses NMSG to transmit data on the higher volume channels. Other channels might use Newline delimited JSON for delivering data, and a few channels deliver in PCAP (Packet Capture) format.

Data Format Information
NMSG Farsight’s Network Message (NMSG) Encapsulation format. See the SIE NMSG User Guide for details.
JSON JavaScript Object Notation format
NDJSON Newline delimited JSON
PCAP Packet Capture format

SIE Access Methods

Data from SIE can be accessed and acquired using the following methods:

For additional information about SIE access methods, please see the SIE Technical Overview document.

Direct Connect

SIE Direct Connect allows a customer to physically connect a server to the Farsight SIE network for maximum data throughput. This can be done in one of two ways:

If a blade server is leased from Farsight, it will be pre-installed with the essential software components needed to acquire, process, compress, buffer, and transfer data from SIE channels to the customer’s data center for additional analysis, enrichment, and storage.

The customer will be given an admin account, with root access, and the password for the root account. This allows the customer to modify the operating system for their specific needs. The creation of any additional accounts on the blade server is the responsibility of the customer. See the SIE Blade User Guide for additional information.

If a customer uses their own server, an order can be submitted for a cross-connect to the SIE switches hosted at select Equinix data centers (Ashburn DC3 and Palo Alto SV8). An FSI account manager can help guide cross-connect provisioning details, hosting, or colocation options.

SIE Remote Access (SRA)

SIE Remote Access (SRA) enables a customer to remotely connect to the Security Information Exchange (SIE) from anywhere on the Internet. SRA provides access to SIE channel data on customer’s local servers, allowing their analysis and processing systems to be located in their own data centers rather than physically co-located at a Farsight’s data center.

Due to the technical limitations of transporting high bitrate SIE channels across the Internet, the SRA access method is not available for all SIE channels. Please reference the SIE Channel Guide for channels that can be accessed using SRA.

SRA uses the Advanced Exchange Access (AXA) transport protocol which enables SRA sessions to perform the following:

The streaming search and filtering capabilities of AXA enables SRA to access and acquire meaningful and relevant data from SIE while avoiding the costs of transporting enormous volumes of data across the Internet.

Note: For high volume channels accessed using SRA, it is expected that customer’s will specify a search or filter for IP addresses and DNS domain names or hostnames of interest. The SRA service will only collect and send data matching the specified criteria across the Internet to the customer.

The SRA session to SIE is encrypted and streamed inside a Transport Layer Security (TLS) tunnel. Authentication and access control for the TLS tunnel is provided by TLS pre-shared keys (PSKs). A customer that chooses to access SIE using SRA must create a key-pair (which will generate two (2) keys; one (1) private and one (1) public key) and send the public key to Farsight. Farsight will configure the customer’s public key to access SIE using SRA and the list of SIE channels that the customer is subscribed to based on contract entitlement.

Look here more information on TLS.

Customer’s have the option to use Farsight’s open source tools or they can write custom AXA applications using the C or Python APIs. Farsight’s SRA tools are freely available.

Farsight has published source code examples that demonstrate how to access SIE using the SRA service and AXA protocol. The example code includes a “tunnel” application that replicates SIE channels on local sockets and creates loopback interfaces or files. This enables the use of any Network Message (NMSG) or Packet Capture (PCAP) based software that can observe or acquire data from an SIE channel using the Direct Connect access method.

SIE Batch

SIE Batch provides on-demand access for downloading data from SIE channels using a RESTful API or web-based interface. You select the channel and duration of time you are interested in, and then download the data for analysis. The duration of available data is dependent on the channel, but is typically the most recent 12-18 hours. SIE Batch allows you to acquire data from SIE channel using two (2) methods:

Configuring the SIE Network Interface

The sie-update python script is required for configuring and connecting the SIE network interfaces on the customer’s server to the SIE switch. This configuration update is for Direct Connect, not SIE Remote Access (SRA) or SIE Batch. The python script configures the required virtual LAN (VLAN) interfaces and updates configuration files needed by libnmsg and nmsgtool. The MAC address of the SIE network interfaces on the customer’s server must be provided to and provisioned in Farsight’s system for sie-update to run properly.

A current version of the sie-update script is available as a Debian/Ubuntu package after installation of Farsight’s package repository. Customers can also run the following apt-get command on a system with the Debian/Ubuntu operating system:

$ apt-get install python-daemon sie-update

For other operating systems, customers can download the script and install it using the following commands:

$ wget -O "/usr/local/bin/sie-update"\
    "https://raw.github.com/farsightsec/sie-update/master/sie-update"
$ chmod +x "/usr/local/bin/sie-update"

For optional “daemon” support, python-daemon must be installed.

$ easyinstall python-daemon # requires python setuptools

For sie-update to run properly, the name of the SIE network interface must be provided on the command line. Systems that interact with SIE must have two (2) network interfaces, one to observe traffic from SIE channels and one that provides connectivity to other networks. You must provide sie-update with the interface to use for it to properly work. It is recommended that sie-update be run in “daemon” mode using the --daemon flag, the script will periodically check for changes and automatically update as necessary. For example, to use sie-update with eth1 interface as the SIE network interface, run:

$ sie-update -i eth1 -d

Multiple interfaces can be specified on the command line like -i eth1 -i eth3. This command must be run at system startup, for instance by adding the following line to the /etc/rc.local script:

$ sie-update -i eth1 -d

Note: Depending on how your environment is configured, you may need to specify the absolute path of the sie-update script.

By default, the sie-update script creates the nmsg alias files in the /etc directory, but this can be overriden by specifying the -e / --etcdir parameter to sie-update. Note: When compiling nmsg from source, --sysconfdir=/etc should be passed to the ./configure script so libnmsg searches the correct directory for alias files, otherwise the configuration files will by default be installed in the /usr/local/etc directory.

$ /usr/local/bin/sie-update -v -i eth1 -e eth3 -e /usr/local/etc

Advanced Exchange Access Toolkit (AXA)

Farsight’s Advanced Exchange Access Toolkit (AXA) enables customers to remotely and securely connect to the SRA (SIE Remote Access) service. The SRA service provides access to channels available from Farsight’s Security Information Exchange (SIE). AXA is a Farsight developed binary protocol used to transport real-time data available from SIE.

AXA uses a streaming API encrypted by TLS for transporting SIE data over the Internet. The AXA protocol uses two (2) streams that transport messages between a customers client, such as sratool, and the SRA service. There is one (1) stream in each direction using a single TCP connection.

Some SIE channels may burst to an extremely high bitrate, potentially more than 500Mbps. AXA has two (2) solutions for high volume channels: 1.) optional filtering and 2.) loss-tolerance are both built into the protocol.

One of the following filtering methods can be used to reduce the volume of data received from SRA.

Note: The AXA protocol is deliberately “lossy”, which means data can potentially be lost. If a customer requests more data than the network can transport, data overruns will occur. To notify customers when this happens, loss markers are reliably transmitted within the AXA stream using the AXA accounting subsystem. Because of this, the AXA protocol must use a reliable stream protocol - which is why AXA connections use TLS over TCP. Note: SIE data can potentially be lost before encapsulation into AXA protocol messages due to network congestion, CPU overload, lack of memory, etc. or other system issues.

Farsight also provides a RESTful middleware layer in front of its AXA service. This service is called the AXA Middleware Daemon (AXAMD) and provides a RESTful capability that adds a streaming HTTP interface on top of the AXA toolkit. This enables web-application developers to interface with SIE using SRA. Farsight also published a command line tool and Python extension library called axamd_client. This toolkit is licensed under the Apache 2.0 license.

Note: “AXA” is an overloaded term and depending on the context, may refer to the following:

In this document, where appropriate, context is provided to disambiguate these situations.

The AXA Toolkit

The Advanced Exchange Access (AXA) toolkit contains tools and a C library to bring Farsight’s real-time data and services directly from the Farsight Security Information Exchange (SIE) to the customers network.

You can find the AXA Toolkit here.

The axa-tools distribution contains the following:

libaxa is the middleware for the AXA protocol and includes capabilities to allow remote SIE data to appear on a local network socket.

For detailed usage of sratunnel, radtunnel, sratool, and radtool, please review the respective man pages included in the distribution.

The AXA Transport Layer

AXA offers three (3) encrypted transports for establishing sessions and tunneling data. One of the following identity / authentication methods is required to use AXA. While all three options provide equal security, Farsight strongly recommends using the APIKEY method due to its ease of setup and use.

Prior to transporting data across a network, AXA compresses all NMSGs using the built-in zlib compression capability. IP packets are not compressed.

NMSG

To acquire, prepare, and transport SIE data, Farsight created an adaptable container wire and file format for storing and transmitting blobs of data called Network Message (NMSG). As its core, NMSG leverages Google Protocol Buffers Version 2 for binary encoding using pre-defined schemas, or in a native packetized format like PCAP.

Other data formats, like JSON or XML, can also be encapsulated in NMSG for consistent transport across Farsight’s Security Information Exchange (SIE) infrastructure and acquired and analyzed by receiving systems.

The adaptable NMSG container format allows for consistent or variable message types. NMSG container data may be streamed to a file or transmitted as UDP datagrams. NMSG containers can contain multiple NMSG messages or a fragment of a message too large to fit in a single container. The data in an NMSG container can also be compressed. Additional capabilities include sequencing and rate-limiting.

More information is available in the Farsight’s Network Message, Volume 1: Introduction to NMSG blog article.

System Requirements

Farsight supports the Debian operating system (OS). For information about the currently supported Debian OS, please see Security Information Exchange (SIE) on Debian.

Installation instructions for Security Information Exchange (SIE) on CentOS / RHEL Linux and FreeBSD are available at the following links:

Note: Installation of SIE software packages from source code can be performed on other operating systems, but may require modifications to properly work.

Data from lower volume SIE channels can be acquired and processed with an Atom server or cloud instance with a 1GHz CPU, a hard disk for local storage, and 1GB or more of RAM.

A server that is configured by Farsight for trial or lease has the following operating system (OS) and hardware specifications.

Component Specification
Operating System Debian 9 (“stretch”)
CPU One (1) x Intel Quad Core Xeon E3 3.40Ghz
Memory 16GB RAM
Storage Two (2) x 2TB 7200RPM drives (configured for RAID 1, 2TB available for customer use)
Internet Access 100Mbps connection to the Internet
SIE Network Connection to SIE network

These hardware specifications are adequate for acquiring, processing, compressing, and buffering data from any SIE channel. However, there may not be enough memory (RAM) to perform intensive processing and analysis on data from SIE channels with the highest data volumes.

For the Direct Connect access method to SIE, the system must have two (2) network interfaces:

  1. SIE broadcast network: Typically a ten (10) gigabit link.
  2. Internet uplink: Typically a one (1) gigabit link that Farsight rate-limits to 100Mbps by default.

Additional Information

Farsight has written several blog articles demonstrating ways to interact with SIE using several of the methodologies and tools described in this document. See the following list of blog articles for more information about effectively using SIE:

About Farsight Security

Farsight Security, Inc. is the world’s leading provider of historical and real-time DNS intelligence solutions. We enable security teams to qualify, enrich, and correlate all sources of threat data and ultimately save time when it is most critical - during an attack or investigation. Our solutions provide enterprise, government, and security industry personnel and platforms with unmatched global visibility, context, and response. Farsight Security is headquartered in San Mateo, California, USA. To learn more about how we can empower your security, threat, and intelligence platforms and security organization with Farsight Security passive DNS (pDNS) and threat intelligence solutions, please visit us at www.farsightsecurity.com or follow us on Twitter at @FarsightSecInc.