Security Information Exchange (SIE) DNS Changes
Channel 214, the SIE DNS Changes channel is a source of DNS intelligence for domains, hostnames, or record data (rdata) observed in DNSDB for the first time or the rdata for a for a domain or hostname has changed. This enables customers to observe and monitor when domains and hostnames become active for the first time or changes for DNS resource records are observed.
DNS Changes is one of several channels that tracks domain observations, creation, and changes. These channels follow:
- Channel 211: Newly Active Domains: Previously seen domains observed in channel 204 after 10 days of inactivity
- Channel 212: Newly Observed Domains (NOD): Base domains1 that have never been observed in DNSDB
- Channel 213: Newly Observed Hostnames (NOH): Hostnames, also known as Fully Qualified Domain Names (FQDNs), that have never been observed in DNSDB. Both the RRname (left-side) and Rdata (right-side) of a DNS resource record (RR) are checked
- Channel 214: DNS Changes: Domains, hostnames, or record data that is unknown to DNSDB, either because the data is for a new domain or hostname or because the record data for a domain or hostname has changed. These changes may include new RR types, new or changed IP addresses, or a change in the authoritative name servers for a domain
1 A “base domain” is one label followed by a suffix. See the Public Suffix List for information on the current list of official suffixes. Suffixes are a superset of the Top Level Domains (TLDs).
These channels use Channel 204 Processed DNS Data, that is used by DNSDB, as their authoritative data source. The DNS data available from channel 204 is after the deduplication and verification phases from the Passive DNS Processing “Waterfall” Model. Domains and hostnames are checked for historic observations in DNSDB back to June 2010.
For more information about Channel 204 Processed DNS Data, please refer to the SIE Technical Overview or SIE Technical Reference guides.
About Security Information Exchange (SIE)
The Security Information Exchange (SIE), from Farsight Security® Inc. (now a part of DomainTools), is a scalable and adaptable real-time data streaming and information sharing platform. SIE collects and provides access to more than 200,000 observations per-second of raw data from its global sensor network. Farsight also applies unique and proprietary methods for improving usability of the data, directly sharing the refined intelligence with SIE customers and DNSDB®, one of the world’s largest passive DNS (pDNS) databases.
The diverse set of data available from SIE includes the following and is relevant and useful for practitioners in various technology roles:
- Raw and processed passive DNS data
- Darknet/darkspace telescope data
- SPAM sources and URLs
- Phishing URLs and associated targeted brands
- Connection attempts from malware-infected systems (as seen by a sinkhole)
- Network traffic blocked by Intrusion Detection Systems (IDS) and firewall devices
Each unique set of data in SIE is known as a channel and the data acquired from a specific channel can be customized to meet the needs of each customer, enabling you to subscribe to and access only the channels needed to solve your problem. A channel in SIE may be the result from analyzing the data or a subset of data from other channels.
Why Passive DNS (pDNS)?
DNS is a critical component of Internet communication and almost all Internet transactions begin with a DNS query and response.
- Visiting a website? Your system uses DNS to resolve the IP address of the hostname for the website you are attempting to access
- Sending an email? Email uses DNS to resolve the IP address of the mail exchange server your message should be delivered to
DNS serves as early warning and detection solution for phishing, spam, malicious and suspicious behaviors, and other attacks. DNS intelligence is considered the only source of “ground truth” information for the Internet.
Passive DNS (pDNS) begins with raw DNS traffic that is observed and collected by passive DNS sensors and contributed to Farsight’s Security Information Exchange (SIE) by pDNS sensor operators. Once the data is sent to SIE, the data then passes through a series of processing phases:
- Deduplication: Channel 207, DNSDB Deduplicated Data
- Verification: Channel 208, DNSDB Verified Data
- Filtering: Channel 204, Processed DNS Data (which used by DNSDB)
The end result is the highest-quality and most comprehensive passive DNS database, DNSDB, of its kind-with more than 100 billion unique DNS resource records since 2010.
Farsight Security’s mission is to make the Internet a safer place. We provide security solutions that empower customers with meaningful and relevant intelligence. This information provides customers with insights about the network configuration of a threat and the surrounding network on the Internet for improving the value and impact of threat intelligence and research.
The Security Information Exchange (SIE), from Farsight Security Inc., is designed with privacy in mind. The passive DNS (pDNS) sensors do not collect Personally Identifiable Information (PII) from client resolvers (also known as stub) by deliberately collecting between recursive resolvers and authoritative servers.
The data from SIE enables security professionals to accurately identify, map, and protect their networks from cybercriminal activity by providing global visibility. It provides immediate access to a real-time global sensor network without the need to develop or deploy your own data collection infrastructure.
About SIE DNS Changes
Channel 212 provides insights about unknown DNS activity, because the base domain or hostname is new, or record data about a domain or hostname has changed. These changes could include new RR types, new or changed IP addresses, or new or changed authoritative name servers.
When is DNS intelligence for NOH sent to channel 214?
The following narrative will inform and guide you in understanding when DNS resource record data is sent to one of the channels that tracks domain observations, creation, and changes.
- Suppose you are creating a new website. When you register a new domain for the website, that DNS registration will be sent to channel 214 DNS Changes
- When you start to create and test the website, accessing it will generate DNS requests. These DNS requests will be sent to channel 212 Newly Observed Domains (NOD) and channel 213 Newly Observed Hostnames (NOH) as the first time the domain or hostname was observed being resolved
- When you are done creating and testing the website, but before it is announced to the public and starts being used, activity on the website and associated DNS requests will stop being observed. If the duration of inactivity is more than 10 days for the domain and then a DNS request is observed, that information will be sent to channel 211 Newly Active Domains. If the duration between DNS requests is less than 10 days, it is considered active and no information is sent
For a good introduction into the opportunities and possibilities of these channels, see New (and Newly-Changed) Fully Qualified Domain Names (FQDNs): A View of Worldwide Changes to the Internet’s DNS* from Black Hat Europe, 2015
Use Cases for SIE DNS Changes
DNS Changes is an important tool to monitor the global DNS for changes that may potentially be associated with malicious activity or part of command and control (C&C) infrastructure. Actively monitoring and correlating changes related to known malicious IP addresses, name servers, domains, and hostnames in near real-time is an effective method for a layered approach to cybersecurity.
Channel 213 DNS Changes also enables customers to monitor for unplanned, unintended, or malicious changes to your own network and DNS architecture. Additionally, customers can monitor for domains or hostnames that use your company’s name or brand to facilitate an attack (e.g. phishing, credential harvesting, malware, etc.), or in a manner that violates your intellectual property.
Channel Information for SIE Channel 214
| Channel Name | DNS Changes |
| Channel Number | 214 |
| Description | Passive DNS observations where some aspect of the query or response was not found when compared to the DNSDB historical database. |
| Schema | SIE:newdomain |
To see current channel traffic volumes and service options for accessing it, please see the Security Information Exchange (SIE) Channel Guide.
Data Format for SIE Channel 214
The DNS Changes channel data uses the SIE NMSG newdomain DNS Query and Response resource record schema that observes and collects data returned from a query. The data available from this channel contains NMSG SIE:newdomain type messages that include the following fields:
The NMSG header includes the following fields:
| KEY | VALUE |
|---|---|
| time | Time when a new / changed base domain, hostname, or record data was observed in Channel 204. |
| vname | Vendor Name, SIE. |
| mname | Message type, newdomain. |
| source | Identifies pDNS operator contributing DNS data, not the sensor. |
| message | Embedded JSON record describing the observed DNS Query and Response RR. |
The embedded NMSG message payload is JSON formatted and includes the following fields:
| KEY | VALUE |
|---|---|
| domain | Base domain of the query observed by pDNS. |
| time_seen | Time that pDNS observed the hostname. |
| bailiwick2 | The domain under which the RRset answer was given. |
| rrname | Domain name of the query observed by pDNS. |
| rrclass | RR CLASS is always “Internet (IN)“, which is decimal value “1“. |
| rrtype | RR TYPE describes the type of RR, e.g., A(1), NS(2), CNAME(5). |
| rdata | Data that describes the RR type, returned as an array. |
| keys | Always empty or null. |
| new_rr | Is any data in the RR or RRset new? True or False |
| new_domain | Is the effective base domain new? True or False |
| new_rrname | Is the hostname (also FQDN) new? True or False |
| new_rrtype | Is this a new RR record type? True or False |
| new_rrset | Is the RRset new? True or False |
2 DNS data is considered “in bailiwick” if the resource record being returned is the response from a name server that is known to be responsible for answering with authoritative information about that domain. See What is a Bailiwick? for additional information.
Note: Time-based strings are in the YYYY-MM-DD HH:MM:SS format. The
month “MM” starts at 01 for January and ends with 12 for December. The
hours “HH” are 00-23, and minutes “MM” and seconds “SS” are 00-59.
The times are recorded at UTC (GMT) and daylight savings time (DST) is not
applicable.
Example Message from SIE Channel 214
Data acquired from Channel 214 DNS Changes is returned in NMSG format when
using the Direct Connect or SIE Remote Access (SRA) access methods. NMSG is an
adaptable container format that allows for consistent or variable message types.
If data is downloaded for Channel 214 using SIE Batch, the data is already
delivered in ND-JSON format, and the nmsgtool step below can be skipped.
The nmsgtool program is a tool for acquiring a variety of different inputs,
like data streams from the network, capturing data from network interfaces,
reading data from files, or even standard input and making NMSG payloads
available to one or more outputs. The nmsgtool program can acquire data from
SIE Channel 214 and convert it to a ND-JSON (newline-delimited JSON) text format
for display or additional processing and analysis. nmsgtool is a program
written by Farsight and released as open source.
See the following pages for instructions on how to install software packages for a specific distribution.
- Security Information Exchange (SIE) on Debian
- Security Information Exchange (SIE) on CentOS 6 / RHEL 6
- Security Information Exchange (SIE) on FreeBSD
After data for Channel 214 has been acquired, written, and saved to a file, you
need to decode it to ND-JSON using nmsgtool. The [-r ch214_changes.nmsg]
option tells nmsgtool to read binary NMSG data from a file, [-c 1] limits
the output to single NMSG payload, and [-J -] displays the record in ND-JSON
format to stdout, which is typically the screen.
$ nmsgtool -r ch214_changes.nmsg -c 1 -J -
{"time":"2020-03-24 16:58:59.248954057","vname":"SIE","mname":"newdomain",
"source":"a1ba02cf","message":{"domain":"example.com.",
"time_seen":"2020-03-24 16:57:18", "bailiwick":"example.com.", "rrname":"448757277.verify.example.com.","rrclass":"IN","rrtype":"CNAME", "rdata":["an.example.com."],"keys":[],"new_domain":false,"new_rrname":true,
"new_rrtype":true,"new_rr":[false],"new_rrset":true}}Once the data has been formatted to ND-JSON, a record from the DNS Changes channel will look similar to the following. The following output can be sent to another tool for additional processing.
{"time":"2020-03-24 16:58:59.248954057","vname":"SIE","mname":"newdomain",
"source":"a1ba02cf","message":{"domain":"example.com.",
"time_seen":"2020-03-24 16:57:18","bailiwick":"example.com.",
"rrname":"448757277.verify.example.com.","rrclass":"IN","rrtype":"CNAME", "rdata":["an.example.com."],"keys":[],"new_domain":false,"new_rrname":true,
"new_rrtype":true,"new_rr":[false],"new_rrset":true}}If you want to display a pretty-printed output of ND-JSON formatted records, we recommend using jq, a lightweight and flexible command-line JSON processor. The open source software package is available on Debian and can be installed using $ sudo apt-get install jq. The output from nmsgtool in JSON format [-J -] can be piped to jq using the following:
$ nmsgtool -r ch214_changes.nmsg -c 1 -J - | jq -r '.'
{
"time": "2020-03-24 16:58:59.248954057",
"vname": "SIE",
"mname": "newdomain",
"source": "a1ba02cf",
"message": {
"domain": "example.com.",
"time_seen": "2020-03-24 16:57:18",
"bailiwick": "example.com.",
"rrname": "448757277.verify.example.com.",
"rrclass": "IN",
"rrtype": "CNAME",
"rdata": [
"an.example.com."
],
"keys": [],
"new_domain": false,
"new_rrname": true,
"new_rrtype": true,
"new_rr": [
false
],
"new_rrset": true
}
}SIE Access Methods
Data from SIE can be accessed and acquired using the following methods:
- Direct Connect: Connect a system to the SIE network. This 1.) requires a server to be installed in a data center where Farsight has a point of presence, and 2.) then ordering a network cross connect between your server and the SIE network. Customers can optionally, and prefer to, lease a blade server from Farsight.
- SIE Remote Access (SRA): Remotely connect to the SIE network using an encrypted tunnel from your workstation or a server in your local data center.
- SIE Batch: Provides on-demand access for downloading data from SIE channels using a RESTful API or web-based interface. You select the channel and duration of time you are interested in, and then download the data for analysis. The duration of available data is dependent on the channel, but is typically the most recent 12-18 hours.
For additional information about SIE access methods, please see the SIE Technical Overview document.
Direct Connect
SIE Direct Connect allows a customer to physically connect a server to the Farsight SIE network for maximum data throughput. This can be done in one of two ways:
- Blade Server: Pre-configured blade servers co-located in one of Farsight’s data centers that can be leased by customers for direct access to SIE channels.
- Customer Server: Customer (owned, managed, and operated) servers that can be installed in one of Farsight’s data centers and physically connected to the SIE network with a network cross-connect.
If a blade server is leased from Farsight, it will be pre-installed with the essential software components needed to acquire, process, compress, buffer, and transfer data from SIE channels to the customer’s data center for additional analysis, enrichment, and storage.
If a customer uses their own server, an order can be submitted for a cross-connect to the SIE switches hosted at select Equinix data centers (Ashburn DC3 and Palo Alto SV8). An FSI account manager can help guide cross-connect provisioning details, hosting, or colocation options.
For additional information about SIE connection methods, please see the SIE Technical Overview document. A Farsight’s sales representatives is happy to share a copy of this document with you. This will help inform and guide you in understanding which connection method will work best for you.
SIE Remote Access (SRA)
SIE Remote Access (SRA) enables a customer to remotely connect to the Security Information Exchange (SIE) from anywhere on the Internet. SRA provides access to SIE channel data on customer’s local servers, allowing their analysis and processing systems to be located in their own data centers rather than physically co-located at a Farsight’s data center.
Due to the technical limitations of transporting high bitrate SIE channels across the Internet, the SRA access method is not available for all SIE channels. Please reference the SIE Channel Guide for channels that can be accessed using SRA.
SRA uses the Advanced Exchange Access (AXA) transport protocol which enables SRA sessions to perform the following:
- Select which SIE channel or channels to monitor and acquire data from
- Define user-specified search or filtering criteria to match IP or DNS traffic
- Control rate-limits and other AXA parameters
The streaming search and filtering capabilities of AXA enables SRA to access and acquire meaningful and relevant data from SIE while avoiding the costs of transporting enormous volumes of data across the Internet.
Note: For high volume channels accessed using SRA, it is expected that customer’s will specify a search or filter for IP addresses and DNS domain names or hostnames of interest. The SRA service will only collect and send data matching the specified criteria across the Internet to the customer.
SIE Batch
SIE Batch provides on-demand access for downloading data from SIE channels using a RESTful API or web-based interface. You select the channel and duration of time you are interested in, and then download the data for analysis. The duration of available data is dependent on the channel, but is typically the most recent 12-18 hours. SIE Batch allows you to acquire data from SIE channel using two (2) methods:
- API: Allows you to write tools to programmatically download data from SIE channels for analysis
- Interactively: Web-based interface to the API that enables you to select and download SIE channel data on-demand
Additional Information
Links to Information in this Document
- New (and Newly-Changed) Fully Qualified Domain Names (FQDNs): A View of Worldwide Changes to the Internet’s DNS* from Black Hat Europe, 2015
- SIE:newdomain DNS Query and Response RR schema
- jq a lightweight and flexible command-line JSON processor
- Description of JSON Lines text format (aka ND-JSON)
Newly Observed Domains and Hostnames Links
- Newly Observed Domains (NOD): threat protection from new domains
- Newly Observed Hostnames (NOH)
- Newly Observed Hostnames
- Introduction to Farsight’s NOD RPZ
- Rejecting Spam and Unwanted Email with NOD and RPZ
