Via the Newly Observed Domains services, Farsight Security® Inc. (now a part of DomainTools) provides its subscribers with the data necessary to identify new domain names and to implement appropriate risk management policies for managing new domain names when little is known about them in traditional network reputation services.

Leveraging its real-time Passive DNS telemetry feed on the Security Information Exchange (SIE), and cross-referencing that data with its industry-leading DNSDB historical database, Farsight offers a range of data solutions under the Newly Observed Domains (NOD) brand that present subscribers with real-time actionable insights on youthful domain names.

Product Architecture

Farsight’s name detection software watches the stream of passive Domain Name System (DNS) data on SIE channel 202, and uses a Bloom filter to determine domain names that are likely to be newly active, or never seen before. A second stage lookup is then performed against the DNSDB database to see if there are any historical entries that match the domain. Based on the outcome the domain name is either emitted on the Newly Active (SIE channel 211) or the Never Seen Before (SIE channel 212) channels and is also stored in snapshot file formats that can be periodically downloaded.

Features and Benefits

  • Faster access to data for GTLD changes than Zone File Access (ZFA). Active domains can be caught within minutes rather than hours to days with ZFA. Median time from registration to first DNS sighting with NOD is under one hour compared to an average of over 12 hours with Zone File Access
  • Coverage for zones, such as ccTLDs, where zone file access is completely unavailable
  • Coverage for de-facto registries including dynamic DNS providers, application service providers, and subdomain resellers

Requirements

  • SIE access for real time access to this data feed. Requires the lease of a SIE Port or a subscription to SIE Remote Access, and the use of Farsight’s NMSG software
  • Internet Access for the remote query of the data store. Requires software that is compatible with the DNS Blacklist and Whitelist query format standardized in section 3 of RFC 5782
  • Internet Access for the download of file snapshots. Requires an Internet connected machine with connectivity to Farsight. File snapshots are presented in two formats, “rbldnsd” and “RPZ”, and made available via rsync over ssh

Service Delivery

Real-time observations are delivered via Farsight Security Information Exchange (SIE) as follows:

  • Domains not previously seen in in passive DNS data over the past 10 days are broadcast on channel 211
  • First sightings of domain names in passive DNS data are broadcast on channel 212
  • Messages broadcast to both channels include contextual information revealing how Farsight discovered the new domain
  • Messages are formatted using the SIE newdomain message type available in sie-nmsg 0.16 and later

Snapshots of observations over a recent period of time are formatted using the rbldnsd file format and the RPZ file format and are delivered via rsync over ssh. These snapshots include a timestamp corresponding to when each domain in the snapshot was first seen.

  • These snapshots may be served locally as a DNS zone using a self-hosted authoritative DNS server that is able to load or convert zones in the rbldnsd file format, or RPZ file format
  • This data is intended to be queried locally, but Farsight operates a rate limited rbldnsd server to aid in evaluation and testing
  • Both A and TXT records in industry-standard format

Service Properties

  • Snapshots of observations are also bucketed by age and formatted into 7 different zone files suitable for use with the RPZ features of some name servers. By loading the Farsight provided RPZ zone into a RPZ compliant name server the new domains can be made to “disappear” from the Internet for users of that name server
  • Coverage: Contents are winnowed from Farsight SIE passive DNS data. The NOD system imports data from Farsight’s DNSDB and from TLD Zone File Access (ZFA) to eliminate all previously known domain names from the first sightings channel. Median time from registration to discovery of GTLD names is less than one hour
  • Response Time: The target response time for a new DNS sighting to notification on SIE channel 212 is three minutes, and five minutes for publication in a downloadable snapshot
  • Data Rate: Farsight currently broadcasts an average of 50000 messages per hour on channel 211 and 1000 messages per hour on channel 212. Peaks can exceed 100000 messages per hour on channel 211 and 20000 messages per hour on channel 212

Data Format

SIE/NMSG Schema

SIE newdomain message type contains the following fields:

DOMAINThe observed domain name
TIME_SEENThe time at which the sensor first observed the domain
RRNAMEThe query from which the domain was extracted
RRTYPEThe type of record data that was extracted
RDATAThe resource records in the observed response

DNSBL Schema

The DNSBL protocol parameters for Newly Observed Domains are as follows:

  • QNAME = domain.V1.BL.FSI-NOD.NET where domain is the domain name to be interrogated
  • QTYPE is either A to retrieve bucketed times or TXT to retrieve specific times. The former is easier to use in contemporary RBL-aware software
  • DNS response packets will contain a single record in the answer section of the requested type or the DNS packet’s RCode field will be set to 3 (NXDOMAIN) if the requested domain is too old or has been previously unseen in passive DNS data
  • For TXT record requests, a successful response will be a set of key value pairs separated by whitespace of the form “first_seen=timestamp” where timestamp is the UNIX timestamp of when the domain name was first seen in passive DNS. The timestamp value is the ASCII encoding of an integer
  • For A record requests, a successful response will be an address in the 127.0.0.1/24 subnet, where each address indicates the relative age of the oldest sighting. Version 1 uses the following age ranges:
0-5 minute127.0.0.2
5-10 minute127.0.0.3
10-30 minute127.0.0.4
30-60 minute127.0.0.5
1-3 hours127.0.0.6
3-12 hours127.0.0.7
12-24 hours127.0.0.8

RBLDNSD File Format

The downloadable zone files will be formatted in a manner compatible with rbldnsd’s “dnset” file format. It will include the following elements:

  • Comments are prefixed with the pound (#) symbol
  • Records are formatted as “domain :bucket:first_seen=timestamp” with domain and : being separated by whitespace
  • Each file will contain a “Start of Authority” and “Name servers” declaration
  • Start of Authority records are formatted as “$SOA ttl primary_ns email_addr serial refresh_time retry_time expiry_time nxdomain_time”
  • The name server list is formatted as “$NS name server*” and contains the list of name servers that host V1.BL.FSI-NOD.NET

All fields are as defined in RFC 1035. The zone’s serial number is the UNIX timestamp of when the file was created.

DNS RPZ File Format

The downloadable zone files will be formatted in a manner compatible with name servers that can load RFC 1035 compliant zones and use them for RPZ purposes. It will include the following elements:

  • Comments are prefixed with the semi-colon (;) symbol
  • Records are formatted as a CNAME for the domain itself, as well as a wild card CNAME for all subdomains. Comments indicate the time the domain was first seen
  • Start of Authority records are formatted as “$SOA ttl primary_ns email_addr serial refresh_time retry_time expiry_time nxdomain_time.” All fields are as defined in RFC 1035. The zone’s serial number is the UNIX timestamp of when the file was created
  • The name server list is formatted as “$NS name server*” and contains the list of name servers that host V1.BL.FSI-NOD.NET

Suggested Applications

  • Feeding Reputation Systems: Process each domain and RDATA set seen in the newly active channel (211). Investigate the domain name and the contents of observer response data by cross-referencing with telemetry or reputation data. Update a reputation database, transmit a real time notification, or count the event for later statistics. Use the fact that a domain has been recently published in the never-before seen channel (212) as a feature in your reputation system. These applications can help increase the responsiveness of a security tool to nimble threats
  • Crawl Director: Web-crawl each domain name published in the newly active channel (211) to find compromised websites, malware, phishing sites, counterfeit products, or links to same. Reduce time-to-detection for any crawling-based defense technology. Cross-reference these domains with customer telemetry data to find hidden malicious content on compromised web sites
  • Brand Protection: Examine each new domain published in the newly active channel (211), using Soundex or similar fuzzy string matching algorithms to find domain names which may infringe upon a brand
  • Incident Response: Analyze DNS query logs, netflow output, web proxy and other service logs. Look up the age of each domain in the Newly Observed Domain DNSBL and prioritize the investigative response for very young domains