Newly Observed Hostnames (NOH) User Guide
Introduction
Farsight Security® Inc.’s (now a part of DomainTools) Newly Observed Hostnames (NOH) leverages Farsight’s Passive DNS database (DNSDB) to determine if a hostname or Fully Qualified Domain Names (FQDNs) is newly seen.
Delivery
NOH is delivered one of two ways:
- As a real time stream of intelligence using the Security Information Exchange (SIE)
- As a CSV file generated hourly using RSYNC and SSH
SIE
For instructions on how to consume NOH as a real time stream, please see SIE documentation.
RSYNC CSV Files
Farsight provides twenty five CSV files of NOH data available via RSYNC + SSH. Twenty four of the files correspond to hourly roll ups of the previous twenty four hours of NOH data. The twenty fifth file is the current hour updated every minute with newly seen hostnames seen within the last sixty seconds.
The file names are created using the format ‘fqdn-YYYYMMDD-HHMM.csv’.
During the provisioning process you will be asked to submit a public SSH key. Once provisioning has been completed you should receive an email similar to the one below with a username specific to you.
Your NOH access has now been provisioned on the Farsight servers.
To access the data, you will use the username of FSI-XXXX-X. You may
find a configuration like the one below useful on your end, placing it
in either ~/.ssh/config, or /etc/ssh/ssh_config:
Host rsync.dns-nod.net
User FSI-XXXX-X
IdentityFile /path/to/sshkey
Port 49222
NOTE: If you do not edit your ssh configuration you will need to use
extended command-line options for rsync.
You may now synchronize files from the rsync.dns-nod.net rsync server
accessed via ssh on port 49222.
Example:
rsync -az -e 'ssh -p 49222 -i/path/to/sshkey' [email protected]:nod/ /srv/nod
NOH files will be found in the subdirectory named 'csv'. New NOH files
will be delivered once an hour and the previous twenty four hours of
NOH files will be available. For this reason we ask you to poll the
server at most once an hour.
If you have any questions, please let me know. Please direct future
support requests to [email protected].
Data Structures
The data set available for NOH differs depending on the delivery mechanism chosen (RSYNC and SSH or SIE). SIE provides the complete data set whereas the CSV file provides a reduced data set. You can see example data structures below per delivery method.
Security Information Exchange (SIE)
Real time stream of newly observed domains for security intelligence and enrichment
"message": {
"domain": "ampproject.net.",
"time_seen": "2017-02-27 18:23:55",
"bailiwick": "ampproject.net.",
"rrname": "d-22069952994228697343.ampproject.net.",
"rrclass": "IN",
"rrtype": "A",
"rdata": [
"216.58.194.78"
],
"keys": [],
"new_rr": []
}Hourly CSV
Columns:
- domain (hostname)
- first_seen (Unix time)
domain,first_seen
ztvxm.pgdbz.com.,1488210985
www.ferselogistic.com.,1488211039
dyaxukuqsm.spotilocal.com.,1488211005
Limits
When using RSYNC to synchronize the CSV files, we recommend running RSYCN once an hour; at maximum do not run RSYNC more than once a minute as the current hour file is not updated any faster than once every sixty seconds.
