Advanced Exchange Access (AXA)
This article is intended to introduce and acquaint the user with Farsight’s (now a part of DomainTools) AXA suite of tools and library code for SIE Remote Access (SRA). It is the “who, what, and why” introduction to AXA.
Farsight Advanced Exchange Access
Advanced Exchange Access (AXA) is a suite of tools and library code that brings the capabilities of the Farsight Security Information Exchange (SIE) right to a remote user’s network. For a proper introduction to AXA, we first need to learn a bit about SIE.
Farsight Security Internet Exchange
Farsight’s cup of tea is real-time security relevant telemetry data. Lots and lots of it. At break-neck speeds. SIE is a highly scalable, data-sharing platform that collects, aggregates, processes and re-broadcasts telemetry data – all in real-time. More concisely, SIE can be thought of as a data clearinghouse enabling contributors and consumers to efficiently share Internet telemetry. The data flowing through SIE includes telemetry representative of real-time observations and threat-oriented feeds from variety of sources, include:
- Passive DNS data from Farsight’s massive global sensor array (DNSDB)
- Darknet scan logs
- Spam data
- Phishing data
- Malware data
- Sanitized Firewall and IDS log data
- Conficker sinkhole activity
- Farsight’s Newly Observed Domains
- Farsight’s Newly Active Domains
All of this data is delivered as directly broadcast UDP datagrams inside our geographically disparate SIE data centers. Each data source is segregated into its own VLAN – channel in Farsight’s parlance. The raw intensity of the data flows in each channel varies depending on the data source and docs factors (some flows are predictably diurnal while others are sensitive to docs influences). Some channels are extremely low bandwidth, averaging 1.5 Kbps, while our raw passive DNS channel average 410 Mbps with peaks hovering at 650 Mbps. All in all, there is a superabundant amount of real-time data lighting up the SIE network exchanges.
How can you, the battle-hardened security analyst, consultant, engineer, level 17 packet prestidigitator gain access to this treasure trove of data? Traditionally, subscribers would co-locate a Linux host inside one of our SIE data centers. While this gives terrific performance and premium access to data, this is not always convenient for customers.
There is another way.
Farsight SIE Remote Access
SIE Remote Access (SRA) is Farsight’s software solution to make SIE content available to remote users. SRA enables SIE channel traffic to be delivered through a TCP stream across the Internet. In order to reduce the bitrates, SRA provides subscribers with the ability to invoke a server-side filtering capability across a set of channels, selecting only that subset of records that match specific domain name / IP address search criteria.
The Advanced Exchange Access suite of tools and library code is the software that implements Farsight’s SRA service.
AXA, what’s in the box?
The AXA suite consists of two Unix command line tools and one C library that daring developers such as yourself can use to build custom SRA applications.
Sratool
sratool is the AXA Swiss Army Knife. It is a versatile tool used to test, debug, or stream AXA connections. It connects to an SRA server, sends protocol messages and displays the responses. It can also tunnel SIE data like sratunnel.
Sratunnel
sratunnel transfers selected SIE data from the remote server to the local network. The connection to the server is created and restored after problems with binary exponential delays between retries.
Libaxa
libaxa is the C programming library that exposes the AXA API to the application programmer.
The Swiss Army Knife
sratool is the AXA Swiss Army knife. It is a versatile tool used to test, debug, inspect, or stream AXA connections. In its most common invocation, sratool connects to an SRA server, issues a few AXA protocol messages, and displays the responses.
So let’s get to it!
Tutorial 1: Watch Newly Observed Domains
This tutorial assumes you’ve signed up to receive Farsight’s Newly Observed Domains (NOD) datafeed to watch newly active domains and ensure your users don’t visit any newly minted – often malicious – domains. It shows you how to examine the Newly Observed Domains (NOD) feed in real time. Commands and their output are listed with discussion below.
1 $ sratool
2 > connect ssh:[email protected]
3 * HELLO srad version 0.2.5 sra-eft AXA protocol 1
4 > 1 watch ch=211
5 1 OK WATCH started
6 > count 5
7 > channel 211 on
8 * OK CHANNEL ON/OFF channel ch211 on
9 1 ch211 SIE newdomain
10 flyinghorse-colorado.com/A: flyinghorse-colorado.com
11 1 ch211 SIE newdomain
12 treatmentforboils.com/NS: treatmentforboils.com
13 1 ch211 SIE newdomain
14 servicedeck.com/NS: servicedeck.com
15 1 ch211 SIE newdomain
16 www.markenmacher.eu/A: markenmacher.eu
17 1 ch211 SIE newdomain
18 recruitniks.com/NS: recruitniks.com
19 packet count limit exceeded
20 > count
21 packet printing stopped by count 1990 packets agoLines 1-3: Invoke sratool, and use the connect command to establish a connection to the SRA server. The connection is managed via SSH, meaning all of the benefits conferred by the SSH protocol are available to sratool. Upon success, the client emits the hello string from the AXA_P_OP_HELLO message which was sent by the server and contains the server’s software version, name, and AXA protocol verison.
Lines 4-5: Inform the server we want to watch SIE channel 211 traffic (this is the NOD channel). The server responds with the current watch status. The watch is the most fundamental sratool command. This is how sratool “signs up” to receive data from the SRA server. As its name implies, watch sets up a watch which is a low-level primitive that tells the SRA server that the client is interested in nmsg messages or IP packets that meet one of the following criteria:
- is to, from, or contains the specified address
- contains the specified domain name
- arrived on the specified SIE channel
- are SIE messages that could not be decoded
A watch is given a tag that is an integer label used to refer to the watch. An SRA server connection or session can have zero or more watches at a time and the user can add or delete watches as needed. Note that sratool allows only a single SRA connection at a time.
Line 6: Using the count command, we inform sratool we only want to see 5 packets. After this number is met, sratool will stop emitting packets to the screen (though traffic may still be flowing from server to client).
Lines 7-8: With the channel command, enable channel 211 (NOD). The current channel status is printed. Another fundamental command to sratool is channel. Issued alone on the command-line, it will emit the entire list of available SIE channels for which the user is provisioned.
Lines 9-19: sratool emits 5 NOD packets as it receives then from the server. Once the packet count limit is reached, emission stops.
Lines 20-21: Issuing the count command with no arguments prints the current count status. In this case, we find 1990 NOD packets have been streamed to the client, but since we exceeded our limit, they were not emitted to the screen.
Tutorial 2: Counts And Limits
Continuing in the session above, let’s tweak a few knobs and press a few buttons.
22 > list watches
23 1 ch=ch211
24 > 1 delete
25 1 OK STOP watch deleted
26 > rate
27 RATE LIMITS
28 unlimited per second; current value=307
29 10 seconds between reports
30 > rate 1
31 RATE LIMITS
32 1 per second; current value=2
33 10 seconds between reportsLines 22-23: The list watches command prints all of the active watches. We’ve still got one going, we’re just not emitting any packets to the screen.
Lines 24-25: We delete the watch by referencing its tag with the delete command.
Lines 26-29: Another handy command, rate allows us to query the rate limiter and control it. Currently, there is no rate limiting in play – packets will be emitted as quickly as they appear. For lower bandwidth channels, like NOD, this is might not be a problem. For the DNSDB channels, which are much higher bandwidth, we’ll want to limit the rate at which those packets are sent by the server to sratool.
Lines 30-33: Using the rate command, we set a rate limit of 1 packet per second. This will come in handy in the last part of the tutorial where we’ll examine DNSDB.
Tutorial 3: Watch For A Specific Domain In Farsight’s Passive Dns Feed
As a bonus, let’s peek at SIE channel 202 traffic, Farsight’s raw passive DNS feed.
34 > 2 watch dns=*.github.com
35 2 OK WATCH started
36 > channel 202 on
37 * OK CHANNEL ON/OFF channel ch202 on
38 2 ch202 base dnsqr response UDP_QUERY_RESPONSE
39 204.13.250.16.53 > 68.105.29.142.17296 IP TTL=58 UDP 86 bytes
40 DNS: raw.github.com IN A qr aa NOERROR 1 ans, 0 auth, 0 add RRs
41 2 ch202 base dnsqr response UDP_QUERY_RESPONSE
42 208.78.71.16.53 > 208.106.17.39.64372 IP TTL=56 UDP 153 bytes
43 DNS: api.github.com IN A qr aa cd NOERROR 1 ans, 4 auth, 0 add RRs
44 2 ch202 base dnsqr response UDP_QUERY_RESPONSE
45 204.13.250.16.53 > 68.105.28.174.52707 IP TTL=58 UDP 89 bytes
46 DNS: malsup.github.com IN A qr aa NOERROR 1 ans, 0 auth, 0 add RRs
47 * MISSED
48 lost 0 input packets, dropped 0 for congestion,
49 121 for per sec limit
50 since 2014/12/08 17:29:38
51 2 ch202 base dnsqr response UDP_QUERY_RESPONSE
52 204.13.250.16.53 > 68.105.28.174.47116 IP TTL=58 UDP 149 bytes
53 DNS: github.com IN A qr aa NOERROR 1 ans, 4 auth, 0 add RRs
Lines 34-35: We set another watch, this time we want to watch for the wild card domain “*.github.com”. Anything matching this domain will be emitted, such as www.github.com and github.com itself.
Lines 36-37: We turn on channel 202, Farsight’s raw passive DNS channel.
Lines 38-53: All domains matching the watch are emitted.
Sratunnel
At the end of our last tutorial, you probably wondered aloud: “…this is cute but I need real-time bulk transfer of SIE data back to my network. Does this technology even exist in the modern world?”
Yes, yes it does.
sratunnel is the workhorse of the AXA family. It is used to transfer SIE data from the remote server to the local network. It is what Farsight uses for production deployment of SIE data to the customer. sratunnel can be thought of as a fast, efficient, and smart conduit for SIE data. Data goes in one end and sratunnel has a variety of nozzles the user can custom fit on the other end to emit the data into different output formats, including:
- NMSGs to a UDP port
- NMSGs to a TCP port
- NMSGs to a file
- pcap to a file
- pcap to a network interface
Tunnel Newly Observed Domains
Now that you know how to use the Newly Observed Domains (NOD) datafeed to watch newly active domains and ensure your users don’t visit any newly minted – often malicious – domains, we are going to show you how to to tunnel the data to your local network for bulk analysis.
NMSG Primer
To consume the data in this tutorial, you’ll need another Farsight implement called nmsgtool. It is a deeply useful all-purpose tool for working with NMSGs (network messages). NMSG is the format Farsight uses to type, structure and package arbirtary data for transit. Much of Farsight’s data is packaged and delivered as NMSG. A detailed discussion of the NMSG suite will be covered in future blog series. For now it’s just important to understand that you can work with NMSGs using nmsgtool. Onward!
This tutorial will show you, gallant Farsight datafeed customer, how to plumb the Newly Observed Domains (NOD) feed from SIE to your local network, in real time. Commands and their output are listed with discussion below.
1 $ sratunnel -s 'ssh:[email protected]' \
2 > -c 211 \
3 > -w ch=211 \
4 > -o nmsg:udp:127.0.0.1,8430
Line 1: Invoke sratunnel. The -s option instructs the tool where and how to connect. The option string should look familiar, it’s the same one used with sratool with the same intent and results. Securely connect via SSH as sra-service to sra-eft.sie-remote.net.
Line 2: The -c option sets the channel you want to stream. We want NOD which is channel 211.
Line 3: The -w option sets the watch. You learned last week that a watch is how to inform the tool what to look for. In this case, everything on channel 211.
Line 4: Finally, we specify -o, which tells sratunnel where to put the data it streams. In the case above, we’ve snapped on a shiney new “NMSGs to localhost on port 8430” nozzle and that’s where we’ll find our output. Well done! You’ve plumbed your first SRA session. Data is aflowin’. Let’s build a small corpus and have a look…
5 $ nmsgtool -l 127.0.0.1/8430 \
6 > -c 20000 \
7 > -o channel-211.txt
8 $ head -8 channel-211.txt
9 [98] [2014-12-16 23:31:06.438992023] [2:5 SIE newdomain] [a1ba02cf] [] []
10 domain: befrenshee.com.
11 time_seen: 2014-12-16 23:28:19
12 rrname: befrenshee.com.
13 rrclass: IN (1)
14 rrtype: NS (2)
15 rdata: ns67.domaincontrol.com.
16 rdata: ns68.domaincontrol.com.
17 $ grep ^domain: channel-211.txt | awk '{print $2}' > NOD.txt
Line 5: We use nmsgtool to connect to the loopback address on port 8430.
Line 6: The -c option specifies a maximum count of payloads to capture.
Line 7: The -o option tells nmsgtool to write presentation output to a file.
Line 8: Let’s examine one entry…
Line 9: Each NMSG datagram contains a fixed-length header containing the message size, a UTC timestamp, the message type, a 32-bit source identifier and optional SIE operator and group codes (both empty in this case).
Line 10: The fresh young domain, hot off the press!
Line 11: How fresh? At the time of this writing, that timestamp of when the domain was observed was just over two minutes old.
Lines 12-16: The DNS meta-data associated with the domain.
Line 17: You’re free to manipulate the data however you see fit. As per the above, you can take the list of 20,000 young domains and feed the file into the young domain crunching automation of your choice… :)
In summary
We learned how to invoke sratunnel and configure it to emit Farsight’s Newly Observed Domain traffic to a local NMSG listener. We then built up a small treasure trove of brand new domains and saw how to use standard Unix shell tools to manipulate the data.
This wraps our introductory series on AXA. In future blog series, we’ll build on this knowledge and explore the AXA C and Python APIs so you can learn to build your own SRA-aware tools. Until next time!
Ok I’m in, where can I get it?
The AXA suite is available for download on Farsight’s GitHub AXA page and available as Debian packages on our Debian package server.
I’ve got the tools, how can I get some data? You need to sign up . We offer several options for accessing our data, including a grant program for those that qualify. Reach out to us!
Additional Information
- SIE Channel Guide https://www.farsightsecurity.com/assets/media/download/fsi-sie-channel-guide.pdf
- DNSDB: Passive DNS data from Farsight’s massive global sensor array https://www.farsightsecurity.com/solutions/dnsdb/
- Newly Observed Domains https://www.farsightsecurity.com/solutions/threat-intelligence-team/newly-observed-domains/
- Newly Active Domains https://www.farsightsecurity.com/solutions/threat-intelligence-team/newly-observed-domains/
- nmsgtool https://github.com/farsightsec/nmsg
- VLAN http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst4500/12-2/25ew/configuration/guide/conf/vlans.html
