This article is intended to introduce and acquaint the user with Farsight’s AXA suite of tools and library code for SIE Remote Access (SRA). It is the “who, what, and why” introduction to AXA.

Farsight Advanced Exchange Access

Advanced Exchange Access (AXA) is a suite of tools and library code that brings the capabilities of the Farsight Security Information Exchange (SIE) right to a remote user’s network. For a proper introduction to AXA, we first need to learn a bit about SIE.

Farsight Security Internet Exchange

Farsight’s cup of tea is real-time security relevant telemetry data. Lots and lots of it. At break-neck speeds. SIE is a highly scalable, data-sharing platform that collects, aggregates, processes and re-broadcasts telemetry data – all in real-time. More concisely, SIE can be thought of as a data clearinghouse enabling contributors and consumers to efficiently share Internet telemetry. The data flowing through SIE includes telemetry representative of real-time observations and threat-oriented feeds from variety of sources, include:

All of this data is delivered as directly broadcast UDP datagrams inside our geographically disparate SIE data centers. Each data source is segregated into its own VLANchannel in Farsight’s parlance. The raw intensity of the data flows in each channel varies depending on the data source and docs factors (some flows are predictably diurnal while others are sensitive to docs influences). Some channels are extremely low bandwidth, averaging 1.5 Kbps, while our raw passive DNS channel average 410 Mbps with peaks hovering at 650 Mbps. All in all, there is a superabundant amount of real-time data lighting up the SIE network exchanges.

How can you, the battle-hardened security analyst, consultant, engineer, level 17 packet prestidigitator gain access to this treasure trove of data? Traditionally, subscribers would co-locate a Linux host inside one of our SIE data centers. While this gives terrific performance and premium access to data, this is not always convenient for customers.

There is another way.

Farsight SIE Remote Access

SIE Remote Access (SRA) is Farsight’s software solution to make SIE content available to remote users. SRA enables SIE channel traffic to be delivered through a TCP stream across the Internet. In order to reduce the bitrates, SRA provides subscribers with the ability to invoke a server-side filtering capability across a set of channels, selecting only that subset of records that match specific domain name / IP address search criteria.

The Advanced Exchange Access suite of tools and library code is the software that implements Farsight’s SRA service.

AXA, what’s in the box?

The AXA suite consists of two Unix command line tools and one C library that daring developers such as yourself can use to build custom SRA applications.

Sratool

sratool is the AXA Swiss Army Knife. It is a versatile tool used to test, debug, or stream AXA connections. It connects to an SRA server, sends protocol messages and displays the responses. It can also tunnel SIE data like sratunnel.

Sratunnel

sratunnel transfers selected SIE data from the remote server to the local network. The connection to the server is created and restored after problems with binary exponential delays between retries.

Libaxa

libaxa is the C programming library that exposes the AXA API to the application programmer.

The Swiss Army Knife

sratool is the AXA Swiss Army knife. It is a versatile tool used to test, debug, inspect, or stream AXA connections. In its most common invocation, sratool connects to an SRA server, issues a few AXA protocol messages, and displays the responses.

So let’s get to it!

Tutorial 1: Watch Newly Observed Domains

This tutorial assumes you’ve signed up to receive Farsight’s Newly Observed Domains (NOD) datafeed to watch newly active domains and ensure your users don’t visit any newly minted – often malicious – domains. It shows you how to examine the Newly Observed Domains (NOD) feed in real time. Commands and their output are listed with discussion below.

1    $ sratool
2    > connect  ssh:sra-service@sra-eft.sie-remote.net
3    * HELLO srad version 0.2.5 sra-eft AXA protocol 1
4    > 1 watch ch=211
5    1 OK WATCH started
6    > count 5
7    > channel 211 on
8    * OK CHANNEL ON/OFF channel ch211 on
9    1 ch211  SIE newdomain
10     flyinghorse-colorado.com/A: flyinghorse-colorado.com
11   1 ch211  SIE newdomain
12     treatmentforboils.com/NS: treatmentforboils.com
13   1 ch211  SIE newdomain
14     servicedeck.com/NS: servicedeck.com
15   1 ch211  SIE newdomain
16     www.markenmacher.eu/A: markenmacher.eu
17   1 ch211  SIE newdomain
18     recruitniks.com/NS: recruitniks.com
19   packet count limit exceeded
20    > count
21        packet printing stopped by count 1990 packets ago

Lines 1-3: Invoke sratool, and use the connect command to establish a connection to the SRA server. The connection is managed via SSH, meaning all of the benefits conferred by the SSH protocol are available to sratool. Upon success, the client emits the hello string from the AXA_P_OP_HELLO message which was sent by the server and contains the server’s software version, name, and AXA protocol verison.

Lines 4-5: Inform the server we want to watch SIE channel 211 traffic (this is the NOD channel). The server responds with the current watch status. The watch is the most fundamental sratool command. This is how sratool “signs up” to receive data from the SRA server. As its name implies, watch sets up a watch which is a low-level primitive that tells the SRA server that the client is interested in nmsg messages or IP packets that meet one of the following criteria:

A watch is given a tag that is an integer label used to refer to the watch. An SRA server connection or session can have zero or more watches at a time and the user can add or delete watches as needed. Note that sratool allows only a single SRA connection at a time.

Line 6: Using the count command, we inform sratool we only want to see 5 packets. After this number is met, sratool will stop emitting packets to the screen (though traffic may still be flowing from server to client).

Lines 7-8: With the channel command, enable channel 211 (NOD). The current channel status is printed. Another fundamental command to sratool is channel. Issued alone on the command-line, it will emit the entire list of available SIE channels for which the user is provisioned.

Lines 9-19: sratool emits 5 NOD packets as it receives then from the server. Once the packet count limit is reached, emission stops.

Lines 20-21: Issuing the count command with no arguments prints the current count status. In this case, we find 1990 NOD packets have been streamed to the client, but since we exceeded our limit, they were not emitted to the screen.

Tutorial 2: Counts And Limits

Continuing in the session above, let’s tweak a few knobs and press a few buttons.

22    > list watches
23    1 ch=ch211
24    > 1 delete
25    1 OK STOP watch deleted
26    > rate
27    RATE LIMITS
28        unlimited per second; current value=307
29        10 seconds between reports
30    > rate 1
31    RATE LIMITS
32        1 per second; current value=2
33        10 seconds between reports

Lines 22-23: The list watches command prints all of the active watches. We’ve still got one going, we’re just not emitting any packets to the screen.

Lines 24-25: We delete the watch by referencing its tag with the delete command.

Lines 26-29: Another handy command, rate allows us to query the rate limiter and control it. Currently, there is no rate limiting in play – packets will be emitted as quickly as they appear. For lower bandwidth channels, like NOD, this is might not be a problem. For the DNSDB channels, which are much higher bandwidth, we’ll want to limit the rate at which those packets are sent by the server to sratool.

Lines 30-33: Using the rate command, we set a rate limit of 1 packet per second. This will come in handy in the last part of the tutorial where we’ll examine DNSDB.

Tutorial 3: Watch For A Specific Domain In Farsight’s Passive Dns Feed

As a bonus, let’s peek at SIE channel 202 traffic, Farsight’s raw passive DNS feed.

34    > 2 watch dns=*.github.com
35    2 OK WATCH started
36    > channel 202 on
37    * OK CHANNEL ON/OFF channel ch202 on
38    2 ch202  base dnsqr response  UDP_QUERY_RESPONSE
39         204.13.250.16.53 > 68.105.29.142.17296  IP TTL=58 UDP 86 bytes
40     DNS: raw.github.com IN A   qr aa  NOERROR  1 ans, 0 auth, 0 add RRs
41    2 ch202  base dnsqr response  UDP_QUERY_RESPONSE
42        208.78.71.16.53 > 208.106.17.39.64372  IP TTL=56 UDP 153 bytes
43    DNS: api.github.com IN A   qr aa cd  NOERROR  1 ans, 4 auth, 0 add RRs
44    2 ch202  base dnsqr response  UDP_QUERY_RESPONSE
45       204.13.250.16.53 > 68.105.28.174.52707  IP TTL=58 UDP 89 bytes
46    DNS: malsup.github.com IN A   qr aa  NOERROR  1 ans, 0 auth, 0 add RRs
47    * MISSED
48        lost 0 input packets, dropped 0 for congestion,
49            121 for per sec limit
50            since 2014/12/08 17:29:38
51    2 ch202  base dnsqr response  UDP_QUERY_RESPONSE
52       204.13.250.16.53 > 68.105.28.174.47116  IP TTL=58 UDP 149 bytes
53    DNS: github.com IN A   qr aa  NOERROR  1 ans, 4 auth, 0 add RRs

Lines 34-35: We set another watch, this time we want to watch for the wild card domain “.github.com”. Anything matching this domain will be emitted, such as *www.github.com and github.com itself.

Lines 36-37: We turn on channel 202, Farsight’s raw passive DNS channel.

Lines 38-53: All domains matching the watch are emitted.

Sratunnel

At the end of our last tutorial, you probably wondered aloud: “…this is cute but I need real-time bulk transfer of SIE data back to my network. Does this technology even exist in the modern world?”

Yes, yes it does.

sratunnel is the workhorse of the AXA family. It is used to transfer SIE data from the remote server to the local network. It is what Farsight uses for production deployment of SIE data to the customer. sratunnel can be thought of as a fast, efficient, and smart conduit for SIE data. Data goes in one end and sratunnel has a variety of nozzles the user can custom fit on the other end to emit the data into different output formats, including:

Tunnel Newly Observed Domains

Now that you know how to use the Newly Observed Domains (NOD) datafeed to watch newly active domains and ensure your users don’t visit any newly minted – often malicious – domains, we are going to show you how to to tunnel the data to your local network for bulk analysis.

NMSG Primer

To consume the data in this tutorial, you’ll need another Farsight implement called nmsgtool. It is a deeply useful all-purpose tool for working with NMSGs (network messages). NMSG is the format Farsight uses to type, structure and package arbirtary data for transit. Much of Farsight’s data is packaged and delivered as NMSG. A detailed discussion of the NMSG suite will be covered in future blog series. For now it’s just important to understand that you can work with NMSGs using nmsgtool. Onward!

This tutorial will show you, gallant Farsight datafeed customer, how to plumb the Newly Observed Domains (NOD) feed from SIE to your local network, in real time. Commands and their output are listed with discussion below.

1    $ sratunnel -s 'ssh:sra-service@sra-eft.sie-remote.net' \
2    > -c 211                                                \
3    > -w ch=211                                             \
4    > -o nmsg:udp:127.0.0.1,8430

Line 1: Invoke sratunnel. The -s option instructs the tool where and how to connect. The option string should look familiar, it’s the same one used with sratool with the same intent and results. Securely connect via SSH as sra-service to sra-eft.sie-remote.net.

Line 2: The -c option sets the channel you want to stream. We want NOD which is channel 211.

Line 3: The -w option sets the watch. You learned last week that a watch is how to inform the tool what to look for. In this case, everything on channel 211.

Line 4: Finally, we specify -o, which tells sratunnel where to put the data it streams. In the case above, we’ve snapped on a shiney new “NMSGs to localhost on port 8430” nozzle and that’s where we’ll find our output. Well done! You’ve plumbed your first SRA session. Data is aflowin’. Let’s build a small corpus and have a look…

5     $ nmsgtool -l 127.0.0.1/8430 \
6     > -c 20000                   \
7     > -o channel-211.txt
8     $ head -8 channel-211.txt
9     [98] [2014-12-16 23:31:06.438992023] [2:5 SIE newdomain] [a1ba02cf] [] []
10    domain: befrenshee.com.
11    time_seen: 2014-12-16 23:28:19
12    rrname: befrenshee.com.
13    rrclass: IN (1)
14    rrtype: NS (2)
15    rdata: ns67.domaincontrol.com.
16    rdata: ns68.domaincontrol.com.
17    $ grep ^domain: channel-211.txt | awk '{print $2}' > NOD.txt

Line 5: We use nmsgtool to connect to the loopback address on port 8430.

Line 6: The -c option specifies a maximum count of payloads to capture.

Line 7: The -o option tells nmsgtool to write presentation output to a file.

Line 8: Let’s examine one entry…

Line 9: Each NMSG datagram contains a fixed-length header containing the message size, a UTC timestamp, the message type, a 32-bit source identifier and optional SIE operator and group codes (both empty in this case).

Line 10: The fresh young domain, hot off the press!

Line 11: How fresh? At the time of this writing, that timestamp of when the domain was observed was just over two minutes old.

Lines 12-16: The DNS meta-data associated with the domain.

Line 17: You’re free to manipulate the data however you see fit. As per the above, you can take the list of 20,000 young domains and feed the file into the young domain crunching automation of your choice… :)

In summary

We learned how to invoke sratunnel and configure it to emit Farsight’s Newly Observed Domain traffic to a local NMSG listener. We then built up a small treasure trove of brand new domains and saw how to use standard Unix shell tools to manipulate the data.

This wraps our introductory series on AXA. In future blog series, we’ll build on this knowledge and explore the AXA C and Python APIs so you can learn to build your own SRA-aware tools. Until next time!

Ok I’m in, where can I get it?

The AXA suite is available for download on Farsight’s GitHub AXA https://github.com/farsightsec/axa page and available as Debian packages on our Debian package server.

I’ve got the tools, how can I get some data? You need to sign up https://www.farsightsecurity.com/order-services/. We offer several options for accessing our data, including a grant program for those that qualify. Reach out to us!

Additional Information

About Farsight Security

Farsight Security, Inc. is the world’s largest provider of historical and real-time DNS intelligence solutions. We enable security teams to qualify, enrich and correlate all sources of threat data and ultimately save time when it is most critical - during an attack or investigation. Our solutions provide enterprise, government and security industry personnel and platforms with unmatched global visibility, context and response. Farsight Security is headquartered in San Mateo, California, USA. Learn more about how we can empower your threat platform and security team with Farsight Security passive DNS solutions at www.farsightsecurity.com or follow us on Twitter: @FarsightSecInc.

This document was originally published on Farsight’s blog as Advanced Exchange Access by Mike Schiffman https://www.farsightsecurity.com/txt-record/2014/12/03/mschiffm-axaintro1of3/