My next class:
Reverse-Engineering Malware: Advanced Code AnalysisOnline | Greenwich Mean TimeOct 28th - Nov 1st 2024

Querying the DShield API from RTIR

Published: 2015-09-03. Last Updated: 2015-09-03 10:01:43 UTC
by Xavier Mertens (Version: 1)
5 comment(s)

A few days ago, Tom wrote a diary(1) about RTIR(2) and its REST API. He explained how the tool can be fulfilled with external data. Being a DShield contributor for years (I submit my firewall logs), I like to search for IP addresses information in the DShield database. By default, RTIR extracts IP addresses from tickets and has an interface to query services like WHOIS servers, to perform a traceroute or to query any third-party website. RTIR being extremely configurable, why not extend it to query the DShield database using the ISC API(3)!

If IP addresses can be queried via the URL "https://isc.sans.edu/ipinfo.html?ip=x.x.x.x", don't do this. First of all for performance reasons but the page cannot be displayed in an iframe (that's the case in RTIR) because it sets the 'X-Frame-Options' to 'SAMEORIGIN'. To query details of an IP address, use the following IP API call:

https://isc.sans.edu/api/ip/x.x.x.x

Results are returned in XML. To integrate DShield lookups into RTIR, follow this procedure.

1. Create a new page called "isc_ipinfo.php" in your Apache server running RTIR (or any available HTTP server). This page will receive the IP address, query the DShield API and reformat (basically) the XML output:

<?php
$ip = $_GET['ip'];
if (!filter_var($ip, FILTER_VALIDATE_IP)) {
        echo "Invalid IP address!";
        exit;
}
$d = simplexml_load_file('https://isc.sans.edu/api/ip/'.$ip);
?>
<table>
<tr><td align="right"><b>IP Address:</b></td><td><?php echo $d->ip; ?>(<a href="https://isc.sans.edu/ipdetails.html?ip=<?php echo $ip ?>" target="_blank">Details</a>)</td></tr>
<tr><td align="right"><b>Network:</b></td><td><?php echo $d->network; ?></td></tr>
<tr><td align="right"><b>AS:</b></td><td><?php echo $d->as; ?></td></tr>
<tr><td align="right"><b>AS Name:</b></td><td><?php echo $d->asname; ?></td></tr>
<tr><td align="right"><b>AS Size:</b></td><td><?php echo $d->assize; ?></td></tr>
<tr><td align="right"><b>Country:</b></td><td><?php echo $d->country; ?></td></tr>
<tr><td align="right"><b>Count:</b></td><td><?php echo $d->count; ?></td></tr>
<tr><td align="right"><b>Attacks:</b></td><td><?php echo $d->attacks; ?></td></tr>
<tr><td align="right"><b>Min Date:</b></td><td><?php echo $d->mindate; ?></td></tr>
<tr><td align="right"><b>Max Date:</b></td><td><?php echo $d->maxdate; ?></td></tr>
<tr><td align="right"><b>Last Updated:</b></td><td><?php echo $d->lastupdated; ?></td></tr>
<tr><td align="right"><b>Abuse Contact:</b></td><td><?php echo $d->asabusecontact; ?></td></tr>
<tr><td align="right"><b>Comment:</b></td><td><?php echo $d->comment; ?></td></tr>
</table>

2. Edit your $RTIRHOME/etc/RTIR_SiteConfig.pm and add the new service in $RTIRIframeResearchToolConfig (pointing to your URL):

Set($RTIRIframeResearchToolConfig, {
    1 => { FriendlyName => 'SANS ISC IP Info', URL => 'http://xxxxxxxx/isc_ipinfo.php?ip=__SearchTerm__' },
    3 => { FriendlyName => 'Google', URL => 'https://encrypted.google.com/search?q=__SearchTerm__' },
    4 => { FriendlyName => 'CVE', URL => 'http://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=__SearchTerm__'},
    5 => { FriendlyName => 'TrustedSource.org', URL => 'http://www.trustedsource.org/query/__SearchTerm__'},
    6 => { FriendlyName => 'McAfee SiteAdvisor', URL => 'http://www.siteadvisor.com/sites/__SearchTerm__'},
    7 => { FriendlyName => 'BFK DNS Logger', URL => 'http://www.bfk.de/bfk_dnslogger.html?query=__SearchTerm__#result'}
} );

3. Restart your RTIR instance and enjoy! You can now query the DShield API:

It's also easy to create new "portlets" to be used in dashboards. As a bonus, let's display the ISC Infocon status in a RTIR dashboard. 

1. Create the new portlet in $RTIRHOME/local/html/Elements. Let's call it "InfoconStatus":

<&|/Widgets/TitleBox, title => loc('SANS ISC Status')&>
<table>
        <tr>
                <td>
                <img src="https://isc.sans.edu/images/status.gif" alt="SANS ISC Infocon Status">
                </td>
        </tr>
</table>
</&>

2. Enable the new portlet in $RTIRHOME/etc/RTIS_SiteConfig.pm:

Set(@RTIR_HomepageComponents, qw(
    QuickCreate
    Quicksearch
    MyAdminQueues
    MySupportQueues
    MyReminders
    RefreshHomepage
    Dashboards
    SavedSearches
    InfoconStatus
    /RTIR/Elements/NewReports
    /RTIR/Elements/UserDueIncidents
    /RTIR/Elements/NobodyDueIncidents
    /RTIR/Elements/DueIncidents
));

3. Restart your RTIR instance and adapt your favorite dashboards:

(1) https://isc.sans.edu/forums/diary/Automating+Metrics+using+RTIR+REST+API/20087/
(2) https://www.bestpractical.com/rtir/
(3) https://isc.sans.edu/api/

Xavier Mertens
ISC Handler - Freelance Security Consultant
rootshell.be
truesec.be

5 comment(s)
My next class:
Reverse-Engineering Malware: Advanced Code AnalysisOnline | Greenwich Mean TimeOct 28th - Nov 1st 2024

Comments

Some addresses gives confusing results:
<ip>
<ip>127.0.0.1</ip>
<paddedip>127.000.000.001</paddedip>
<count>202222</count>
<attacks>1</attacks>
<maxdate>2015-09-04</maxdate>
<mindate>2014-11-15</mindate>
<updated>2015-09-04 04:02:47</updated>
<comment/>
<asabusecontact>hostmaster@nic.ad.jp</asabusecontact>
<as>17676</as>
<asname>GIGAINFRA Softbank BB Corp.</asname>
<ascountry>JP</ascountry>
<assize>40136086</assize>
<network>126.255.0.0/16</network>
</ip>

<ip>
<ip>10.0.0.1</ip>
<paddedip>010.000.000.001</paddedip>
<count/>
<attacks/>
<maxdate/>
<mindate/>
<updated/>
<comment/>
<asabusecontact>abuse@level3.com</asabusecontact>
<as>3356</as>
<asname>
<![CDATA[ LEVEL3 - Level 3 Communications, Inc. ]]>
</asname>
<ascountry>US</ascountry>
<assize>53716672</assize>
<network>8.255.146.0/24</network>
</ip>
I don't know why RFC1912 and loopback IP addresses are in the DShield DB indeed.
On the other side, it's not relevant to query information about them.
I fixed the badly identified RFC 1918 addresses. The issue was that if an IP was not routed, the closest (but smaller) routable IP address space was shown. Should be fixed now. You may still see cached data for a bit.

As for importing RFC1918 networks: I recently stopped doing it. I was a bit conflicted about it. In some ways, I want to import the data "as received", and people do send RFC1918 traffic. But since this is most likely internal data that should not have been sent, I stopped importing it. Old data will be kept for now.
Would be nice to do it also for other systems such as MISP... :)
Indeed, I added this in my todo/wishlist.

Diary Archives