Published: 2022-08-08

JSON All the Logs!

My recent obsession has been creating all of my logs in JSON format. The reasons for that are pretty simple: I like to log with Elasticsearch, so creating JSON formatted logs makes working with Elasticsearch easier. Command line tools like 'jq' make parsing JSON logs on the command line simpler than "good old" standard Syslog format and a string of 'cut,' 'sed,' and 'awk' commands. 

Before going into examples, first a few caveats when it comes to creating JSON logs:

Proper Output Encoding

Make sure you properly encode special characters in JSON. For example, characters like '{, ''},' quotes, slashes, and such need to be escaped. Some log generation tools will do it for you (see examples below)

Proper and Consistent Types

This comes down to quotes or no quotes around numbers. If you want a value to be treated as a number: Do not enclose it in quotes. But if a value isn't always a number (consider what happens if the field doesn't exist), it should be enclosed in quotes. For example, port numbers are... usually... numbers. But what if it is an ICMP packet and there are no ports? Or, let's say a firewall log showing a fragment that isn't the first fragment getting blocked? I have seen people use "0" as a port, which works, but can lead to confusion.

Consistent Names

Downstream processing is a lot easier if the source IP address is always called "srcip" (or whatever name you come up with). This will reduce the effort needed to normalize naming later. Of course, depending on the software originating the logs, you do not always have a choice.

Your Downstream Log Processing Pipeline

I mentioned Elasticsearch above, and it loves JSON. But many systems process logs and yours may not be ready to process JSON. Converting from JSON to the format of choice may still be more straightforward than writing hundreds of different Logstash rules, but if you are already set up for whatever format you use, the change may not be worth it.

Creating JSON logs can be pretty simple, depending on your software. Here are a few examples:


The Nginx web server has a helper function to escape JSON properly. To output logs in JSON format, use something like this:

log_format main escape=json '{"host": $http_host", "srcip": "$remote_addr", time_local": "$time_local", "request": "$request" ...}';


Apache uses a similar log format directive. But there is no simple JSON escape. I have to test Apache a bit more to see if it will escape various characters appropriately.

LogFormat "{ \"time\":\"%t\", \"srcip\":\"%a\", \"host\":\"%V\", \"request\":\"%U\", \"query\":\"%q\",...}" json


Recent versions of rsyslog do support JSON encoded logs. You will have to define a custom template like:

template(name="outfmt" type="list" option.jsonf="on") {
                  dateFormat="rfc3339" format="jsonf")
                  name="hostname" format="jsonf")
                  name="syslogseverity-text" caseConversion="upper" format="jsonf")
                  name="syslogfacility-text" format="jsonf")
                  name="syslogtag" format="jsonf")
                  name="app-name" format="jsonf")
                  name="msg" format="jsonf")

see: https://rainer.gerhards.net/2018/02/simplifying-rsyslog-json-generation.html


For any software written in Python, there is a JSON-logging module (see https://pypi.org/project/json-logging/ )


For the Bind name server, I only found limited JSON support in the statistics channel. It is an optional feature (with XML being another option)

see: https://bind9.readthedocs.io/en/latest/chapter10.html?highlight=json#optional-features

Missing Pieces

So far, the two items I am missing the most are firewall logs (BSD or iptables/nftalbes) and Postfix or other mail servers. Got any other tricks? Let us know.

[comments are sadly not working right now. Hope to have them back today/tomorrow]


Johannes B. Ullrich, Ph.D. , Dean of Research, SANS.edu



Published: 2022-08-04

TLP 2.0 is here

Earlier this week, the global Forum of Incident Response and Security Teams – or FIRST, as it is commonly known – published a new version of its Traffic Light Protocol standard[1]. The Traffic Light Protocol (TLP) is commonly used in the incident response community, as well as in the wider security space, to quickly and in a standardized way indicate any limitations on further sharing of any transferred information.

Since different organizations and security teams around the world use differing (and not necessarily compatible) standards for information classification, it can be difficult to quickly share any sensitive information with anyone outside the organization without also appending the entire information classification standards of the source organization that specifies how/whether the recipient may use and further share the information. This is where the TLP comes in and why it is quite valuable, since it provides everyone with a common, easy to understand and easy to use information classification scheme. One only has to indicate (in an e-mail subject, on a first slide of a presentation or document, in spoken exchange, …) that the information that is about to be shared has specific TLP label, and the recipient should be able to immediately understand how they may use it and with whom (if anyone) they may share it.

The new version of the standard brings several important changes, the most visible one having to do with the classification labels. In its previous iteration[2], the TLP consisted of the following four labels that governed how the transferred information may be shared:

  • TLP: WHITE – Disclosure of information is not limited
  • TLP: GREEN – Limited disclosure, recipients can spread information within their community
  • TLP: AMBER – Limited disclosure of information, restricted to participants’ organizations only
  • TLP: RED – Not for disclosure, information restricted to exchange participants only

In the 2.0 version of the standard, TLP: WHITE has been renamed TLP: CLEAR and a new TLP: AMBER+STRICT label was added. Some changes have also been made to the definitions and the overall language as well as to some other minor areas, which should help minimize any uncertainty in the meaning of different labels.

You may find the entire standard on the FIRST website, but in general, the new classification basically gives the following meaning to each of the labels.

If you use TLP in your daily activities, it would be advisable to start using its new iteration as soon as possible, since the 2.0 version of the standard is now considered authoritative.

[1] https://www.first.org/tlp/
[2] https://www.first.org/tlp/v1/

Jan Kopriva
Nettles Consulting


Published: 2022-08-03

l9explore and LeakIX Internet wide recon scans.

Earlier today, I noticed a scan for an odd set of vulnerabilities from

Apache directory traversal issue
GET /cgi-bin/.%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/etc/hosts HTTP/1.1


Looking for secrets/fingerprinting
GET /config.json HTTP/1.1
GET /.DS_Store HTTP/1.1
GET /.env HTTP/1.1
GET /.git/config HTTP/1.1
GET / HTTP/1.1
GET /idx_config/ HTTP/1.1
GET /info.php HTTP/1.1
GET /.json HTTP/1.1
GET /login.action HTTP/1.1
GET /server-status HTTP/1.1
GET /s/lkx/_/;/META-INF/maven/com.atlassian.jira/jira-webapp-dist/pom.properties HTTP/1.1
GET /telescope/requests HTTP/1.1


Connecting to a non-TLS server with TLS

Based on these URLs, I would call this more "fingerprinting" or "vulnerability discovery" and less so an actual exploit, even though you may argue that this will, of course, set the attacker up for exploits, in particular, if sensitive secrets are leaked in the configuration files. Some of them are oddly specific, like the directory traversal attempt. 

So far, I have found three different IPs scanning "the internet" for these URLs:

%%ip: (Google Cloud)
%%ip: (probe-ny001.rand0.leakix.org, Digitalocean)
%%ip: (NovinVPS)

All requests used the same user agent: l9explore/1.2.2. This user agent, like the second IP address in the list above, is associated with the internet scanning platform Leakix. The scanner itself is open source, so possibly the other IPs just run it outside of Leakix. But it maybe a user agent to add to a blocklist to keep the noise down on your systems. Also review that you are not exposing any sensitive information via the URLs listed above. Leakix isn't the only one scanning for them.

IP addresses associated with LeakIX can be found in our "research" feed:

https://isc.sans.edu/api/threatcategory/research/?json. (for all researchers scanning the internet)
https://isc.sans.edu//api/threatlist/leakix?json (just leakix)

Johannes B. Ullrich, Ph.D. , Dean of Research, SANS.edu


Published: 2022-08-02

Increase in Chinese "Hacktivism" Attacks

With the US Speaker of the House, Nancy Pelosi, approaching an unusually high-level visit to China, various reports indicate an increase in military saber-rattling and a ramp-up of attacks against networks in Taiwan and the US.

So far, we have more anecdotal evidence vs. "real data." But some of the initial indicators we have seen:

  •  A slight increase in scans for "nuisance vulnerabilities" like Word Press from Chinese consumer IP addresses.
  • Reports of small/medium application-specific DDoS attacks similar to what our site has seen starting Friday
  • A small (not quite significant based on preliminary data) increase in ssh scanning from Chinese consumer IP addresses.

Chinese hacktivists have a history of picking up on government sentiment communicated in local news reports [1]. They will often show their patriotism by attacking various "unfriendly" websites. The targets are often somewhat random, and the attacks are not coordinated. But even a home user with a small botnet can harness significant "firepower" to take down many websites without dedicated DDoS protection. And, of course, sometimes they get lucky scanning for simple vulnerabilities. If a few million (probably more than a few thousand) "kids" are brute forcing passwords, they may just get lucky and find one.

What do you need to do?

Not much at this point. Monitor and be ready for a DDoS attack. In particular, if your website or company has a higher profile in China or is associated with the US government (this includes contractors, related organizations, and news sites reporting about the visit).

For example, the Taiwan president's website experienced a DDoS attack of approximately 200 times the regular traffic [2]. I do not consider this a "huge" attack and something likely within the capabilities of a few hacktivists getting together. A more organized "government-sponsored" DDoS attack would likely involve tools like "Great Cannon" (sometimes also called red-ion-cannon) that can harness a much larger attack power [3].

Please use our "contact us" form to report any attacks you are seeing.

[1]  https://scholarworks.lib.csusb.edu/cgi/viewcontent.cgi?referer=https://www.google.com/&httpsredir=1&article=1413&context=etd
[2] https://m.facebook.com/story.php?story_fbid=pfbid0oetXRVEQ2dj7Vd1kTzC32FhdMLdyuoQJAYf6baYJDghKKVCBMERfUgXhP72U4obVl&id=100044311095166&m_entstream_source=timeline
[3] https://citizenlab.ca/2015/04/chinas-great-cannon/


Johannes B. Ullrich, Ph.D. , Dean of Research, SANS.edu


Published: 2022-08-02

A Little DDoS in the Morning - Followup

I love it when people read and learn from what I am writing here. And it looks like whoever is behind the little DDoS from Friday did just that. I removed our filters yesterday after the attack stopped, and today see similar traffic ... but... now with different user agents ;-). At this point, the traffic isn't causing any performance issues, so I will let them go for now.

Here is a small sample of user agents involved. The disadvantage for them is now that some of these User-Agents are so unusual that they again easily stick out. The source IPs are still all Chinese. FWIW: I got some questions if the source IPs could be spoofed: No. They are sending complete HTTP requests, so they are not spoofed. But they could be some kind of proxy on compromised machines/devices.

Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36
Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36
Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36
Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.2117.157 Safari/537.36
Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.2309.372 Safari/537.36
Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36
Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-HK) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1623.0 Safari/537.36
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.17 Safari/537.36
Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; Media Center PC 4.0; SLCC1; .NET CLR 3.0.04320)

I feel like this will become a little series of posts.

Johannes B. Ullrich, Ph.D. , Dean of Research, SANS.edu


Published: 2022-08-01

A Little DDoS In the Morning

Friday morning (at least it wasn't Friday afternoon), we got an alert that our database and web servers exceeded the expected load. Sometimes, this "happens." Often it is just some user innocently flooding our API with requests. We do use quite a bit of caching and such for requests, but it can happen that things pile up at the wrong time. So I took a look at the logs. In these cases, I first look at the top IPs sending requests to our API. The first IP that stood out was %ip: At the time, it had sent about 6,000 requests in 3 hrs. Nothing that would typically cause problems. But the requests themselves didn't make much sense. A small sample:

GET /api/ip/2001:14ba:1f8:c600:94ab:7958:3347:1faa?json HTTP/1.1
GET /api/ip/2001:14ba:7ff:d00:3575:c285:aaaf:4c5e?json HTTP/1.1
GET /api/ip/2001:14bb:150:2f72:7d26:2f5:b2f9:490a?json HTTP/1.1
GET /api/ip/2001:14bb:150:2f72:88c3:36b3:c5b8:507a?json HTTP/1.1
GET /api/ip/2001:14bb:150:2f72:d1f:d7cc:d8b4:6e36?json HTTP/1.1
GET /api/ip/2001:14bb:150:2f72:f196:6cce:b08e:ec1b?json HTTP/1.1
GET /api/ip/2001:14bb:170:ca0:4ca1:cc47:9f4f:5f9f?json HTTP/1.1
GET /api/ip/2001:14bb:170:ca0:901d:76cb:f861:67f4?json HTTP/1.1
GET /api/ip/2001:14bb:170:ca0:a4e7:9ed:8b8f:b8e?json HTTP/1.1

This API endpoint doesn't offer IPv6 data. We do not have enough IPv6 data yet to make this work well. Secondly, it looks like they are "scanning" random IPv6 addresses. There were likely a few more gazillion queries coming to complete this scan. In other words: This will take a while and doesn't make much sense. The user-agent they used curl/7.29.0pointed to a script and didn't comply with our guidelines. So I started by blocking this IP address, but the load on our systems remained high. 

So I started to look at other logs and ran into a flood of requests for https://dshield.org/ipinfo.html. These requests went to dshield.org, not isc.sans.edu, but both sites share the same backend. 

This is where things got a bit more interesting. A quick sample of the logs again:, /ipinfo.html?ip=, /ipinfo.html?ip=, /ipinfo.html?ip=, /ipinfo.html?ip=, /ipinfo.html?ip=, /ipinfo.html?ip=, /ipinfo.html?ip=, /ipinfo.html?ip=, /ipinfo.html?ip=, /ipinfo.html?ip=

This snapshot is after the attack was in progress for a while. The IP addresses appeared to be incrementing. Targeting "ipinfo.html" was likely because this is one of our slower pages. It is even slower if you hit random non-existing IPs. All requests used the same user agent again:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ Safari/537.36

The user agent looks normal for a slightly older version of Google Chrome. So nothing unusual. By spreading the requests over multiple IPs, it was more challenging to identify the offending requests, but the user-agent was helpful. Looking at the signature of requests to ipinfo.html from this user agent, we counted about 2 million requests against dshield.org but less than 100 against the same URL at isc.sans.edu, the more popular website. So we got a culprit.

More than 100k source IPs participated in the attack. But the most requests we got from a particular IP address were about 350. So there was no clear "winner" to block easily. But looking at the IP addresses, I noticed they came from the same subnets. Summarizing by /16s, only 506 subnets remained. And some of them originated a significant number of requests. 117.69/16 was the most active network, followed by 27.156/16.

Based on whois data: - China Telecom - Broadband network, the city of fuzhou domain, Fujian Province

It turned out the requests causing problems originated exclusively from China. So I started blocking requests from China in our proxy/waf that protects DShield. The load on our servers almost immediately recovered. I already had country lookups configured on the server. We are using Nginx as a proxy, and blocking by country is pretty straightforward:

First, load the geoip_country lookup with data from Maxmind

geoip_country /usr/share/GeoIP/GeoIP.dat;

next, set an "$allowed_country" variable based on the country of origin returned by the country lookup (this will make it easier to block multiple countries)

map $geoip_country_code $allowed_country {
         default yes;
         CN no;

Finally, as part of the location directive, return an error if the country is not allowed

location / {
          if ($allowed_country = no) {
                return 444;

(I started with 666, but figured for China, 444 is more appropriate).

How much traffic was it overall? Here is a quick graph of the traffic over the weekend from our proxy:

Graph of DDoS traffic from Friday July 29th to Monday August 1st.

Note how it all stopped this morning. Likely whoever started it came into work and noticed the attack no longer having any effect. I will keep you updated as to what we see next.

Lessons learned:

  • Always nice to have a few scripts ready to summarize your logs quickly. Either on the command line or in your fancy log monitoring system. 
  • Often, you need to make a quick decision to block an attack accepting some false positives but allowing most of your traffic to succeed. It buys you time and breathing room to find a more specific block.
  • I'm not too fond of country-level blocks. But in this case, it kept things going again, and we will remove the block soon after the attack stops.

Some final words:

  • This was not a significant DDoS attack. It was pretty small. In the end, you have to "buy your way out of a DDoS attack." For DShield.org, I am willing to accept a bit of downtime like this if I can learn from it.
  • Who was behind this attack? I have no idea. All requests came from China. But this could be Chinese hacktivists or just "for the LOLs."
  • Were there > 100k systems attacking us? I doubt it. The rate was too low for 100k systems. A lot of the IPs were mobile systems. I suspect they had very dynamic IP addresses, and some devices may show up with multiple IP addresses. My best guess is that these were infected mobile gateways. We see a lot of Mirai-like activity against systems like this. But who knows? I haven't had a chance to look into this closer.
  • Am I giving away too much by writing this up? I doubt it, and our mission is to share information and to educate. You do not hear much about these little nuisance DDoS attacks in the news, but they are a big problem for many small businesses.

For a write-up about a more interesting Chinese DDoS in the past, see this attack that involved the Chinese Great Firewall.

[our comments are currently broken, unrelated to the topic of this post. We will have them back shortly. You can reach us via our "Contact Us" form.]

Johannes B. Ullrich, Ph.D. , Dean of Research, SANS.edu