Last Updated: 2014-10-30 16:40:03 UTC
by Rob VandenBrink (Version: 1)
Often the start of a problem and it's solution is receiving a call from a manger, project manager or other non-technical decision maker. You'll know going in that the problem is absolutely real, but the information going in might be a total red herring.
Some classic examples are:
"The network is slow – I ran a speed test, we should being seeing 10x the speed."
This is almost always a math error. The speed was measured in Bytes (upper case B), instead of bits (lower case B). Multiply by 8 and things should look better.
“the network is slow – our new web server takes 30 seconds to load the lead page”
As most of you know, in a modern gigabit network, even on a busy network there just isn’t anything on the network that will add a 30 second delay. 30 seconds in particular would have me checking for DNS issues first, especially for a new host or service. However, in this case, the client was loading their entire Java application (including the business logic) before the login page. The “appdev” answer to this would be to load the login page first, then load the app asynchronously in the background. The security answer to this is to question why you would load the application logic to an untrusted workstation on a hostile network (public internet).
The network is slow – it must be a broadcast storm.
It’s exceedingly rare to see a broadcast storm. Plus if the switches are configured correctly, if a broadcast storms does occur, it should be contained to a single Ethernet port, and it should either be rate limited or the port should be shut down, depending on your configuration.
When a non-technical person says “broadcast storm”, it really could mean anything that affects performance. Almost always it will end up being something server side – DNS misconfigurations are a common thing (10-30 second delays on the first request), but it could also be an oversubscribed virtual infrastructure, coding errors, out of memory conditions, errors in programming, anything really.
The firewall is blocking our traffic
In some cases, especially if there is an egress filter, this can be the case. However, in many other cases it could be something else entirely. We recently worked on an issue where an AS400 (iSeries now I guess) was not connecting to the server. It turned out that the certificate needed for the connection was incorrect - the vendor had sent us a cert for a different site entirely. Wireshark did a great job in this case of saying "LOOK HERE- THE PROBLEM IS HERE" by giving us a "Bad Certificate" error - in bright red - in the main view.
We need port 443 open, in both directions
This is NEVER the case, but is commonly seen in vendor documentation. Either you need an outbound port (possibly an update to the egress filter), or an inbound port open. There are very few “in both directions” requirements - special cases like IPSEC VPN’s encapsulated in UDP (NAT-T) for instance will have both a source and destination port of udp/500. In most cases, when the requirement is “in both directions” or “bidirectional”, it’s a bit of a treasure hunt to figure out what they mean (usually it’s outbound).
The moral of the story? I guess the first one is that if somebody tells you that the problem is the network, 70% of the time it’s not the network. More importantly though, is that if you get a business problem from a business person, it’s not something to minimize. You might not be able to count on all the information you get going in, but if they tell you something is slow or not usable, it’s their system, they are usually correct in at least identifying that the problem is real.
Please, use our comment form and fill us in on any recent false positives from a non-technical source that you've seen. Extra points if it was a real problem, but the initial info started you off in the wrong direction.
Last Updated: 2014-10-30 02:38:37 UTC
by Rob VandenBrink (Version: 1)
Recently we seem to have a theme of new bugs in old code - first (and very publically) openssl and bash. This past week we've had a bunch more, less public but still neat bugs.
First, a nifty bug in strings - CVE-2014-8485, with more details here http://lcamtuf.blogspot.ca/2014/10/psa-dont-run-strings-on-untrusted-files.html
a problem in wget with ftp: https://community.rapid7.com/community/metasploit/blog/2014/10/28/r7-2014-15-gnu-wget-ftp-symlink-arbitrary-filesystem-access
and now the ftp client (found first in BSD) - http://cxsecurity.com/issue/WLB-2014100174
These all share some common ground, where data that the code legitimately should be processing can be crafted to execute an arbitrary command on the target system. The other common thing across these as that these utilities are part of our standard, trusted toolkit - we all use these every day.
Who knew? Coders who wrote stuff in C back in the day didn't always write code that knew how much was too much of a good thing. Now that we're all looking at problems with bounds checking on input data, expect to see at least a couple more of these!
If you have more information or corrections regarding our diary, please share.