January 11, 2004

Information Security Stillness: Tuning for Tranquility

Think for a second about your digital defences. When you are probed, do you know? When you are attacked, are you aware? Do you even care?

Over the last five years there has been a paradigm shift. Where in the past it was quite easy to review and audit logs for attacks, it is becoming much more difficult as of late. Why is this? Well, it is something I call the "noise poise". Far to many security tools report any activity that they see, without understanding the context that they are in. The result? Far too much noise overwhelms the information security professional, making it more difficult to really see what is going on. Intrusion detection sensors are the worst for this, because they trigger so many false positives that when a true attack occurs, many people miss it.

This can be fixed. What we have to do is tune our tools for silence, eliminating the regular false positives and putting ourselves in a state of tranquility, or stillness. This way we can truly listen, knowing that any noise made should be investigated further.

What do I mean? Consider network probing. Do you really care if some script kiddie runs nmap against your firewall? Probably not. Especially if your firewall is configured properly. How about the IIS related traversal attacks against your Apache server? (Snort reports I currently get hit with over 100 IIS attacks a day on my personal Apache server) Doubtful. Useless noise. The idea is to tune any security checkpoint you have to give you a sense of stillness. This may take time. A good way to do this is to go through a series of steps to get to that state of tranquility:

  1. Understand what it is that your security tool can log, and what it can’t. By first understanding its capabilities, you can then learn how to better utilize it.
  2. Understand what the security tool does in the context of your layered defensive posture. You will probably have internal firewalls protecting critical resources tuned more sensitively than you perimeter firewall against certain kinds of probes. Know your enemy to know thyself.
  3. Once you understand the landscape the tool is defending against, set it to its most sensitive setting, generating a plethora of logs so you can really see what it can trigger. Remember, many of the alerts will be false positives, and that is ok at this stage.
  4. Apply filters and adjust the logging settings to remove acceptable events. If possible, have these events stored in secondary event logs, allows for correlation later if needed. This is where the power of tools like Perl can really come in handy. It can help to filter commonly accepted events and sift through the noise for you. How did I know I get 100 IIS attacks a day on my Apache server? Because I have a Perl script that goes through my Snort reports and sifts out this information for me, giving me a single line telling me of the attack instead of having 100 separate event entries. I could just as easily not look for this sort of attack (being that I refuse to run IIS)... but I have my reasons for wanting to know when this sort of attack occurs.
  5. At this stage, the waters begin to calm. Events that are triggered now can be more scrutinized. Some events may not be malicious, but they may not fit in with your information security policies. Misconfigured networked devices may throw events, or unapproved software may attempt to do tasks that are unknown to the administrator. This is the time where you can fix your defences. You shouldn’t be trying to filter these events with your security tools, but rather fix the actual problem at the source. Either fix or remove the offending soft/hardware, or modify the security policy. Never just accept the activity as normal. False positives should be stomped out where possible.
  6. Stand in stillness. Listen for the enemy. Trust in your alarms, and verify the events.

There is a fine balance when doing this. If you don’t filter, you will be overwhelmed with logging events, and the attack will probably get through along with a plethora of extraneous logs and false alarms. Filter too much, and the attack will slip through undetected. Only you can decide how to set a balance between the two. When doing so though, consider this:
  • Remember the Rule of Immediate and Proper Response. You should be prepared to respond to anything. When someone yells "The sky is falling" (and they always do) you have to be able to quickly filter out the nutjobs from the sincere. You can only do this if you are working in stillness, looking for where the ripples are in you defences. When done right, you can respond with "I already know".
  • Know your Five W's. Ensure your filters take into account who is accessing what resource from where, why, and when. Consider what is considered normal activity during office hours, and when it is not. Should someone be accessing the corporate financial data at 2 in the morning? Should VPN access be allowed after hours? Your policy will dictate what is acceptable, and it will also allow you to tune the sensitivity of your filters accordingly. Where possible… use security tools that are intelligent enough to apply time of day conditional sensors and filters.
  • Respect the Rule of Change Management when modifying your logging facilities. No single person should be allowed to adjust, or more importantly turn off, security event logging without management understanding WHY. Intentionally ignoring security alerts is a bad idea. And leaving such critical data to a single person may breach the Rule of Trust.

Let me explain WHY ignoring security alerts is a bad idea. It all comes down to the human condition. The weakest link is the human factor… and there is a major reason for this. Too many false positives in any environment will eventually be ignored. Perhaps you heard of such a fable when you were a child. If you recall, a boy “cried wolf” one to many times, and was gobbled up. The same is true in the security world. There are stories about how in World War II, the French resistance would purposely trigger security defences three or four times in a night to annoy the Germans into turning off their safeguards, or giving up on checking the alarms all together because they were tired of false alarms. Then they would sneak by undetected. Or the hacker that would continiously trigger the IDS sensor...causing the admin to turn it off because it continiously paged him all night every night for a week. (Coincidentally... this specific example occured in a few well published attacks)

Don’t ever relax your security in this manner. When you least expect it… expect it.

Be vigilant. Be safe. And be still.

Posted by SilverStr at January 11, 2004 04:21 PM | TrackBack
Comments

But Master, while I am but a simple and foolish grasshopper, I do not know what the best way to filter the noise is. I truely wish to reach the nirvana that you have in understanding of my network alerts, but I am not as learned as you are master, and I beg you to help me find the path.

Posted by: Arcterex at January 12, 2004 08:55 AM

I like the post and the idea behind it, but is this really possible with current Intrusion Detection devices and host logs? I have finally been getting the Snort alerts for the enterprise I protect down to the level of less than 100 per day, on a good day. Most of those are false alerts or normal traffic I can dismiss with a glance. But I need the alarms that are being triggered to be there for anomolous traffic too. If I filter much more, I run the possibility of false negatives, not getting alerts when there really is an incident.

The point you make about the human element being the weakest link is valid, but human beings can also be the strongest link. Only a human being can really make the decision if an alarm is realy or not. You need to know your network, know what is common traffic, know what is acceptable traffic, and, most importantly, know what is bad traffic. There is, to the best of my knowledge, no other way to get to this point of network defense than a lot of time spent pouring over your logs.

I'd rather have to spend a little extra time going over the logs and be pretty sure I saw everything the bad guys are throwing at me, rather than have that nervous thought that maybe I'm missing something. But that's just me.

Posted by: Martin McKeay at January 13, 2004 06:56 AM

Martin,

You are absolutely right. Points very well made. Think about when you first installed snort. The vast amounts of events were just nuts. You have already tuned your IDS sensors to a stage that you are willing to accept. I would bet that you are down over 50% or more in the amount of events that you are now reviewing since the tuning... and I could gather that from your experiences in your network, that the "normal traffic" you are seeing can be tweaked out with a further filter... your brain. I would point out that if you know a good portion of the traffic is normal traffic consider having a parser seperate that out into a second log... allowing for the "not normal" to stand out. Personally I have all my logs imported into a sql database source through cron, and can create queries to further refine my reporting as well as to provide agrregate data.

I like your view about how people are also the STRONGEST point. This is a ying and yang thing. And very true. No tool is as good as an human that can interpret the data. Computers are actually very dumb. They only do what you tell them. Nothing more. People have to refine the results the computer generate to get a real picture and understanding of what is going on.

Spending a little time reviewing logs isn't a bad thing. But streamlining that process to any degree only helps to stand in the stillness. Imagine if you didn't tune your snort filters and had to review thousands of events daily. Chances are you would just as easily miss something as if you filtered to much generating false negatives.

As I said in the original post... its a fine balancing act. Only you can decide how to strike a balance between the two.

Excellent insight here. Thanks for posting!

Posted by: SilverStr at January 13, 2004 08:10 AM

one of the easiest ways to do this is to graph the data you measure. this isn't as easy as it sounds, you do have to learn how to measure in the process (ie its not sufficient to graph snort alerts per hour). *but* it does mean you can learn to measure everything, do trending (via aggregation) and look for deviations. graphing this means that you can intuitively spot a trend: a rapid upsurge in bandwidth and firewall hits for a single port suggest something akin to a worm. a sudden upsurge in port 25 traffic means a spammer is active or a mail worm is active. things like that.

tools i use include rrdtool and aguri and a lot of custom scripts to graph the data and present it. i had a large set of awk and grep scripts (partially described in a sysadmin magazine article i wrote some years ago) which could help me accumulate gobs of data into a simple report. i could spot slow scans or legitimate problems in addition to the regular stuff, all in under ten minutes a day.

you're right in that you have to tune, but measuring helps you tune. record this and do it automatically. record everything, classify everything, do various cross products and the like. this is, after all, what i do for a living. watch everything by keeping track at a coarse level and then having access to the data at a finer level when you need it.

tools to look at: use aguri (time and space based aggregation); use argus (builds flows from packet traces, useful summaries); learn perl or awk or python; learn rrdtool; if you have access to netflow sources (via routers, switches, or pcap tools), use cflowd from caida or the flow-tools from osu.

Posted by: jose nazario at January 13, 2004 08:23 PM