Log Analysis Professionals

SANS Security Laboratory “Thought Leaders” Article

Posted by Andrew Hay under Log Analysis Professionals

Stephen Northcutt, of SANS Institute fame, recently recognized me as a Thought Leader in the area of log management. I’m quite humbled to be included with the likes of our own Log Analysis Professional contributors Dr. Anton Chuvakin and Ron Gula (among others).

The interview has been posted on the SANS Technology Institute site here. This has certainly made my week

How to Make Logs Sexy … Again!?

Posted by Anton Chuvakin under Log Analysis Professionals

Now, some people hate logging, because logs are too hard to deal with (enable, collect, store and especially understand and interpret). However, there is a whole other group of fairly intelligent people who “hate logs:” the organizers of some well-known technical security conferences. The experience of many of my colleagues (and competitors!) and myself proves that a log-related talk will NOT be accepted to ANY technical security conference nowadays. Now, some were generous enough to explain why. Others were not (screw them and no link :-)).

But let me rant about this one a bit. First, it is always a possibility that they dislike me not logs:-) - this is easily disproved, however, since some of my colleagues had the same exact experience. Do they dislike vendors talking about logs? Nah, this isn’t it either - most of my conference presentations had nothing to do with LogLogic, even though they are about logs. Some of my friends (and this blog readers) tried to suggest that an audience of such events “knows everything there is to know about logs.” This is not true since - gasp!- nobody knows everything there is to know about logs: they hide way too many mysteries (with useful answers!) to discount them like that. Another one I’ve heard is that “real hackers don’t get logged -> logs are useless”, which is also silly: this is true only if you take a very narrow view of logs (e.g. NIDS alerts),; clearly, everybody is logged by the firewalls, servers, apps, etc. The challenge is not a lack of data, but too much data and not enough time and tools.

But we are about to “hit paydirt” with this question…

Tool? Did I just mention tools? This opens the last and final, deeply evil reason for such “log-hate”: one of the conference organizers mentioned that, in his opinion, there is nothing new in the field of log analysis since regex-match-based alerting (and regex-based parsing into database tables).

And you know what?

Drum roll….

He was actually somewhat right.

Indexing did come in the world of logging, but, personally, I don’t find it to be a huge feat of human ingenuity (even though it is definitely useful). I also think we are not doing enough with index data (and I definitely intend to change that…)

In addition, there was A LOT of academic research on the subject, from the SRI EMERALD in the 80s (and even earlier) to today, but many of the papers I’ve seen sit on the “hilarious side of useless”…

So, we need a campaign “Making Logs Sexy Again!” (the term “sexy logs” and - yet uncreated - picture of a dancing sexy log are copyright Andrew Hay :-))

How to Fight “Log Apathy”?

Posted by Anton Chuvakin under Log Analysis Professionals

So, I was talking to this small log management vendor the other day and he confided to me that his product faces fierce competition in his target market (which is, important to note, small to medium companies with 10-100 systems): and this competition is apathy.

More specifically, his prospects either just blow him off by saying “pah, who needs logging!” or they profess their undying love of all things logging - and then still don’t buy his product (which is priced, shall we say, “to go”)

Admittedly - and somewhat tongue-in-cheek, these are the same companies that form the core of today’s botnets (due to various reasons including their scarce resources) and enable RBN to deliver high-quality malicious services to criminal enterprises worldwide. Still, if you happen to have thoughts along the line of “who needs logs?” or “ah, logging? it will come later!”, you really deserve a nice fat check from RBN and other malicious “hacking” syndicates since it is extremely likely that your overall attitude towards security is just as misguided…

But how to progress from such … what was before the Stone Age? … Sharpened Stick Age? to modernity? Most companies go through the following stages in regards to their logging:

1. Deep log ignorance: “Logs? What are those?”
2. Shallow log ignorance: “Later…later…later… #37 on the TODO list.”
3. Log collection: “We gather and store dead log data…cold.”
4. Log searching: “We will dig into the pile when we have to … hopefully never!”
5. Log analysis and reporting: “We know our logs - and what they mean”

(also see my post “Natural Flow of Log Management” for some specifics)

Of course, compliance (PCI DSS and others) helped move people from 1. and 2. to 3., but, sadly, people often get stuck at 3. (just collection) or 4. (collection + maybe search) and never progress to Logging Enlightenment of 5.

Yes, PCI DSS and other regulations mandate not just log collection, not just dead cold log storage, but also log review (daily, in case of PCI DSS Requirement 10), but “review” happens to be the item that gets overlooked all too often.

Why is that?

I think the reason is that log analysis is still too hard and still not automated enough for an average organization. Yes, I did see some corporations that built their own log analysis systems that - surprise! - exceeded the best available from the vendors [at the time]. However, a typical company IT department would not have Ph.D. poring through hardcore text mining research papers in order to improve their home-grown log analysis AI. They expect the vendors to eat the logs, chew on them for a bit - and then spit out the answers.

Are we there yet? No, but we will be!!!

Welcome Peter Giannoulis to the Log Analysis Professionals Blogger Roster

Posted by Andrew Hay under Log Analysis Professionals

I’d like to welcome Peter Giannoulis to the Log Analysis Professionals stable of professional bloggers.

Peter is an information security consultant in Toronto, Ontario. Over the last 9 years Peter has been involved in the design and implementation of client defenses using many different security technologies. He is also skilled in vulnerability and penetration testing having taken part in hundreds of assessments. Peter has been involved with SANS and GIAC for quite some time as an Authorized Grader for the GSEC certification, courseware author, exam developer, Advisory Board member, Stay Sharp instructor and is currently a Technical Director for the GIAC family of certifications. He currently maintains The Academy - www.theacademy.ca, which is the first information security video website that assists organizations in implementing and troubleshooting some of the most popular security products. Peter’s current certifications include: GSEC, GCIH, GCIA, GCFA, GCFW, GREM, GSNA, CISSP, CCSI, INFOSEC, CCSP, & MCSE.

Welcome Raffael Marty to the Log Analysis Professionals Blogger Roster

Posted by Andrew Hay under Log Analysis Professionals

I’d like to welcome Raffael “Raffy” Marty to the Log Analysis Professionals stable of professional bloggers.

As chief security strategist and senior product manager, Raffy is customer advocate and guardian - expert on all things security and log analysis at Splunk. With customers, he uses his skills in data visualization, log management, intrusion detection, and compliance to solve problems and create solutions. Inside Splunk, he is the conduit for customer issues, new ideas and market requirements to the development team. Fully immersed in industry initiatives, standards efforts and activities, Raffy lives and breathes security and
visualization. His passion for visualization is evident in the many presentations he gives at conferences around the world.

Active in standard committees like CEE (common event expression) and OVAL (open vulnerability and assessment language), he is also creator of automation tools Thor and AfterGlow, founder of the security visualization portal http://secviz.org, and contributing author to a number of books on security and visualization. Before coming to Splunk he managed the solutions team at ArcSight, was an IT security consultant for PriceWaterhouse Coopers, and was a member of the Global Security Analysis Lab at IBM Research, where he participated in various intrusion detection related research projects.

Welcome Shyaam Sundhar to the Log Analysis Professionals Blogger Roster

Posted by Andrew Hay under Log Analysis Professionals

I’d like to welcome Shyaam Sundhar to the Log Analysis Professionals stable of professional bloggers.

Shyaam Sundhar is a security analyst at Symantec MSS. He has been working as an analyst for almost two years, where he was a threat analyst and intrusion signature writer in the previous job. Shyaam, has a background in information security, computer security and information assurance though academia. He holds a Master’s in Information Security from the George Washington University, Master’s Certificate in Computer Security and Information Assurance from the George Washington University and graduate level security certificate in Computer Security from Stanford University. With active professional membership with ACM, ACFE, ISACA and IACSP, he has been actively participating in the community in a very stealthy way. He holds professional certs such as, GPCI, GCDS, GLDR, SSP-CNSA, SSP-MPA, SSP-GHD, GREM, GHTQ, GWAS, GIPS and GCFA. He is a board member at IARIA research group where he has participated as TPC, Chair and Co-Chair of several IEEE conferences related to Security. His profile can be found at http://www.linkedin.com/in/intrusion.

Welcome Ron Gula to the Log Analysis Professionals Blogger Roster

Posted by Andrew Hay under Log Analysis Professionals

I’d like to welcome Ron Gula to the Log Analysis Professionals stable of professional bloggers.

Ron Gula was the original author of the Dragon IDS and CTO of Network Security Wizards which was acquired by Enterasys Networks. At Enterasys, Mr. Gula was Vice President of IDS Products and worked with many top financial, government, security service providers and commercial companies to help deploy and monitor large IDS installations. Mr. Gula was also the Director of Risk Mitigation for US Internetworking and was responsible for intrusion detection and vulnerability detection for one of the first application service providers. Mr. Gula worked for BBN and GTE Internetworking where he conducted security assessments as a consultant, helped to develop one of the first commercial network honeypots and helped develop security policies for large carrier-class networks. Mr. Gula began his career in information security while working at the National Security Agency conducting penetration tests of government networks and performing advanced vulnerability research. Mr. Gula has a BS from Clarkson University and an MSEE from the University of Southern Illinois. Ron Gula was the recipient of the 2004 Techno Security Conference “Industry Professional of the Year” award.

Welcome Harlan Carvey to the Log Analysis Professionals Blogger Roster

Posted by Andrew Hay under Log Analysis Professionals

I’d like to welcome Harlan Carvey to the Log Analysis Professionals stable of professional bloggers.

Harlan is a nerd who does incident response and computer forensics work, and is based out of the Metro DC area. In an effort to demonstrate just how much of a nerd he is, Harlan has authored three books on incident response, computer forensics, and Perl scripting, all for the Windows platform. His dislikes include being required to use EnCase for analysis, and he enjoys moonlit walks on the beach, Registry analysis, Perl’s ‘use’ and ‘require’ pragmas, and a frothy ale with a nice copper color.

Welcome Daniel Cid to the Log Analysis Professionals Blogger Roster

Posted by Andrew Hay under Log Analysis Professionals

I’d like to welcome Daniel Cid to the Log Analysis Professionals stable of professional bloggers.

Daniel Cid is the creator and main developer of the OSSEC HIDS (Open Source Security Host Intrusion Detection System). Daniel has been working in the security area for many years, with a special interest in intrusion detection, log analysis and secure development. He is currently working at Q1 Labs Inc. as a software engineer. In the past, he worked at Sourcefire, NIH and Opensolutions. Daniel holds several industry certifications including the CCNP, GCIH, and CISSP.

Against Log Silos!

Posted by Anton Chuvakin under Log Analysis Professionals

While the world of logging is full of inconsistencies and troubles (e.g. ugly logs!), there is one that beats many others: siloed approach to logs!

There is little that I hate more than siloed approach to logs. A situation where your security team “owns” network IDS logs, the network team has firewall and router logs (as well as all SNMP traps) and the system administrator has the logs from servers and desktop, is not only sad, counterproductive, inefficient, and wasteful, but it is also dangerous.

Where does the siloed approach to logs (when they are divided by both technical and political chasms) break down most painfully? In the case of an incident response, of course! An incident response is where, instead of one query across all logs and all log sources (or whatever needed subset of logs or log sources), you would end up running around, begging, connecting, waiting, swearing, waiting, downloading logs, digging in many places at once, waiting, grep, suffering with many UIs, swearing some more, etc. You would be doing all this, instead of connecting to your shiny new log management system and running a few reports, doing drilldowns, and searching across the relevant logs!

Where else does it break down? Compliance of course! Most regulations and mandates don’t call out logs by the log source type, instead they call all logs equally. Therefore having one system to verify the compliance status is much more productive compared to digging in many systems.

Ideally, you’d break down the silo walls by deploying a log management platform across the entire organization and then controlling access to every team requiring access to the system for logs, using the interface or a web API. Apart from being a trend (e.g. see recent ESG report), it will make your IT and security operations that much more efficient - and pleasant!

On the other hand, what is bizarre, is some newer vendors, claiming to do log management, actually work to propagate, not combat, the siloed approach. For example, selling the tool for $5,000 to each of the various teams within the organization IMHO must be made illegal because it builds walls, not bridges; digs holes and overall “silo-izes” your IT operation…