Log Analysis Professionals

How to Make Logs Sexy … Again!?

Now, some people hate logging, because logs are too hard to deal with (enable, collect, store and especially understand and interpret). However, there is a whole other group of fairly intelligent people who “hate logs:” the organizers of some well-known technical security conferences. The experience of many of my colleagues (and competitors!) and myself proves that a log-related talk will NOT be accepted to ANY technical security conference nowadays. Now, some were generous enough to explain why. Others were not (screw them and no link :-)).

But let me rant about this one a bit. First, it is always a possibility that they dislike me not logs:-) - this is easily disproved, however, since some of my colleagues had the same exact experience. Do they dislike vendors talking about logs? Nah, this isn’t it either - most of my conference presentations had nothing to do with LogLogic, even though they are about logs. Some of my friends (and this blog readers) tried to suggest that an audience of such events “knows everything there is to know about logs.” This is not true since - gasp!- nobody knows everything there is to know about logs: they hide way too many mysteries (with useful answers!) to discount them like that. Another one I’ve heard is that “real hackers don’t get logged -> logs are useless”, which is also silly: this is true only if you take a very narrow view of logs (e.g. NIDS alerts),; clearly, everybody is logged by the firewalls, servers, apps, etc. The challenge is not a lack of data, but too much data and not enough time and tools.

But we are about to “hit paydirt” with this question…

Tool? Did I just mention tools? This opens the last and final, deeply evil reason for such “log-hate”: one of the conference organizers mentioned that, in his opinion, there is nothing new in the field of log analysis since regex-match-based alerting (and regex-based parsing into database tables).

And you know what?

Drum roll….

He was actually somewhat right.

Indexing did come in the world of logging, but, personally, I don’t find it to be a huge feat of human ingenuity (even though it is definitely useful). I also think we are not doing enough with index data (and I definitely intend to change that…)

In addition, there was A LOT of academic research on the subject, from the SRI EMERALD in the 80s (and even earlier) to today, but many of the papers I’ve seen sit on the “hilarious side of useless”…

So, we need a campaign “Making Logs Sexy Again!” (the term “sexy logs” and - yet uncreated - picture of a dancing sexy log are copyright Andrew Hay :-))

Posted in Log Analysis Professionals |

How to Fight “Log Apathy”?

So, I was talking to this small log management vendor the other day and he confided to me that his product faces fierce competition in his target market (which is, important to note, small to medium companies with 10-100 systems): and this competition is apathy.

More specifically, his prospects either just blow him off by saying “pah, who needs logging!” or they profess their undying love of all things logging - and then still don’t buy his product (which is priced, shall we say, “to go”)

Admittedly - and somewhat tongue-in-cheek, these are the same companies that form the core of today’s botnets (due to various reasons including their scarce resources) and enable RBN to deliver high-quality malicious services to criminal enterprises worldwide. Still, if you happen to have thoughts along the line of “who needs logs?” or “ah, logging? it will come later!”, you really deserve a nice fat check from RBN and other malicious “hacking” syndicates since it is extremely likely that your overall attitude towards security is just as misguided…

But how to progress from such … what was before the Stone Age? … Sharpened Stick Age? to modernity? Most companies go through the following stages in regards to their logging:

1. Deep log ignorance: “Logs? What are those?”
2. Shallow log ignorance: “Later…later…later… #37 on the TODO list.”
3. Log collection: “We gather and store dead log data…cold.”
4. Log searching: “We will dig into the pile when we have to … hopefully never!”
5. Log analysis and reporting: “We know our logs - and what they mean”

(also see my post “Natural Flow of Log Management” for some specifics)

Of course, compliance (PCI DSS and others) helped move people from 1. and 2. to 3., but, sadly, people often get stuck at 3. (just collection) or 4. (collection + maybe search) and never progress to Logging Enlightenment of 5.

Yes, PCI DSS and other regulations mandate not just log collection, not just dead cold log storage, but also log review (daily, in case of PCI DSS Requirement 10), but “review” happens to be the item that gets overlooked all too often.

Why is that?

I think the reason is that log analysis is still too hard and still not automated enough for an average organization. Yes, I did see some corporations that built their own log analysis systems that - surprise! - exceeded the best available from the vendors [at the time]. However, a typical company IT department would not have Ph.D. poring through hardcore text mining research papers in order to improve their home-grown log analysis AI. They expect the vendors to eat the logs, chew on them for a bit - and then spit out the answers.

Are we there yet? No, but we will be!!!

Posted in Log Analysis Professionals |

The Windows Registry as a Log File

You’re probably looking at the title of this blog post and thinking…what? What’s he talking about? Well, as an incident analyst (which includes performing forensic examinations), I many times have to attempt to determine user login times, user activity on the system (applications run, files opened or modified), USB removable storage devices connected to the system (what type of device, and when it was connected), as well as a wide variety of other bits of information. On Windows systems, an excellent resource for this information is the Windows Registry, which is described by MS as “a central hierarchical database used in Microsoft Windows…used to store information that is necessary to configure the system for one or more users, applications and hardware devices.” (KB256986).

At its core, the Registry consists of different types of cells, the two primary types that we’re interested in being keys and values. Registry keys appear in Registry viewers such as RegEdit as the folders in the left-hand pane, and values appear in the right-hand pane along with their associated data.

Registry keys have a value associated with them called their “LastWrite time”. This is an 8-byte FILETIME object that records the date and time that the key was last modified (in this way, it is analogous to a files last modification time). Therefore, if a value or subkey is added to or deleted from a key, then the LastWrite time will be updated accordingly. (NOTE: As of yet, I have been unable to locate a publicly accessible API for modifying a key’s LastWrite time to an arbitrary value; there are, however, publicly available APIs for modifying file MAC times.)

Registry values do not have timestamps built into their structures the way that keys do, but many values include FILETIME objects embedded within their binary data. For example, beneath the UserAssist keys, many of the values found within the Count subkey have 16 bytes of binary data associated with them, and the last 8 bytes is a FILETIME object, indicating when that action was last taken. In addition, there are several values whose data is simply a 4-byte Unix time value.

The key to viewing the Registry as a log file is to understand what actions within the system cause various Registry keys and values (particularly values that record a timestamp) to update their timestamp value. In this way, we can then associate that action with a specific date and time.

For example, when a user accesses a file via one of the many graphical user interface (GUI) applications available for Windows, the application will maintain a list of the recently accessed files, so that those files can be quickly accessed at a later date, simply by opening the application and clicking on the File menu item…in many cases, the list of recently accessed files appears at or near the bottom of the drop-down menu. This is referred to as a “most recently used” or MRU list. In many Windows applications, the MRU list is maintained within a key, and the values within the key are updated each time a file is accessed with that application. Therefore, the most recently accessed file (as maintained by either the value naming scheme, or within an MRUList or MRUListEx value) can be directly associated with the key’s LastWrite time. By understanding the modification technique for various applications on the system, and then querying all of those keys and values, a forensic analyst can develop a timeline of when specific files were accessed by a user.

I mentioned the UserAssist keys earlier…beneath these keys, the Count subkeys contain a wealth of information about user activities that can be used by a forensic analyst to associate a specific user action (double-clicking an application icon, navigating the Start->Programs menu to launch an application, etc.) to the last time that that action had been performed. For example, this type of Registry analysis has been used to clearly demonstrate that a user account was used to install, launch, and then uninstall a password cracking application on a compromised host. While there were no easily discernible indications of this activity within the file system itself, the contents of the user account’s NTUSER.DAT hive file clearly showed not only that this activity occurred, but when.

There are a wide variety of Registry keys within the various hive files that can be used to answer a number of questions. For example, Registry keys and values can be used to determine which USB removable storage devices had been connected to a system (as well as the date/time that they were last disconnected from the system), when the system had last been connected to particular wireless SSIDs, the last time users had connected to remote systems and network shares, etc.

The Registry maintains a great deal of configuration information about the system, as well as about user activities. The fact that the Registry also maintains timestamps for a great number of actions and modifications that occur on the system clearly demonstrates that for a forensic analyst the Registry is a prolific and detailed log file. What’s currently missing for forensic analysts is a tool that provides that information in an easy-to-view and understand manner. This is an issue I will be addressing, through the use of a specific tool (dubbed “RegRipper”) in my next blog post.

Thanks!

Posted in The Art of Log Analysis |

Common Field List

Under the umbrella of the common event expression (CEE) effort, we just posted a proposal for a common field list for log files.

At this point, we are really interested in getting feedback from the community! Have a look at the post on the CEE list and the list itself. Let us know, on the CEE discussion list (CEE-DISCUSSION-LIST@LISTS.MITRE.ORG), what you like and what you don’t like about the approach!

A little more context on the field list can be found here also: Common Event Syntax and more about CEE is outlined here: CEE / CEF event interoperability standards.

Posted in The Art of Log Analysis |

Welcome Peter Giannoulis to the Log Analysis Professionals Blogger Roster

I’d like to welcome Peter Giannoulis to the Log Analysis Professionals stable of professional bloggers.

Peter is an information security consultant in Toronto, Ontario. Over the last 9 years Peter has been involved in the design and implementation of client defenses using many different security technologies. He is also skilled in vulnerability and penetration testing having taken part in hundreds of assessments. Peter has been involved with SANS and GIAC for quite some time as an Authorized Grader for the GSEC certification, courseware author, exam developer, Advisory Board member, Stay Sharp instructor and is currently a Technical Director for the GIAC family of certifications. He currently maintains The Academy - www.theacademy.ca, which is the first information security video website that assists organizations in implementing and troubleshooting some of the most popular security products. Peter’s current certifications include: GSEC, GCIH, GCIA, GCFA, GCFW, GREM, GSNA, CISSP, CCSI, INFOSEC, CCSP, & MCSE.

Posted in Log Analysis Professionals |

The Launch of The Academy Website

The Academy (http://www.theacademy.ca) officially launches its web site today providing instructional videos for the information security community. For the first time ever, the average user to the most seasoned industry expert will be able to watch instructional videos on how to install popular products, address common configuration issues, and troubleshoot difficult problems. The Academy is a user driven community and videos are created at the request of its members. Vendors can also leverage the site to showcase the features and capabilities of their products. The Academy is an ideal place to find and share knowledge with others practicing or interested in the information security field.

Yours truly will be contributing as many log related videos as possible so that people understand how to properly make those crazy blinking boxes they have in their racks send logs.

Posted in The Art of Log Analysis |

Welcome Raffael Marty to the Log Analysis Professionals Blogger Roster

I’d like to welcome Raffael “Raffy” Marty to the Log Analysis Professionals stable of professional bloggers.

As chief security strategist and senior product manager, Raffy is customer advocate and guardian - expert on all things security and log analysis at Splunk. With customers, he uses his skills in data visualization, log management, intrusion detection, and compliance to solve problems and create solutions. Inside Splunk, he is the conduit for customer issues, new ideas and market requirements to the development team. Fully immersed in industry initiatives, standards efforts and activities, Raffy lives and breathes security and
visualization. His passion for visualization is evident in the many presentations he gives at conferences around the world.

Active in standard committees like CEE (common event expression) and OVAL (open vulnerability and assessment language), he is also creator of automation tools Thor and AfterGlow, founder of the security visualization portal http://secviz.org, and contributing author to a number of books on security and visualization. Before coming to Splunk he managed the solutions team at ArcSight, was an IT security consultant for PriceWaterhouse Coopers, and was a member of the Global Security Analysis Lab at IBM Research, where he participated in various intrusion detection related research projects.

Posted in Log Analysis Professionals |

Segregating Your Logging for Availability

Although not a new concept, I thought I’d remind people of the benefits of sending your security, system, and application logs across a segregated network to maintain availability. Consider the following scenario:

Your network is experiencing a horrible worm outbreak that is eating up critical bandwidth as it attempts to spread from host to host. All of this malicious traffic is causing your firewalls, intrusion systems, routers, switches, and servers to feverishly log every worm related event they possibly can. The corporate security policy dictates that all event logs are to be sent to a log management server so that no logs are lost.

This remote logging, although small when you consider the size of an individual log (A UDP syslog packet cannot exceed 1024 bytes), does impact available network bandwidth. This is especially true if thousands of logs per second are being transported to a log management or log storage solution through the same path as your regular traffic (1000 logs per second X 1024 bytes = 1,024,000 additional bytes per second — worst case of course). Similarly, a Denial of Service (DoS) or Distributed Denial of Service (DDoS) could also adversely impact the rate at which regular data, and associated logged events, flows through your network infrastructure.

You could investigate implementing QoS rules for your logs on your existing network but all this does is dedicate already sparse network resources to your logging traffic. This is a good solution if your main concern is the availability of your logs but it does nothing to help reserve bandwidth for your network traffic during the outbreak.

If you designate a separate and distinct network segment for the transmission of your logs, you can keep your critical network bandwidth available for regular operation while you mitigate the outbreak. This can be as simple as configuring an additional interface on your device for logging or as complex as creating distinct VLANs for the logging traffic (which, in all honesty, isn’t all that complex a task). Although this is a great solution for maintaining the availability of both your network traffic and your logging traffic, there is an associated infrastructure cost (switches, network interfaces, rack space, power, administration overhead, etc.). Hopefully you can see how this short-term investment will pay off in the long term.

I haven’t discussed the business case for segregating your logging for security reasons but I will in a future article. Segregating your logging for availability, however, is definitely something to think about when you’re planning your logging infrastructure

Posted in The Art of Log Analysis |

What Logging You MUST Do For PCI DSS Compliance?

Somebody asked me a few days ago: EXACTLY what logging we absolutely MUST do for PCI DSS compliance? This is actually not as simple!

The honest answer to the above question is that there is no list of what EXACTLY you MUST be logging due to PCI or, pretty much, any other recent “compliance thingy” (as we all know, PCI DSS rules are more specific than most others). However, the above does NOT mean that you CAN log nothing.

Is this bizarre or what? Yes, it is

But that is exactly why vendors and consultants tell you what you SHOULD be logging. There is no easy “MUST-log-this” list; it is pretty much up to individual auditor, consultant, vendor, engineer, etc to interpret (again, not simply ‘read’, but interpret!) the PCI DSS guidance (e.g. Requirement 10 that is dedicated to logging) in your own environment.

Logging vendor’s field engineers do interpret it for their customers; I provided my own interpretation in my PCI book, etc. But, there is still no MUST list; just the following route:

PCI DSS guidance -> consultant, vendor engineer, etc -> your very own logging recommendations.

A few folks wondered: why not ask the auditor? Well, these critters will tell you whether “yours is OK” (rarely) or “Oh, no!” (frequently), but will certainly not write your logging policy for you. With them, the best approach is: define your logging policy, then show to auditor, if they are happy - now you know what you MUST do this time.

As a final word: still, I dislike the above compliance-induced daze as much as the next guy. I much prefer that people think what they want from their logs as well as how they need to use them and then log that!

Technorati tags: logging, log management, compliance, PCI

Posted in The Art of Log Analysis |

Welcome Shyaam Sundhar to the Log Analysis Professionals Blogger Roster

I’d like to welcome Shyaam Sundhar to the Log Analysis Professionals stable of professional bloggers.

Shyaam Sundhar is a security analyst at Symantec MSS. He has been working as an analyst for almost two years, where he was a threat analyst and intrusion signature writer in the previous job. Shyaam, has a background in information security, computer security and information assurance though academia. He holds a Master’s in Information Security from the George Washington University, Master’s Certificate in Computer Security and Information Assurance from the George Washington University and graduate level security certificate in Computer Security from Stanford University. With active professional membership with ACM, ACFE, ISACA and IACSP, he has been actively participating in the community in a very stealthy way. He holds professional certs such as, GPCI, GCDS, GLDR, SSP-CNSA, SSP-MPA, SSP-GHD, GREM, GHTQ, GWAS, GIPS and GCFA. He is a board member at IARIA research group where he has participated as TPC, Chair and Co-Chair of several IEEE conferences related to Security. His profile can be found at http://www.linkedin.com/in/intrusion.

Posted in Log Analysis Professionals |