Log Analysis Professionals

Some Interesting SIEM Articles

Posted by Andrew Hay under Log Analysis Professionals

I found these two SIEM related blog posts today and thought them worth checking out:

• Decurity Blog | SIEM: A Lessons Learned Overview
• CSI: Security Provoked | Case Study: Implementing SIEM in the Israeli e-Government

Log Analysis and Visualization Workshop in Boston

Posted by Raffael Marty under The Art of Log Analysis

“Log Analysis and Security Visualization” is a two-day training class held on March 9th and 10th 2009 in Boston during the SOURCE Boston conference that addresses the data management and analysis challenges of today’s IT environments.

The students will leave this class with the knowledge to visualize and manage their own IT data. They will learn the basics of log analysis, learn about common data sources, get an overview of visualization techniques, and learn how to generate visual representations of IT data for a number of different use-cases from DoS and worm detection to compliance reporting. The training is filled with hands-on exercises utilizing DAVIX, the open-source data analysis and visualization platform.

The trainer is the author of the book Applied Security Visualization and has been working on log analysis for many years.

Register today to secure your spot.

Applied Security Visualization - The Book

Posted by Raffael Marty under The Art of Log Analysis

Are you analyzing log files? Are you overwhelmed with the amount of data to look at? Are you interested in learning about the following topics?

• Transform complex data into crystal-clear visual representations
• Iterate your graphs to deliver even better insight for taking action
• Assess threats to your network perimeter, as well as threats imposed by insiders
• Use visualization to manage risks and compliance mandates more successfully
• Visually audit both the technical and organizational aspects of information and network security
• Intimately understand the data sources that are essential for effective visualization
• Choose the most appropriate graphs and techniques for your IT data
• Compare and master today’s most useful tools for security visualization

This is what the book, Applied Security Visualization is all about. You find some more information about the book, such as press coverage on the Applied Security Visualization page on secviz.org. There you will find a couple of pod casts and interviews about the book. If you are interested in listening in on some of the press coverage:

SANS Tool Talk Webcast: “Log Management: No Longer Optional”

Posted by Andrew Hay under The Art of Log Analysis

I’ll be presenting a SANS Tool Talk Webcast entitled “Log Management: No Longer Optional” on Tuesday, June 2nd at 1pm EST.

About the session:
Both network and security professionals agree - a log management solution is no longer optional. It’s now a required tool in their arsenal.

Unfortunately, many of their log management projects have failed because the solution they chose was unable to support the size and scope of the deployment and/or effectively deliver useful results.

During this webcast Andrew Hay will discuss important considerations when selecting and deploying a log management solution for your organization and how to avoid some of the pitfalls.

Join this webcast and learn about:

• Drivers of log management, including security best practices and regulatory mandates
• Architectural considerations for supporting large distributed enterprise networks
• Deployment considerations for supporting a multi-vendor network
• Correlation considerations to effectively make sense of enterprise-wide network & security events
• Advanced security management considerations to improve an organization’s ability to detect more complex integrated network threats
• Reporting, auditing and forensics considerations that support compliance initiatives

Sign up for the webcast here.

WASL ‘08 Call for Papers

Posted by Andrew Hay under The Art of Log Analysis

This is really cool news! The First USENIX Workshop on the Analysis of System Logs (WASL ‘08) is happening on December 7th, 2008 in San Diego, CA. About the event:

System logs contain a wide variety of information about system status and health, including events from various applications, daemons, and drivers, as well as sampled information such as resource utilization statistics. As such, these logs represent a rich source of information for the analysis and diagnosis of system problems and prediction of future system events. However, their lack of organization and the general lack of semantic consistency among the information from various software and hardware vendors means that most of this information content is wasted. Today’s most popular log analysis technique is to use regular expressions either to detect events of interest or to filter the log so that a human operator can examine it manually. Clearly, this captures only a fraction of the information available in these logs and does not scale to the large systems common in business and supercomputing environments.

The call for papers is now open and can be accessed here: http://www.usenix.org/events/wasl08/cfp/cfp.html

Topics
This workshop will focus on novel techniques for extracting operationally useful information from existing logs and on methods to improve the information content of future logs. Topics include but are not limited to:

• Reports on publicly available sources of sample log data
• Log anonymization
• Log feature detection and extraction
• Prediction of malfunction or misuse based on log data
• Statistical techniques to characterize log data
• Applications of Natural-Language Processing (NLP) to logs
• Scalable log compression
• Log comparison techniques
• Methods to enhance and standardize log semantics
• System diagnostic techniques
• Log visualization
• Analysis of services (problem ticket) logs

Deadline and Submission Instructions
Submitted papers must be no longer than 8 single-spaced 8.5″ x 11″ pages, including figures, tables, and references; two-column format, using 10-point type on 12-point (single-spaced) leading; and a text block 6.5″ wide x 9″ deep. Author names and affiliations should appear on the title page.

Papers must be in PDF and must be submitted via the Web submission form, which will be available here soon.

Authors will be notified of acceptance by September 26, 2008. Authors of accepted papers will produce a final PDF and the equivalent HTML by October 28, 2008. All papers will be available online to attendees prior to the workshop and will be available online to everyone starting on December 7, 2008.

SANS Security Laboratory “Thought Leaders” Article

Posted by Andrew Hay under Log Analysis Professionals

Stephen Northcutt, of SANS Institute fame, recently recognized me as a Thought Leader in the area of log management. I’m quite humbled to be included with the likes of our own Log Analysis Professional contributors Dr. Anton Chuvakin and Ron Gula (among others).

The interview has been posted on the SANS Technology Institute site here. This has certainly made my week

How to Make Logs Sexy … Again!?

Posted by Anton Chuvakin under Log Analysis Professionals

Now, some people hate logging, because logs are too hard to deal with (enable, collect, store and especially understand and interpret). However, there is a whole other group of fairly intelligent people who “hate logs:” the organizers of some well-known technical security conferences. The experience of many of my colleagues (and competitors!) and myself proves that a log-related talk will NOT be accepted to ANY technical security conference nowadays. Now, some were generous enough to explain why. Others were not (screw them and no link :-)).

But let me rant about this one a bit. First, it is always a possibility that they dislike me not logs:-) - this is easily disproved, however, since some of my colleagues had the same exact experience. Do they dislike vendors talking about logs? Nah, this isn’t it either - most of my conference presentations had nothing to do with LogLogic, even though they are about logs. Some of my friends (and this blog readers) tried to suggest that an audience of such events “knows everything there is to know about logs.” This is not true since - gasp!- nobody knows everything there is to know about logs: they hide way too many mysteries (with useful answers!) to discount them like that. Another one I’ve heard is that “real hackers don’t get logged -> logs are useless”, which is also silly: this is true only if you take a very narrow view of logs (e.g. NIDS alerts),; clearly, everybody is logged by the firewalls, servers, apps, etc. The challenge is not a lack of data, but too much data and not enough time and tools.

But we are about to “hit paydirt” with this question…

Tool? Did I just mention tools? This opens the last and final, deeply evil reason for such “log-hate”: one of the conference organizers mentioned that, in his opinion, there is nothing new in the field of log analysis since regex-match-based alerting (and regex-based parsing into database tables).

And you know what?

Drum roll….

He was actually somewhat right.

Indexing did come in the world of logging, but, personally, I don’t find it to be a huge feat of human ingenuity (even though it is definitely useful). I also think we are not doing enough with index data (and I definitely intend to change that…)

In addition, there was A LOT of academic research on the subject, from the SRI EMERALD in the 80s (and even earlier) to today, but many of the papers I’ve seen sit on the “hilarious side of useless”…

So, we need a campaign “Making Logs Sexy Again!” (the term “sexy logs” and - yet uncreated - picture of a dancing sexy log are copyright Andrew Hay :-))

How to Fight “Log Apathy”?

Posted by Anton Chuvakin under Log Analysis Professionals

So, I was talking to this small log management vendor the other day and he confided to me that his product faces fierce competition in his target market (which is, important to note, small to medium companies with 10-100 systems): and this competition is apathy.

More specifically, his prospects either just blow him off by saying “pah, who needs logging!” or they profess their undying love of all things logging - and then still don’t buy his product (which is priced, shall we say, “to go”)

Admittedly - and somewhat tongue-in-cheek, these are the same companies that form the core of today’s botnets (due to various reasons including their scarce resources) and enable RBN to deliver high-quality malicious services to criminal enterprises worldwide. Still, if you happen to have thoughts along the line of “who needs logs?” or “ah, logging? it will come later!”, you really deserve a nice fat check from RBN and other malicious “hacking” syndicates since it is extremely likely that your overall attitude towards security is just as misguided…

But how to progress from such … what was before the Stone Age? … Sharpened Stick Age? to modernity? Most companies go through the following stages in regards to their logging:

1. Deep log ignorance: “Logs? What are those?”
2. Shallow log ignorance: “Later…later…later… #37 on the TODO list.”
3. Log collection: “We gather and store dead log data…cold.”
4. Log searching: “We will dig into the pile when we have to … hopefully never!”
5. Log analysis and reporting: “We know our logs - and what they mean”

(also see my post “Natural Flow of Log Management” for some specifics)

Of course, compliance (PCI DSS and others) helped move people from 1. and 2. to 3., but, sadly, people often get stuck at 3. (just collection) or 4. (collection + maybe search) and never progress to Logging Enlightenment of 5.

Yes, PCI DSS and other regulations mandate not just log collection, not just dead cold log storage, but also log review (daily, in case of PCI DSS Requirement 10), but “review” happens to be the item that gets overlooked all too often.

Why is that?

I think the reason is that log analysis is still too hard and still not automated enough for an average organization. Yes, I did see some corporations that built their own log analysis systems that - surprise! - exceeded the best available from the vendors [at the time]. However, a typical company IT department would not have Ph.D. poring through hardcore text mining research papers in order to improve their home-grown log analysis AI. They expect the vendors to eat the logs, chew on them for a bit - and then spit out the answers.

Are we there yet? No, but we will be!!!

The Windows Registry as a Log File

Posted by Harlan under The Art of Log Analysis

You’re probably looking at the title of this blog post and thinking…what? What’s he talking about? Well, as an incident analyst (which includes performing forensic examinations), I many times have to attempt to determine user login times, user activity on the system (applications run, files opened or modified), USB removable storage devices connected to the system (what type of device, and when it was connected), as well as a wide variety of other bits of information. On Windows systems, an excellent resource for this information is the Windows Registry, which is described by MS as “a central hierarchical database used in Microsoft Windows…used to store information that is necessary to configure the system for one or more users, applications and hardware devices.” (KB256986).

At its core, the Registry consists of different types of cells, the two primary types that we’re interested in being keys and values. Registry keys appear in Registry viewers such as RegEdit as the folders in the left-hand pane, and values appear in the right-hand pane along with their associated data.

Registry keys have a value associated with them called their “LastWrite time”. This is an 8-byte FILETIME object that records the date and time that the key was last modified (in this way, it is analogous to a files last modification time). Therefore, if a value or subkey is added to or deleted from a key, then the LastWrite time will be updated accordingly. (NOTE: As of yet, I have been unable to locate a publicly accessible API for modifying a key’s LastWrite time to an arbitrary value; there are, however, publicly available APIs for modifying file MAC times.)

Registry values do not have timestamps built into their structures the way that keys do, but many values include FILETIME objects embedded within their binary data. For example, beneath the UserAssist keys, many of the values found within the Count subkey have 16 bytes of binary data associated with them, and the last 8 bytes is a FILETIME object, indicating when that action was last taken. In addition, there are several values whose data is simply a 4-byte Unix time value.

The key to viewing the Registry as a log file is to understand what actions within the system cause various Registry keys and values (particularly values that record a timestamp) to update their timestamp value. In this way, we can then associate that action with a specific date and time.

For example, when a user accesses a file via one of the many graphical user interface (GUI) applications available for Windows, the application will maintain a list of the recently accessed files, so that those files can be quickly accessed at a later date, simply by opening the application and clicking on the File menu item…in many cases, the list of recently accessed files appears at or near the bottom of the drop-down menu. This is referred to as a “most recently used” or MRU list. In many Windows applications, the MRU list is maintained within a key, and the values within the key are updated each time a file is accessed with that application. Therefore, the most recently accessed file (as maintained by either the value naming scheme, or within an MRUList or MRUListEx value) can be directly associated with the key’s LastWrite time. By understanding the modification technique for various applications on the system, and then querying all of those keys and values, a forensic analyst can develop a timeline of when specific files were accessed by a user.

I mentioned the UserAssist keys earlier…beneath these keys, the Count subkeys contain a wealth of information about user activities that can be used by a forensic analyst to associate a specific user action (double-clicking an application icon, navigating the Start->Programs menu to launch an application, etc.) to the last time that that action had been performed. For example, this type of Registry analysis has been used to clearly demonstrate that a user account was used to install, launch, and then uninstall a password cracking application on a compromised host. While there were no easily discernible indications of this activity within the file system itself, the contents of the user account’s NTUSER.DAT hive file clearly showed not only that this activity occurred, but when.

There are a wide variety of Registry keys within the various hive files that can be used to answer a number of questions. For example, Registry keys and values can be used to determine which USB removable storage devices had been connected to a system (as well as the date/time that they were last disconnected from the system), when the system had last been connected to particular wireless SSIDs, the last time users had connected to remote systems and network shares, etc.

The Registry maintains a great deal of configuration information about the system, as well as about user activities. The fact that the Registry also maintains timestamps for a great number of actions and modifications that occur on the system clearly demonstrates that for a forensic analyst the Registry is a prolific and detailed log file. What’s currently missing for forensic analysts is a tool that provides that information in an easy-to-view and understand manner. This is an issue I will be addressing, through the use of a specific tool (dubbed “RegRipper”) in my next blog post.

Thanks!

Common Field List

Posted by Raffael Marty under The Art of Log Analysis

Under the umbrella of the common event expression (CEE) effort, we just posted a proposal for a common field list for log files.

At this point, we are really interested in getting feedback from the community! Have a look at the post on the CEE list and the list itself. Let us know, on the CEE discussion list (CEE-DISCUSSION-LIST@LISTS.MITRE.ORG), what you like and what you don’t like about the approach!

A little more context on the field list can be found here also: Common Event Syntax and more about CEE is outlined here: CEE / CEF event interoperability standards.