Issue 41530045: ASL BSD Plaintext for Mac OS X.

	Unified diffs	Side-by-side diffs	Delta from patch set	Stats (+453 lines, -0 lines)			Patch
A	plaso/parsers/mac_syslog.py	View	1 2 3 4 5	1 chunk	+237 lines, -0 lines	10 comments	Download
A	plaso/parsers/mac_syslog_test.py	View	1 2 3 4 5	1 chunk	+168 lines, -0 lines	8 comments	Download
A	test_data/mac_bsd_syslog	View	1 2 3 4 5	1 chunk	+48 lines, -0 lines	0 comments	Download

Messages

Total messages: 18

Expand All Messages | Collapse All Messages

Joaquin Moreno

Well, this is a proof of concept because it is not able to read all ...

10 years, 4 months ago (2013-12-25 04:44:59 UTC) #3

Well, this is a proof of concept because it is not able to read all the BSD
parser logs, have some issues and it is not fast. I have done a small script
with 1/3 than the Plaso parser and it is able to parse everything. I don't know
how to do it in a proper way in Plaso. I have checked event, Parse and
text_parser, but I couldn't understand how it suppose to deal with a logs with
more than one line per entry.

This is the script that can parse every ASL BSD file:

#!/usr/bin/python
# -*- coding: utf-8 -*-

import pyparsing
import sys

# PLASO CODE
def PyParseIntCast(dummy_string, dummy_location, tokens):
  for index, token in enumerate(tokens):
    try:
      tokens[index] = int(token)
    except ValueError:
      logging.error(u'Unable to cast [{}] to an int, returning -1'.format(
          token))
      tokens[index] = 0

MONTH = pyparsing.Word(
    pyparsing.string.uppercase, pyparsing.string.lowercase, exact=3)
INTEGER = pyparsing.Word(pyparsing.nums).setParseAction(PyParseIntCast)
TWO_DIGITS = pyparsing.Word(pyparsing.nums, exact=2).setParseAction(
    PyParseIntCast)
ONE_OR_TWO_DIGITS = pyparsing.Word(
    pyparsing.nums, min=1, max=2).setParseAction(PyParseIntCast)
TIME = pyparsing.Group(
    TWO_DIGITS + pyparsing.Suppress(':') + TWO_DIGITS +
    pyparsing.Suppress(':') + TWO_DIGITS)
PID= pyparsing.Word(pyparsing.nums, min=1, max=5).setParseAction(PyParseIntCast)
 
  
BSD_PLAINTEXT = (
    MONTH.setResultsName('month') +
    ONE_OR_TWO_DIGITS.setResultsName('day') +
    TIME.setResultsName('time') +
    pyparsing.Word(pyparsing.printables).setResultsName('computer_name') +
    pyparsing.CharsNotIn(u'[').setResultsName('agent') +
    pyparsing.Literal(u'[').suppress() +
    PID.setResultsName('pid') +
    pyparsing.Literal(u']').suppress() + pyparsing.Literal(u':') +
    pyparsing.SkipTo(pyparsing.lineEnd).setResultsName('message'))

# Repeated line.
REPEATED_LINE = (
    MONTH.setResultsName('month') +
    ONE_OR_TWO_DIGITS.setResultsName('day') +
    TIME.setResultsName('time') +
    pyparsing.Literal(u'---').suppress() +
    pyparsing.CharsNotIn(u'---').setResultsName('repeated_line') +
    pyparsing.Literal(u'---').suppress())  
  
previous_structure = None

def RawToUTF8(text):
  try:
    text = text.decode('utf-8')
  except UnicodeDecodeError:
    print "Mierda y cuchara"
    text = text.decode('utf-8', 'ignore')
  return text

def ParseLine(line):  
  print u'Timestamp: {} {} {}, Host: {}, Agent: {} (Pid: {}).'.format(
      line.month, line.day, line.time, line.computer_name, 
      line.agent.split(' ', 1)[1], line.pid)
  print u'Message: {}'.format(line.message)
  
try:
  f = open(sys.argv[1])
except:
  print 'No file, no senorito, no file.'
cont = 1
for line in f:
  try:
    data = BSD_PLAINTEXT.parseString(line)  
    if previous_structure != None:
      ParseLine(previous_structure)
    data.message = RawToUTF8(data.message)
    previous_structure = data
  except:
    try:
      data = REPEATED_LINE.parseString(line)
      ParseLine(previous_structure)
      previous_structure.month = data.month
      previous_structure.day = data.day
      previous_structure.time = data.time
      ParseLine(previous_structure)
    except:
      previous_structure.message += RawToUTF8(line)
ParseLine(previous_structure)
f.close()

Joaquin Moreno

New version completely re-programmed. It is not so clear code, but it can parse all ...

10 years, 4 months ago (2013-12-26 00:37:00 UTC) #5

kiddi

Few comments this time, but I think you should not use the base parser but ...

10 years, 4 months ago (2013-12-30 19:29:37 UTC) #6

Joaquin Moreno

Syslog using Pyparsing and multiline from text_parsed instead of regular expression. I have detected some ...

10 years, 3 months ago (2014-01-02 03:43:20 UTC) #8

Joaquin Moreno

On 2014/01/02 18:50:42, kiddi wrote: > so are you removing the old syslog parser for ...

10 years, 3 months ago (2014-01-02 18:52:37 UTC) #10

kiddi

I'm reviewing the multi line parser right now and looking at this implementation as a ...

10 years, 3 months ago (2014-01-06 23:10:09 UTC) #11

Joaquin Moreno

First of all, sorry for the delay. Normal syslog works well using the pyparsing, can ...

10 years, 2 months ago (2014-02-07 23:05:03 UTC) #14

kiddi

Few comments, been a while since I looked at this CL, sorry for the extremely ...

10 years, 1 month ago (2014-03-26 21:54:43 UTC) #17

Any news/updates?

Expand All Messages | Collapse All Messages