Gen log.py

From CEDPS

Here is a simple python script that will generate a syslog-ng compatible UDP packet that also conforms to the CEDPS "Best Practices" format. This can be used within a site to generate logs about just about anything, and send them to a central collector.

• gen_log.py

Sample Use (send to host netlogger.lbl.gov on port 5141):

gen_log.py -h netlogger.lbl.gov -p 5141 -e nagios.alert msg="host down"
gen_log.py -h netlogger.lbl.gov -p 5141 -e nagios.alert msg="load is too high"
gen_log.py -h netlogger.lbl.gov -p 5141 -e job.start jobname=myprog args="-a -x -f fname"
gen_log.py -h netlogger.lbl.gov -p 5141 -e job.end host=hostname jobname=myprog status=0
gen_log.py -h netlogger.lbl.gov -p 5141 -e job.end host=hostname jobname=myprog status=-1 msg="job failed, disk full"

This program automatically generates the "Best Practice" format timestamp and event. For example, sample 3 above generates:

2007-09-18T13:38:11+07:00 myhost.lbl.gov cedps-logger: ts=2007-09-18T20:38:11.621540Z event=job.start progname=myprog args="-a -x -f fname"

Then you can use a syslog-ng server configured like this to collect log messages:

options {
  time_sleep(50);  # polling interval, in ms (helps reduce CPU)
  create_dirs(yes);  # create output directories
  use_fqdn(yes);  # use fully qualified domain names
  ts_format(iso);  # use ISO8601 timestamps (syslog-ng 2.0 only)
  #
  flush_timeout(500); # in ms
  #
  stats_freq(3600); 
};
#
# define the source: any host sending to port 5141
source network {
  udp(port(5141));
  internal();  # internal syslog-ng messages
};
#
# Define the destination, automatically creating new directories 
#    for each month and new host.
destination gridlogs {
    file ("/var/log/site/$YEAR.$MONTH/site.$HOST.log"
          perm(0644) dir_perm(0755) create_dirs(yes)
         template("$ISODATE $HOST $MSG\n") );
};
#
log { source(network);
    destination(gridlogs); flags (flow-control);
};

More information on installing and configuring syslog-ng can be found at http://www.cedps.net/wiki/index.php/Syslog-ng