Gridftp-syslogng

From CEDPS

Contents 1 Overview 2 Sender configuration 3 Receiver configuration 4 Server adminstration

Overview

Here are sample configurations for a syslog-ng sender to forward GridFTP logs to a remote syslog-ng receiver.

A text-cartoon of the configuration in mind:

+----------------------+                 +----------------+
| GridFTP server       |                 |my.host.org     |
|    |                 |                 |                |
|    |       syslog-ng |   ('''''''''')  |    syslog-ng   |
|    v        / sender-+---( Internet )--+--> receiver    |
| GridFTP logs         |   (,,,,,,,,,,)  |      |         |
| /path/to/gridftp.log |                 |      v         |
+----------------------+                 | /path/to/logs/ |
                                         +----------------+

Sender configuration

# Example syslog-ng sender for GridFTP logs
# 
# Author: Dan Gunter <dkgunter@lbl.gov>
# Date: 2009-08-28

# Global options
# --------------
options {
   time_sleep(500);  # polling interval, in ms (make this once per second)
   use_fqdn(yes);    # use fully qualified domain names
   ts_format(iso);   # use ISO8601 timestamps

   # for normal load
   flush_lines (10); # number of lines to buffer before writing to disk
   log_fifo_size(100);

   stats_freq(3600);  # number of seconds between syslog-ng internal stats events; these are useful
                      # for ensuring syslog-ng is not getting overloaded
};

# Sources
# -------
# XXX: Replace /path/to/gridftp.log with actual path
source gridftp_log { file ("/path/to/gridftp.log" follow-freq(1) flags(no-parse) log_prefix('gridftp_log ') ); };

source syslog_ng { internal(); };
source test_src  { unix-stream("/tmp/syslog-ng-test");  };

# Destinations
# ------------
# XXX: Replace my.host.org with the remote host
destination remote_collector   { tcp("my.host.org" port(5145) ); };
# Local copy, if desired
# destination syslog_ng_dest { file ("/export/data/syslog-ng/syslog-ng.log" perm(0644) ); };

# Log = Source + Destination
# --------------------------
log { source(gridftp_log); destination(remote_collector); flags(flow-control); };

# for syslog-ng debugging
# log { source(syslog_ng); destination(syslog_ng_dest); };

Receiver configuration

# Global options
# --------------
options {
    time_sleep(100);  # polling interval, in ms (helps reduce CPU)
    create_dirs(yes);  # create output directories
    use_fqdn(yes);  # use fully qualified domain names
    ts_format(iso);  # use ISO8601 timestamps (syslog-ng 2.0 only)

    # for normal load
    flush_lines (10); # number of line to buffer before writing to disk
    log_fifo_size(100);

    # for heavy load
    #flush_lines (1000); # number of line to buffer before writing to disk
    #log_fifo_size(1000);

    flush_timeout(500); # in ms

    stats_freq(3600);  # number of seconds between syslog-ng internal stats events; these are useful
                        # for ensuring syslog-ng is not getting overloaded
};

# Sources
# -------
source network {
    tcp(port(5145) max-connections(100));
#    udp(port(5145));
};
source syslog_ng { internal(); };

# Destinations
# ------------
# automatically create new directories each month and for each new host
destination gridlogs {
      file ("/path/to/logs/${YEAR}.${MONTH}/${HOST}.vdt.log"
            perm(0644) dir_perm(0755) create_dirs(yes)
           template("$ISODATE $HOST $MSG\n") template_escape(no) );
};

destination syslog_ng_dest { file ("/tmp/syslog-ng-receiver.log" perm(0644) ); };

# Log definitions
# ---------------
log { source(network); destination(gridlogs); flags (flow-control); };

# for syslog-ng debugging
log { source(syslog_ng); destination(syslog_ng_dest); };

Server adminstration

The syslog-ng server can get "stuck" sometimes, for reasons I don't entirely understand. If you are forwarding log files (i.e. the sender configuration above), do not use the /etc/init.d/syslog-ng restart command unless you want to re-send all your log files from the start. Instead, send a kill -HUP, like this:

ps axw | grep "syslog[-]ng" | cut -f1 -d' ' | xargs kill -HUP