Syslog-ng
From CEDPS
This page contains instructions for using syslog-ng to collect Grid log files from several hosts to a central location.
First we recommend syslog-ng 2.0 over syslog-ng 1.6 because of the new ISO date option, which is needed for logging across multiple time zones. To download, go to: http://www.balabit.com/downloads/files/syslog-ng/sources/stable/src/
Here are some sample syslog-ng 2.0 configuration files. These sample configuration assumes that you prefer not to replace the default syslog with syslog-ng. This means that one does not require root access to install syslog-ng.
Contents |
Sample package for VDT
See this page.
Sample syslog-ng configuration file for host running Grid Services
options {
time_sleep(50); # polling interval, in ms (helps reduce CPU)
use_fqdn(yes); # use fully qualified domain names
ts_format(iso); # use ISO8601 timestamps
#
# for normal load
flush_lines (10); # number of line to buffer before writing to disk
log_fifo_size(100);
#
# for heavy load
#flush_lines (1000); # number of line to buffer before writing to disk
#log_fifo_size(1000);
#
stats_freq(3600); # number of seconds between syslog-ng internal stats events; these are useful
# for ensuring syslog-ng is not getting overloaded
};
#
source gridftp_log { file ("/mypath/gridftp.log" follow-freq(1) flags(no-parse) log_prefix('gridftp_log ') ); };
source gatekeeper_log { file ("/mypath/gatekeeper.log" follow-freq(1) flags(no-parse) log_prefix('gatekeeper_log ') ); };
# syslog-ng internal logs; useful for testing syslog-ng config
source syslog_ng { internal(); };
#
# define the Forwarding Destination
destination gridlog_dst {
tcp("myloghost.grid.gov" port(5141));
};
destination syslog_ng_dst {
file ("/tmp/syslog-ng.log" perm(0644) );
};
#
# forward sources to destination
log { source(gridftp_log); destination(gridlog_dst); flags(flow-control); };
log { source(gatekeeper_log); destination(gridlog_dst); flags(flow-control); };
#
# for syslog-ng debugging
log {
source(syslog_ng); destination(syslog_ng_dst);
};
Sample syslog-ng config file for central Log capture host
options {
time_sleep(50); # polling interval, in ms (helps reduce CPU)
create_dirs(yes); # create output directories
use_fqdn(yes); # use fully qualified domain names
ts_format(iso); # use ISO8601 timestamps (syslog-ng 2.0 only)
#
# for normal load
flush_lines (10); # number of line to buffer before writing to disk
log_fifo_size(100);
#
# for heavy load
#flush_lines (1000); # number of line to buffer before writing to disk
#log_fifo_size(1000);
flush_timeout(500); # in ms
#
stats_freq(3600);
};
#
# define the source: any host sending to port 5141
source network {
tcp(port(5141) max-connections(500));
internal(); # internal syslog-ng messages
};
#
# Define the destination, automatically creating new directories
# for each month and new host.
destination gridlogs {
file ("/tmp/grid/logfiles/$YEAR.$MONTH/gridlog.$HOST.log"
perm(0644) dir_perm(0755) create_dirs(yes)
template("$ISODATE $HOST $MSG\n") );
};
#
log { source(network);
destination(gridlogs); flags (flow-control);
};
Testing your setup
To verify this all works, try:
echo "test message" > /path/gridftp.log
Then look to see if the message arrives on the server. You may need to do several messsages before they are flushed to the server.
syslog-ng 2.0.x build hints for Linux
You can just download the OSG configuration:
pacman -get OSG:Syslog-ng
Or you can build from source
# if /usr/lib/libglib-2.0.so is missing, do this first apt-get install libglib2.0-dev or yum install glib2-devel # download wget http://www.balabit.com/downloads/files/syslog-ng/sources/stable/src/eventlog-0.2.7.tar.gz wget http://www.balabit.com/downloads/files/syslog-ng/sources/stable/src/syslog-ng-2.0.9.tar.gz tar xvzf eventlog-0.2.7.tar.gz tar xvzf syslog-ng-2.0.9.tar.gz cd eventlog-0.2.7 ./configure --prefix=/path_to_install make make install # cd ../syslog-ng-2.0.9 setenv EVTLOG_CFLAGS "-I/path_to_install/include/eventlog" setenv EVTLOG_LIBS "-L/path_to_install/lib/ -levtlog" ./configure --prefix=/path_to_install make make install
SSL-wrapped syslog-ng instructions
For instructions on sending syslog-ng over a SSL socket see: http://gentoo-wiki.com/HOWTO_create_a_logserver_with_syslog-ng and http://www.stunnel.org/examples/syslog-ng.html
For More Information
