Log pipeline configurations

From CEDPS

NetLogger log pipeline deployment configurations, organized by system

Contents 1 PDSF 2 LBNL ITB cluster 2.1 Running the pipeline 2.2 OSG location 2.3 Log locations 2.4 Collector 2.5 NetLogger 2.6 NetLogger pipeline files 2.7 MySQL

PDSF

All hosts are in the domain .nersc.gov

Collector host: osp

• Syslog-ng receiver
• Pipeline
• MySQL

Log senders:

• pdsfgrid2: Bestman
• pdsfgrid4: SGE, Globus

LBNL ITB cluster

All hosts are in the domain .lbl.gov unless otherwise noted.

• krusty - Runs the NL pipeline, therefore also: the syslog-ng collector and mysql
• natasha - Condor and syslog-ng-sender to krusty
• grolsh - Globus: gatekeeper, container, accounting and gridftp_auth & syslog-ng-sender to krusty
• oliver - bestman srm & syslog-ng-sender to krusty
• zip - bro & syslog-ng-sender to krusty
• newbio.cs.wisc.edu - condor_bp and syslog-ng-sender to krusty

Running the pipeline

nl_pipeline -c /opt/netlogger/pipeline/etc -i 120

• The '-c' option tells where to find the configuration files
• The '-i' option is the refresh time, in seconds, for the configuration. This can safely be left out.

OSG location

Home: /opt/osg

This is a symlink to /opt/itb-0.92 [3/26/09]

Log locations

grolsch syslog-ng sender configuration:

source globus_condor_log { file ("/opt/osg/globus/var/globus-condor.log" follow-freq(1) flags(no-parse) log_prefix('globus_condor_log ') ); };
source gridftp_auth_log { file ("/opt/osg/globus/var/log/gridftp-auth.log" follow-freq(1) flags(no-parse) log_prefix('gridftp_auth_log ') ); };
source gridftp_log { file ("/opt/osg/globus/var/gridftp.log" follow-freq(1) flags(no-parse) log_prefix('gridftp_log ') ); };
source globus_gatekeeper_log { file ("/opt/osg/globus/var/globus-gatekeeper.log" follow-freq(1) flags(no-parse) log_prefix('globus_gatekeeper_log ') ); };
source container_real_log { file ("/opt/osg/globus/var/container-real.log" follow-freq(1) flags(no-parse) log_prefix('container_real_log ') ); };
source container_log { file ("/opt/osg/globus/var/container.log" follow-freq(1) flags(no-parse) log_prefix('container_log ') ); };
source accounting_log { file ("/opt/osg/globus/var/accounting.log" follow-freq(1) flags(no-parse) log_prefix('accounting_log ') ); };

Collector

Host: krusty

• Syslog-ng receiver
• MySQL
• NetLogger pipeline

NetLogger

• Home: /opt/netlogger
• User: netlogger
• Env vars:
  • PATH = /home/portnoy/u7/netlogger/trunk/python/scripts:/home/portnoy/u7/netlogger/trunk/python/scripts/pegasus: ... :/home/portnoy/u7/netlogger/scripts:/home/portnoy/u7/netlogger/bin
  • PYTHONPATH = /home/portnoy/u7/netlogger/trunk/python

NetLogger pipeline files

• /opt/netlogger/pipeline (files user:group => netlogger:users)
  • etc/
    • nl_loader.conf
    • nl_logging.conf
    • ..etc..
  • var/
    • log/
      • nl_{loader,parser,pipeline}.log
    • run/
      • nl_{loader,parser}.state
      • nl_{loader,parser}.pid
  • my.cnf (for MySQL; perms are 600)
  • parsed_logs/
    • parsed.bp.N -- NetLogger files to be loaded into DB
  • watch (shell script to look at progress)

MySQL

• NetLogger user: cedps
• Password: See my.cnf
• Database: itb

mysql --defaults-extra-file=/opt/netlogger/pipeline/my.cnf itb