Osp pipeline details
From CEDPS
Netlogger pipeline details for osp
Contents |
Installation
Netlogger is installed on osp into /opt/netlogger as root.
To install current trunk code:
- Run as nluser:
cd ~/netlogger-trunk/python svn update
- Run as root:
cd ~nluser/netlogger-trunk/python python setup.py install --home=/opt/netlogger
Basic setup
- The nluser user's ~/.bashrc sets:
export NETLOGGER_HOME=/opt/netlogger
PATH=${PATH}:${NETLOGGER_HOME}/bin
export PYTHONPATH=${NETLOGGER_HOME}/lib/python
- The base dir of the pipeline config is ~nluser/pipeline
Running and monitoring
Here's some commands on how to start, monitor and stop the pipeline.
All of the following command are assumed to run in the ~nluser/pipeline dir as the nluser user.
Is the pipeline running?
$ ps -p `cat ./var/run/nl_*.pid` PID TTY STAT TIME COMMAND 19851 ? Ss 0:00 /usr/local/bin/python /opt/netlogger/bin/nl_pipeline -c ./etc -v 19862 ? Ssl 12:02 /usr/local/bin/python /opt/netlogger/bin/nl_parser --daemon --pipeline-port=25253 --pidfile=... 19874 ? Ssl 2:58 /usr/local/bin/python /opt/netlogger/bin/nl_loader --daemon --pipeline-port=25252 --pidfile=...
if you don't see all 3 components, then something is not running
Stopping the pipeline
As user nluser, simply kill the nl_pipeline process.
$ kill `cat ./var/run/nl_pipeline.pid`
Starting the pipeline
As user nluser:
$ nl_pipeline -c ./etc -v
Monitoring the pipeline
There are many ways to do this, a simple one is to use the custom written watch script there in the pipeline dir:
$ ./watch
This will print out lots of info about netlogger state files and events in the DB.
More detailed investigation would then follow by comparing the values in the state files vs the size of the files that are being followed by the pipeline components. It might also be necessary to get onto the collector machines to see if there may be sysnlog-ng issues...
