Pegasus/Kickstart Log Processing

From CEDPS

Jump to: navigation, search

This is a HOWTO Guide for using NetLogger to parse and load Kickstart logs into mySQL for analysis

Contents

Download/install NetLogger Python Tools

Note that NetLogger python tools require python 2.5 or higher. 

svn co https://cvs.globus.org/repos/netlogger/trunk/python
python setup.py build
# possibly as root:
python setup.py install

Install / Configure mySQL

Something like this, depending on your OS: 

apt-get install mysql-server
apt-get install mysql-client
apt-get install python-mysqldb

Parsing and Loading Kickstart files

Parsing: Since the parser opens all the files at once, sets of over 200 or so files will cause a "too many open files" error.

To avoid this, use find piped through xargs with the -L option: 

find . -name "merge_scec-*.out" | xargs -L 100 nl_parser -m pegasus -p one_event=1 > merged.log

Loading into MySQL

First set up your environment: 

export PYTHONPATH=/home/shared/NetLogger/lib/python
MANPATH=$MANPATH:/home/shared/NetLogger/man
PATH=/home/shared/NetLogger/bin:$PATH

The following command will run the NL DB loader on the file CyberShake_LBP_35.bplog creating a new database called 'pegasus' if it doens't already exist, in mysql running on localhost, using defaults as defined in your ~/.my.cnf file. 

nl_loader -u mysql://localhost -p read_default_file="~/.my.cnf" -p db=pegasus -i CyberShake_LBP_35.bplog

Sample Analysis

see SQL and R script examples at: Pegasus Sample Queries

Bug Reports

You can file bug reports at: https://crd.lbl.gov/mantis/main_page.php (Email bltierney@lbl.gov or dkgunter@lbl.gov to request access)

Retrieved from "http://www.cedps.net/index.php/Pegasus/Kickstart_Log_Processing"
Personal tools