Gridftp-netlogger
From CEDPS
Contents |
Overview
As part of the CEDPS Troubleshooting effort, we added the ability to analyze I/O instrumentation to NetLogger, and integrated this with the Globus GridFTP client and server. This lets the end-user easily determine if the network or the disk is the bottleneck for their file transfer.
Please let us know what you think of this new feature, and how it could be improved.
Interpreting the Results
There is a new command line option for globus-url-copy, "-nlb", which will generate output such as this:
Total instantaneous throughput: disk read = 17022.2 Mbits/s disk write = 26630.8 Mbits/s net read = 509.0 Mbits/s net write = 1053.4 Mbits/s Bottleneck: network
The instantaneous throughput is the average of, for each read or write, the number of bytes divided by the time spent blocking on the system call. Because this does not include the time waiting for data or space to be available, the instantaneous throughputs are in general higher than the overall throughput of the transfer: they are primarily useful for comparison and not as absolute numbers.
You should ignore the "net write" value, as the time to write to the network is strongly influenced by the time to read from disk, and system and TCP buffer artifacts.
Possible "Bottleneck:" results are:
- network: somewhere in the network
- disk read: sender's disk
- disk write: receiver's disk
- unknown: disk/network are about the same and/or highly variable
Installation Instructions
Build and Install GridFTP
NetLogger is bundled with Globus starting with Globus v4.2, so downloading NetLogger separately is no longer needed. These instructions have been tested and verified to work with Globus 4.2rc2 (available at: http://dev.globus.org/wiki/Globus_Toolkit/4.2.0). This ability is not enabled by default, however. To enable NetLogger's bottleneck detection, use the --enable-netlogger flag:
wget http://www-unix.globus.org/ftppub/gt4/4.2.0/installers/src/gt4.2.0-all-source-installer.tar.gz tar xvfz gt4.2.0-all-source-installer.tar.gz mkdir GL cd gt4.2.0-all-source-installer ./configure --prefix=`pwd`/../GL --enable-netlogger make gridftp globus_xio_netlogger_driver # Note that this step can take up to 1 hour. make install gridftp globus_xio_netlogger_driver
Configure GridFTP server
Use -log-module stdio_ng -log-level info,warn,error -logfile <filename> to generate "best practice" logs.
Use -fs-whitelist netlogger,file -dc-whitelist netlogger,gsi,tcp to do NetLogger I/O instrumentation
To run from xinetd, add something like this to: /etc/xinetd.d/gsiftp:
server = /opt/globus/sbin/globus-gridftp-server server_args = -i -log-module stdio_ng -log-level info,warn,error -logfile /var/log/gridftp.log -fs-whitelist netlogger,file -dc-whitelist netlogger,gsi,tcp
Alternatively, you can use the file $GLOBUS_LOCATION/etc/gridftp.conf to configure this:
inetd 1 log_level ERROR,WARN,INFO log_module stdio_ng fs-whitelist netlogger,file dc-whitelist netlogger,gsi,tcp log_single /opt/globus/var/log/gridftp-auth.log log_transfer /opt/globus/var/log/gridftp.log
Please note: The "fs-whitelist" and "dc-whitelist" options will not work in gridftp.conf in gridftp versions before 4.2.1
Or, for testing, you can run from the server from command line:
setenv GLOBUS_LOCATION /your_install_loc/GL
source $GLOBUS_LOCATION/etc/globus-user-env.csh
grid-proxy-init
$GLOBUS_LOCATION/sbin/globus-gridftp-server -log-module stdio_ng -log-level info,warn,error \
-logfile /tmp/gridftp.log -fs-whitelist netlogger,file -dc-whitelist netlogger,gsi,tcp
If you run the server like this, you'll need to add the -subject flag to globus-url-copy, e.g.:
sbj=`cat ~/.gridmap | cut -f2 -d'"'` globus-url-copy -subject "$sbj" ...etc..
For more information on configuring the GridFTP server see:
http://www.globus.org/toolkit/docs/development/4.2-drafts/data/gridftp/admin/index.html
Testing
To test, first telnet to the gridftp server port to verify that the server args actually work, and that firewalls or tcp_wrappers aren't blocking anything.
Then try using the client:
source $GLOBUS_LOCATION/etc/globus-user-env.csh #or source $GLOBUS_LOCATION/etc/globus-user-env.sh
then
grid-proxy-init globus-url-copy gsiftp://testhost:port/etc/group file:///dev/null globus-url-copy -vb gsiftp://testhost:port/tmp/200M.test.file file:///dev/null
Now try out the new NetLogger bottleneck detection (-nlb option) (note: currently this only works with 3rd party transfers):
globus-url-copy -p 2 -nlb -vb -len 500M gsiftp://host1:port/path/myfile gsiftp://host2:port/dev/null
This should output something like the following:
Total instantaneous throughput: disk read = 17022.2 Mbits/s disk write = 26630.8 Mbits/s net read = 509.0 Mbits/s net write = 1053.4 Mbits/s Bottleneck: network
