Monitor hundreds of linux ethernet interfaces remotly and agentless

Posted:   |   More posts about go linux sysadmin

There are plenty of tools to monitor network interfaces on a linux machine (es. iftop, iptraf). One day I needed a way to monitor network parameters of tens of linux firewalls remotly and without installing anything on targets. After evaluated existent solutions I wrote rim.

How it works

rim gets relevant network interfaces' data (Rx-Kb/s, Tx-Kb/s, Rx-Pckts/s, Tx-Pckts/s, Rx-Drp/s, Tx-Drp/s, Rx-Err/s, Tx-Err/s) reading /proc filesystem. The agentless fashion is reached thanks to ssh transport layer.

rim use concurrency to realize parallelism and dinamically adapts number or workers to the number of usable cores. On a multicore machine it could be easily retrieves data from hundreds of servers in seconds.

Installation

Sadly there are still no prebuilt packages but Go makes installing software from source very easy. The trickiest part is to install a Go environment which will not be explained here but official docs are very good.

Once you have got Go installed all you have to do is:

$ go get github.com/eraclitux/rim

Usage

Tip: how easily copy you ssh key

$ ssh-copy-id root@<target_host>

The best way is to use ssh-agent. This implies that you have to copy your public key to all machines that you must controll.

Create a file (es. my_hosts.txt) containing the target hosts, one per line es:

firewall-1.mynet.com
firewall-2.mynet.com
firewall-3.mynet.com
firewall-4.mynet.com:2222
...

You can now get the data from remote hosts (you must add $GOPATH/bin to $PATH to invoke rim like this):

$ rim -f my_hosts.txt

Sample output:

.               Host   Interface  Rx-Kb/s  Tx-Kb/s  Rx-Pckts/s  Tx-Pckts/s    Rx-Drp/s    Tx-Drp/s    Rx-Err/s    Tx-Err/s
firewall-4.mynet.com          lo        0        0           0           0           0           0           0           0
firewall-2.mynet.com        eth0    16105   124971        9439       13267           0           0           0           0
firewall-4.mynet.com        eth2   214952    36429       24761       18859         100           0           0           0
firewall-1.mynet.com         br0        0        0           0           0           0           0           0           0
firewall-3.mynet.com        eth1    15101    85914        8129       10024           0           0           0           0
firewall-2.mynet.com        eth3        0        0           1           0           0           0           0           0
firewall-1.mynet.com       veth0        0        0           0           0           0           0           0           0

Sorting results

Sorting capabilities will be added in future releases meanwhile you can pipe the output to sort. For example to spot which interfaces are receiving most Kb per second:

$ rim -f my_hosts.txt -n | sort -n -r -k3

Reading results

Every network has its own peculiar characteristics and only continuous observation (or fancy machine learning technics) can spot problems but dropped packets are always bad. In the sample output above eth2 @ firewall-4.mynet.com is dropping 100 packets per second in receiveing queue. This usually means that the load on machine is too high (you can check it with top) and all the services that are using that interface are suffering bad performances.