Monitor hundreds of linux ethernet interfaces remotly and agentless
Posted: | More posts about go linux sysadmin
There are plenty of tools to monitor network interfaces on a linux machine (es. iftop, iptraf). One day I needed a way to monitor network parameters of tens of linux firewalls remotly and without installing anything on targets. After evaluated existent solutions I wrote rim.
How it works
rim gets relevant network interfaces' data (Rx-Kb/s, Tx-Kb/s, Rx-Pckts/s, Tx-Pckts/s, Rx-Drp/s, Tx-Drp/s, Rx-Err/s, Tx-Err/s) reading /proc filesystem. The agentless fashion is reached thanks to ssh transport layer.
rim use concurrency to realize parallelism and dinamically adapts number or workers to the number of usable cores. On a multicore machine it could be easily retrieves data from hundreds of servers in seconds.
Sadly there are still no prebuilt packages but Go makes installing software from source very easy. The trickiest part is to install a Go environment which will not be explained here but official docs are very good.
Once you have got Go installed all you have to do is:
$ go get github.com/eraclitux/rim
The best way is to use ssh-agent. This implies that you have to copy your public key to all machines that you must controll.
Create a file (es. my_hosts.txt) containing the target hosts, one per line es:
firewall-1.mynet.com firewall-2.mynet.com firewall-3.mynet.com firewall-4.mynet.com:2222 ...
You can now get the data from remote hosts (you must add $GOPATH/bin to $PATH to invoke rim like this):
$ rim -f my_hosts.txt
. Host Interface Rx-Kb/s Tx-Kb/s Rx-Pckts/s Tx-Pckts/s Rx-Drp/s Tx-Drp/s Rx-Err/s Tx-Err/s firewall-4.mynet.com lo 0 0 0 0 0 0 0 0 firewall-2.mynet.com eth0 16105 124971 9439 13267 0 0 0 0 firewall-4.mynet.com eth2 214952 36429 24761 18859 100 0 0 0 firewall-1.mynet.com br0 0 0 0 0 0 0 0 0 firewall-3.mynet.com eth1 15101 85914 8129 10024 0 0 0 0 firewall-2.mynet.com eth3 0 0 1 0 0 0 0 0 firewall-1.mynet.com veth0 0 0 0 0 0 0 0 0
Sorting capabilities will be added in future releases meanwhile you can pipe the output to sort. For example to spot which interfaces are receiving most Kb per second:
$ rim -f my_hosts.txt -n | sort -n -r -k3
Every network has its own peculiar characteristics and only continuous observation (or fancy machine learning technics) can spot problems but dropped packets are always bad. In the sample output above eth2 @ firewall-4.mynet.com is dropping 100 packets per second in receiveing queue. This usually means that the load on machine is too high (you can check it with top) and all the services that are using that interface are suffering bad performances.