See Also: Technical Documentation
Max has been working on monitoring statistics from with the wifi router using collectd and RRD.
Max discussed with ChrisJ about system wide monitoring... recommends Icinga.
Monitoring Server
See Also: Mesh/Icinga 2 Installed on small frame Dell PC, service tag: 2FDSGC1, green tape with info on the front.
- OS: Linux monitor 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u3 x86_64 GNU/Linux
- Access:
- monitor or monitor.local - (as of 10.30.2014, not accessible by this host name, perhaps dns is not correctly defined)
- on 09.18.2014 - 192.168.50.15
- login sudoroom:sudomesh
- http://192.168.50.15/icinga2-classicui - icinga web ui
- http://192.168.50.15/ - graphite web ui
- github repo: not yet. soon.
Sensu: Depricated - requires a client application service and not useful for our needs for monitoring mesh nodes...
- The sensu web page can be accessed internally at:
- http://192.168.42.65:8080/# (as of 2014.07.25 no services defined)
- user: admin
- pw: secret
List of hosts
Running nmap (09.18.2014) to see if I can find a list of hosts which we might want to be monitored...
Most of these are probably laptops connected to the internet.
cjefferies$ nmap -sn 192.168.50.0/24 Starting Nmap 6.46 ( http://nmap.org ) at 2014-09-18 22:04 PDT Nmap scan report for 192.168.50.6 Host is up (0.099s latency). Nmap scan report for 192.168.50.15 Host is up (0.062s latency). Nmap scan report for 192.168.50.20 Host is up (0.063s latency). Nmap scan report for 192.168.50.39 Host is up (0.13s latency). Nmap scan report for 192.168.50.49 Host is up (0.087s latency). Nmap scan report for 192.168.50.60 Host is up (0.011s latency). Nmap scan report for 192.168.50.61 Host is up (0.022s latency). Nmap scan report for 192.168.50.86 Host is up (0.034s latency). Nmap scan report for 192.168.50.87 Host is up (0.047s latency). Nmap scan report for 192.168.50.88 Host is up (0.047s latency). Nmap scan report for 192.168.50.100 Host is up (0.00025s latency). Nmap scan report for 192.168.50.101 Host is up (0.096s latency). Nmap done: 256 IP addresses (12 hosts up) scanned in 23.23 seconds
- 192.168.50.15 - monitoring server
RRD
- http://oss.oetiker.ch/rrdtool/
- also look at whisper, a light weight rrd - http://graphite.wikidot.com/whisper
collectd
Sensu - Deprecated
See our notes about the Sensu install: Sensu Page
About Sensu - From Sensu site
Sensu is often described as the "monitoring router". Essentially, Sensu takes the results of "check" scripts run across many systems, and if certain conditions are met; passes their information to one or more "handlers". Checks are used, for example, to determine if a service like Apache is up or down. Checks can also be used to collect data, such as MySQL query statistics or Rails application metrics. Handlers take actions, using result information, such as sending an email, messaging a chat room, or adding a data point to a graph. There are several types of handlers, but the most common and most powerful is "pipe", a script that receives data via standard input. Check and handler scripts can be written in any language, and the community repository continues to grow!
Fun Sensu facts:
- Written in Ruby, using EventMachine.
- Has great test coverage with continuous integration via Travis CI.
- Can use existing Nagios plugins.
- Configuration is all in JSON.
- Has a message-oriented architecture, using RabbitMQ and JSON payloads.
- Packages are "omnibus", for consistency, isolation, and low-friction deployment.
- Sensu is designed for modern infrastructures and to be driven by configuration management tools, designed for the "cloud".
Icinga
Interesting article: [Setting up Icinga on a Debian Server to Remotely Monitor an OpenWrt Router]
Icinga is a Nagios fork which, as of Fall 2013, has more development involvement. Icinga is the central system that pings other systems like openWRT. Icinga gathers the data and can track and send notifications when values drift beyond normal tolerances. On the remote hosts it is required to install nrpe and a basic set of nrpe plugins. The article referenced above shows how it is possible to install nrpe on openwrt through the openwrt web interface. After that one can ssh into the router and configure it.
Once the router is configured it is necessary to configure the central Icinga server with:
- the IP address of each node it will track
- the host groups
- the services that are to be monitored.
OpenWRT Package: nrpe
Package: nrpe Version: 2.12-4 Depends: libc, librt, libpthread, libopenssl, libwrap Source: feeds/packages/admin/nrpe SourceFile: nrpe-2.12.tar.gz SourceURL: @SF/nagios Section: admin Architecture: ib42x0 Installed-Size: 19018 Filename: nrpe_2.12-4_ib42x0.ipk Size: 19801 MD5Sum: f36019344c747a1e88f5aab50776bd4e Description: The NRPE addon is designed to allow you to execute Nagios plugins on remote Linux/Unix machines. The main reason for doing this is to allow Nagios to monitor "local" resources (like CPU load, memory usage, etc.) on remote machines. Since these public resources are not usually exposed to external machines, an agent like NRPE must be installed on the remote Linux/Unix machines.
Icinga 2
Icinga 2 is a rewrite of Icinga with a cleaner implementation and configuration structure. It will be able to run SNMP calls to nodes which run mini-snmpd and be able to send performance data to charting apps like graphite.
See Install notes at: Icinga 2
Charting
Graphite
- http://www.slideshare.net/reyjrar/graphite-overview
- http://graphite.wikidot.com/whisper
- http://graphite.wikidot.com/
- http://graphite.readthedocs.org/en/latest/config-local-settings.html
- 09.18.2014, ChrisJ started installing this on the monitor server. Not finished...
- 10.02.2014 - another night... continuing.
- installed on host: monitor:/opt/graphite/
- install tutorial: https://tipstricks.itmatrix.eu/installing-icinga2-in-debian-wheezy/
Grafana
- a cool dashboard that you can install under apache (maybe ngnx) and crete a dashboard of data streamed from graphite
- limited to line charts, no meters, gauges, etc.
Installed in: /opt/grafana
As of 10.30.2014 - net yet working... has an issue connecting to Elasticsearch to save settings in grafana. https://github.com/grafana/grafana/issues/330
Setup
Elasticsearch
- requires elasticsearch - https://xenforo.com/community/threads/how-to-basic-elasticsearch-installation-debian-ubuntu.26163/
http://192.168.50.15:9200 - returns something like this:
{
"status" : 200,
"name" : "Exterminator",
"version" : {
"number" : "1.0.3",
"build_hash" : "61bfb72d845a59a58cd9910e47515665f6478a5c",
"build_timestamp" : "2014-04-16T14:43:11Z",
"build_snapshot" : false,
"lucene_version" : "4.6"
},
"tagline" : "You Know, for Search"
}
Graphene
Cricket
motd
____ __ __ ___ __
/ __/ __ __ ___/ / ___ / |/ / ___ ___ / /
_\ \ / // // _ / / _ \ / /|_/ / / -_) (_-< / _ \
/___/ \_,_/ \_,_/ \___//_/ /_/ \__/ /___//_//_/
______________________________________________________
SudoMesh Monitoring Server - Fall, 2014
_______ __ _______ __
| __|.--.--..--| |.-----.| | |.-----..-----.| |--.
|__ || | || _ || _ || || -__||__ --|| |
|_______||_____||_____||_____||__|_|__||_____||_____||__|__|
SudoMesh Monitoring Server - Fall, 2014
____ __ __
/\ _`\ /\ \ /'\_/`\ /\ \
\ \,\L\_\ __ __ \_\ \ ___ /\ \ __ ____ \ \ \___
\/_\__ \ /\ \/\ \ /'_` \ / __`\ \ \ \__\ \ /'__`\ /',__\ \ \ _ `\
/\ \L\ \ \ \ \_\ \/\ \L\ \ /\ \L\ \ \ \ \_/\ \ /\ __/ /\__, `\ \ \ \ \ \
\ `\____\ \ \____/\ \___,_\\ \____/ \ \_\\ \_\\ \____\\/\____/ \ \_\ \_\
\/_____/ \/___/ \/__,_ / \/___/ \/_/ \/_/ \/____/ \/___/ \/_/\/_/
SudoMesh Monitoring Server - Fall, 2014
____ _ __ __ _
/ ___| _ _ __| | ___ | \/ | ___ ___ | |__
\___ \ | | | | / _` | / _ \ | |\/| | / _ \ / __| | '_ \
___) | | |_| | | (_| | | (_) | | | | | | __/ \__ \ | | | |
|____/ \__,_| \__,_| \___/ |_| |_| \___| |___/ |_| |_|
SudoMesh Monitoring Server - Fall, 2014