Difference between revisions of "Mesh/Monitoring"

From Sudo Room
Jump to navigation Jump to search
 
(20 intermediate revisions by one other user not shown)
Line 1: Line 1:
See Also: [[Mesh#Technical_Documentation | Technical Documentation]]
See Also: [[Mesh#Technical_Documentation | Technical Documentation]],  [[Mesh/Bandwidth Quotas | Bandwidth Quotas]]


[[Category:Monitoring]]
[[Category:Monitoring]]
[[Category:Technical]]
[[Category:Technical]]


Max has been working on monitoring statistics from with the wifi router using collectd and RRD.
This is all currently out of date. The currently implemented monitor server is running on VPS, which is tunneled into the network.
It's running [https://monitor.sudomesh.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=45 Cacti] and [https://monitor.sudomesh.org/smokeping/smokeping.cgi?target=Mesh Smokeping] as well as gathering logs.


Max discussed with ChrisJ about system wide monitoring...  recommends Icinga.  
When I have the time, I'll update the details here, but I've documented the setup as provisioning code here: https://github.com/max-b/mesh-playbooks - [[User:Maxb]]


== Monitoring Server ==
 
== The Basics ==
See Also: [[Mesh/Icinga 2]]
See Also: [[Mesh/Icinga 2]]
Installed on small frame Dell PC, service tag: 2FDSGC1, green tape with info on the front.
 
=== The Hardware ===
Small frame Dell PC, service tag: 2FDSGC1, green tape with info on the front.
 
=== The Software ===
* OS:  Linux monitor 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u3 x86_64 GNU/Linux
* OS:  Linux monitor 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u3 x86_64 GNU/Linux
* Access:
** monitor or monitor.local - (as of 10.30.2014, not accessible by this host name, perhaps dns is not correctly defined)
** on 09.18.2014 - 192.168.50.15
** login sudoroom:sudomesh


* http://192.168.50.15/icinga2-classicui - icinga web ui
=== Use It ===
* http://192.168.50.15/ - graphite web ui
 
ssh to:
* monitor or monitor.local - (as of 10.30.2014, not accessible by this host name, perhaps dns is not correctly defined)
* 192.168.50.15
* login: sudoroom:sudomesh
 
Icinga web UI:
* http://192.168.50.15/icinga2-classicui (login: icingaadmin:mojave)


* github repo: not yet. soon.
Charting:
* Graphite - http://192.168.50.15/
* Grafana - http://192.168.50.15/grafana
* Mesh Charts - http://192.168.50.15/grafana/#/dashboard/db/sudomesh  (works best in firefox)


----
Github repo:  
Sensu: Depricated  - requires a client application service and not useful for our needs for monitoring mesh nodes...
* https://github.com/TinajaLabs/sudomesh_icinga
* The sensu web page can be accessed internally at:
** http://192.168.42.65:8080/# (as of 2014.07.25 no services defined)
** user: admin
** pw: secret


== List of hosts ==
== List of hosts ==
Running nmap (09.18.2014) to see if I can find a list of hosts which we might want to be monitored...
Run nmap (09.18.2014) to see a list of hosts which we might want to monitor...


Most of these are probably laptops connected to the internet.
Most of these are probably laptops connected to the internet.
<pre>
nmap -sn 192.168.50.0/24
cjefferies$ nmap -sn 192.168.50.0/24
Starting Nmap 6.46 ( http://nmap.org ) at 2014-09-18 22:04 PDT
Nmap scan report for 192.168.50.6
Host is up (0.099s latency).
Nmap scan report for 192.168.50.15
Host is up (0.062s latency).
Nmap scan report for 192.168.50.20
Host is up (0.063s latency).
Nmap scan report for 192.168.50.39
Host is up (0.13s latency).
Nmap scan report for 192.168.50.49
Host is up (0.087s latency).
Nmap scan report for 192.168.50.60
Host is up (0.011s latency).
Nmap scan report for 192.168.50.61
Host is up (0.022s latency).
Nmap scan report for 192.168.50.86
Host is up (0.034s latency).
Nmap scan report for 192.168.50.87
Host is up (0.047s latency).
Nmap scan report for 192.168.50.88
Host is up (0.047s latency).
Nmap scan report for 192.168.50.100
Host is up (0.00025s latency).
Nmap scan report for 192.168.50.101
Host is up (0.096s latency).
Nmap done: 256 IP addresses (12 hosts up) scanned in 23.23 seconds
</pre>


* 192.168.50.15 - monitoring server
* 192.168.50.15 - monitoring server
*
*


== RRD ==
== Icinga ==
* http://oss.oetiker.ch/rrdtool/
Interesting article: [http://www.smallbusinesstech.net/more-complicated-instructions/nagios/setting-up-nagios-on-a-debian-server-to-remotely-monitor-an-openwrt-router Setting up Icinga on a Debian Server to Remotely Monitor an OpenWrt Router]
* also look at whisper, a light weight rrd - http://graphite.wikidot.com/whisper
*


== collectd ==
Icinga is a Nagios fork which, as of Fall 2013, has more development involvement.  Icinga is the central system that pings other systems like openWRT.  Icinga gathers the data and can track and send notifications when values drift beyond normal tolerances.  On the remote hosts it is required to install '''nrpe''' and a basic set of '''nrpe plugins'''.  The article referenced above shows how it is possible to install nrpe on openwrt through the openwrt web interface.  After that one can ssh into the router and configure it.
* http://collectd.org/


== Sensu - Deprecated ==
Once the router is configured it is necessary to configure the central Icinga server with:
See our notes about the Sensu install: [[Mesh/Sensu_Page | Sensu Page]]
* the IP address of each node it will track
* the host groups
* the services that are to be monitored.


=== About Sensu - From Sensu site ===
== Icinga 2 ==
Icinga 2 is a rewrite of Icinga with a cleaner implementation and configuration structure.  It will be able to run SNMP calls to nodes which run mini-snmpd and be able to send performance data to charting apps like graphite.


Sensu is often described as the "monitoring router". Essentially, Sensu takes the results of "check" scripts run across many systems, and if certain conditions are met; passes their information to one or more "handlers". Checks are used, for example, to determine if a service like Apache is up or down. Checks can also be used to collect data, such as MySQL query statistics or Rails application metrics. Handlers take actions, using result information, such as sending an email, messaging a chat room, or adding a data point to a graph. There are several types of handlers, but the most common and most powerful is "pipe", a script that receives data via standard input. Check and handler scripts can be written in any language, and the community repository continues to grow!
See Install notes at: [[Icinga 2]]


Fun Sensu facts:


* Written in Ruby, using EventMachine.
== Access to remote hosts ==
* Has great test coverage with continuous integration via Travis CI.
Icinga can use various methods to monitor remote hosts, NRPE, SNMP, etc.
* Can use existing Nagios plugins.
* Configuration is all in JSON.
* Has a message-oriented architecture, using RabbitMQ and JSON payloads.
* Packages are "omnibus", for consistency, isolation, and low-friction deployment.
* Sensu is designed for modern infrastructures and to be driven by configuration management tools, designed for the "cloud".


== Icinga ==
=== IMCP ===
Interesting article: [[http://www.smallbusinesstech.net/more-complicated-instructions/nagios/setting-up-nagios-on-a-debian-server-to-remotely-monitor-an-openwrt-router Setting up Icinga on a Debian Server to Remotely Monitor an OpenWrt Router]]
The ping service; for determining if the host is reachable


Icinga is a Nagios fork which, as of Fall 2013, has more development involvement.  Icinga is the central system that pings other systems like openWRT.  Icinga gathers the data and can track and send notifications when values drift beyond normal tolerancesOn the remote hosts it is required to install '''nrpe''' and a basic set of '''nrpe plugins'''.  The article referenced above shows how it is possible to install nrpe on openwrt through the openwrt web interface.  After that one can ssh into the router and configure it.
=== SNMP ===
 
We will probably start off with simple SNMP monitoringIt return very basic info but it does not require too much setup on the remote hosts.
Once the router is configured it is necessary to configure the central Icinga server with:
* the IP address of each node it will track
* the host groups
* the services that are to be monitored.


=== OpenWRT Package: nrpe ===
=== OpenWRT Package: nrpe ===
NRPE requires a daemon running on the remote host and a number of mostly bash scripts for specialized info.  Bash scripts can be developed to read disk stats, memory usage, etc.  There are about 30 scripts which come with NRPE.
<pre>
<pre>
Package: nrpe
Package: nrpe
Line 123: Line 94:
</pre>
</pre>


== Icinga 2 ==
== Alternative tools ==
Icinga 2 is a rewrite of Icinga with a cleaner implementation and configuration structure. It will be able to run SNMP calls to nodes which run mini-snmpd and be able to send performance data to charting apps like graphite.
 
=== RRD ===
* http://oss.oetiker.ch/rrdtool/
* also look at whisper, a light weight rrd - http://graphite.wikidot.com/whisper
*
 
=== collectd ===
* http://collectd.org/
 
 
=== Sensu - Deprecated ===
See our notes about the Sensu install: [[Mesh/Sensu_Page | Sensu Page]]
 
'''Requires a client application service and not useful for our needs for monitoring mesh nodes...'''
 
* The sensu web page can be accessed internally at:
** http://192.168.42.65:8080/# (as of 2014.07.25 no services defined)
** user: admin
** pw: secret
 
==== About Sensu - From Sensu site ====
 
Sensu is often described as the "monitoring router". Essentially, Sensu takes the results of "check" scripts run across many systems, and if certain conditions are met; passes their information to one or more "handlers". Checks are used, for example, to determine if a service like Apache is up or down. Checks can also be used to collect data, such as MySQL query statistics or Rails application metrics. Handlers take actions, using result information, such as sending an email, messaging a chat room, or adding a data point to a graph. There are several types of handlers, but the most common and most powerful is "pipe", a script that receives data via standard input. Check and handler scripts can be written in any language, and the community repository continues to grow!
 
Fun Sensu facts:


See Install notes at: [[Icinga 2]]
* Written in Ruby, using EventMachine.
* Has great test coverage with continuous integration via Travis CI.
* Can use existing Nagios plugins.
* Configuration is all in JSON.
* Has a message-oriented architecture, using RabbitMQ and JSON payloads.
* Packages are "omnibus", for consistency, isolation, and low-friction deployment.
* Sensu is designed for modern infrastructures and to be driven by configuration management tools, designed for the "cloud".


== Charting ==
== Charting ==
Line 135: Line 136:
* http://graphite.wikidot.com/
* http://graphite.wikidot.com/
* http://graphite.readthedocs.org/en/latest/config-local-settings.html
* http://graphite.readthedocs.org/en/latest/config-local-settings.html
 
* graphite admin:  http://192.168.50.15/admin - sudomesh:sudomesh


* 09.18.2014, ChrisJ started installing this on the monitor server.  Not finished...
* 09.18.2014, ChrisJ started installing this on the monitor server.  Not finished...
Line 143: Line 144:


* install tutorial: https://tipstricks.itmatrix.eu/installing-icinga2-in-debian-wheezy/
* install tutorial: https://tipstricks.itmatrix.eu/installing-icinga2-in-debian-wheezy/
<gallery>
File:http://192.168.50.15/render?width=400&from=-2hours&until=now&height=250&target=icinga.localhost.ping4.rta&_uniq=0.43809576937928796&title=icinga.localhost.ping4.rta|Caption1
</gallery>


==== Sample live charts rendered from Graphite ====
http://192.168.50.15/render?width=400&from=-2hours&until=now&height=250&target=icinga.localhost.ping4.rta&_uniq=0.43809576937928796&title=icinga.localhost.ping4.rta&.png
http://192.168.50.15/render?width=400&from=-2hours&until=now&height=250&target=icinga.localhost.ping4.rta&_uniq=0.43809576937928796&title=icinga.localhost.ping4.rta&.png
<br>Round trip average ping over the last 2 hours:


=== Grafana ===
=== Grafana ===
Line 163: Line 163:
http://192.168.50.15/grafana
http://192.168.50.15/grafana


http://192.168.50.15/grafana/#/dashboard/db/sudomesh - works best in firefox...


==== Elasticsearch ====
==== Elasticsearch ====
Line 190: Line 191:
* http://cricket.sourceforge.net/
* http://cricket.sourceforge.net/
*
*


== motd ==
== motd ==

Latest revision as of 13:36, 30 December 2015

See Also: Technical Documentation, Bandwidth Quotas

This is all currently out of date. The currently implemented monitor server is running on VPS, which is tunneled into the network. It's running Cacti and Smokeping as well as gathering logs.

When I have the time, I'll update the details here, but I've documented the setup as provisioning code here: https://github.com/max-b/mesh-playbooks - User:Maxb


The Basics

See Also: Mesh/Icinga 2

The Hardware

Small frame Dell PC, service tag: 2FDSGC1, green tape with info on the front.

The Software

  • OS: Linux monitor 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u3 x86_64 GNU/Linux

Use It

ssh to:

  • monitor or monitor.local - (as of 10.30.2014, not accessible by this host name, perhaps dns is not correctly defined)
  • 192.168.50.15
  • login: sudoroom:sudomesh

Icinga web UI:

Charting:

Github repo:

List of hosts

Run nmap (09.18.2014) to see a list of hosts which we might want to monitor...

Most of these are probably laptops connected to the internet.

nmap -sn 192.168.50.0/24
  • 192.168.50.15 - monitoring server

Icinga

Interesting article: Setting up Icinga on a Debian Server to Remotely Monitor an OpenWrt Router

Icinga is a Nagios fork which, as of Fall 2013, has more development involvement. Icinga is the central system that pings other systems like openWRT. Icinga gathers the data and can track and send notifications when values drift beyond normal tolerances. On the remote hosts it is required to install nrpe and a basic set of nrpe plugins. The article referenced above shows how it is possible to install nrpe on openwrt through the openwrt web interface. After that one can ssh into the router and configure it.

Once the router is configured it is necessary to configure the central Icinga server with:

  • the IP address of each node it will track
  • the host groups
  • the services that are to be monitored.

Icinga 2

Icinga 2 is a rewrite of Icinga with a cleaner implementation and configuration structure. It will be able to run SNMP calls to nodes which run mini-snmpd and be able to send performance data to charting apps like graphite.

See Install notes at: Icinga 2


Access to remote hosts

Icinga can use various methods to monitor remote hosts, NRPE, SNMP, etc.

IMCP

The ping service; for determining if the host is reachable

SNMP

We will probably start off with simple SNMP monitoring. It return very basic info but it does not require too much setup on the remote hosts.

OpenWRT Package: nrpe

NRPE requires a daemon running on the remote host and a number of mostly bash scripts for specialized info. Bash scripts can be developed to read disk stats, memory usage, etc. There are about 30 scripts which come with NRPE.

Package: nrpe
Version: 2.12-4
Depends: libc, librt, libpthread, libopenssl, libwrap
Source: feeds/packages/admin/nrpe
SourceFile: nrpe-2.12.tar.gz
SourceURL: @SF/nagios
Section: admin
Architecture: ib42x0
Installed-Size: 19018
Filename: nrpe_2.12-4_ib42x0.ipk
Size: 19801
MD5Sum: f36019344c747a1e88f5aab50776bd4e
Description:  The NRPE addon is designed to allow you to execute Nagios plugins on
 remote Linux/Unix machines.  The main reason for doing this is to allow
 Nagios to monitor "local" resources (like CPU load, memory usage, etc.)
 on remote machines.  Since these public resources are not usually
 exposed to external machines, an agent like NRPE must be installed on
 the remote Linux/Unix machines.

Alternative tools

RRD

collectd


Sensu - Deprecated

See our notes about the Sensu install: Sensu Page

Requires a client application service and not useful for our needs for monitoring mesh nodes...

  • The sensu web page can be accessed internally at:

About Sensu - From Sensu site

Sensu is often described as the "monitoring router". Essentially, Sensu takes the results of "check" scripts run across many systems, and if certain conditions are met; passes their information to one or more "handlers". Checks are used, for example, to determine if a service like Apache is up or down. Checks can also be used to collect data, such as MySQL query statistics or Rails application metrics. Handlers take actions, using result information, such as sending an email, messaging a chat room, or adding a data point to a graph. There are several types of handlers, but the most common and most powerful is "pipe", a script that receives data via standard input. Check and handler scripts can be written in any language, and the community repository continues to grow!

Fun Sensu facts:

  • Written in Ruby, using EventMachine.
  • Has great test coverage with continuous integration via Travis CI.
  • Can use existing Nagios plugins.
  • Configuration is all in JSON.
  • Has a message-oriented architecture, using RabbitMQ and JSON payloads.
  • Packages are "omnibus", for consistency, isolation, and low-friction deployment.
  • Sensu is designed for modern infrastructures and to be driven by configuration management tools, designed for the "cloud".

Charting

Graphite

  • 09.18.2014, ChrisJ started installing this on the monitor server. Not finished...
  • 10.02.2014 - another night... continuing.
  • installed on host: monitor:/opt/graphite/

Sample live charts rendered from Graphite

render?width=400&from=-2hours&until=now&height=250&target=icinga.localhost.ping4.rta&_uniq=0.43809576937928796&title=icinga.localhost.ping4.rta&.png
Round trip average ping over the last 2 hours:

Grafana

http://grafana.org/

  • a cool dashboard that you can install under apache (maybe ngnx) and crete a dashboard of data streamed from graphite
  • limited to line charts, no meters, gauges, etc.

Installed in: /opt/grafana

As of 10.30.2014 - net yet working... has an issue connecting to Elasticsearch to save settings in grafana. https://github.com/grafana/grafana/issues/330

Setup

http://192.168.50.15/grafana

http://192.168.50.15/grafana/#/dashboard/db/sudomesh - works best in firefox...

Elasticsearch

http://192.168.50.15:9200 - returns something like this:

{
"status" : 200,
"name" : "Exterminator",
"version" : {
"number" : "1.0.3",
"build_hash" : "61bfb72d845a59a58cd9910e47515665f6478a5c",
"build_timestamp" : "2014-04-16T14:43:11Z",
"build_snapshot" : false,
"lucene_version" : "4.6"
},
"tagline" : "You Know, for Search"
}

Graphene

Cricket

motd

We can set a boot message for the /etc/motd file.

These samples from http://patorjk.com/software/taag/#p=testall&h=0&v=0&f=Mer&t=SudoMesh


   ____           __        __  ___             __ 
  / __/ __ __ ___/ / ___   /  |/  / ___   ___  / / 
 _\ \  / // // _  / / _ \ / /|_/ / / -_) (_-< / _ \
/___/  \_,_/ \_,_/  \___//_/  /_/  \__/ /___//_//_/
______________________________________________________
    SudoMesh Monitoring Server - Fall, 2014



 _______            __         _______                __    
|     __|.--.--..--|  |.-----.|   |   |.-----..-----.|  |--.
|__     ||  |  ||  _  ||  _  ||       ||  -__||__ --||     |
|_______||_____||_____||_____||__|_|__||_____||_____||__|__|
    SudoMesh Monitoring Server - Fall, 2014                                                          


 ____                  __                                          __         
/\  _`\               /\ \             /'\_/`\                    /\ \        
\ \,\L\_\    __  __   \_\ \     ___   /\      \      __     ____  \ \ \___    
 \/_\__ \   /\ \/\ \  /'_` \   / __`\ \ \ \__\ \   /'__`\  /',__\  \ \  _ `\  
   /\ \L\ \ \ \ \_\ \/\ \L\ \ /\ \L\ \ \ \ \_/\ \ /\  __/ /\__, `\  \ \ \ \ \ 
   \ `\____\ \ \____/\ \___,_\\ \____/  \ \_\\ \_\\ \____\\/\____/   \ \_\ \_\
    \/_____/  \/___/  \/__,_ / \/___/    \/_/ \/_/ \/____/ \/___/     \/_/\/_/
     SudoMesh Monitoring Server - Fall, 2014

  ____                _           __  __                _     
 / ___|   _   _    __| |   ___   |  \/  |   ___   ___  | |__  
 \___ \  | | | |  / _` |  / _ \  | |\/| |  / _ \ / __| | '_ \ 
  ___) | | |_| | | (_| | | (_) | | |  | | |  __/ \__ \ | | | |
 |____/   \__,_|  \__,_|  \___/  |_|  |_|  \___| |___/ |_| |_|
     SudoMesh Monitoring Server - Fall, 2014