Mesh/Technical Overview

From Sudo Room
Revision as of 19:41, 17 July 2014 by Jwentwistle (talk | contribs) (created the page)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Technical overview

The mesh is made up mostly of wifi routers using Atheros chipsets and running our own firmware based on OpenWRT, B.A.T.M.A.N. Advanced and wlan slovenja's tunneldigger. We're using 2.4 GHz routers for indoor and street-level coverage and 5 GHz 802.11n routers for high-bandwidth and long distance roof to roof links. Most of our outdoor equipment is from Ubiquiti. We refer to the wifi routers as mesh nodes, or simply nodes.

Node-owners can choose to connect the nodes to their existing LAN using ethernet. If they have Internet access, they can share a portion of it with the mesh. The amount of bandwidth shared is limited with 'tc'. It is chosen at node-configuration time and can be changed using the simple built-in web admin interface.

Detailed Documentation


Wifi networks and IP assignment

The nodes each run three wifi networks (three SSIDs on the same physical wifi interface):

  • peoplesopen.net is an open access point. Most people will use the network by connecting to this.
  • pplsopen.net-node2node is an ad-hoc network that the nodes use to mesh with each other
  • A private wifi network is named by the node owner (or a name is generated) and uses WPA2-PSK.

If a node-owner is sharing internet, then the node will create a layer 2 (L2TP) tunnel to a VPN server on the Internet using tunneldigger. batman-adv will connect over this tunnel to other nodes on the mesh, so the mesh can route traffic over the internet if no wifi path to another node is available (e.g. other nodes are physically too far away). When people connect to the peoplesopen.net access point and try to access the Internet, the traffic will flow through the VPN, and the source IP of requests will appear to be the VPN with the sudo mesh organization listed as the abuse contact.

The nodes run DHCP servers and each have a /24 IPv4 subnet in the 10.0.0.0/8 range that is statically assigned by coordination between mesh groups and individuals hosting and administrating their own nodes on People's Open Network (currently only the sudo mesh organization). If a user connects to the peoplesopen.net access point on a node that isn't sharing internet, then batman-adv intercepts the DHCP request and forwards the request to another node on the network that has Internet connectivity (see the gw_mode option for batman-adv).

The private network does not limit bandwidth and provides access to both direct access to the Internet (if the node owner has hooked the node up to the Internet) and access to the mesh. Each node's private network runs on 172.30.0.0/16 and uses NAT between the private network and the mesh. It does not accept any new incoming connections from the mesh onto the 172.30.0.0/16 subnet.

Node flashing and configuration

One of our medium-term goals is to be able to sell nodes on our website and minimize the amount of work required to re-flash/configure the nodes and provide documentation for the user. To facilitate this, our current process for new nodes is:

  • A new node is flashed either automatically (using e.g. ubiquiti-flasher or merakiflasher) or manually with the sudowrt firmware.
  • The node is plugged into a server running our node-configurator software.
  • A sudo mesh volunteer pulls up https://nodeconf.local and uses a web interface to fill out contact info for the node owner, initial bandwidth sharing limits and private wifi SSID.
  • The node-configurator generates SSH keys, SSH root password, web admin password and private wifi password, then it configures the node, saves the info in the node database and shuts down the node.
  • The node-configurator automatically prints a sticker containing some basic info including wifi and web admin passwords.
  • The sudo mesh volunteer attaches the sticker to the nodes power supply and puts the node back in the box with a set of instructions for how to install and use the node.
  • The node is shipped to the new node owner!

The node-configurator has both a server and a client component. The newly flashed sudowrt nodes automatically run the node-configurator client when they boot, and the client uses DNS-SD and mDNS to find node-configurator servers on the local network. The node then connects to the server using SSL and the server is ready to configure the node. The node-configurator server talks to Avahi using DBUS to announce itself using DNS-SD. The server is written in Python using Twisted and the client is written in Lua using luasec, and uses the mdnssd-min utility to provide DNS-SD and mDNS.

The node-configurator includes a webserver and management web app. The web app talks to the server and connected nodes using websockets.

Node management

All nodes set up by sudo mesh automatically allow root access using an SSH key held by a few trusted sudo mesh organizers. This is to allow us to update the firmware and troubleshoot network issues. We inform node-owners of this fact and tell them how to prevent sudo mesh from accessing their nodes, but also indicate that they should be ready to manage their own node if they choose to do this.

We don't yet have a solution for node monitoring but we're expecting to use the new version of wlan slovenja's nodewatcher software.

We don't yet have an automatic update solution in place, but it will work similarly to the node-configurator:

  • Any number of node-updater servers announce themselves on the mesh and whether or not an update is available.
  • The nodes run a future version of mdnssd-min as a daemon that keeps a currently list of node-updaters.
  • Once every N hours +/- a random factor, if any node-updaters have updates available, all nodes connect to a randomly chosen node-updater and request an update.
  • The node-updaters send the nodes an ipk file with the update and the nodes check the signature and install it if it's signed by a trusted authority.
Retrieved from "https://sudoroom.org/mediawiki/index.php?title=Mesh/Technical_Overview&oldid=8412"