Difference between revisions of "Mesh/Technical Overview"

From Sudo Room
Jump to navigation Jump to search
(created the page)
 
 
(18 intermediate revisions by 5 users not shown)
Line 1: Line 1:
== Technical overview ==
[[Category:Stale]]
'''This page has been marked as [[:Category:Stale|stale]], as it is outdated to the point of providing significant misinformation. Please update it before linking other pages here.'''


The mesh is made up mostly of wifi routers using Atheros chipsets and running [https://github.com/sudomesh/sudowrt-firmware our own firmware] based on [http://openwrt.org/ OpenWRT], [http://www.open-mesh.org/projects/batman-adv/wiki/ B.A.T.M.A.N. Advanced] and wlan slovenja's [https://github.com/sudomesh/tunneldigger tunneldigger]. We're using 2.4 GHz routers for indoor and street-level coverage and 5 GHz 802.11n routers for high-bandwidth and long distance roof to roof links. Most of our outdoor equipment is from Ubiquiti. We refer to the wifi routers as mesh nodes, or simply nodes.
[[File:Sudomesh home and extender node illustration extra-simplified.png|thumb|Illustration showing how a sudomesh home node can connect to a window-mounted extender node. The device with two antennas is the home node and the device on the outside of the window is an extender node.]]


Node-owners can choose to connect the nodes to their existing LAN using ethernet. If they have Internet access, they can share a portion of it with the mesh. The amount of bandwidth shared is limited with 'tc'. It is chosen at node-configuration time and can be changed using the simple built-in web admin interface.
[[File:Sudomesh home and extender node illustration.png|thumb|Illustration showing a more complicated setup. Here the node operator has connected their personal computer, a small Raspberry Pi computer and two extender nodes: One on the outside of the window and one on the roof (not shown but a can be seen going up and out of the frame). This is definitely an advanced node operator.]]
 
[[File:Sudomesh home node ethernet ports.png|thumb|A close-up of the wired network ports on the home node and their use.]]
 
== Home and extender nodes ==
 
''Note: You can also read the [[home node|non-technical explanation of home nodes and extender nodes]].''
 
Our mesh has two types of nodes: Home nodes and extender nodes.
 
The home node is a 5 ethernet port wifi router that runs the full sudowrt firmware. A network of home nodes can make a mesh network if they are within the same proximity e.g 200 meters and line-of-sight of each other. Beyond the home nodes broadcast area, an extender node needs to be connected to a home node to communicate to distant neighbours. An extender node runs a minimal firmware and does nothing on its own, but if its connected to a home node it will serve as an extra radio and extend the reach of the network.
 
The node operator can choose to put a home node in their living room and if they decide they want a high speed directional link to a neighbour then they can put e.g. a Nanobridge M5 running the extender node firmware on their roof and connect it via ethernet to the home node. Or they could put a Nanostation M2 pointed out of their window and extend the signal further out into the street or to nearby neighbours. Each extender node extends all three networks (explained in the next section).
 
For home nodes we are currently supporting these off-the-shelf router [https://sudoroom.org/wiki/Mesh/WalkThrough#Download.2FBuild models]. We are not planning to support anything else for the time being, though if you have a good reason please let us know. For extender nodes we are currently supporting the Ubiquiti M series [https://sudoroom.org/wiki/Mesh/Flashing_extender_nodes#Flash_the_Extender_Node routers].
 
The mesh mainly uses wifi routers with Atheros chipsets and running [https://github.com/sudomesh/sudowrt-firmware our own firmware] based on [http://openwrt.org/ OpenWRT], [http://www.pps.univ-paris-diderot.fr/~jch/software/babel/ Babel] and wlan slovenja's [https://github.com/sudomesh/tunneldigger tunneldigger].
 
Node-owners can choose to connect the nodes to their existing internet connection using ethernet. If they have Internet access, they can share a portion of it with the mesh. The amount of bandwidth shared is limited with the tool 'tc'. It is chosen at node-configuration time and can be changed using the simple built-in web admin interface.
 
== The three wifi networks and IP assignment ==
 
The nodes each run three wifi networks (three SSIDs on the same physical wifi interfaces):
 
* The Open network with SSID peoplesopen.net is an open access point. Most people will use the network by connecting to this.
* The Mesh network with SSID pplsopen.net-node2node is an ad-hoc network that the nodes use to mesh with each other using the Babel mesh protocol.
* A private wifi network SSID that is named by the node owner (or a name is generated by makenode) and uses WPA2-PSK.
 
All three of these networks are available on both 2.4 and 5 GHz on the home node. They are also all three available on any extender nodes plugged in to the home node. The three different networks are extended over a single ethernet cable using VLANs.
 
If a node-owner is sharing internet, then the node will create a layer 2 (L2TP) tunnel to a VPN server on the Internet using tunneldigger. Babel (the mesh routing protocol) will connect over this tunnel to other nodes on the mesh, so the mesh can route traffic over the internet if no wifi path to another node is available (e.g. other nodes are physically too far away). When people connect to the peoplesopen.net access point and try to access the Internet, the traffic will flow through the VPuN, and the source IP of requests will appear to be the VPuN with the sudo mesh organization listed as the abuse contact.
 
The nodes run DHCP servers and each have a /24 IPv4 subnet in the 100.64.0.0/10 range that is statically assigned by coordination between mesh groups and individuals hosting and administrating their own nodes on People's Open Network (currently only the sudo mesh organization).
 
The private network does not limit bandwidth and provides access to both direct access to the Internet (if the node owner has hooked the node up to the Internet) and access to the mesh. Each node's private network runs on 172.30.0.0/16 and uses NAT between the private network and the mesh. It does not accept any new incoming connections from the mesh onto the 172.30.0.0/16 subnet.
 
== Accessing Home Node via SSH ==
 
Your home node is accessible via the [https://wiki.openwrt.org/doc/uci/dropbear Dropbear] Secure Shell (SSH) server on port 22.
 
The 4 Ethernet ports that are provided with your router should be configured as follows:
 
# Private network with DHCP of 172.30.0.x network addresses
# Public network with DHCP of 100.64.x.x network addresses (SudoMesh network)
# NotDHCP for Extender 1
# NotDHCP for Extender 2
 
This may not apply to your router. For instance, the TP-Link N750 uses the opposite port assignment, with port 3 for private network, and port 4 for public SudoMesh/PeoplesOpen.net network.
 
=== Network Settings ===
 
=== Default Build Configuration ===
 
The IP address of your home node is <tt>172.22.0.1</tt> prior to configuration via the [https://github.com/sudomesh/makenode makenode] utility. You can SSH into the node as <tt>root</tt> using the password 'meshtheplanet' after setting up your computer to use an IP on the 172.22.0.0/16 network.
 
=== Post Makenode Configuration ===
 
The private network configuration uses a 172.30.0.0/16 network with DHCP available for connected clients. You can configure your workstation to use the following manual network configuration so that you can SSH into your home node.
 
* IP Address: 172.30.0.9 (or anything other than 172.30.0.1)
* Netmask: 255.255.255.0
* Gateway: 172.30.0.1
 
See Network Configuration Guides: [https://sudoroom.org/wiki/Mesh/Network%20Configuration%20for%20Linux Linux] [https://sudoroom.org/wiki/Mesh/Network%20Configuration%20for%20MacOS%20X Mac]
 
The IP of your home node on the private network is <tt>172.30.0.1</tt>, with the root password you specified when running <tt>makenode</tt> to configure it.
 
  ssh root@172.30.0.1
  The authenticity of host '172.30.0.1 (172.30.0.1)' can't be established.
  RSA key fingerprint is b8:9d:4a:2f:1b:f5:e1:ae:b8:19:5b:70:92:8b:7f:34.
  Are you sure you want to continue connecting (yes/no)?
 
After accepting the key by entering 'yes' and pressing ENTER, it will ask you for the root password.
 
=== SSH Keys ===
 
If you'd like to add your ssh key to the router (instead of using a root password), add it to the <tt>configs/authorized_keys</tt> file. You'll see that there are 3 other keys there for our developers. You can remove them if you'd like, but they're currently the only way we can provide remote support. During the alpha test phase we ask that you consider whether you are able to do diagnostics/debugging yourself before you remove them.
 
== Node flashing and configuration ==
 
One of our medium-term goals is to be able to sell nodes on our website and minimize the amount of work required to re-flash/configure the nodes and provide documentation for the user. To facilitate this, our current process for new nodes is:
 
* A new node is flashed manually with the [https://github.com/sudomesh/sudowrt-firmware sudowrt] firmware.
* The node is then configured by a sudo mesh volunteer using our [https://github.com/sudomesh/makenode makenode] software.
* Makenode generates SSH keys, a SSH root password, a web admin password and a private wifi password. Then it configures the node, saves the info in the [https://github.com/sudomesh/node-database node database] and shuts down the node.
* Makenode then automatically [https://github.com/sudomesh/ql570 prints a sticker] containing some basic info including wifi and web admin passwords.
* The sudo mesh volunteer attaches the sticker to the nodes power supply and puts the node back in the box with a set of instructions for how to install and use the node.
* The node is shipped to the new node owner!
 
== Node management ==
 
All nodes set up by sudo mesh automatically allow root access using an SSH key held by a few trusted sudo mesh organizers. This is to allow us to update the firmware and troubleshoot network issues. We inform node-owners of this fact and tell them how to prevent sudo mesh from accessing their nodes, but also indicate that they should be ready to manage their own node if they choose to do this.
 
== Node Monitoring ==
 
Currently, the peoples open network can be monitored in real time with an [https://peoplesopen.herokuapp.com heroku app].
Our exit nodes communicate to this URL using a monitor.sh script. This relays information about the number of active routes and number of active gateways. If the exit node doesn't check-in for longer than 2 minutes, it is assumed to be down.


==Detailed Documentation==
==Detailed Documentation==
*[[Mesh/Tracking]] - Problems and solutions related to tracking, logging and anonymity
*[[Mesh/Tracking]] - Problems and solutions related to tracking, logging and anonymity
*[[Mesh/WalkThrough]] - Easy how-to for setting up OpenWRT & BATMAN-adv on Ubiquiti routers
*[[Mesh/WalkThrough]] - Easy how-to for setting up OpenWRT
**[[Mesh/Flashing Extender Antennas]] - How-to for Ubiquiti routers
**[[Mesh/OpenWRT]] - Experiences with the OpenWRT router firmware
**[[Mesh/OpenWRT]] - Experiences with the OpenWRT router firmware
**[[Mesh/BATMAN-adv]] - Experiences with the batman-adv mesh protocol
*[[Mesh/Firmware]] - Everything related to our firmware
*[[Mesh/Firmware]] - Everything related to our firmware
**[[Mesh/Firmware/Overview]] - High level view
**[[Mesh/Firmware/Flash]] - How to install the sudowrt firmware based on openwrt with supported hardware
**[[Mesh/Firmware/Flash]] - How to install the sudowrt firmware based on openwrt with supported hardware
**[[Mesh/Firmware/Splash page]] - Splash page / Captive Portal
**[[Mesh/Firmware/Splash page]] - Splash page / Captive Portal
Line 16: Line 114:
**[[Mesh/Firmware/Zeroconf]] - Issues related to mDNS and DNS-SD
**[[Mesh/Firmware/Zeroconf]] - Issues related to mDNS and DNS-SD
**[[Mesh/Firmware/MTU_issues]] - MTU issues
**[[Mesh/Firmware/MTU_issues]] - MTU issues
**[[Mesh/Firmware/Generating]] - How do we generate firmwares on demand
**[[Mesh/Firmware/Web Admin Development]] - Web admin development
**[[Mesh/Firmware/Web Admin Development]] - Web admin development
*[[Mesh/Network topology]] - High-level explanation of the mesh network structure
*[[Mesh/Network topology]] - High-level explanation of the mesh network structure
Line 33: Line 130:
* [[Mesh/Install]] - physical installation notes/stories
* [[Mesh/Install]] - physical installation notes/stories
* [[Mesh/Diagrams]] - technical diagrams to support new outreach materials
* [[Mesh/Diagrams]] - technical diagrams to support new outreach materials
== Wifi networks and IP assignment ==
The nodes each run three wifi networks (three SSIDs on the same physical wifi interface):
* peoplesopen.net is an open access point. Most people will use the network by connecting to this.
* pplsopen.net-node2node is an ad-hoc network that the nodes use to mesh with each other
* A private wifi network is named by the node owner (or a name is generated) and uses WPA2-PSK.
If a node-owner is sharing internet, then the node will create a layer 2 (L2TP) tunnel to a VPN server on the Internet using tunneldigger. batman-adv will connect over this tunnel to other nodes on the mesh, so the mesh can route traffic over the internet if no wifi path to another node is available (e.g. other nodes are physically too far away). When people connect to the peoplesopen.net access point and try to access the Internet, the traffic will flow through the VPN, and the source IP of requests will appear to be the VPN with the sudo mesh organization listed as the abuse contact.
The nodes run DHCP servers and each have a /24 IPv4 subnet in the 10.0.0.0/8 range that is statically assigned by coordination between mesh groups and individuals hosting and administrating their own nodes on People's Open Network (currently only the sudo mesh organization). If a user connects to the peoplesopen.net access point on a node that isn't sharing internet, then batman-adv intercepts the DHCP request and forwards the request to another node on the network that has Internet connectivity (see the gw_mode option for batman-adv).
The private network does not limit bandwidth and provides access to both direct access to the Internet (if the node owner has hooked the node up to the Internet) and access to the mesh. Each node's private network runs on 172.30.0.0/16 and uses NAT between the private network and the mesh. It does not accept any new incoming connections from the mesh onto the 172.30.0.0/16 subnet.
== Node flashing and configuration ==
One of our medium-term goals is to be able to sell nodes on our website and minimize the amount of work required to re-flash/configure the nodes and provide documentation for the user. To facilitate this, our current process for new nodes is:
* A new node is flashed either automatically (using e.g. [https://github.com/sudomesh/ubiquiti-flasher ubiquiti-flasher] or [https://github.com/sudomesh/merakiflasher merakiflasher]) or manually with the [https://github.com/sudomesh/sudowrt-firmware sudowrt] firmware.
* The node is plugged into a server running our [https://github.com/sudomesh/node-configurator node-configurator] software.
* A sudo mesh volunteer pulls up https://nodeconf.local and uses a web interface to fill out contact info for the node owner, initial bandwidth sharing limits and private wifi SSID.
* The node-configurator generates SSH keys, SSH root password, web admin password and private wifi password, then it configures the node, saves the info in the [https://github.com/sudomesh/node-database node database] and shuts down the node.
* The node-configurator automatically [https://github.com/sudomesh/ql570 prints a sticker] containing some basic info including wifi and web admin passwords.
* The sudo mesh volunteer attaches the sticker to the nodes power supply and puts the node back in the box with a set of instructions for how to install and use the node.
* The node is shipped to the new node owner!
The node-configurator has both a [https://github.com/sudomesh/node-configurator server] and a [https://github.com/sudomesh/node-configurator-client client] component. The newly flashed sudowrt nodes automatically run the node-configurator client when they boot, and the client uses DNS-SD and mDNS to find node-configurator servers on the local network. The node then connects to the server using SSL and the server is ready to configure the node. The node-configurator server talks to Avahi using DBUS to announce itself using DNS-SD. The server is written in Python using Twisted and the client is written in Lua using luasec, and uses the [https://github.com/sudomesh/mdnssd-min mdnssd-min] utility to provide DNS-SD and mDNS.
The node-configurator includes a webserver and management web app. The web app talks to the server and connected nodes using websockets.
== Node management ==
All nodes set up by sudo mesh automatically allow root access using an SSH key held by a few trusted sudo mesh organizers. This is to allow us to update the firmware and troubleshoot network issues. We inform node-owners of this fact and tell them how to prevent sudo mesh from accessing their nodes, but also indicate that they should be ready to manage their own node if they choose to do this.
We don't yet have a solution for node monitoring but we're expecting to use the new version of wlan slovenja's nodewatcher software.
We don't yet have an automatic update solution in place, but it will work similarly to the node-configurator:
* Any number of node-updater servers announce themselves on the mesh and whether or not an update is available.
* The nodes run a future version of mdnssd-min as a daemon that keeps a currently list of node-updaters.
* Once every N hours +/- a random factor, if any node-updaters have updates available, all nodes connect to a randomly chosen node-updater and request an update.
* The node-updaters send the nodes an ipk file with the update and the nodes check the signature and install it if it's signed by a trusted authority.

Latest revision as of 20:30, 16 June 2018

This page has been marked as stale, as it is outdated to the point of providing significant misinformation. Please update it before linking other pages here.

Illustration showing how a sudomesh home node can connect to a window-mounted extender node. The device with two antennas is the home node and the device on the outside of the window is an extender node.
Illustration showing a more complicated setup. Here the node operator has connected their personal computer, a small Raspberry Pi computer and two extender nodes: One on the outside of the window and one on the roof (not shown but a can be seen going up and out of the frame). This is definitely an advanced node operator.
A close-up of the wired network ports on the home node and their use.

Home and extender nodes

Note: You can also read the non-technical explanation of home nodes and extender nodes.

Our mesh has two types of nodes: Home nodes and extender nodes.

The home node is a 5 ethernet port wifi router that runs the full sudowrt firmware. A network of home nodes can make a mesh network if they are within the same proximity e.g 200 meters and line-of-sight of each other. Beyond the home nodes broadcast area, an extender node needs to be connected to a home node to communicate to distant neighbours. An extender node runs a minimal firmware and does nothing on its own, but if its connected to a home node it will serve as an extra radio and extend the reach of the network.

The node operator can choose to put a home node in their living room and if they decide they want a high speed directional link to a neighbour then they can put e.g. a Nanobridge M5 running the extender node firmware on their roof and connect it via ethernet to the home node. Or they could put a Nanostation M2 pointed out of their window and extend the signal further out into the street or to nearby neighbours. Each extender node extends all three networks (explained in the next section).

For home nodes we are currently supporting these off-the-shelf router models. We are not planning to support anything else for the time being, though if you have a good reason please let us know. For extender nodes we are currently supporting the Ubiquiti M series routers.

The mesh mainly uses wifi routers with Atheros chipsets and running our own firmware based on OpenWRT, Babel and wlan slovenja's tunneldigger.

Node-owners can choose to connect the nodes to their existing internet connection using ethernet. If they have Internet access, they can share a portion of it with the mesh. The amount of bandwidth shared is limited with the tool 'tc'. It is chosen at node-configuration time and can be changed using the simple built-in web admin interface.

The three wifi networks and IP assignment

The nodes each run three wifi networks (three SSIDs on the same physical wifi interfaces):

  • The Open network with SSID peoplesopen.net is an open access point. Most people will use the network by connecting to this.
  • The Mesh network with SSID pplsopen.net-node2node is an ad-hoc network that the nodes use to mesh with each other using the Babel mesh protocol.
  • A private wifi network SSID that is named by the node owner (or a name is generated by makenode) and uses WPA2-PSK.

All three of these networks are available on both 2.4 and 5 GHz on the home node. They are also all three available on any extender nodes plugged in to the home node. The three different networks are extended over a single ethernet cable using VLANs.

If a node-owner is sharing internet, then the node will create a layer 2 (L2TP) tunnel to a VPN server on the Internet using tunneldigger. Babel (the mesh routing protocol) will connect over this tunnel to other nodes on the mesh, so the mesh can route traffic over the internet if no wifi path to another node is available (e.g. other nodes are physically too far away). When people connect to the peoplesopen.net access point and try to access the Internet, the traffic will flow through the VPuN, and the source IP of requests will appear to be the VPuN with the sudo mesh organization listed as the abuse contact.

The nodes run DHCP servers and each have a /24 IPv4 subnet in the 100.64.0.0/10 range that is statically assigned by coordination between mesh groups and individuals hosting and administrating their own nodes on People's Open Network (currently only the sudo mesh organization).

The private network does not limit bandwidth and provides access to both direct access to the Internet (if the node owner has hooked the node up to the Internet) and access to the mesh. Each node's private network runs on 172.30.0.0/16 and uses NAT between the private network and the mesh. It does not accept any new incoming connections from the mesh onto the 172.30.0.0/16 subnet.

Accessing Home Node via SSH

Your home node is accessible via the Dropbear Secure Shell (SSH) server on port 22.

The 4 Ethernet ports that are provided with your router should be configured as follows:

  1. Private network with DHCP of 172.30.0.x network addresses
  2. Public network with DHCP of 100.64.x.x network addresses (SudoMesh network)
  3. NotDHCP for Extender 1
  4. NotDHCP for Extender 2

This may not apply to your router. For instance, the TP-Link N750 uses the opposite port assignment, with port 3 for private network, and port 4 for public SudoMesh/PeoplesOpen.net network.

Network Settings

Default Build Configuration

The IP address of your home node is 172.22.0.1 prior to configuration via the makenode utility. You can SSH into the node as root using the password 'meshtheplanet' after setting up your computer to use an IP on the 172.22.0.0/16 network.

Post Makenode Configuration

The private network configuration uses a 172.30.0.0/16 network with DHCP available for connected clients. You can configure your workstation to use the following manual network configuration so that you can SSH into your home node.

  • IP Address: 172.30.0.9 (or anything other than 172.30.0.1)
  • Netmask: 255.255.255.0
  • Gateway: 172.30.0.1

See Network Configuration Guides: Linux Mac

The IP of your home node on the private network is 172.30.0.1, with the root password you specified when running makenode to configure it.

 ssh root@172.30.0.1
 The authenticity of host '172.30.0.1 (172.30.0.1)' can't be established.
 RSA key fingerprint is b8:9d:4a:2f:1b:f5:e1:ae:b8:19:5b:70:92:8b:7f:34.
 Are you sure you want to continue connecting (yes/no)? 

After accepting the key by entering 'yes' and pressing ENTER, it will ask you for the root password.

SSH Keys

If you'd like to add your ssh key to the router (instead of using a root password), add it to the configs/authorized_keys file. You'll see that there are 3 other keys there for our developers. You can remove them if you'd like, but they're currently the only way we can provide remote support. During the alpha test phase we ask that you consider whether you are able to do diagnostics/debugging yourself before you remove them.

Node flashing and configuration

One of our medium-term goals is to be able to sell nodes on our website and minimize the amount of work required to re-flash/configure the nodes and provide documentation for the user. To facilitate this, our current process for new nodes is:

  • A new node is flashed manually with the sudowrt firmware.
  • The node is then configured by a sudo mesh volunteer using our makenode software.
  • Makenode generates SSH keys, a SSH root password, a web admin password and a private wifi password. Then it configures the node, saves the info in the node database and shuts down the node.
  • Makenode then automatically prints a sticker containing some basic info including wifi and web admin passwords.
  • The sudo mesh volunteer attaches the sticker to the nodes power supply and puts the node back in the box with a set of instructions for how to install and use the node.
  • The node is shipped to the new node owner!

Node management

All nodes set up by sudo mesh automatically allow root access using an SSH key held by a few trusted sudo mesh organizers. This is to allow us to update the firmware and troubleshoot network issues. We inform node-owners of this fact and tell them how to prevent sudo mesh from accessing their nodes, but also indicate that they should be ready to manage their own node if they choose to do this.

Node Monitoring

Currently, the peoples open network can be monitored in real time with an heroku app. Our exit nodes communicate to this URL using a monitor.sh script. This relays information about the number of active routes and number of active gateways. If the exit node doesn't check-in for longer than 2 minutes, it is assumed to be down.

Detailed Documentation