Mesh/Network topology

wifi topology

We use 2.4 ghz 802.11g or 802.11n wifi gear with omni or semi-directional antennas to provide connectivity to devices such as laptops and smartphones at street level and within buildings. We are currently using mostly Ubiquiti Picostation 2 HP and Ubiquiti Bullet 2 HP routers for the outdoors. For the indoors we will likely use TP-Link TL-WR703N routers.

A high-speed wireless backbone for the mesh will be provided by 5 ghz 802.11n hardware, usually with point to point or point to multipoint connections mounted in high places such as on rooftops, flagpoles or antenna towers. We currently have a variety of Ubiquiti M5 routers such as airgrids, nanobridges, nanostations and a rocket.

All of the outdoor gear will be Power over Ethernet (PoE), requiring only a single cable for network and power connectivity.

mesh topology

All routers run the batman-adv mesh routing protocol. This is a layer 2 protocol (operating at the ethernet layer). The street-level 2.4 ghz routers should ideally be able to function in the event that e.g. an earthquake takes out all of the point to point and point to multipoint rooftop nodes (more alignment sensitive) and the mesh should remain functional, though it could become segmented.

The relays (see the internet connectivity section) also run batman-adv, so mesh traffic can flow from one part of the mesh, through the internet, through a relay, and into another part of the mesh if some of the mesh nodes are connected to the internet.

internet connectivity

There are four primary types of devices in the mesh:

• Clients: E.g. smart phones or laptops connected to the mesh.

These do not run the meshing protocol.

• Mesh nodes: Wifi routers running OpenWRT.

Some are rooftop backbone nodes and some are street-level or in-home nodes.

• Relays: Professionally hosted servers that relay mesh traffic over the internet.

These run the meshing protocol. Mesh nodes are connected with L2TP tunnels to them.

• Exit nodes: Co-located servers that appear as the source IP for packets from mesh to internet.

Both relays and exit nodes serve as a layer of protection between people sharing their internet connections with the mesh.

Some mesh routers will be hosted in homes that already have internet connections. If an internet connection is available, a mesh router will open an L2TP tunnel (using the tunneldigger software) to several relay nodes over the internet connection. A relay could be e.g. a VPS without a bandwidth cap. The relays all run batman-adv and function as part of the mesh through the L2TP tunnels to the mesh nodes. Each relay will have a connection to an exit nodes. The relays allow segments of the mesh that are not connected with wifi to be connected over the internet.

Each relay is connected to one exit node (tunnel type not yet decided). It does NAT (IP Masquerading) on traffic coming from the mesh and headed for the internet. All traffic coming from the mesh and going to the wider internet goes through an exit node. The source IP of data coming from the mesh thus appears as the IP of one of the exit nodes. This provides a layer of protection such that e.g. abuse complaints will be sent to the mesh organization instead of the individuals who donate some of their internet bandwidth to the mesh.

Until network has an AS, only one exit node should be made and multiple relay nodes should connect to that exit node (Tunneldigger software can be reused for that). Otherwise clients can have issues when routing protocol decides to move from one exit node to another. But it is true that batman-adv has some protection against that, so that once a client decides for a gateway, it should be more or less sticky, no?

It is important that nodes are connecting to relays and relays to exit nodes and that no IPs of those connecting to relays and exit nodes is stored.