Mesh/Distributed Services

The mesh network should provide communication services, so that it offers value to its users beyond mere connectivity to the Internet. What can be done to help it strengthen ties among Oakland residents, and to benefit the community? This wiki page is here to collect ideas about what services could/should be offered.

Services (such as file sharing, email, contacts, calendar, and phone) should be provided based on existing free/open source software. Orientation & setup could be built into the introductory process for new mesh users.

Some general aspects worth considering:

• Security
• Integration into a user account distributed database with master-master replication and low latency (ex. Cassandra).

Mesh Connected Hosts

Hosts wishing to offer services can simply connect to the public network port of the home node they've setup to connect to the SudoMesh network. Follow this services guide for a step-by-step tutorial of setting up a service on the network.

Externally Hosted

Hosts that are not directly connected to a SudoMesh node can make themselves available on the SudoMesh network by establishing a VPN connection into the network via the Exit node (exit.sudomesh.org). This requires use of the tunneldigger client. As of Ubuntu 14.4, the required kernel modules supporting L2TP are not included, so you must build a custom kernel that includes the necessary modules, or use the Debian distribution instead. Plans are in place to eventually use a more efficient tunneling method using Foo Over UDP, which will eliminate this L2TP complication.

Your host will need to use a static IP assigned from the mesh network. Support for obtaining only a static IP assignment will need to be added to the makenode utility to accomplish this.

Active Services

The service browser is unfinished, but as we're working on building out the network and setting up services on the network, we can document what we currently have running here for now:

• SudoPad, an instance of etherpad-lite: this collaborative, real-time notetaking application is now live at https://peoplesopen.net/pad
• PeoplesOpen Chat, an instance of Rocket.Chat, a Slack-like alternative chat service, is now live at https://peoplesopen.net/chat

Architecture: Centralized & decentralized services

A central philosophical and practical consideration is how services will be provided. Do we fundamentally want the mesh to be a centralized entity which offers certain services to all members? Or a platform on which individual members can offer services? Or a hybrid?

As it stands now (summer 2015), the mesh offers some opportunity for individual members to offer services. For instance, it's possible for anybody on the mesh to offer services that are accessible elsewhere on the mesh. Should there be (for instance) a central, community-curated file server? Or should it just be up to anybody who wants to, to offer up file shares?

Are there certain kinds of sharing we want to discourage or prohibit, and if so, what will be the social and technical means for enforcement? How will we respond if people start openly sharing copyrighted music and movies? Proprietary software? Child porn? It's worthwhile to think through these scenarios, as a successful network that lacks any kind of governance would become an attractive resource for those whose illegal activities cannot be pursued in the open on the Internet. (An example of this dynamic: State Supreme Court rules against Backpage.com in sex-trafficking suit, Seattle Times, Sept. 2015)

Storage / File Systems

There are several options for a distributed file system and each have there advantages and disadvantages. The distributed file system can be used to offer file storage or reduce latency across the network.

• Lustre - Used by sixty percent of the top 100 websites.
  • Performance: Amazingly fast! I can assert that Lustre can serve a lot of streams and that encoding speed is not affected by accessing files via Lustre.
  • POSIX compatibility: Very good!. No need to modify applications to use luster.
  • Replication, Load Balancing and Fail Over: Very bad!. For replication load balancing we and fail over we need to rely on other software such as virtual IPs and DRDB.
  • Installation: The worst!. Impossible to install by mere mortals. Requires a very specific combination of kernel, lustre patches and tweaks to get it working. And current luster patches usually work with old kernels that are incompatible with new hardware/software.

• MogileFS
  • Performance: Good for small files but not usable for medium to large files. This is mostly due to HTTP overhead since all files are send/receive via HTTP requests that encode all data in base64 adding a 33% overhead to each file.
  • POXIS compatibility: Non existent. All applications require to be modified to use MogileFS that renders it useless for streaming/encoding since most streaming servers and encoding tools do not understand the protocol.
  • Replication and failover out of the box and load balancing can be implemented in the application by accessing more than one tracker at a time.
  • Installation: Relatively easy and ready to use packages exist in most distributions. The only difficulty I found was setting the database master-slave to eliminate the single point of failure.

• GlusterFS
  • Performance: Very bad for streaming. I cannot reach more than a few Mbps in a 10Gbps network. Clients and Server CPU skyrockets on heavy writes. For encoding works because the CPU is saturated before the network and I/O.
  • POXIS: Almost compatible. The tools I use can access gluster mounts as normal folders in disk but in some edge cases things start causing problems. Check gluster mailing lists and you will see there are a lot of problems.
  • Replication, Failover and Load balancing: The best! if they actually worked. Gluster is very new and it has a lot of bugs and performance problems.
  • Installation is too easy. The management command line is amazing and setting replicated, striped and distributed volumes among several servers can not be any easier.

(source of above information)

• Hadoop (HDFS) - slow and has a lot of proprietary derivatives (MapR, GPFS, and other alternatives)
• XtreemFS - slow and we need more research on it.
• Cassandra File System (CFS) - proprietary software made by DataStax and included in the Enterprise Edition of their software.
• Network File Sytem (NFS) - comparable to Lustre, but the IO bandwidth doesn't scale.

Non-distributed file systems There's the option to use a modern file system with rsync. Comparison of ZFS and BTRFS A Quick Introduction to Modern Filesystems

Databases and Servers

These are decentralized databases and servers that could ultimately be optimized to store data nearest to where people access it. A significant factors with mesh networks, that as "size and complexity continues to grow, mesh networks that contain multiple hops become increasingly vulnerable to problems such as bandwidth degradation, radio interference and network latency." There are several other methods of improving network latency, but all means should be tested and implemented to improve services.

• cassandra - a decentralized database with asynchronous masterless replication allowing low latency operations for all clients. Cassandra has Hadoop integration, with MapReduce support.
• CouchDB - I can tell you that it quickly becomes complicated to write multi-user app with CouchDB. Decent access control means that you need a database per user and likely a few extra databases. You then have to set all of these databases up to synchronize between each other which is further complicated by the limitations on what information is available by the limitations on how synchronization rules and filters can be written. There seem to be no good tools for managing/visualizing these complex relationships so you end up having to roll your own.
• irmin - a distributed database with built-in snapshot, branch and revert mechanisms. (similar to git)
• levelDB - a fast and lightweight key/value database library.
• mongoDB - an open-source document database, a NoSQL database. Written in C++.
• node.js - an HTTP server written in javascript.
• PouchDB - a javascript based couchDB.
• scuttlebutt - a data synchronization protocol, or a peer-to-peer replicable data structure.
• tahoe-lafs - a decentralized database.

Communications

• Discourse - Discussion forum in the form of infinite scrolling feeds. Built on Ruby on Rails and ember.js
• radicale - a CalDAV (calendar) and CardDAV (contacts) server.
• calypso - Radicale fork with a git back-end (by Keith Packard of X11 fame)
• Village telco - an initiative to build low-cost community telephone network hardware and software that can be set up in minutes anywhere in the world. No mobile phone towers or land lines are required, it uses the internet. It has a Simple Unified Dashboard (SPUD) for mesh networks.
• Ripple (XRP) - a math-based crypto-currency designed to work seamlessly with the Internet. Powered by a global network of computers; a fast, direct, and secure way to send payments on the web. It can be used to prevent malious attacks on servers.
• OpenBTS - open source cell phone tower software.
• OpenMAMA - a high performance middleware agnostic messaging API that interfaces with a variety of message oriented middleware systems
• an encrypted asynchronous messaging system to replace email
• a module to have collaborative documents, voting, and groups (ie. groupware)
  • Read information about the peer-to-peer voting scheme.
• a module for peer to peer file sharing.
• a module for object-orientated tagging, shareable feeds.
• a module for feeds and posts, such as Trsst.
• Frontline SMS - a free, open-source desktop application that allows you to send texts.

Mapping

• NodeWatcher - an open source network planning, deployment, monitoring and maintanance platform. Provides stats on the uptime and status of every node, mapping them, contact information, access control levels
• NodeShot - an easy to use wireless community map server.
• OpenWiFiMap - a database and map for free network WiFi routers (freifunk and others, too!).
• LibreMap - a scalable, global and decentralized router database and map visualization for community networks, such as guifi, ninux, etc.
• Tidepools - collaborative local mapping software
• Edible Cities - a food mapping project
• Bulletin Board / Local Classified Ads
  • Community Asset Mapping (ex. Mycelia)

Education

• LibraryBox - "LibraryBox v2.0 is a combination of a router (a variety of hardware will work), USB drive, and software that, when combined, give you a small, low powered webserver. The webserver acts like a captive portal, and delivers files that are stored on the USB drive. To use LibraryBox, you simply connect to the wifi SSID "LibraryBox" and launch a browser. Attempting to visit any webpage will push you to the LibraryBox homepage on the device, which has information about the project, and links on the menu for downloads. You can browse the contents of the Shared folder, and download any files you'd like. The device you are using does have to be able to understand the download (for instance, downloading an EPUB ebook to a Kindle won't really work)."

Protocols

• Tent - a method to store all your data in one place that you control. Instead of your digital life being spread across many services.
• Docker - an open-source project to easily create lightweight, portable, self-sufficient containers from any application.
• MaidSafe - an encrypted, fully distributed data management service. This network manages static and dynamic data as well as communications. (charges money)
• Telehash - "Telehash is a 100% open, secure mesh networking technology with these design principles: full end-to-end encryption, all the time; strict privacy: no content, identity, or metadata is ever revealed to third parties; maximum app/device compatibility: suitable for embedded, mobile, and web usage; making privacy the easy choice for developers; flexible transport protocols, for compatibility with existing layers; native implementations for the widest possible variety of languages/platforms. The telehash team includes collaborators from around the world, many of whom were the principal architects of XMPP. It is intended as a next-generation protocol advancing the original goals of Jabber."

Social Platforms

• Diaspora - a decentralized social network
• Identi.ca - a high-performance Open Source social engine.
• Pump - a stream server that does most of what people really want from a social network.
• Twister - a decentralized, peer to peer microblogging platform

Utilities and Misc

• Avahi - a mDNS responder that allows you to broadcast services
• Local Wiki - create a local wikipedia (eg; http://oaklandwiki.org)
• a module for sensor data (eg; Temperature, Seismic activity, air pollution)
• PiplMesh - an open source social networking and info portal for wireless networks used as a welcome page when users connect to the network.
• Neocities (git) - create free user sites
• Crabgrass - a project out of riseup labs, a ruby rails application for community organizing. It's not the easiest to run your own instance, but very easy to sign up to their server (we.riseup.net)