Bit451

decentralized / distributed anonymous p2p media network

Project maintained by Bit451 Hosted on GitHub Pages — Theme by mattgraham

Table of Contents

Bit451

Bit451 is an open source decentralized / distributed anonymous p2p media network. It has been characterized as “YouTube meets BitTorrent meets Bitcoin”.

logo

Synopsis

Bit451 is comprised of multiple open source projects working together as one: a video browsing interface, a mesh of file transfer protocols/networks with streaming content capabilities, and a specialized distributed database, are its primary components. You, as a Bit451 user, will have the option to simply access the network directly from your browser, no additional setup required - or, to run a light, streamlined service to help host and contribute to the Bit451 network. Website creators can also easily tap into and serve Bit451 to their visitors using the same light setup configured to their liking.

As for the Bit451 network itself, it will have all the features you have come to know and love: user accounts/profiles, favorites, ratings, subscriptions, playlists, search, recommendations, and much more. The network is community-based, so the users and content you interact with is determined by your preferences and the users you trust. If you don’t specifically know or “trust” any users starting out, don’t worry, you’re not alone and the default settings will work just great. Bit451 will also introduce an automatable cryptocurrency payment plugin system, so that content consumers and producers can buy, sell, and host content for their favorite coins.

For broader reuse, Bit451 will create:

At this point you may be wondering, that all sounds great but what’s so wrong with YouTube?. (Yes, Bit451 can even read your mind! Pretty amazing so far, right?..)

Feedback on any and all aspects of Bit451 are strongly desired and encouraged! Please join in to Discuss, Donate, and Contribute!

Inspiration

The name Bit451 was inspired by Fahrenheit 451, a dystopian novel by Ray Bradbury published in 1953 which presents a future society where books are outlawed and “firemen” burn any that are found. A small community, intent on preserving the books, memorizes them to be recreated for future generations. The proper pronunciation of the number is “four five one” according to Bradbury.

YouTube

The primary catalyst behind this idea is the increasing prevalence of issues and concerns with YouTube:

To be sure, YouTube has on the whole provided an important service to the internet community, and has the right to do as it wishes within its domain. But hasn’t the time has come to unleash the freedom and creativity of all of us into the content sphere?!

Many of these complaints could also extend to other current media powerhouses (Vimeo, Soundcloud, etc), but YouTube garners the most enthusiasm and is the prime focus.

BitTorrent

BitTorrent is a technology before its time, the full potential of which has likely not yet been fully realized. Streaming BitTorrent content is the next step in its evolution, and along with it, the Bit451 network.

Bitcoin

Bitcoin has sparked the internet’s imagination. It has enabled not only new forms of commerce, but of communication, transmission, etc, in ways never dreamed of before, and Bit451 will take that to the next level.

Similar Technologies and Projects

WeTube

WeTube is a “Decentralized p2p, free, uncensored media provider”. While this project is a step in the right direction, it also raises multiple concerns:

WeTube may find its strengths in some of these areas, and hopefully it will; however, the overall vision for the project simply differs significantly from Bit451.

Development and Technical Details

Methodology

The main Components can each be separate projects to be forked and merged. Bounties will be divvied out proportionally based on completion of each project; for example, if Murray completes 50%, then Lew forks Murray’s project and completes the other 50%, Murray and Lew will each receive 50% of the bounty. Unit tests are strongly encouraged.

When submitting a code change (including to readme or other documentation!), send a pull request, then post it in the primary Bit451 discussion area for community review; technical discussion of the pull request is encouraged to take place in the pull request discussion area. After it passes and is accepted, a waiting period of approximately one week will occur before bounties are paid out so that other developers can review the code. If it is found that code has been copied from another developer without credit, bounties will not be paid to the plagiarizer, but rather to the original developer.

Draw.io is the recommended online service for UML. Suitable UML diagram XMLs and their PNGs should be added to /uml.

Bit451.org will be used for the GitHub Pages (github.io) Project Page to serve this readme from the base Bit451 project, via the automatic generator, using Jekyll. Bit451.com will be used for the Project Page to serve (via submodule) the live Bit451 Portal, since it is static/browser-based. Bit451.net use is TBD. These are initial planned uses.

Components

Phase 1 Development

The goal of this phase is to create a standardized, usable, aesthetic, browser-based frontend which can access the content network.

Portal

The browser/viewport/frontend. This is the interface used to access and interact with content from the network. This should be usable on a local machine (via a local engine) as well as hosted from a server, where it can be additionally configured/customized.

GreenAddress

GreenAddress is a sleek web wallet with the advantages of a local software wallet like Electrum: “The safer Bitcoin wallet that puts you in control”.

Bit451’s fundamental structure is based around user accounts, and accounts are built on a deterministic wallet-type scheme, as GreenAddress is (see Account Creation / Recovery for more on this). GreenAddress also has a very aesthetic interface and is JavaScript-based, making it the optimal starting point for Bit451.

Portal Action Item

Fork GreenAddress to Bit451 Portal. Transform it to a media frontend, and integrate programmatic interfaces for File Transfer networks. Bit451.com will serve the Portal directly from its GitHub dev repo, in its live state.

Alternative:

MediaDrop

MediaDrop is a modular video, audio, and podcast publication platform which can be extended with plugins: “The Web’s Open Source Video Platform”.

File Transfer

These are the protocols by which content files will be transmitted. BitTorrent, Storj, Freenet, I2P, Tor, MaidSafe, Bitcloud, Tahoe-LAFS, Dropbox, etc are all examples of candidates. File Transfer networks used in Bit451 will need a streaming capability if it does not already exist. This ability to tie into many different existing and new networks has many advantages including greater user choice, established roots, wide user base, infinite content waiting to be brought to the masses, development community, etc. The end result is a P2P file sharing network crossover methodology/capabilities to bridge and integrate multiple different protocols.

A protocol-agnostic, generic interface will be available to be implemented for each protocol (in the Portal).

Portal Gateway

This is the content bridge between the File Transfer networks and the Portal (as seen here, for BitTorrent; some File Transfer protocols may not require this if the functionality already exists).

WebTorrent

WebTorrent is a streaming torrent client for the browser using WebRTC. The Project Goal is to build a browser BitTorrent client that requires no install (no plugin/extension/etc.) and fully-interoperates with the regular BitTorrent network.

Since WebTorrent is web-first, it’s simple for users who do not understand .torrent files, magnet links, NATs, etc. By making BitTorrent easier, it will be accessible to new swathes of users who were previously intimidated, confused, or unwilling to install a program on their machine to participate.

Portal Gateway Action Item

Fork WebTorrent to Bit451 Portal Gateway - BitTorrent, if not yet complete, and contribute to completion of the project, implementing any additional functionality if necessary.

Alternative:

Btapp.js

The BitTorrent project Btapp.js can be used to bring the BitTorrent network to the browser. This project integrates with the BitTorrent Torque browser plugin or a number of other common software clients. For the purposes of this project let us assume that the Torque browser plugin is used, as it is the simplest way for the user to click-and-go.

A demo site/project called oneHash exists to show some Btapp.js capabilities. It works directly hosted from the Github project, however, the domain is expired. You can still try it here, or you can edit your computer’s hosts file and point onehash.com to Github’s IP (currently 192.30.252.153) as noted in Setting up a custom domain with GitHub Pages.

oneHash demonstrates the ability to stream torrent content via the browser. Currently only audio streaming is functional; video streaming is not yet completed as noted in the Code Snippets page. If this is still the case by the time Bit451 development begins, it will need to be implemented.

Phase 1 Results

Via the static web files hosted locally or on any website (including the live Github project page at Bit451.com), users can now tap into and stream any media file on the File Transfer networks using the Bit451 Portal interface.

Phase 2 Development

The goal of this phase is to implement attributes of the Bit451 overlay network. The database uses trusted timestamping to verify data changes (most recently timestamped data trumps). A thin client will be implemented to connect the Bit451 network to the File Transfer networks and to allow Bit451 to be run server-side so site owners can provide their visitors with client-less access to the Bit451 network by retrieving/transcoding/caching/serving content.

Bit451 Service

This is the user connection to the Bit451 network. It can optionally also enable servers in retrieving/transcoding/caching content from the File Transfer networks and serving it directly to their visitors in the browser, replacing the need for potential users to use network-specific clients and download data to their computer.

Bit451 servers can configure which content they wish to serve: from any users in their network, from Subscriptions only, from Favorites only, etc. Video transcoding/serving is already built into the Bit451 Portal system, so it simply needs to be adapted for Bit451’s architecture.

As much as possible this should be decoupled from the Bit451 components’ specific architectures. The ability to auto-update should also be built-in.

A similar GUI client (see examples below) can be used as a starting point, and stripped down to what’s needed.

Database

The core of the Bit451 overlay network is its distributed data system. Options to store and access this data: 1) distributed database (DDB), 2) distributed hash table (DHT). Option 1, DDB, is the Bit451 method of choice. It will be used in a conceptually new and exciting manner for P2P networks, the end result being a P2P distributed database which can easily be reused by other P2P apps. The replication method devised is modeled on the concept of communities and community-based trust.

OrientDB

OrientDB is web ready (natively supports HTTP/RESTful protocol/JSON), cross-platform, embeddable (with local mode to bypass the server), has a footprint of only about 1MB for the full server, is well-supported/documented, supports drivers for JavaScript (among others), database- and record-level security, and distributed architecture.

Database Functionality

A primary functionality to be implemented for Bit451 is a security layer to: sign and verify records, selectively determine which records to query from/replicate, identify/block malicious nodes, etc.

Bit451 will embed the database in a thin wrapper client and interact via its JavaScript driver. Each node acts as a server and represents an Account. Each record should be signed with a hash of ALL its fields (including ID, timestamp, and Account ID, etc) and public and private key, and then when verifying simply remove the signature (sig) and check against it. The client should enforce incoming data integrity, rate limit/protect against DoS attacks, etc (basically act as database gatekeeper).

Database connections should also be authenticated so that, for example, a hosted Bit451 installation cannot rely on a single naive user as its direct database connection/server. The user should also have the option to allow unauthenticated connections, i.e. to be an open host.

The last known IP/port for each node will, at least in this Phase, be stored in the Account record. Each time a node reconnects it will update its IP/port in its Account record (if it has changed), sign it, and replicate it to the network. It will also update its own TCP/IP member list after replicating from the network. This way the network always knows the most current connection to each node.

Accounts can also verify themselves to their web domain to engender trust (like a certification, or “cert”). For example, if the user of Account 123 is affiliated with foo.com, a DNS TXT record of “123” could be set at bit451.foo.com; then, the Account’s cert field set to bit451.foo.com. Now when viewing Account 123, an Ajax DNS lookup is done at bit451.foo.com, and when successful, foo.com is shown as the certified domain.

The database structures used are typically regarded as database “tables” which have “fields”; in OrientDB these are actually called classes, and have properties. Records are accessible network-wide but security is record-level.

Each created Bit451 account will have self-owned classes, both vertices and edges. These will be replicated (read-only) across nodes on a determined basis: for each account a node networks with, said account’s classes are replicated on that node.

Accounts can also Recommend other Accounts. Whereas Subscribe denotes a personal preference (“I want to watch this Account”), Recommend denotes an interpersonal reference (“I think others should watch this Account”). Likewise, Accounts can Mimic another Account’s Recommendations, meaning that it will include all of that Account’s Recommendations in its network (minus any Accounts it has ignored). Therefore, Subscribe will replicate one Account, and Mimic will replicate many Accounts.

The nodes which replicate an account are its Replication Network (R-Net). The nodes which an account replicates are its Transmission Network (T-Net), since it transmits them. Each R-Net and T-Net will include at least one node (itself). Note that *-Nets usually aren’t necessarily bi-directional; for example, if Account 1 subscribes to Account 2, and Account 2 subscribes to Account 3, then node 1’s T-Net is {1,2} and R-Net is {1}, node 2’s T-Net is {2,3} and R-Net is {1,2}, and node 3’s T-Net is {3} and R-Net is {2,3}.

A replication Hook will be used to replicate only records from one’s T-Net (aggregation of one’s Subscriptions, Recommendations, and Mimics, minus Ignores).

Bootstrapping

Because of the community-oriented nature of Bit451 networks, “bootstrapping” is not, in the conventional sense, necessary. Rather, new users will simply Subscribe (network with) and/or Mimic (network with connections of) the Account(s) they trust. Ideally a new user can simply Mimic a trusted Account from their community, and will instantly join the network of all of that Account’s Recommendations.

Account Creation / Recovery

In the event that a private key is compromised, there should be a recovery method. Traditional methods such as email aren’t suitable, and a new ID can’t simply be generated since the account is tied to the ID.

The solution is a master public key (MPK)/hardened child public key scheme. The MPK will be the Account ID. Then, a child public key is used to sign records. The lowest key should be used first.

To recover the account, the user simply “burns” their current key. Records are then signed with the next-highest key. When the replication filter verifies records, it must first simply verify that the signing key is valid for the Account ID, and then, the highest key should win. This means the user has the ability to burn as many keys as needed; their account is never unrecoverable, since the use of hardened keys means that a compromised child private key does not compromise the master private key.

In order to resolve the event that a user’s whole account is simply lost rather than compromised (ex: computer crash), accounts should be generated from a deterministic wallet seed, or master private key, similar to the Electrum seed. When creating a new account, the user is highly encouraged to record their seed somewhere safe. Bit451 Accounts, then, are somewhat comparable to Bitcoin wallets, at least insofar as functionality within the client. Also see BIP32.

See Web Wallets links below for implementations.

Data Integrity

Incoming replication data will hit a Hook which will verify each record’s sig and, if forged, add the IP to its blacklist.

Deletions are a special case: records should have their “deleted” bit set by the owner; the deleted bit should always be queried against. Records should not be actually deleted from the database, since deleted records cannot be verified by the replication filter after deletion.

Since Bit451 data is stored in a DDB, this is relatively simple, as searches are just a query algorithm (DHT searches would be more complex).

Database Classes

Key

-> referenced/linked relationships

Accounts

Each Account represents a user, similar to profiles or channels. Properties include ID, name, about, favorites[]->, ratings[]->, subscriptions[]->, recommendations[]->, ignores[]->, mimics[]->, address (IP/port), cert (DNS TXT record), created, updated, deleted?, key, sig.

Content

Each Content represents a user-created reference to content on a P2P file sharing network. Properties include ID, Account ID, address{} (object denoting network, ID, and any other data required for that network), name, description, tags[], type{video, audio, image, text, other}, created, updated, deleted?, key, sig.

Any user can, of course, add any existing network content they wish to their Bit451 account. Bit451 will also introduce a brief template (with flags) to add to your network content metadata (i.e. torrent) which essentially verifies you as the owner.

Database Action Item

Fork OrientDB to Bit451 Database. Implement any default config changes (group name/password, database name/password, etc) and other necessary modifications.

Database Notes

Phase 2 Results

At this point users can log into and participate in Bit451 via a small app on their local machine, or they can guest browse Bit451 content via hosted Bit451 sites.

Phase 3 Development

The goal of this phase is to complete/enhance/granularize the Bit451 functionality and user experience. More technical details of this phase will be fleshed out as development progresses.

Anonymity

Provide ability to disable last known IP/port field in Account records, essentially allowing nodes to push/replicate data without being able to be connected to, as well as ability to encrypt traffic. There are plenty of reference points starting from anonymous P2P, I2P (end-to-end encryption), Tor, etc. This refers only to the Bit451 Database data, of course.

The media content itself comes from the various P2P file sharing networks, so anonymity at this point is the responsibility of each network and/or the client used. The user has the option to select the networks they wish to access.

Payments

A plugin API should be implemented to enable plugins (short scripts) to be written for cryptocurrencies and other payment systems, which can tie into various Bit451 hooks, allowing automated hosting agreements and payments.

Each Content record (and, of course, its data) is hosted only on those nodes which choose to do so. This can include:

Etc. In this way, content is hosted to the extent which it is demanded.

Database Additions

Reputation

Community interactions, such as Payments, require a level of trust. Reputation will enable users to judge the reliability of other users within their community. A good/bad/neutral rating could be associated with each Payment made via Bit451.

Hashtags

Tagging and searching of Content and Accounts via Hashtags.

Playlists

Can be stored locally initially, and eventually via optionally (paid?) remotely hosted protocol.

Similar Accounts / Content

Graph depth can be explored to find similar/related data to recommend to the user.

Comments

Account: Properties include comments[]->, moderators[]->, commentsMode (mode bits for: Account-level, Content-level, defer?).

Content: Properties include comments[]->, moderators[]->, commentsMode (mode bits for: Content-level, defer?)

Bit451 Lite

Provides the ability to fully participate in Bit451 actions (creating/modifying accounts/content, etc) directly via the browser using JavaScript, so the user is not required to run the Bit451 Service to access the network.

Phase 3 Results

Bit451 complete. It is now a fully-featured, browser-based, distributed media network.

Post-Development

At this point we proceed with any new additions, bug fixes, etc. Some additional ideas include:

UML

Bit451 technology stack

Bit451 technology stack

Bit451 Portal technology stack

Bit451 Portal technology stack

Bit451 network "community" example

Bit451 network “community” example

Contribute

There is essentially little-to-no “official” development team, as development is crowdsourced, ad hoc, and decentralized. In fact, the community as a whole is the dev team. This means that anyone is able to contribute and collaborate in whatever ways work best for them. With the existence of Github, code can be forked and built upon any number of times by any number of contributors until a working solution is created. Solutions will then be audited, tested, and assessed by the community before being accepted.

The primary reason to contribute to Bit451 is simply because you believe in the idea and want to make it happen; but an additional reason is that, to the extent possible, Bit451 will be bounty-driven. A significant amount of Bitcoin donations raised (as much as possible after paying other costs) will be used for Bitcoin bounties on Bit451’s projects, both directly and via Bountysource.

If a deeply established, respected, and community-loved open source dev(s) were to volunteer for the role of official Bit451 code gatekeeper, and the Bit451 community overwhelmingly supported it, that would be cool.

Now let us go and ignite a firestorm, planting the technological seedlings of the future over a trail of obsolete overgrowth!

Roadmap / Timeline

3 Phases; the goal is ~1 month per phase (after initial discussion/planning phase). It should be completed by the end of the year. Yes, this is ambitious, but very much possible!

Goals and Features

Discuss

Donate

Architect

Calen Fretts is a schemer, dreamer, believer, fighter, doer, troublemaker, idealist, voluntaryist, and many other -ists and -ers. Basically he’s probably just crazy.

License

MIT.

Bit451. Set. Go!

http://Bit451.org

https://github.com/Bit451

Bit451 should be used morally.

“Here’s to the crazy ones. The misfits. The rebels. The troublemakers. The round pegs in the square holes. The ones who see things differently. They’re not fond of rules. And they have no respect for the status quo. You can quote them, disagree with them, glorify or vilify them. But the only thing you can’t do is ignore them. Because they change things. They push the human race forward. And while some may see them as the crazy ones, we see genius. Because the people who are crazy enough to think they can change the world, are the ones who do.”