Newswire – Collaborative real-time content delivery
Why Newswire? Newswire is a peer-to-peer, fully decentralized system that brings news to your desktop, within seconds after it is published. This technology gives the community the power to weave a collaborative infrastructure for the delivery of essential information to individuals in a robust, scalable and secure way. Newswire is a survivable system which will deliver news to subscribers even if large parts of the infrastructure are under attack or stress.
The development of Newswire followed discussions after the 9/11 attacks when flash-crowd effects made it impossible for many to reach essential news sites. Email and weblogs proved more effective technologies in that situation, although both still suffer from the centralized risks of overload and single-point-of-failure. A more robust approach is to use a peer-to-peer structuring of the system and provide strong incentives for people to collaborate on the delivery of information to everyone by giving up a bit of bandwidth and CPU cycles.
Newswire has the additional effect that it can significantly reduce the load on the websites carrying real-time news information by providing hints about when information has changed. This would change the frequent redundant poll behavior seen by many news sites into more effective visits.
The Newswire Technology What is the technology that makes Newswire so special? At the core is epidemic communication and state management to maintain distributed knowledge about subscriptions, participant network capabilities and forwarding load. The epidemic technology was first developed by Alan Demers and friends at PARC, but has been completely revised at Cornell in the past years
The structuring into zones and virtual hierarchies is based on the small worlds phenomenon. When you organize participants into small groups of local participants with some knowledge of other remote nodes you can construct very effective routes in a decentralized, autonomous manner.
These technologies are used to build a loosely coupled, dynamic overlay network, that continuously monitors network load and capacities to achieve a fair load.
Subscription information stored in Bloom Filters is aggregated such that simple forwarding decisions can be made anywhere in the network based on the location of the publication and the direction where possible subscriber are localed.
Vogels pointed to half a dozen really great papers in All Things Distributed: Epidemic Computing at Cornell
… that having to scale up your system is no longer a burden, but it becomes an advantage and you can deliver on the true promise of scale-out: more nodes means a more robust system.
Also, he noted: All Things Distributed: Wired on the Scalability of Feed Aggregators
We have had some legal issues with the deployment of the newswire beta, and a new core communication component is being developed. But I would be happy to break open the ideas in the system and work with others to see whether this can be a good experimentation platform.