November 29, 2004

MapReduce for decentralized computation

Decentralization By: ams

I just added a technical note to our Wiki extending the work of Dean and Ghemawat on MapReduce, a support library for programs that take advantage of large clusters such as those at Google. The fundamental problems of writing distributed systems like these — latency, naming (or memory access), partial failures, and concurrency are toy versions of most of the key problems of decentralized systems. (Agency conflict is the one problem that distinguishes decentralized systems from distributed systems, and it dramatically worsens the other four.)

It occurred to me that MapReduce has the capability to handle some of the agency problems of decentralized systems, particularly those that affect reliability, in a remarkably simple way:

In the usual case, where these functions are deterministic, they can be executed on two administratively-independent servers, and the results (which, in the Google case, are merely files) can be compared. If they differ, the same results can be recomputed on more administratively-independent servers to see which ones were correct.

  • blog

  • companies & initiatives

  • December 2019
    M T W T F S S
    « May    
     1
    2345678
    9101112131415
    16171819202122
    23242526272829
    3031  
  • archive

  • categories