Is Zookeeper becoming the standard cluster manager?
Thursday, March 15, 2012 at 01:51PM I have been intrigued lately about the growth of Zookeeper as a general purpose cluster manager. Zookeeper started as a clone of Chubby, a google project to maintain distributed locks. Zookeeper is the core metadata filesystem for Hadoop. Zookeeper maintains information on cluster participants, services and status - it's a cluster manager.
There have been some other uses in the past – Neo4j and Rackspace – but things seem to be accelerating lately.
Netflix’s Curator project is quite interesting – particulary the Service Discovery work as more general purpose workflow coordination.
Twitter is using Mesos on top of Zookeeper for meta data but also to instantiate services.
More influencial is Storm. Built on top of Mesos. Nathan's presentation at Strata 2012 was jammed.
LinkedIn wrote Norbert last year – seems very light weight but they are doing something similar to what Twitter is doing.
The most popular session about Zookeeper at 2012 Hadoop summit?
Dynamic Reconfiguration of Apache Zookeeper
Zookeeper is definitely not just for map reduce jobs anymore...