Apache Mesos
Apache Mesos is a retired cluster manager (now in the Apache Attic) that provided efficient resource isolation and sharing across distributed applications or frameworks. It abstracted CPU, memory, storage, and other compute resources from machines, enabling fault-tolerant and elastic distributed systems. Mesos exposed comprehensive HTTP APIs for schedulers, operators, executors, and agents.
APIs
Apache Mesos Operator HTTP API
The Mesos Operator HTTP API provides a POST-based API at /api/v1 on both master and agent nodes for cluster administration including health checks, state queries, resource reser...
Apache Mesos Scheduler HTTP API
The Mesos Scheduler HTTP API at /api/v1/scheduler enables framework schedulers to subscribe to resource offers, launch tasks, kill tasks, reconcile status, and manage framework ...
Features
Abstracts CPU, memory, storage, and other compute resources from physical machines across the entire cluster.
Framework schedulers receive resource offers from Mesos and decide how to use them, enabling coexistence of diverse workloads.
Proven to scale to tens of thousands of nodes with fault-tolerant replicated master using ZooKeeper.
Native Docker and AppC container image support for running containerized workloads.
Comprehensive HTTP API supporting JSON and Protobuf encoding for schedulers, operators, executors, and agents.
Fault-tolerant master failover via ZooKeeper with automatic leader election and state recovery.
Static and dynamic resource reservations for frameworks and roles with quota management.
Built-in maintenance window scheduling for graceful draining and reactivation of agent nodes.
Use Cases
Run multiple distributed frameworks including Hadoop, Spark, and Kafka on shared cluster resources.
Schedule and manage containerized workloads across a datacenter with resource isolation.
Run Apache Spark, Hadoop MapReduce, and other big data frameworks on Mesos-managed resources.
Host microservices workloads with Marathon framework providing long-running service scheduling on Mesos.
Integrations
Run Hadoop MapReduce jobs on Mesos-managed cluster resources.
Apache Spark supports Mesos as a cluster manager for distributed job execution.
Kafka brokers can be scheduled and managed on Mesos clusters.
ZooKeeper provides leader election and state storage for Mesos master high availability.
Native Docker container image and runtime support for containerized workload execution.
Elasticsearch can be deployed and managed as a framework on Mesos clusters.