Apache Avro
Apache Avro is a data serialization system that provides rich data structures, a compact binary format, and container files for storing persistent data. Avro uses JSON for defining data types and protocols, and serializes data in a compact binary format.
APIs
Apache Avro Schema Format
JSON Schema for validating Apache Avro schema definitions. Covers all Avro types including primitive types (null, boolean, int, long, float, double, bytes, string), complex type...
Features
Avro requires schemas to be defined in JSON before serialization, enabling strong typing and schema validation.
Avro supports backward, forward, and full schema compatibility through aliases, defaults, and type promotions.
Avro serializes data in a compact binary format without field names, reducing payload size significantly.
Supports primitive types, complex types (records, enums, arrays, maps, unions, fixed), and logical types (date, time, decimal, UUID).
Official implementations in Java, Python, C, C++, C#, PHP, Ruby, and Rust with broad ecosystem support.
Avro Object Container Files (OCF) embed the schema with the data for self-describing data files.
Avro defines an RPC protocol mechanism using schemas for both request and response messages.
Apache Kafka ecosystem uses Avro as a primary serialization format with the Confluent Schema Registry.
Use Cases
Serialize Kafka events with Avro schemas stored in a Schema Registry for high-throughput data pipelines.
Store large datasets in Avro container files in Hadoop-compatible storage with embedded schema metadata.
Use Confluent Schema Registry to manage schema versions and enforce compatibility across producers and consumers.
Define message contracts between microservices using Avro schemas for type-safe data exchange.
Process large volumes of structured data with Apache Spark, Hive, or Flink using Avro as the interchange format.
Integrations
Native serialization format for Kafka messages via the Confluent Schema Registry and Kafka clients.
Spark SQL and DataFrames support reading and writing Avro files natively.
Hive tables can be backed by Avro container files with schema stored in the Hive Metastore.
Centralized schema management service for validating and evolving Avro schemas in Kafka ecosystems.
Flink supports Avro for serialization and deserialization of streaming data.
Avro is a native storage format supported by the Hadoop ecosystem for distributed processing.