Cloud Storage and Data Acquisition

aid: cloud-storage-and-data-acquisition
name: Cloud Storage and Data Acquisition
description: >-
  Cloud Storage and Data Acquisition is a topic profile in the API
  Evangelist Network covering APIs and tooling for ingesting, moving,
  and persisting bulk and streaming data into cloud-resident storage.
  It groups object storage services, data-lake foundations, managed
  ingestion pipelines, change-data-capture connectors, transfer
  appliances, and data-broker APIs that provide source material for
  cloud-storage workloads. The topic is intended as an entry point
  for developers and architects evaluating how data lands in cloud
  storage from on-premises systems, SaaS APIs, IoT devices, public
  data sources, and partner exchanges.
type: Index
image: https://kinlane-productions.s3.amazonaws.com/apis-json/apis-json-logo.jpg
url: https://raw.githubusercontent.com/api-evangelist/cloud-storage-and-data-acquisition/refs/heads/main/apis.yml
created: '2024-01-01'
modified: '2026-04-23'
specificationVersion: '0.19'
kind: topic
tags:
  - Bulk Transfer
  - Change Data Capture
  - Cloud Storage
  - Data Acquisition
  - Data Ingestion
  - Data Lake
  - ETL
  - Object Storage
  - Pipelines
  - Streaming
apis:
  - aid: cloud-storage-and-data-acquisition:object-storage
    name: Object Storage Surface
    tags:
      - Object Storage
      - REST
    image: https://kinlane-productions2.s3.amazonaws.com/apis-json/apis-json-logo.jpg
    humanURL: https://aws.amazon.com/s3/
    properties:
      - url: https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html
        type: Documentation
      - url: https://cloud.google.com/storage/docs/json_api/v1
        type: Documentation
      - url: https://learn.microsoft.com/en-us/rest/api/storageservices/blob-service-rest-api
        type: Documentation
    description: >-
      The object-storage surface includes Amazon S3, Google Cloud
      Storage, and Azure Blob Storage REST APIs. These APIs provide
      the canonical landing zone for cloud data acquisition and are
      the most common targets for ingestion pipelines.
  - aid: cloud-storage-and-data-acquisition:streaming-ingest
    name: Streaming Ingest Surface
    tags:
      - Pub/Sub
      - Real-time
      - Streaming
    image: https://kinlane-productions2.s3.amazonaws.com/apis-json/apis-json-logo.jpg
    humanURL: https://aws.amazon.com/kinesis/data-streams/
    properties:
      - url: https://docs.aws.amazon.com/kinesis/latest/APIReference/
        type: Documentation
      - url: https://cloud.google.com/pubsub/docs/reference/rest
        type: Documentation
      - url: https://learn.microsoft.com/en-us/rest/api/eventhub/
        type: Documentation
    description: >-
      The streaming-ingest surface covers Amazon Kinesis Data Streams,
      Google Cloud Pub/Sub, and Azure Event Hubs. These services
      ingest high-volume event streams and make them durable for
      downstream landing into object storage and analytics systems.
  - aid: cloud-storage-and-data-acquisition:managed-ingestion
    name: Managed Ingestion Pipelines
    tags:
      - Data Pipelines
      - ETL
      - Managed
    image: https://kinlane-productions2.s3.amazonaws.com/apis-json/apis-json-logo.jpg
    humanURL: https://aws.amazon.com/glue/
    properties:
      - url: https://docs.aws.amazon.com/glue/latest/webapi/Welcome.html
        type: Documentation
      - url: https://cloud.google.com/dataflow/docs/reference/rest
        type: Documentation
      - url: https://learn.microsoft.com/en-us/rest/api/datafactory/
        type: Documentation
    description: >-
      Managed ingestion pipelines such as AWS Glue, Google Cloud
      Dataflow, and Azure Data Factory expose REST APIs for
      orchestrating extract-transform-load and extract-load-transform
      jobs that land source data in cloud storage.
  - aid: cloud-storage-and-data-acquisition:change-data-capture
    name: Change Data Capture Connectors
    tags:
      - CDC
      - Connectors
      - Replication
    image: https://kinlane-productions2.s3.amazonaws.com/apis-json/apis-json-logo.jpg
    humanURL: https://debezium.io/
    properties:
      - url: https://debezium.io/documentation/
        type: Documentation
      - url: https://docs.aws.amazon.com/dms/
        type: Documentation
    description: >-
      Change Data Capture connectors (Debezium, AWS DMS, Fivetran,
      Striim) replicate row-level changes from operational databases
      to cloud storage and warehouses, providing a low-latency feed
      for analytics and data-lake hydration.
  - aid: cloud-storage-and-data-acquisition:transfer-services
    name: Bulk Transfer Services
    tags:
      - Bulk Transfer
      - Migration
      - Offline
    image: https://kinlane-productions2.s3.amazonaws.com/apis-json/apis-json-logo.jpg
    humanURL: https://aws.amazon.com/datasync/
    properties:
      - url: https://docs.aws.amazon.com/datasync/latest/userguide/API_Reference.html
        type: Documentation
      - url: https://cloud.google.com/storage-transfer/docs/reference/rest
        type: Documentation
      - url: https://learn.microsoft.com/en-us/azure/databox/
        type: Documentation
    description: >-
      Bulk transfer services (AWS DataSync and Snow family, Google
      Storage Transfer Service, Azure Data Box) move large datasets
      from on-premises and edge locations into cloud storage over
      network or via offline appliances.
  - aid: cloud-storage-and-data-acquisition:data-marketplaces
    name: Data Marketplaces
    tags:
      - Datasets
      - Marketplace
      - Open Data
    image: https://kinlane-productions2.s3.amazonaws.com/apis-json/apis-json-logo.jpg
    humanURL: https://aws.amazon.com/data-exchange/
    properties:
      - url: https://aws.amazon.com/data-exchange/
        type: Documentation
      - url: https://cloud.google.com/marketplace
        type: Documentation
      - url: https://registry.opendata.aws/
        type: Open Data Registry
    description: >-
      Data marketplaces and open-data registries expose third-party
      datasets through REST APIs, subscription delivery, and shared
      buckets. They are an increasingly common acquisition channel
      for cloud-storage data lakes.
common:
  - type: Topic
    url: https://apievangelist.com/topics/cloud-storage-and-data-acquisition/
  - type: API Evangelist
    url: https://apievangelist.com/
  - type: Network
    url: https://network.apievangelist.com/
  - type: GitHub
    url: https://github.com/api-evangelist
  - type: JSONLD
    url: json-ld/cloud-storage-and-data-acquisition-context.jsonld
  - type: Spectral
    url: rules/cloud-storage-and-data-acquisition-rules.yml
maintainers:
  - FN: Kin Lane
    email: [email protected]