Ceph

Definition

Ceph is a unified, distributed storage system that provides object, block, and file storage from a single cluster. It is designed to provide excellent performance, reliability, and scalability while eliminating single points of failure.

Ceph is hosted by the CNCF and is widely used for cloud infrastructure (OpenStack, Kubernetes) and large-scale storage deployments.

Ceph Components

Component Description
OSD (Object Storage Daemon) Stores data, handles replication, recovery, and rebalancing
MON (Monitor) Maintains cluster map (OSD map, CRUSH map, MGR map)
MGR (Manager) Monitors and manages Ceph cluster state, exposes metrics
MDS (Metadata Server) Metadata for CephFS (file system only, not needed for RBD/RGW)
CRUSH (Controlled Replication Under Scalable Hashing) Algorithm that determines data placement across OSDs

Ceph Storage Types

Type Protocol Use Case
RBD (RADOS Block Device) iSCSI/NVMe-oF Block storage for VMs (OpenStack, Kubernetes)
RGW (RADOS Gateway) S3/Swift API Object storage (S3-compatible)
CephFS NFS/SMB/POSIX File system (NFS gateway, direct mount)

Ceph Architecture

Client
    ↓
Ceph Cluster
├── MONitors (quorum-based cluster map)
├── MGRs (metrics, monitoring)
├── OSDs (stores data, handles replication)
│   ├── Primary OSD
│   ├── Replica OSD 1
│   ├── Replica OSD 2
└── MDS (CephFS only)

Ceph vs Traditional Storage

Feature Ceph Traditional SAN Traditional NAS
Scalability Petabytes+ Limited (vendor lock-in) Limited
Cost Open-source (no licensing) Expensive (vendor) Moderate
Hardware Commodity x86 Proprietary Proprietary
Redundancy Automatic (CRUSH) RAID (single point) RAID
APIs S3, Swift, block, file Vendor-specific NFS, SMB
Complexity High Low Moderate

Ceph in Kubernetes

  • RBD CSI driver: Persistent volumes backed by RBD
  • CephFS CSI driver: Persistent volumes backed by CephFS
  • RGW: Object storage for Kubernetes workloads
  • rook: Kubernetes operator for Ceph

References