What is Ceph?#

Ceph is an open-source, software-defined storage platform that provides highly scalable, distributed storage for large-scale systems. It is designed to work on commodity hardware, making it cost-effective and flexible for handling massive amounts of data.

Ceph’s architecture provides object storage, block storage, and file storage within a unified system. It is widely used in cloud infrastructure, data centers, and for large-scale workloads due to its fault tolerance, high availability, and scalability.

Key Features of Ceph:#

  1. Unified Storage:

    • Supports object storage (via RADOS Gateway), block storage (via RBD - RADOS Block Device), and file storage (via CephFS).

  2. Scalability:

    • Scales horizontally across thousands of nodes to store petabytes of data.

  3. Fault Tolerance:

    • Redundant data replication or erasure coding ensures no single point of failure.

  4. High Availability:

    • Ceph’s self-healing and self-managing capabilities provide continuous availability of data.

  5. Open Source:

    • It is part of the Linux ecosystem and is used in conjunction with OpenStack, Kubernetes, and other cloud systems.

Core Components:#

  1. OSD (Object Storage Daemon):

    • Stores data, handles data replication, and recovery.

  2. MON (Monitor):

    • Manages the cluster state, including mapping, health, and configuration.

  3. MDS (Metadata Server):

    • Manages metadata for CephFS (the file system).

  4. RADOS (Reliable Autonomic Distributed Object Store):

    • The foundation of Ceph’s storage system, enabling distributed and fault-tolerant object storage.

  5. RBD (RADOS Block Device):

    • Provides block storage support for VMs and applications.

  6. RGW (RADOS Gateway):

    • Allows object storage access via protocols like S3 and Swift.

Use Cases:#

  • Cloud Infrastructure (e.g., OpenStack storage backend)
  • Big Data and Analytics Storage
  • Media Streaming
  • Backup and Archive Solutions
  • Virtual Machine Storage
  • Kubernetes Persistent Storage

Ceph’s ability to combine performance, scalability, and fault tolerance makes it a strong choice for modern IT infrastructures.

Deploy CEPH to Proxmox VE#

To deploy Ceph in a Proxmox VE (Virtual Environment) cluster, follow these steps. Proxmox has native integration with Ceph, making it relatively simple to set up for distributed storage.

Prerequisites:#

  1. Proxmox VE Cluster:
    • Ensure you have a Proxmox VE cluster with at least 3 nodes.
    • Nodes must be on the same network.
  2. Hardware Requirements:
    • Minimum 3 nodes for redundancy.
    • Dedicated SSDs/HDDs for Ceph storage (separate from Proxmox OS disk).
    • Adequate CPU, RAM, and network speed (preferably 10GbE for Ceph traffic).
  3. Networking:
    • A dedicated network for Ceph storage traffic.
    • Static IP addresses for cluster nodes.

Step-by-Step Deployment:#

1. Enable Ceph Repository:#

Log in to your Proxmox VE cluster nodes and enable the Ceph repository:

echo "deb http://download.proxmox.com/debian/ceph-quincy bookworm main" > /etc/apt/sources.list.d/ceph.list
apt update

2. Install Ceph on Each Node:#

Install Ceph packages:

apt install ceph ceph-common

3. Initialize Ceph on the Cluster:#

On one of the nodes (usually the Proxmox cluster master), run:

pveceph init --network <storage-network-subnet>
  • Replace <storage-network-subnet> with your Ceph network, e.g., 192.168.10.0/24.

This initializes the Ceph monitor (MON) network.

4. Create Monitor(s) (MON):#

Ceph monitors maintain the state of the cluster. Add a monitor on each node:

pveceph mon create

Repeat this command on at least 3 nodes.

5. Add Ceph Manager (MGR):#

The Ceph manager daemon helps monitor and manage the cluster:

pveceph mgr create

6. Configure Object Storage Daemons (OSDs):#

An OSD stores the actual data. Before creating OSDs:

  • Make sure disks for Ceph are unmounted and free of partitions.
  • Use dedicated disks (e.g., /dev/sdb, /dev/sdc).

Create an OSD using the command:

pveceph osd create /dev/sdb

Replace /dev/sdb with your target disk. Repeat for all storage disks on all nodes.

7. Check Ceph Cluster Health:#

To verify cluster status:

ceph status

or

pveceph status

You should see the cluster with MONs, MGRs, and OSDs running.

8. Add a Ceph Pool:#

A Ceph pool provides storage for your VMs and containers. Create a pool:

pveceph pool create <pool-name> --size 3
  • Replace <pool-name> with a name (e.g., vm-storage).
  • --size 3 ensures 3 copies of data for redundancy.

9. Integrate Ceph Storage into Proxmox VE:#

  1. Go to the Proxmox VE web interface.
  2. Navigate to Datacenter > Storage.
  3. Add a new storage:
    • Type: RBD (Ceph).
    • Name: Enter a name.
    • Monitor Host(s): IP addresses of your Ceph MON nodes.
    • Pool: Select the pool created earlier.

Save the settings.

10. Verify Storage:#

  • Go to Storage in the Proxmox interface.
  • You should see the Ceph storage pool ready for use.
  • Start creating VMs or containers on the Ceph storage.

Monitoring the Cluster:#

  • Use the Proxmox Ceph Dashboard: Navigate to Node > Ceph in the web UI.
  • CLI commands for monitoring: 
    ceph status
ceph osd tree
ceph df

Best Practices:#

  1. Use a dedicated network (10GbE preferred) for Ceph traffic.
  2. Enable HA (High Availability) for virtual machines.
  3. Monitor OSD and disk health regularly.