Setting Up and Troubleshooting Rook-Ceph on Minikube

Introduction

Storage is a critical component in Kubernetes environments, and Rook-Ceph is a powerful storage orchestrator that brings Ceph into Kubernetes clusters. In this post, we’ll go through deploying Rook-Ceph on Minikube, setting up CephFS, and resolving common issues encountered during the process.

Prerequisites

Before we begin, ensure you have the following installed on your machine:

  • Minikube (running Kubernetes v1.32.0)
  • Kubectl
  • Helm

If you don’t have Minikube installed, you can start a new cluster with:

minikube start --memory=4096 --cpus=2 --disk-size=40g

Enable required Minikube features:

minikube addons enable default-storageclass
minikube addons enable storage-provisioner

Step 1: Install Rook-Ceph with Helm

Add the Rook Helm repository:

helm repo add rook-release https://charts.rook.io/release
helm repo update

Install Rook-Ceph Operator

helm install rook-ceph rook-release/rook-ceph --namespace rook-ceph --create-namespace

Verify the operator is running:

kubectl -n rook-ceph get pods

Expected output:

NAME                            READY   STATUS    RESTARTS   AGE
rook-ceph-operator-59dcf6d55b   1/1     Running   0          1m

Step 2: Deploy the Ceph Cluster

Now, create a CephCluster resource to deploy Ceph in Kubernetes.

Create ceph-cluster.yaml

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: my-ceph-cluster
  namespace: rook-ceph
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v18.2.0
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
    allowMultiplePerNode: true
  storage:
    useAllNodes: true
    useAllDevices: true
Click Here to Copy YAML

Apply the Ceph Cluster Configuration

kubectl apply -f ceph-cluster.yaml

Check the CephCluster status:

kubectl -n rook-ceph get cephcluster

Expected output (once ready):

NAME              DATADIRHOSTPATH   MONCOUNT   AGE   PHASE     MESSAGE      HEALTH
my-ceph-cluster   /var/lib/rook     3          5m    Ready     Cluster OK   HEALTH_OK

Step 3: Set Up CephFS (Ceph Filesystem)

Create cephfs.yaml

apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: my-cephfs
  namespace: rook-ceph
spec:
  metadataServer:
    activeCount: 1
    activeStandby: true
  dataPools:
    - replicated:
        size: 3
  metadataPool:
    replicated:
      size: 3
Click Here to Copy YAML

Apply CephFS Configuration

kubectl apply -f cephfs.yaml

Verify the filesystem:

kubectl -n rook-ceph get cephfilesystem

Expected output:

NAME        ACTIVEMDS   AGE   PHASE
my-cephfs   1           2m    Ready

Step 4: Create a StorageClass for CephFS

Create cephfs-storageclass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-cephfs
provisioner: rook-ceph.cephfs.csi.ceph.com
parameters:
  clusterID: rook-ceph
  fsName: my-cephfs
  pool: my-cephfs-data0
  mounter: fuse
reclaimPolicy: Delete
Click Here to Copy YAML

Apply the StorageClass Configuration

kubectl apply -f cephfs-storageclass.yaml

List available StorageClasses:

kubectl get storageclass

Step 5: Create a PersistentVolumeClaim (PVC)

Create pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-cephfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: rook-cephfs
Click Here to Copy YAML

Apply the PVC Configuration

kubectl apply -f pvc.yaml

Check if the PVC is bound:

kubectl get pvc

Expected output:

NAME            STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
my-cephfs-pvc   Bound    pvc-xyz  1Gi        RWX            rook-cephfs    1m

Step 6: Deploy a Test Pod Using CephFS Storage

Create cephfs-test-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: cephfs-app
spec:
  containers:
  - name: app
    image: busybox
    command: ["/bin/sh", "-c", "sleep 3600"]
    volumeMounts:
    - mountPath: "/mnt"
      name: mypvc
  volumes:
  - name: mypvc
    persistentVolumeClaim:
      claimName: my-cephfs-pvc
Click Here to Copy YAML

Deploy the Test Pod

kubectl apply -f cephfs-test-pod.yaml

Check if the pod is running:

kubectl get pods

Troubleshooting Issues

Issue: CephCluster Stuck in “Progressing” State

Run:

kubectl -n rook-ceph describe cephcluster my-ceph-cluster

If you see an error like:

failed the ceph version check: the version does not meet the minimum version "18.2.0-0 reef"

Solution:

  1. Ensure you are using Ceph v18.2.0 in ceph-cluster.yaml.
  2. Restart the Ceph operator: kubectl -n rook-ceph delete pod -l app=rook-ceph-operator

Issue: CephCluster Doesn’t Delete

If kubectl delete cephcluster hangs, remove finalizers:

kubectl -n rook-ceph patch cephcluster my-ceph-cluster --type='merge' -p '{"metadata":{"finalizers":null}}'

Then force delete:

kubectl -n rook-ceph delete cephcluster my-ceph-cluster --wait=false

Conclusion

Setting up Rook-Ceph on Minikube provides a robust storage solution within Kubernetes. However, issues such as Ceph version mismatches and stuck deletions are common. By following this guide, you can successfully deploy CephFS, configure a persistent storage layer, and troubleshoot issues effectively.

Would you like a deep dive into RBD or Object Storage in Rook-Ceph next? Let me know in the comments!👇

Leave a comment