15.5. Custom Operators
Table of Contents
Why Build Custom Operators?
Custom Operators extend OpenShift and Kubernetes with domain-specific automation. While catalog Operators (from Red Hat or vendors) cover common platforms and services, custom Operators let you:
- Encode your organization’s operational knowledge (runbooks, SOPs) as code.
- Provide a self-service API for internal platforms (databases, ML platforms, specialized HPC services).
- Standardize deployment patterns so development teams consume higher-level abstractions instead of raw Kubernetes primitives.
In short, a custom Operator turns your “how we run X in production” into a native OpenShift resource that anyone can create and manage declaratively.
Typical Use Cases for Custom Operators
Some common patterns where custom Operators are valuable:
- Internal platforms and PaaS features
- “Internal DB-as-a-Service” (PostgreSQL, MySQL, Redis, etc.).
- Managed message brokers, caches, or search (Kafka, RabbitMQ, ElasticSearch).
- Company-standard application stacks (e.g., Java microservice with sidecars, logging, and monitoring baked in).
- Complex, stateful, or lifecycle-heavy applications
- Distributed databases and storage systems.
- Clustered middleware (e.g., identity providers, API gateways).
- Systems requiring careful rolling updates, backup/restore, or scale operations.
- Environment and compliance enforcement
- Automatic injection of organization-required security settings, sidecars, labels, or network policies.
- Enforcement of naming conventions or deployment policies via admission-style logic.
- HPC and data/AI workloads
- Provisioning of GPU-enabled job environments.
- MPI cluster launchers or job orchestration flows.
- Data pipeline orchestration with specialized resource coordination.
Core Building Blocks of a Custom Operator
A custom Operator is built around a few key concepts:
Custom Resource Definitions (CRDs)
A Custom Resource Definition (CRD) adds a new API type to the cluster, such as:
DatabaseCacheClusterMpiJobModelServingEndpoint
Each CRD defines:
- API group and version (e.g.,
database.example.com/v1alpha1). - Kind (e.g.,
PostgresInstance). - Schema (fields like
spec.size,spec.storage,spec.backupPolicy).
Once the CRD is installed, users can create Custom Resources (CRs) with oc or YAML, just like native Kubernetes/OpenShift objects.
Controllers and Reconciliation
A custom Operator typically includes one or more controllers that:
- Watch for changes to:
- Custom Resources of a given kind.
- Related native resources (Pods, Deployments, PVCs, etc.).
- Reconcile actual cluster state towards the desired state from a CR’s
spec.
The reconciliation pattern usually follows:
- Trigger: A CR is created/updated/deleted or related resources change.
- Observe: The controller reads current state (CR + dependent objects).
- Decide: Determine what needs to change.
- Act: Create/update/delete Kubernetes/OpenShift objects.
- Record status: Update the CR
statusto reflect progress or failures.
Operator Packaging and Distribution
Custom Operators are typically packaged for easy installation:
- As an Operator bundle that can be managed by the Operator Lifecycle Manager (OLM).
- With associated ClusterServiceVersion (CSV) metadata describing:
- Provided APIs (CRDs).
- Required permissions (RBAC).
- Dependencies on other Operators.
- Upgrade strategy and install modes.
This packaging enables:
- Cluster-wide or namespace-scoped installation.
- Versioned upgrades through the OpenShift web console or
oc. - Integration with OperatorHub (internal or external).
Design Considerations for Custom Operators
When designing a custom Operator, there are architectural and API-level decisions that determine how usable and maintainable it will be.
API Design for Custom Resources
Think of your CRD as a product interface:
- Audience: Who will use this CRD?
- Platform engineers? Application developers? Data scientists?
- Abstraction level:
- High-level “I want a
ProductionPostgreswith 3 replicas” vs. exposing low-level tuning knobs. - Schema clarity:
- Use clear field names and types.
- Separate required vs. optional fields.
- Provide sane defaults whenever possible.
Patterns that help:
- Use
specfor desired configuration,statusfor observed state. - Include status fields such as:
status.phase(e.g.,Pending,Ready,Degraded).status.conditions(withtype,status,reason,message).- Important computed fields like connection endpoints, credentials references, etc.
Scoping and Multi-Tenancy
Decide whether your Operator:
- Runs cluster-wide (manages CRs in many namespaces).
- Runs namespace-scoped (limited to a specific project).
Consider:
- Security: Which ServiceAccounts and RBAC rules are needed?
- Multi-tenancy: Should different teams own different instances? Should quotas/limits apply per namespace?
- Resource ownership: Ensure that resources created by the Operator follow clear naming and labels for traceability.
Declarative vs. Imperative Behavior
Operators should:
- Favor declarative interfaces: “what” users want, not “how.”
- Encode the imperative steps (ordering, retries, error recovery) within the Operator logic, not in user-facing YAML.
For example, a CR:
apiVersion: database.example.com/v1
kind: PostgresInstance
metadata:
name: app-db
spec:
size: small
storage:
capacity: 100Gi
highAvailability: trueThe Operator handles:
- Creating StatefulSets, PVCs, Services.
- Coordinated rolling upgrades.
- Automatic failover procedures.
- Backups and restore hooks.
Users only set high-level goals; the Operator decides the action plan.
Implementation Approaches and Tooling
There are multiple ways to implement custom Operators; the choice affects language, tooling, and complexity.
Operator SDK
The Operator SDK (from the Operator Framework) is the standard toolkit for building Operators running on OpenShift. It supports different programming models:
- Go-based Operators
- Full control and flexibility.
- Use
controller-runtimelibraries and patterns. - Best suited for complex, performance-sensitive Operators.
- Ansible-based Operators
- Use Ansible playbooks and roles as reconciliation logic.
- Good for teams with strong Ansible expertise and existing automation.
- Lower barrier to entry for straightforward provisioning patterns.
- Helm-based Operators
- Wrap an existing Helm chart in Operator reconciliation logic.
- Useful when your application is already managed via Helm.
- Less flexible for sophisticated, stateful behavior.
Operator SDK provides:
- Project scaffolding.
- CRD/CV generation and validation.
- Local testing tools.
- Integration with OLM packaging.
Other Frameworks and Patterns
While Operator SDK is most common on OpenShift, you may also encounter:
- Kubebuilder (Go-based; underlying tech for Operator SDK).
- Java, Python, or .NET Operators
- Using language-specific libraries (e.g., Java Operator SDK, Kopf in Python).
- Appropriate if your team prefers a particular language ecosystem.
When using non-standard frameworks, ensure:
- You can containerize and deploy your controller to OpenShift.
- You still follow Kubernetes reconciliation and CRD best practices.
- You can integrate with OLM packaging if you want OperatorHub distribution.
Lifecycle Management of Custom Operators
Building an Operator is only part of the story; you must manage its full lifecycle in production.
Versioning and Upgrades
Key considerations:
- Version both CRDs and Operator logic:
- Use API versioning (
v1alpha1,v1beta1,v1) as your interface stabilizes. - Backward compatibility:
- Support older CR versions during migrations when possible.
- Provide conversion webhooks if you need in-place upgrades between API versions.
- Upgrade strategy:
- Use OLM’s channels (e.g.,
stable,fast) to roll out new versions. - Carefully test upgrade paths, especially for stateful services.
Reliability and Error Handling
A robust custom Operator should:
- Handle transient errors with retries and backoff.
- Avoid tight reconciliation loops that overload the API server.
- Clearly represent error states in CR
statusfields. - Provide actionable error messages for users (not just “failed”).
Consider:
- Idempotent operations: reconciliation should be safe to run repeatedly.
- Protective checks: guard against destructive operations (e.g., unintended data deletion on spec change).
Observability for Operators
To operate and debug custom Operators:
- Emit structured logs with correlation identifiers (e.g., CR name/namespace).
- Expose metrics:
- Reconciliation durations and error counts.
- Number of managed resources.
- Integrate with:
- Cluster logging and monitoring.
- Alerting (for persistent errors, reconciliation failures, or resource anomalies).
This allows SRE/platform teams to treat the Operator itself as a first-class, observable service.
Security and Permissions
Custom Operators require RBAC permissions to manage resources on users’ behalf. Poorly scoped permissions can introduce risk.
RBAC Scoping
Design permission sets carefully:
- Use the principle of least privilege:
- Only grant verbs and resources the Operator actually needs.
- Differentiate:
- Namespaced Operators: scoped to specific namespaces.
- Cluster-scoped Operators: restricted to specific CRDs and core resources necessary for their function.
Review:
- ServiceAccount used by the Operator.
- ClusterRole and RoleBindings granted to that ServiceAccount.
Managing Sensitive Data
If your Operator handles secrets (e.g., DB credentials, TLS keys):
- Use Kubernetes Secrets for storage, not CR
spec. - Avoid logging sensitive values.
- Ensure status fields do not expose credentials.
- Support integration with external secret management solutions where appropriate.
Example Design Pattern: Application Stack Operator
To illustrate how components fit together, consider a simplified pattern for a “WebApp” Operator that standardizes an application stack for development teams.
- CRD:
WebAppwith fields: spec.imagespec.replicasspec.databaseRefspec.ingressHost- Operator behavior:
- On
WebAppcreation: - Create
Deploymentfor the app. - Create or reference a DB CR (e.g.,
PostgresInstance). - Create a
ServiceandRoute. - On updates:
- Perform rolling updates of the Deployment.
- Adjust replicas.
- Update Route if host changes.
- On deletion:
- Clean up application Deployment, Service, and Route.
- Optionally delete or retain the DB instance based on a policy in
spec.
Developers then submit a single WebApp YAML instead of manually wiring multiple resources, while platform teams encode best practices (resource limits, labels, security profiles, logging sidecars) inside the Operator.
Best Practices for Custom Operators
When building custom Operators for OpenShift, follow these guidelines:
- Keep CRDs focused and stable
- Avoid exposing every tunable as a field; start with essential ones.
- Evolve API versions as usage grows; don’t break existing users without migration paths.
- Leverage OpenShift-native features
- Use Routes, SCCs, and OpenShift-specific objects where appropriate.
- Align with cluster-wide observability and logging.
- Integrate with OpenShift authentication/authorization patterns where relevant.
- Automate testing
- Unit tests for reconciliation logic.
- Integration tests on an ephemeral cluster (e.g., using kind, CRC, or test OpenShift clusters).
- Upgrade tests across Operator versions, particularly for stateful workflows.
- Document and support your APIs
- Provide examples and clear field descriptions.
- Explain typical failure modes and remediation steps.
- Treat custom CRDs as public contracts with your internal users.
- Plan for deprecation
- Mark fields or CR versions as deprecated when needed.
- Provide migration recommendations and tooling when APIs change.
By carefully designing, implementing, and operating custom Operators with these practices, you can turn OpenShift into a powerful, self-service platform tailored to your organization’s applications and operational standards.
Views: 0
Where to Move
Move chapter:
- ☰ 1. Course Overview
- ☰ 1.1. What is OpenShift
- ☰ 1.2. Why OpenShift is used in modern IT and cloud environments
- ☰ 1.3. OpenShift vs Kubernetes
- ☰ 1.4. Typical OpenShift use cases
- ☰ 1.5. Structure and goals of the course
-
- ☰ 2. Containers and Cloud-Native Fundamentals
- ☰ 2.1. What is a container
- ☰ 2.2. Containers vs virtual machines
- ☰ 2.3. Introduction to Docker and container images
- ☰ 2.4. Container registries
- ☰ 2.5. Basics of cloud-native applications
-
- ☰ 3. Kubernetes Fundamentals
- ☰ 3.1. What is Kubernetes
- ☰ 3.2. Kubernetes architecture
- ☰ 3.3. Control plane components
- ☰ 3.4. Worker node components
- ☰ 3.5. Kubernetes objects and resources
- ☰ 3.6. Declarative configuration model
-
- ☰ 4. OpenShift Architecture
- ☰ 4.1. Overview of OpenShift architecture
- ☰ 4.2. Control plane in OpenShift
- ☰ 4.3. Nodes and machine roles
- ☰ 4.4. OpenShift Operators
- ☰ 4.5. OpenShift networking model
- ☰ 4.6. OpenShift storage model
-
- ☰ 5. OpenShift Installation and Deployment Models
- ☰ 5.1. OpenShift deployment options
- ☰ 5.2. Installer-Provisioned Infrastructure
- ☰ 5.3. User-Provisioned Infrastructure
- ☰ 5.4. Managed OpenShift services
- ☰ 5.5. Single-node OpenShift
- ☰ 5.6. Cluster lifecycle management
-
- ☰ 6. Accessing and Using OpenShift
- ☰ 6.1. OpenShift web console
- ☰ 6.2. OpenShift CLI (oc)
- ☰ 6.3. Authentication and authorization
- ☰ 6.4. Projects and namespaces
- ☰ 6.5. Resource quotas and limits
-
- ☰ 7. Building and Deploying Applications
- ☰ 7.1. Application deployment concepts
- ☰ 7.2. Source-to-Image (S2I)
- ☰ 7.3. Container-based deployments
- ☰ 7.4. DeploymentConfigs and Deployments
- ☰ 7.5. Rolling updates and rollbacks
-
- ☰ 8. Networking in OpenShift
- ☰ 8.1. Services and service discovery
- ☰ 8.2. Routes and ingress
- ☰ 8.3. Network policies
- ☰ 8.4. Load balancing
- ☰ 8.5. External access to applications
-
- ☰ 9. Storage in OpenShift
- ☰ 9.1. Persistent storage concepts
- ☰ 9.2. Persistent Volumes
- ☰ 9.3. Persistent Volume Claims
- ☰ 9.4. StorageClasses
- ☰ 9.5. Dynamic provisioning
- ☰ 9.6. Stateful applications
-
- ☰ 10. Configuration and Secrets Management
- ☰ 10.1. ConfigMaps
- ☰ 10.2. Secrets
- ☰ 10.3. Environment variables
- ☰ 10.4. Managing sensitive data
- ☰ 10.5. Best practices for configuration
-
- ☰ 11. Scaling and High Availability
- ☰ 11.1. Horizontal pod autoscaling
- ☰ 11.2. Vertical scaling
- ☰ 11.3. Self-healing and pod restarts
- ☰ 11.4. High availability concepts
- ☰ 11.5. Multi-zone and multi-region clusters
-
- ☰ 12. Monitoring, Logging, and Observability
- ☰ 12.1. Built-in monitoring stack
- ☰ 12.2. Metrics and alerts
- ☰ 12.3. Logging architecture
- ☰ 12.4. Distributed tracing
- ☰ 12.5. Troubleshooting applications
-
- ☰ 13. Security in OpenShift
- ☰ 13.1. Security model overview
- ☰ 13.2. Role-Based Access Control
- ☰ 13.3. Security Context Constraints
- ☰ 13.4. Image security and scanning
- ☰ 13.5. Network security
- ☰ 13.6. Compliance and auditing
-
- ☰ 14. CI/CD and DevOps with OpenShift
- ☰ 14.1. DevOps concepts
- ☰ 14.2. CI/CD pipelines overview
- ☰ 14.3. OpenShift Pipelines (Tekton)
- ☰ 14.4. GitOps with Argo CD
- ☰ 14.5. Integrating OpenShift with external CI tools
-
- ☰ 15. Operators and Platform Services
- ☰ 15.1. What are Operators
- ☰ 15.2. Operator Lifecycle Manager
- ☰ 15.3. Installing and managing Operators
- ☰ 15.4. Common platform Operators
- ☰ 15.5. Custom Operators
-
- ☰ 16. Upgrades, Maintenance, and Operations
- ☰ 16.1. OpenShift upgrade process
- ☰ 16.2. Backup and restore
- ☰ 16.3. Node maintenance
- ☰ 16.4. Cluster troubleshooting
- ☰ 16.5. Capacity planning
-
- ☰ 17. OpenShift on HPC and Specialized Workloads
- ☰ 17.1. Running batch workloads on OpenShift
- ☰ 17.2. OpenShift and GPUs
- ☰ 17.3. MPI and parallel workloads in containers
- ☰ 17.4. Hybrid HPC and cloud workflows
-
- ☰ 18. OpenShift in Practice
- ☰ 18.1. Typical OpenShift workflows
- ☰ 18.2. Development to production lifecycle
- ☰ 18.3. Case studies from industry
- ☰ 18.4. Common pitfalls and best practices
-
- ☰ 19. Future Directions and Ecosystem
- ☰ 19.1. Kubernetes ecosystem overview
- ☰ 19.2. OpenShift roadmap
- ☰ 19.3. Serverless and event-driven workloads
- ☰ 19.4. AI and data platforms on OpenShift
-
- ☰ 20. Final Project and Hands-On Exercises
- ☰ 20.1. Deploying a containerized application
- ☰ 20.2. Implementing CI/CD with OpenShift
- ☰ 20.3. Securing and scaling an application
- ☰ 20.4. Monitoring and troubleshooting report
-
Comments
Please login to add a comment.
Don't have an account? Register now!