5.3. User-Provisioned Infrastructure
Table of Contents
Key Characteristics of User-Provisioned Infrastructure (UPI)
In a User-Provisioned Infrastructure (UPI) deployment, you — not the OpenShift installer — are responsible for creating, configuring, and managing the underlying infrastructure on which OpenShift runs. The installer only lays down the OpenShift control plane and worker nodes on top of resources you have already prepared.
In contrast to Installer-Provisioned Infrastructure (IPI), UPI:
- Does not create or manage infrastructure resources (VMs, load balancers, DNS, etc.) for you.
- Relies on pre-existing networking, storage, and compute.
- Gives you full control over architecture, topology, and integration with existing environments.
- Often requires more manual work and expertise, but allows highly customized and non-standard deployments.
Common reasons to choose UPI:
- Strict enterprise governance and change-control processes.
- Existing on-prem data centers or private clouds with their own automation.
- Special networking, security, or storage requirements.
- Unsupported or partially supported platforms where you must tailor everything.
Supported Platforms and Typical Use Cases
UPI is supported on most major platforms where OpenShift runs. Typical environments include:
- Bare metal data centers
- VMware vSphere and other virtualization platforms
- Cloud providers (AWS, Azure, GCP, etc.) where you need to integrate with existing network/security layout rather than use IPI patterns
- Nutanix and other private cloud stacks
- Edge and remote sites with constrained or custom infrastructure
Common use cases:
- Integrating OpenShift into an existing corporate network with pre-defined VLANs, firewalls, and routing.
- Using proprietary load balancers, storage arrays, or network security appliances.
- Meeting compliance requirements that prohibit the installer from creating infrastructure.
- Migrating from an existing cluster or sharing infrastructure with other platforms.
UPI Deployment Workflow Overview
UPI follows a more manual, multi-step process. While details differ by platform, the general workflow is:
- Prepare DNS, networking, and certificates
- Create the required DNS records:
api.<cluster>.<domain>(control plane API)*.apps.<cluster>.<domain>(routes/wildcard for applications)- Ensure network connectivity for:
- Control plane <-> workers
- Nodes <-> registry, mirrors, and external dependencies
- Plan and implement load balancing for:
- API (
:6443) to control plane nodes - Machine Config Server (
:22623) if applicable - Optional: Ingress controllers (HTTP/HTTPS)
- Prepare compute resources
- Create VMs or bare-metal servers for:
- Bootstrap node (temporary)
- Control plane (master) nodes
- Worker nodes
- Allocate CPU, memory, and storage according to sizing guidance.
- Attach appropriate networks (management, storage, application, etc.).
- Generate installation assets
- Use
openshift-installto generate manifests and ignition files: install-config.yaml(input)bootstrap.ignmaster.ignworker.ign- Optionally customize manifests for:
- Network configuration
- Machine config
- Additional cluster settings
- Provision nodes with ignition
- Provide ignition files to nodes using:
- Cloud-init or platform-specific metadata services.
- ISO images or PXE boot with embedded/hosted ignition.
- Web or storage hosting for ignition files.
- Ensure each node gets the correct ignition:
- Bootstrap node uses
bootstrap.ign. - Control plane nodes use
master.ign. - Workers use
worker.ign. - Monitor bootstrap and installation
- Use
openshift-install wait-for bootstrap-completeto monitor. - Once bootstrap is complete:
- Remove the bootstrap node from load balancers.
- Decommission the bootstrap node.
- Approve node certificates and finalize cluster
- Approve pending CSRs (if required) so nodes join the cluster.
- Validate that:
- All control plane nodes are
Ready. - Worker nodes are registered and schedulable.
- Run
openshift-install wait-for install-completeto finish. - Post-install tasks
- Configure:
- Storage (StorageClasses, CSI drivers).
- Ingress controllers and TLS.
- Identity providers and RBAC.
- Integrate with monitoring, logging, and backup systems already present on your infrastructure.
Detailed Responsibilities in a UPI Deployment
Compared to IPI, you must take explicit ownership of several infrastructure components.
DNS and Load Balancing
You manage all DNS and load balancing, typically by:
- Creating A/AAAA or CNAME records for:
api.<cluster>.<domain>pointing to load balancer fronting control plane nodes.*.apps.<cluster>.<domain>pointing to ingress router endpoints (either load balancer or node IPs).- Configuring load balancers:
- For the API: layer 4 TCP load balancing over port 6443 to all control plane nodes.
- For Machine Config Server: port 22623 to control plane nodes during install.
- For Ingress: layer 4 or layer 7 for HTTP/HTTPS, targeting worker nodes running ingress pods.
You must also handle:
- Health checks and failover policies.
- TLS offload decisions (PASSTHROUGH vs EDGE vs REENCRYPT at ingress).
- HA for the load balancer itself if needed.
Compute and Operating System Preparation
You create and lifecycle-manage the machines that form the cluster:
- VM templates or golden images:
- Preload the supported RHCOS or RHEL version (depending on method).
- Ensure required firmware, virtualization features, and drivers.
- Bare metal servers:
- Configure BIOS/firmware for virtualization extensions, boot order, and PXE if used.
- Integrate with provisioning systems (e.g., PXE, IPMI, Redfish).
- Sizing and placement:
- Place control plane and worker nodes across hypervisors, racks, or availability zones for resilience.
- Plan NUMA, CPU pinning, or huge pages if needed for special workloads.
Networking and Security Integration
You must integrate OpenShift networking into your environment:
- Define VLANs or network segments for:
- Node management.
- Application traffic.
- Storage traffic.
- Configure:
- Firewalls to allow internal and external communication needed by OpenShift.
- Routing between cluster networks and corporate networks where appropriate.
- Integrate with existing:
- Proxies or egress gateways.
- Network security tools (IPS/IDS, firewalls, microsegmentation solutions).
Cluster networking itself (SDN/OVN-Kubernetes, etc.) is configured by OpenShift, but its integration with the broader environment is your responsibility.
Storage and Persistent Volumes
Unlike IPI, UPI often requires you to:
- Deploy and configure storage backends:
- External SAN/NAS (e.g., NFS, iSCSI, Fibre Channel).
- CSI drivers for enterprise storage arrays.
- Cloud-native storage if in a private cloud.
- Create and tune StorageClasses for:
- Default workloads.
- Performance-sensitive or compliance-sensitive workloads.
- Ensure:
- Proper multipathing, zoning, and access controls.
- Redundancy and backup policies consistent with enterprise standards.
Bootstrap Process Management
The bootstrap node plays a central role in cluster creation but is temporary:
- You provision the bootstrap node with:
- Correct ignition (
bootstrap.ign). - Network access to control plane nodes and required external services.
- You add the bootstrap node to:
- Load balancing for API and possibly Machine Config endpoints.
- After
bootstrap-complete: - Remove it from API/MCS load balancing.
- Shut down and delete the bootstrap resources.
Managing this lifecycle cleanly is specific to UPI and is often automated with scripts or infrastructure-as-code.
Infrastructure-as-Code and Automation in UPI
Because UPI is highly manual by default, teams often standardize it with automation:
- Terraform/Ansible/PowerCLI/others to:
- Create VMs and configure networks.
- Set up load balancers, DNS, and firewall rules.
- Provision bootstrap/control-plane/worker nodes in repeatable patterns.
- Version-controlled configuration:
install-config.yaml, ignition sources, platform-specific config all live in Git.- Changes to topology or parameters are reviewed and tracked.
- Reusable modules/roles:
- Common patterns for multi-cluster deployments.
- Shared modules for DNS, load balancers, and storage integration.
This approach makes UPI clusters more reproducible and maintainable, especially at scale.
Comparison: When to Choose UPI vs IPI
While an earlier chapter covers deployment options conceptually, from a practical standpoint you typically choose UPI when:
- You must integrate with:
- Existing load balancers, DNS, and IPAM.
- Complex or non-standard network/security models.
- You want maximum control over:
- Node placement, sizing, and hardware features.
- Storage architecture and array usage.
- Upgrade, backup, and monitoring integration patterns.
- You operate on platforms where:
- IPI is not available or not aligned with your enterprise standards.
You trade off:
- Pros:
- Full flexibility and deep integration with legacy or specialized infrastructure.
- Alignment with strict governance and compliance.
- Ability to standardize on existing tooling.
- Cons:
- More initial effort and complexity.
- Need for strong infrastructure and automation skills.
- More responsibility for correctness and reliability of each layer.
Common Pitfalls and Practical Tips for UPI
Some frequent issues and mitigations specific to UPI:
- Incorrect or missing DNS entries
- Symptom: API unreachable, installer timeouts.
- Mitigation: Validate with
dig/nslookup; ensure both forward and reverse lookup where required. - Misconfigured load balancers
- Symptom: API connectivity flaps; cluster install hangs.
- Mitigation: Confirm ports 6443 and 22623 targets and health checks; avoid L7 interference with TLS handshake for API.
- Firewall blocks
- Symptom: Nodes not joining cluster; image pulls failing.
- Mitigation: Audit all required ports between:
- Control plane/worker nodes.
- Nodes and external registries, RH repositories, or mirrors.
- Bootstrap node not cleaned up
- Symptom: Confusing extra node, extra cost/resource usage.
- Mitigation: Explicitly decommission after
bootstrap-complete. - Lack of automation
- Symptom: Repeated manual errors, difficult upgrades or rebuilds.
- Mitigation: Codify entire UPI flow with automation tools, even for small environments.
Planning, documenting, and automating your UPI design significantly reduces operational risk and makes the deployment model sustainable over time.
Views: 0
Where to Move
Move chapter:
- ☰ 1. Course Overview
- ☰ 1.1. What is OpenShift
- ☰ 1.2. Why OpenShift is used in modern IT and cloud environments
- ☰ 1.3. OpenShift vs Kubernetes
- ☰ 1.4. Typical OpenShift use cases
- ☰ 1.5. Structure and goals of the course
-
- ☰ 2. Containers and Cloud-Native Fundamentals
- ☰ 2.1. What is a container
- ☰ 2.2. Containers vs virtual machines
- ☰ 2.3. Introduction to Docker and container images
- ☰ 2.4. Container registries
- ☰ 2.5. Basics of cloud-native applications
-
- ☰ 3. Kubernetes Fundamentals
- ☰ 3.1. What is Kubernetes
- ☰ 3.2. Kubernetes architecture
- ☰ 3.3. Control plane components
- ☰ 3.4. Worker node components
- ☰ 3.5. Kubernetes objects and resources
- ☰ 3.6. Declarative configuration model
-
- ☰ 4. OpenShift Architecture
- ☰ 4.1. Overview of OpenShift architecture
- ☰ 4.2. Control plane in OpenShift
- ☰ 4.3. Nodes and machine roles
- ☰ 4.4. OpenShift Operators
- ☰ 4.5. OpenShift networking model
- ☰ 4.6. OpenShift storage model
-
- ☰ 5. OpenShift Installation and Deployment Models
- ☰ 5.1. OpenShift deployment options
- ☰ 5.2. Installer-Provisioned Infrastructure
- ☰ 5.3. User-Provisioned Infrastructure
- ☰ 5.4. Managed OpenShift services
- ☰ 5.5. Single-node OpenShift
- ☰ 5.6. Cluster lifecycle management
-
- ☰ 6. Accessing and Using OpenShift
- ☰ 6.1. OpenShift web console
- ☰ 6.2. OpenShift CLI (oc)
- ☰ 6.3. Authentication and authorization
- ☰ 6.4. Projects and namespaces
- ☰ 6.5. Resource quotas and limits
-
- ☰ 7. Building and Deploying Applications
- ☰ 7.1. Application deployment concepts
- ☰ 7.2. Source-to-Image (S2I)
- ☰ 7.3. Container-based deployments
- ☰ 7.4. DeploymentConfigs and Deployments
- ☰ 7.5. Rolling updates and rollbacks
-
- ☰ 8. Networking in OpenShift
- ☰ 8.1. Services and service discovery
- ☰ 8.2. Routes and ingress
- ☰ 8.3. Network policies
- ☰ 8.4. Load balancing
- ☰ 8.5. External access to applications
-
- ☰ 9. Storage in OpenShift
- ☰ 9.1. Persistent storage concepts
- ☰ 9.2. Persistent Volumes
- ☰ 9.3. Persistent Volume Claims
- ☰ 9.4. StorageClasses
- ☰ 9.5. Dynamic provisioning
- ☰ 9.6. Stateful applications
-
- ☰ 10. Configuration and Secrets Management
- ☰ 10.1. ConfigMaps
- ☰ 10.2. Secrets
- ☰ 10.3. Environment variables
- ☰ 10.4. Managing sensitive data
- ☰ 10.5. Best practices for configuration
-
- ☰ 11. Scaling and High Availability
- ☰ 11.1. Horizontal pod autoscaling
- ☰ 11.2. Vertical scaling
- ☰ 11.3. Self-healing and pod restarts
- ☰ 11.4. High availability concepts
- ☰ 11.5. Multi-zone and multi-region clusters
-
- ☰ 12. Monitoring, Logging, and Observability
- ☰ 12.1. Built-in monitoring stack
- ☰ 12.2. Metrics and alerts
- ☰ 12.3. Logging architecture
- ☰ 12.4. Distributed tracing
- ☰ 12.5. Troubleshooting applications
-
- ☰ 13. Security in OpenShift
- ☰ 13.1. Security model overview
- ☰ 13.2. Role-Based Access Control
- ☰ 13.3. Security Context Constraints
- ☰ 13.4. Image security and scanning
- ☰ 13.5. Network security
- ☰ 13.6. Compliance and auditing
-
- ☰ 14. CI/CD and DevOps with OpenShift
- ☰ 14.1. DevOps concepts
- ☰ 14.2. CI/CD pipelines overview
- ☰ 14.3. OpenShift Pipelines (Tekton)
- ☰ 14.4. GitOps with Argo CD
- ☰ 14.5. Integrating OpenShift with external CI tools
-
- ☰ 15. Operators and Platform Services
- ☰ 15.1. What are Operators
- ☰ 15.2. Operator Lifecycle Manager
- ☰ 15.3. Installing and managing Operators
- ☰ 15.4. Common platform Operators
- ☰ 15.5. Custom Operators
-
- ☰ 16. Upgrades, Maintenance, and Operations
- ☰ 16.1. OpenShift upgrade process
- ☰ 16.2. Backup and restore
- ☰ 16.3. Node maintenance
- ☰ 16.4. Cluster troubleshooting
- ☰ 16.5. Capacity planning
-
- ☰ 17. OpenShift on HPC and Specialized Workloads
- ☰ 17.1. Running batch workloads on OpenShift
- ☰ 17.2. OpenShift and GPUs
- ☰ 17.3. MPI and parallel workloads in containers
- ☰ 17.4. Hybrid HPC and cloud workflows
-
- ☰ 18. OpenShift in Practice
- ☰ 18.1. Typical OpenShift workflows
- ☰ 18.2. Development to production lifecycle
- ☰ 18.3. Case studies from industry
- ☰ 18.4. Common pitfalls and best practices
-
- ☰ 19. Future Directions and Ecosystem
- ☰ 19.1. Kubernetes ecosystem overview
- ☰ 19.2. OpenShift roadmap
- ☰ 19.3. Serverless and event-driven workloads
- ☰ 19.4. AI and data platforms on OpenShift
-
- ☰ 20. Final Project and Hands-On Exercises
- ☰ 20.1. Deploying a containerized application
- ☰ 20.2. Implementing CI/CD with OpenShift
- ☰ 20.3. Securing and scaling an application
- ☰ 20.4. Monitoring and troubleshooting report
-
Comments
Please login to add a comment.
Don't have an account? Register now!