User-Provisioned Infrastructure

Key Characteristics of User-Provisioned Infrastructure (UPI)

In a User-Provisioned Infrastructure (UPI) deployment, you — not the OpenShift installer — are responsible for creating, configuring, and managing the underlying infrastructure on which OpenShift runs. The installer only lays down the OpenShift control plane and worker nodes on top of resources you have already prepared.

In contrast to Installer-Provisioned Infrastructure (IPI), UPI:

• Does not create or manage infrastructure resources (VMs, load balancers, DNS, etc.) for you.
• Relies on pre-existing networking, storage, and compute.
• Gives you full control over architecture, topology, and integration with existing environments.
• Often requires more manual work and expertise, but allows highly customized and non-standard deployments.

Common reasons to choose UPI:

• Strict enterprise governance and change-control processes.
• Existing on-prem data centers or private clouds with their own automation.
• Special networking, security, or storage requirements.
• Unsupported or partially supported platforms where you must tailor everything.

Supported Platforms and Typical Use Cases

UPI is supported on most major platforms where OpenShift runs. Typical environments include:

• Bare metal data centers
• VMware vSphere and other virtualization platforms
• Cloud providers (AWS, Azure, GCP, etc.) where you need to integrate with existing network/security layout rather than use IPI patterns
• Nutanix and other private cloud stacks
• Edge and remote sites with constrained or custom infrastructure

Common use cases:

• Integrating OpenShift into an existing corporate network with pre-defined VLANs, firewalls, and routing.
• Using proprietary load balancers, storage arrays, or network security appliances.
• Meeting compliance requirements that prohibit the installer from creating infrastructure.
• Migrating from an existing cluster or sharing infrastructure with other platforms.

UPI Deployment Workflow Overview

UPI follows a more manual, multi-step process. While details differ by platform, the general workflow is:

1. Prepare DNS, networking, and certificates
• Create the required DNS records:
• api.<cluster>.<domain> (control plane API)
• *.apps.<cluster>.<domain> (routes/wildcard for applications)
• Ensure network connectivity for:
• Control plane <-> workers
• Nodes <-> registry, mirrors, and external dependencies
• Plan and implement load balancing for:
• API (:6443) to control plane nodes
• Machine Config Server (:22623) if applicable
• Optional: Ingress controllers (HTTP/HTTPS)
2. Prepare compute resources
• Create VMs or bare-metal servers for:
• Bootstrap node (temporary)
• Control plane (master) nodes
• Worker nodes
• Allocate CPU, memory, and storage according to sizing guidance.
• Attach appropriate networks (management, storage, application, etc.).
3. Generate installation assets
• Use openshift-install to generate manifests and ignition files:
• install-config.yaml (input)
• bootstrap.ign
• master.ign
• worker.ign
• Optionally customize manifests for:
• Network configuration
• Machine config
• Additional cluster settings
4. Provision nodes with ignition
• Provide ignition files to nodes using:
• Cloud-init or platform-specific metadata services.
• ISO images or PXE boot with embedded/hosted ignition.
• Web or storage hosting for ignition files.
• Ensure each node gets the correct ignition:
• Bootstrap node uses bootstrap.ign.
• Control plane nodes use master.ign.
• Workers use worker.ign.
5. Monitor bootstrap and installation
• Use openshift-install wait-for bootstrap-complete to monitor.
• Once bootstrap is complete:
• Remove the bootstrap node from load balancers.
• Decommission the bootstrap node.
6. Approve node certificates and finalize cluster
• Approve pending CSRs (if required) so nodes join the cluster.
• Validate that:
• All control plane nodes are Ready.
• Worker nodes are registered and schedulable.
• Run openshift-install wait-for install-complete to finish.
7. Post-install tasks
• Configure:
• Storage (StorageClasses, CSI drivers).
• Ingress controllers and TLS.
• Identity providers and RBAC.
• Integrate with monitoring, logging, and backup systems already present on your infrastructure.

Detailed Responsibilities in a UPI Deployment

Compared to IPI, you must take explicit ownership of several infrastructure components.

DNS and Load Balancing

You manage all DNS and load balancing, typically by:

• Creating A/AAAA or CNAME records for:
• api.<cluster>.<domain> pointing to load balancer fronting control plane nodes.
• *.apps.<cluster>.<domain> pointing to ingress router endpoints (either load balancer or node IPs).
• Configuring load balancers:
• For the API: layer 4 TCP load balancing over port 6443 to all control plane nodes.
• For Machine Config Server: port 22623 to control plane nodes during install.
• For Ingress: layer 4 or layer 7 for HTTP/HTTPS, targeting worker nodes running ingress pods.

You must also handle:

• Health checks and failover policies.
• TLS offload decisions (PASSTHROUGH vs EDGE vs REENCRYPT at ingress).
• HA for the load balancer itself if needed.

Compute and Operating System Preparation

You create and lifecycle-manage the machines that form the cluster:

• VM templates or golden images:
• Preload the supported RHCOS or RHEL version (depending on method).
• Ensure required firmware, virtualization features, and drivers.
• Bare metal servers:
• Configure BIOS/firmware for virtualization extensions, boot order, and PXE if used.
• Integrate with provisioning systems (e.g., PXE, IPMI, Redfish).
• Sizing and placement:
• Place control plane and worker nodes across hypervisors, racks, or availability zones for resilience.
• Plan NUMA, CPU pinning, or huge pages if needed for special workloads.

Networking and Security Integration

You must integrate OpenShift networking into your environment:

• Define VLANs or network segments for:
• Node management.
• Application traffic.
• Storage traffic.
• Configure:
• Firewalls to allow internal and external communication needed by OpenShift.
• Routing between cluster networks and corporate networks where appropriate.
• Integrate with existing:
• Proxies or egress gateways.
• Network security tools (IPS/IDS, firewalls, microsegmentation solutions).

Cluster networking itself (SDN/OVN-Kubernetes, etc.) is configured by OpenShift, but its integration with the broader environment is your responsibility.

Storage and Persistent Volumes

Unlike IPI, UPI often requires you to:

• Deploy and configure storage backends:
• External SAN/NAS (e.g., NFS, iSCSI, Fibre Channel).
• CSI drivers for enterprise storage arrays.
• Cloud-native storage if in a private cloud.
• Create and tune StorageClasses for:
• Default workloads.
• Performance-sensitive or compliance-sensitive workloads.
• Ensure:
• Proper multipathing, zoning, and access controls.
• Redundancy and backup policies consistent with enterprise standards.

Bootstrap Process Management

The bootstrap node plays a central role in cluster creation but is temporary:

• You provision the bootstrap node with:
• Correct ignition (bootstrap.ign).
• Network access to control plane nodes and required external services.
• You add the bootstrap node to:
• Load balancing for API and possibly Machine Config endpoints.
• After bootstrap-complete:
• Remove it from API/MCS load balancing.
• Shut down and delete the bootstrap resources.

Managing this lifecycle cleanly is specific to UPI and is often automated with scripts or infrastructure-as-code.

Infrastructure-as-Code and Automation in UPI

Because UPI is highly manual by default, teams often standardize it with automation:

• Terraform/Ansible/PowerCLI/others to:
• Create VMs and configure networks.
• Set up load balancers, DNS, and firewall rules.
• Provision bootstrap/control-plane/worker nodes in repeatable patterns.
• Version-controlled configuration:
• install-config.yaml, ignition sources, platform-specific config all live in Git.
• Changes to topology or parameters are reviewed and tracked.
• Reusable modules/roles:
• Common patterns for multi-cluster deployments.
• Shared modules for DNS, load balancers, and storage integration.

This approach makes UPI clusters more reproducible and maintainable, especially at scale.

Comparison: When to Choose UPI vs IPI

While an earlier chapter covers deployment options conceptually, from a practical standpoint you typically choose UPI when:

• You must integrate with:
• Existing load balancers, DNS, and IPAM.
• Complex or non-standard network/security models.
• You want maximum control over:
• Node placement, sizing, and hardware features.
• Storage architecture and array usage.
• Upgrade, backup, and monitoring integration patterns.
• You operate on platforms where:
• IPI is not available or not aligned with your enterprise standards.

You trade off:

• Pros:
• Full flexibility and deep integration with legacy or specialized infrastructure.
• Alignment with strict governance and compliance.
• Ability to standardize on existing tooling.
• Cons:
• More initial effort and complexity.
• Need for strong infrastructure and automation skills.
• More responsibility for correctness and reliability of each layer.

Common Pitfalls and Practical Tips for UPI

Some frequent issues and mitigations specific to UPI:

• Incorrect or missing DNS entries
• Symptom: API unreachable, installer timeouts.
• Mitigation: Validate with dig/nslookup; ensure both forward and reverse lookup where required.
• Misconfigured load balancers
• Symptom: API connectivity flaps; cluster install hangs.
• Mitigation: Confirm ports 6443 and 22623 targets and health checks; avoid L7 interference with TLS handshake for API.
• Firewall blocks
• Symptom: Nodes not joining cluster; image pulls failing.
• Mitigation: Audit all required ports between:
• Control plane/worker nodes.
• Nodes and external registries, RH repositories, or mirrors.
• Bootstrap node not cleaned up
• Symptom: Confusing extra node, extra cost/resource usage.
• Mitigation: Explicitly decommission after bootstrap-complete.
• Lack of automation
• Symptom: Repeated manual errors, difficult upgrades or rebuilds.
• Mitigation: Codify entire UPI flow with automation tools, even for small environments.

Planning, documenting, and automating your UPI design significantly reduces operational risk and makes the deployment model sustainable over time.