GitOps with Argo CD

Core GitOps Concepts in the Context of Argo CD

GitOps in OpenShift means that Git is the source of truth for application and (often) cluster configuration, and Argo CD is the controller that continuously reconciles the cluster to match what is in Git.

Specific to Argo CD, this implies:

• Declarative desired state
• Applications, manifests, Helm charts, and Kustomize configurations live in Git.
• Argo CD itself is configured declaratively (e.g., Application CRs).
• Continuous reconciliation
• Argo CD watches Git and the cluster.
• If the cluster drifts from Git, Argo CD attempts to bring it back to the desired state (or reports the drift).
• Pull-based delivery
• OpenShift clusters pull configuration from Git via Argo CD, rather than CI pipelines pushing manifests into the cluster.
• This improves security (clusters do not need open ingress for CI tools) and auditability.
• Single source of truth
• Git history is the audit log: who changed what, when, and why (via commit messages and pull requests).
• Rollbacks become Git operations: reverting to a previous commit and letting Argo CD reconcile.

Argo CD Architecture on OpenShift

Argo CD is typically installed into a dedicated namespace (commonly openshift-gitops when using Red Hat’s OpenShift GitOps Operator). Key components to understand:

• API server (argocd-server)
• Provides UI, CLI (argocd), and API endpoints.
• Integrates with OpenShift OAuth for authentication (via the Operator by default).
• Repository server (argocd-repo-server)
• Clones Git repositories and renders manifests (Helm, Kustomize, plain YAML, etc.).
• Handles templating so the controller works with final Kubernetes manifests.
• Application controller (argocd-application-controller)
• Core reconciliation engine.
• Compares desired state (from Git) to live state (in OpenShift cluster) and applies changes.
• Redis / caching components (implementation detail)
• Used for caching and performance; usually managed for you by the Operator.

On OpenShift, the OpenShift GitOps Operator:

• Deploys a supported Argo CD distribution.
• Sets up integration with:
• OpenShift OAuth (SSO into the Argo CD UI).
• OpenShift RBAC (namespaced and cluster-scoped control).
• Provides a GitOpsService or ArgoCD custom resource for configuring Argo CD instances.

Defining Applications with Argo CD

The central object in Argo CD is the Application custom resource. It links where to get configuration (Git, path, revision) and where to deploy it (cluster and namespace).

Basic `Application` Structure

Typical fields in an Application manifest:

• spec.source
• repoURL: Git repository URL
• targetRevision: branch, tag, or commit (e.g., main, release-1.0)
• path: directory in repo containing the manifests
• Optional tooling: helm, kustomize, directory options, etc.
• spec.destination
• server: Kubernetes API server URL (use https://kubernetes.default.svc for the in-cluster OpenShift cluster).
• namespace: target namespace for the deployment.
• spec.project
• Logical grouping for applications (Argo CD AppProject), used to define policies and permissions.
• spec.syncPolicy
• automated: enable auto-sync.
• syncOptions: behavior such as pruning resources, creating namespaces, etc.
• retry: how Argo CD retries failed syncs (if enabled).

Example: Minimal `Application` for OpenShift

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-sample-app
  namespace: openshift-gitops           # Argo CD instance namespace
spec:
  project: default
  source:
    repoURL: 'https://github.com/example-org/my-sample-app-config.git'
    targetRevision: main
    path: overlays/production
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: my-sample-app
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Key points specific to OpenShift:

• Namespace creation:
  OpenShift often requires that projects/namespaces are pre-created or managed via additional automation. syncOptions can allow Argo CD to create namespaces, but in stricter environments, namespaces might be managed separately via an “infrastructure” repo.
• Security context / SCCs:
  Application manifests must comply with OpenShift Security Context Constraints. Those are usually not managed by the app repo itself but by platform or security repos.

Managing Multiple Environments with GitOps

With Argo CD, environment separation is modeled in Git and Argo CD configuration, rather than hard-coded in CI scripts.

Common Environment Patterns

1. Separate directories in a single repo

Example structure:

   config-repo/
     base/
       deployment.yaml
       service.yaml
     overlays/
       dev/
         kustomization.yaml
         patch-resources.yaml
       staging/
         kustomization.yaml
         patch-resources.yaml
       prod/
         kustomization.yaml
         patch-resources.yaml

• Each overlay becomes an Argo CD Application targeting a different namespace (my-app-dev, my-app-staging, my-app-prod).
• Kustomize overlays can hold environment-specific settings.

2. Separate repos per environment
• my-app-config-dev.git, my-app-config-prod.git, etc.
• Sometimes used in highly regulated environments to separate access.
3. App-of-Apps pattern
• A “root” or “umbrella” Application points at a repo that defines many child Applications.
• Used to bootstrap entire environments (namespaces, operators, apps) from one Git entry point.

Example: App-of-Apps for an OpenShift Environment

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: env-prod-root
  namespace: openshift-gitops
spec:
  project: default
  source:
    repoURL: 'https://github.com/example-org/cluster-config.git'
    targetRevision: main
    path: environments/prod
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: openshift-gitops
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

The environments/prod directory typically contains YAML definitions of child Applications (e.g. apps, logging stack, monitoring, etc.).

Integrating Argo CD into OpenShift-Based CI/CD

In the broader CI/CD pipeline, Argo CD focuses on the CD part. The CI system (e.g. OpenShift Pipelines, Jenkins, GitHub Actions) and Argo CD collaborate via Git:

1. CI builds an image
• CI builds and pushes container images into a registry (e.g. ImageStream in OpenShift, external registries).
• CI updates configuration in a Git repo (e.g., update image tag in a Helm values.yaml or Kustomize patch).
2. Git commit triggers reconciliation
• The CI pipeline creates a pull request with the new image tag.
• Once the PR is reviewed and merged, the target Git branch updates.
3. Argo CD detects the Git change
• Argo CD notices the new commit.
• If auto-sync is enabled, it applies new manifests to the target environment; otherwise, a manual sync is triggered.
4. Status flows back
• Argo CD shows sync/health status in its UI and via its API/CLI.
• CI or other tools may query Argo CD to confirm that deployment succeeded.

Typical interactions with OpenShift features:

• ImageStreams: Argo CD can manage ImageStream definitions, but the actual image tags are updated by CI.
• Routes: Manifest changes to Route resources in Git are automatically applied by Argo CD, controlling external exposure.
• ConfigMaps/Secrets: Non-sensitive configuration can be Git-managed; sensitive values are often injected via other mechanisms.

Security and Access Control with Argo CD on OpenShift

With Argo CD integrated into OpenShift, security and RBAC should be designed to align with both Git and cluster permissions.

Authentication and SSO

• When installed via the OpenShift GitOps Operator:
• Argo CD uses OpenShift OAuth by default.
• Users log in using the same credentials they use for oc and the OpenShift console.

Authorization and RBAC

There are two layers:

1. Argo CD RBAC (application-level)
• Controls who can:
• Create/edit/delete Applications.
• Sync or rollback applications.
• Configuration is typically done via the Argo CD ConfigMap (argocd-rbac-cm) or via the GitOps Operator’s custom resource.
2. OpenShift RBAC (cluster-level)
• Controls what the service account used by Argo CD can do in each namespace.
• A common pattern:
• Platform team grants Argo CD service accounts edit or similar roles in selected namespaces.
• Argo CD is not given cluster-admin; permissions are scoped appropriately.

Git Credentials and Repositories

• Argo CD needs access to Git repositories:
• SSH keys or HTTPS tokens stored as Secrets.
• On OpenShift, these secrets are created in the Argo CD namespace; the Operator can assist with config.
• For private Git servers (on-prem or corporate GitHub/GitLab), network connectivity and certificates must be ensured.

Application Health, Sync Status, and Drift Detection

Argo CD provides two key status dimensions for each application:

• Sync status
• Synced: live cluster state matches Git.
• OutOfSync: resources in cluster differ (e.g., drift, manual changes, or new commit not yet applied).
• Health status
• Healthy, Progressing, Degraded, etc., based on resource readiness (Deployments, StatefulSets, etc.).

Drift detection is especially relevant in OpenShift:

• Manual modifications via oc or the web console will show as drift.
• With selfHeal: true, Argo CD can automatically revert manual changes to enforce Git as the source of truth.

Handling OpenShift-Specific Resources with GitOps

Argo CD can manage any Kubernetes-style resource, including those specific to OpenShift. Typical categories:

• OpenShift application-level resources
• Route, BuildConfig, ImageStream, DeploymentConfig, etc.
• Recommended for green-field GitOps workflows: prefer Kubernetes-native resources (Deployment) where functionally appropriate, but GitOps works with both.
• Platform and cluster configuration
• Cluster-wide operators, logging, monitoring, security policies, etc. can be managed as separate Argo CD applications (often by a platform team):
• Operator Subscriptions and OperatorGroups.
• ClusterRole, ClusterRoleBinding, or specific CRDs for platform services.
• Multi-cluster GitOps with OpenShift
• A single Argo CD instance on one OpenShift cluster can manage other clusters by adding them as destinations (spec.destination.server fields).
• Used in hub-and-spoke architectures or when managing both test and prod clusters from one GitOps control plane.

Typical Workflows for Teams Using Argo CD on OpenShift

Developer Workflow

1. Modify application code and configuration (usually in separate repos).
2. Commit code -> CI pipeline runs, builds new image.
3. CI updates config repo (image tag, environment parameters) via pull request.
4. After review and merge, Argo CD syncs changes into the target OpenShift namespace.
5. Developer verifies the deployment via:
• Argo CD UI (sync/health status).
• OpenShift console (Pods, Routes, logs).
• Application tests.

Platform / SRE Workflow

1. Define cluster configuration (logging, monitoring, security baselines) in Git.
2. Represent these as Argo CD applications (often using the App-of-Apps pattern).
3. Use Argo CD to bootstrap new OpenShift clusters:
• Install core operators.
• Create shared namespaces and quotas.
• Apply default NetworkPolicy, LimitRange, etc.
4. Manage upgrades by updating Git definitions and allowing Argo CD to roll changes out.

Best Practices for Using Argo CD in OpenShift Environments

• Separate config from code
• Maintain separate repos for application code and OpenShift/Kubernetes configuration when appropriate.
• Let CI update the config repo, rather than mixing logic and manifests.
• Use pull requests for all changes
• Even platform changes go through PRs, enabling review and compliance.
• Avoid making manual changes via oc that are not committed to Git.
• Scope Argo CD permissions
• Avoid cluster-admin for Argo CD.
• Use dedicated service accounts per environment or application group when necessary.
• Namespace and project conventions
• Standardize naming (e.g., team-app-env patterns).
• Decide which namespaces Argo CD is allowed to manage vs those reserved for platform components.
• Health checks and sync waves
• Use Argo CD’s hooks and sync phases carefully (e.g., pre-sync jobs to run migrations) if needed.
• Ensure OpenShift-specific readiness conditions (e.g., Route availability) are reflected in your monitoring and tests.
• Observability of GitOps
• Integrate Argo CD metrics and logs into your OpenShift monitoring and logging stacks.
• Watch for frequent sync failures or unsynced apps as indicators of underlying issues.

By combining Git as the single source of truth, Argo CD as the reconciliation engine, and OpenShift as the runtime platform, teams get a consistent, auditable, and automated path from configuration changes in Git to running applications and platform services in the cluster.