Skip to main content

Platform Team Workflow

Purpose: For platform engineers, documents the PR-based GitOps workflow for managing deployments, including tag management, overlay structure, environment promotion, and rollback.

Prerequisites​

  • openCenter cluster with FluxCD reconciling from the customer GitOps repository
  • Git access to the customer repository (push to feature branches, merge to main)
  • opencenter CLI installed
  • flux CLI installed

Repository Structure​

The customer GitOps repository uses a two-tier overlay model:

applications/
├── base/                          # ← from openCenter-gitops-base (read-only)
│   └── services/
│       ├── cert-manager/
│       ├── kyverno/
│       └── ...
└── overlays/
    ├── dev-cluster/               # ← cluster-specific overrides
    │   └── services/
    │       ├── fluxcd/            # Kustomization resources
    │       ├── sources/           # GitRepository sources
    │       └── cert-manager/     # Helm value overrides
    ├── staging-cluster/
    └── prod-cluster/

FluxCD on each cluster reconciles from applications/overlays/<cluster-name>/.

PR-Based Workflow​

All changes to the cluster state go through pull requests. Direct pushes to main are blocked.

1. Create feature branch
2. Make changes (add service, modify values, update config)
3. Run local validation: opencenter cluster validate --manifests
4. Push branch and open PR
5. CI validates manifests (kubeconform, kustomize build)
6. Peer review
7. Merge to main
8. FluxCD reconciles within 1–5 minutes

gitops-base Tag Management​

The openCenter-gitops-base repository provides base manifests. The customer repository references it via a GitRepository source:

# applications/overlays/<cluster>/services/sources/opencenter-base.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: opencenter-gitops-base
  namespace: flux-system
spec:
  interval: 15m
  url: ssh://git@github.com/opencenter-cloud/openCenter-gitops-base.git
  ref:
    tag: 2026.01    # ← Pin to a release tag

To upgrade:

  1. Review the release notes for the target tag.
  2. Update the tag field in the GitRepository source.
  3. Open a PR. CI validates that the new base is compatible.
  4. Merge. FluxCD pulls the new base and reconciles.

Promoting Changes Across Environments​

Promotion follows the branch-per-environment model:

# 1. Develop in dev overlay
git checkout -b feat/add-redis
# Edit applications/overlays/dev-cluster/services/redis/...
git add . && git commit -m "feat: add Redis to dev"
git push -u origin feat/add-redis
# Open PR → merge to main → dev cluster reconciles

# 2. Promote to staging
git checkout -b promote/redis-staging
cp -r applications/overlays/dev-cluster/services/redis \
      applications/overlays/staging-cluster/services/redis
# Adjust values (replicas, resource limits) for staging
git add . && git commit -m "feat: promote Redis to staging"
git push -u origin promote/redis-staging
# Open PR → merge → staging cluster reconciles

# 3. Promote to production
git checkout -b promote/redis-prod
cp -r applications/overlays/staging-cluster/services/redis \
      applications/overlays/prod-cluster/services/redis
# Adjust values for production (HA replicas, production secrets)
git add . && git commit -m "feat: promote Redis to production"
git push -u origin promote/redis-prod
# Open PR → merge → prod cluster reconciles

Rollback Procedures​

# Find the commit that introduced the bad change
git log --oneline applications/overlays/prod-cluster/

# Revert the specific commit
git revert <commit-hash>
git push origin main

# FluxCD reconciles the reverted state within the next interval
flux reconcile kustomization customer-app --with-source

Option 2: Pin to Previous gitops-base Tag​

If a base upgrade caused issues:

# Revert the tag in the GitRepository source
spec:
  ref:
    tag: 2025.12    # ← Previous known-good tag

Option 3: Suspend Reconciliation (Emergency)​

# Stop FluxCD from applying changes while you investigate
flux suspend kustomization <name> -n flux-system

# Fix the issue in Git
# ...

# Resume
flux resume kustomization <name> -n flux-system

Verification​

After any change merges:

# Check all Kustomizations are Ready
kubectl get kustomizations -n flux-system

# Check for failed HelmReleases
kubectl get helmreleases -A --field-selector=status.conditions[0].type!=Ready

# View reconciliation events
flux events --for Kustomization/<name>

Troubleshooting​

SymptomCauseFix
Kustomization stuck ReconcilingInvalid manifest in overlayRun kustomize build applications/overlays/<cluster>/ locally
HelmRelease UpgradeFailedIncompatible Helm values after base upgradeCheck flux events --for HelmRelease/<name> and fix values
Changes not appearingWrong overlay path or FluxCD pausedVerify spec.path in Kustomization matches directory structure
Drift revertedManual kubectl change overwritten by FluxCDCommit the desired state to Git instead