GitOps

GitOps is a modern approach to managing and automating infrastructure and application deployments using Git as the single source of truth. It combines the power of version control with declarative infrastructure and application definitions to provide a reliable and auditable way to manage and deploy changes.

With GitOps, all configuration and deployment instructions are stored in a Git repository, allowing teams to track changes, collaborate, and roll back deployments easily. This approach promotes a Git-centric workflow, where changes made to the repository trigger automated deployments, ensuring consistency and reproducibility across environments.

RAIL is leveraging a PULL-style GitOps model, where the cluster pulls the desired state from a Git repository and applies it to the cluster. This model is in contrast to a PUSH-style model, where a CI/CD system pushes changes to the cluster.

The GitOps Repository

The GitOps Repository is a Git repository that contains all the configuration and deployment instructions for a team’s applications. It is the single source of truth for the team’s infrastructure and application deployments. The repository is organized into directories, with a base directory for the application and resources available for all clusters, and a directory for each cluster where the application is deployed. Using Kustomize, Flux will monitor the subdirectories for each cluster and apply the configuration accordingly.

The GitOps Repository is organized into the following directories:

├── .sops.pub.asc                   # A public PGP key you can use to sign secrets that will be decoded in the clusters by SOPS (mozilla Secret   OPerationS)
├── .sops.yaml                      # A configuration file that tells SOPS which values to encrypt
├── base                            # This level contains apps and resources that's available for use in all clusters
│   ├── example-from-docs           # This level represents this app and its set of resources. It must include kuztomization.yaml that includes   the other manifest files
│   │   ├── deployment.yaml
│   │   ├── ingress.yaml
│   │   ├── kustomization.yaml      # This file includes the other manifest files
│   │   └── service.yaml
│   └── README.md                   # Love your READMEs, and feed and exercise them regulary and keep them lean.
├── osl1-test                       # This directory level representes the configuration thats unique for a specific cluster
│   ├── example-from-docs           # This directory level represents configuration for this app in the osl1-test cluster
│   └── RAIL                        # This is where the fluxcd controller finds deployments for this cluster
├── README.md
└── bgo1-prod                       # You guessed it, this is another cluster, for your production workloads
    ├── RAIL                        # ...and this is where the fluxcd controller finds deployments for this specific cluster
    │    └── some_app
    │        └── kustomization.yaml # Define a namespace and a fluxcd kustomization here in order to deploy   some_app
    └── some_app                    #
        └── kustomization.yaml      # This file includes the base directory and the other manifest files,   enabling the resources in the cluster

Kustomizations

In order to make sense of the GitOps Repository structure we need to understand what Kubernetes Kustomizations are.

First we need to establish some basic Kubernetes terms:

Resource

A somewhat abstract concept of the stuff might be realized or represented by a Kubernetes cluster.

Resource Object

An JSON-style object that describes the desired state of a Resource as well as attributes that describe its current status. Resource Objects are what the Kubernetes API returns and can mutate from its resource endpoints. Resource Objects are identified by the triplet of attributes (kind, metadata.namespace, metadata.name). The field apiVersion is also mandatory and determines together with kind how the rest of the attributes are to be interpreted.

Resource Definition

The schema that specifies what attributes can be present for a Resource Object of a specific kind and apiVersion.

Resource Configuration

An object that represent the desired state of an identified Resource. It is customary for many resource definitions to express the desired state within the attribute named spec. The Resource Configuration is a partial Resource Object.

Manifest

An YAML file (or stream) that contains one or more YAML documents that each represent a single Resource Configuration. Multiple YAML documents in a YAML file are separated by 3 dashes "---\n".

This is the structure of each YAML document (representing a Resource Configuration) in a Manifest:

apiVersion: v1
kind: ResourceType
metadata:
  name: something
  namespace: adm-it-xxx
spec:
  ...

A Kustomization is a directory in the file system that contains a file called kustomization.yaml that contains a single YAML document with the attributes:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - manifest.yaml
  - kustomization_directory
  - ...

The resources attribute is a list of paths which can either reference a Manifest file (with .yaml extension) or another Kustomization directory. The result of this is a new Manifest containing the concatenations of all the referenced Manifests. There might be other attributes in the kustomization.yaml file that modifies attributes within the resource configurations contained or adds new resource configurations. Read the documentation for kustomize for details on how to kustomize your Manifests.

You can run the command kubectl kustomize <directory> to stream the generated Manifest to stdout.

GitOps Repository Structure

Each RAIL Team has an identifier on the form adm-it-xxx and they are given a GitOps repository and a Kubernetes namespace (in each RAIL cluster) with the same name. RAIL clusters synchronizes the resource configurations in the team’s namespace from the kustomization found in the <cluster-name>/RAIL directory in the main branch of the team’s GitOps repository.

The component within RAIL that does this synchronization is called Flux, and more specifically the Flux Kustomization Controller. Its operation and status can be monitored by watching the Flux Kustomization resource objects within your team’s namespace. Run kubectl get ks or kubectl describe ks to list it or display details.

If the <cluster-name>/RAIL directory does not contain a kustomization.yaml file, then Flux works as if it created one by running the equivalent of the command kustomize create --autodetect --recursive in that directory. The effect is that all manifests and kustomizations in that directory and all subdirectories are automatically included.

It is recommended to run each application that the RAIL Team manages in their own sub-namespace. We achieve this by setting up the sub-namespaces with their own Flux Kustomization configurations from the bootstrap kustomization:

<cluster-name>/RAIL # contains configurations for the `adm-it-xxx` namespace
<cluster-name>/RAIL/app1.yaml   # sets up the `adm-it-xxx-app1` namespace
<cluster-name>/RAIL/app2.yaml   # sets up the `adm-it-xxx-app2` namespace
<cluster-name>/app1 # contains configurations for the `adm-it-xxx-app1` namespace
<cluster-name>/app1/ingress.yaml
<cluster-name>/app1/deploy.yaml
<cluster-name>/app2 # contains configurations for the `adm-it-xxx-app2` namespace
<cluster-name>/app2/ingress.yaml
<cluster-name>/app2/deploy.yaml

In addition teams usually want to run multiple instances of the same application on the RAIL clusters without duplicating the configuration. We achieve this by moving the app configuration to the base directory and setting up kustomizations that import it and tweaks some settings for each instance.

For instance we end up with the following structure when we want to run multiple instances of app1:

base
base/app1
base/app1/kustomization.yaml
base/app1/ingress.yaml
base/app1/deploy.yaml
<cluster-name>/RAIL
<cluster-name>/RAIL/app1.yaml
<cluster-name>/RAIL/app1-test.yaml
<cluster-name>/RAIL/app2.yaml
<cluster-name>/app1
<cluster-name>/app1/kustomization.yaml # imports `../../base/app1`
<cluster-name>/app1-test
<cluster-name>/app1-test/kustomization.yaml # imports `../../base/app1`
<cluster-name>/app2
<cluster-name>/app2/ingress.yaml
<cluster-name>/app2/deploy.yaml

Protect secrets with SOPS

Secrets can be added to namespaces by creating encrypted manifests in the GitOps Repository. Secrets encrypted with the SOPS tool using the published public PGP-key will be automatically decrypted inside the RAIL-cluster by Flux.

Preparation to do before you can start using sops:

• install the gnupg package — which gives you access to the gpg-command line utility

• install the sops package — which gives you access to the sops-command line utility

• run gpg --import .sops.pub.asc from the root of the GitOps Repository

The last step makes the published public PGP-key for RAIL available when SOPS invokes gpg to encrypt variables within files. The root of the GitOps repository also contains a .sops.yaml file that tells the sops tool to encrypt files within this directory with this key.

Create a YAML-file (for instance a manifest containing a Secret) with values that you want to encrypt, like:

apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
stringData:
  APP_CLIENT_ID: ...
  APP_CLIENT_SECRET: ...

If you save this file as app-secret.yaml, then you can encrypt it by running:

sops encrypt --in-place app-secret.yaml

Check that the secrets are now replaced with a string that looks like ENC[...unreadable stuff...] and that a sops attribute was appended to the top level structure of the file. While testing run the command without the --in-place option and inspect that you get the expected output on stdout.

Commit the file to Git with the assurance that no one besides the RAIL platform can retrieve the secret, even if they gain access to the repository.

Attention

It is handy to instruct sops how to decrypt files with keys that are shared between members of your team. This allow you to change secrets in the encrypted files directly by invoking sops edit app-secret.yaml

One way to do this at UiB is to set up a shared Azure Key Vault with a key and add this line to the .sops.yaml-file at the root of your GitOps repository: azure_keyvault: https://uib-adm-it-xxx.vault.azure.net/keys/sops-rail SOPS will then manage to use this vault key after you run az login, which requires that the azure-cli package is installed to make the az command available that provide the command line interface to the Azure Cloud.

SOPS with secretGenerator

One problem with Secrets and ConfigMaps resources is that workloads that depend on them do not automatically reload when the resource change. The kustomization.yaml files have a solution for this; the secretGenerator and the configMapGenerator elements. These make kustomize generate secrets and configmaps with names that get a hash-suffix that reflects the content in these resources. The workload manifest with references to these resources are automatically patched with references to the hashed names, which cause these manifests to change, and thus the corresponding workload resources to reload, when secrets and configmaps change.

We can use SOPS with the secretGenerator with some caveats:

• You can’t use literals with encrypted content as the kustomization.yaml file itself is not decrypted before it’s used.

• For envs make the referenced files have .env extension

• For files make both the local name and the referenced file name have the same extension; one of .env, .ini, .yaml or .json is known to work.

• Encrypt the referenced .env, .ini, .yaml or .json files with sops encrypt --in-place <filename>.

Example kustomization.yaml file that demonstrates use of secretGenerator:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yaml
secretGenerator:
- name: app-files
  files:
  - foo.ini=foo.ini
  - bar.json=bar.json
- name: app-vars
  envs:
  - secrets.env

This generates two secrets named app-files-<suffix> and app-vars-<suffix>. In the deploy.yaml file you can reference these as if the names where app-files and app-vars, but the generated Kustomization will have these names patched to match the suffixed names.

You protect the secrets themselves by encrypting them with these commands:

sops encrypt --in-place foo.ini
sops encrypt --in-place bar.json
sops encrypt --in-place secrets.env

To make this work you might need to update the top level .sops.yaml file with entries that also match extensions like .ini and .env.

Remove an app from RAIL

To remove an app from RAIL all that is needed is to remove all YAML manifest files that declare the app and then push the update to the GitOps repository. In actuality only the declaration of the sub-namespace needs to be removed but for clarity it is best to remove all YAML files that belong to the app. If you have some common manifests declared in base for test and production it is best to wait removing them from base until you are ready to remove from both test and production.

If you for some reason want to delete the namespace manually, run kubectl -n adm-it-xxx delete subns adm-it-xxx-app. Be aware that if the corresponding manifest has not been removed in GitOps, then the namespace will be recreated in RAIL after a short while.