DRG infrastructure

Infrastructure is split into shared resources and independent services.

Environment configuration

Environment configuration is assembled from every .yaml and .yml file under environments/<environment>/, in lexical order. Each file must contain unique top-level sections; duplicate sections fail validation. The homelab OpenStack settings live in environments/homelab/openstack/, while the remaining shared environment settings are kept in platform.yaml.

To validate or materialize the normalized configuration locally:

python3 shared/scripts/environment_config.py --environment homelab --check
python3 shared/scripts/environment_config.py --environment homelab

The generated file is written to .runtime/<environment>/platform.yaml and is not committed. Kolla globals are generated from this normalized configuration at runtime; platform/openstack/kolla/globals.yml is not a source file.

Layout

shared/
  ansible/      Reusable Ansible roles
  scripts/      Common helper scripts
  terraform/    Reusable Terraform modules

services/
  gitlab/
    terraform/  GitLab-only Terraform root
    ansible/    GitLab-only Ansible playbook
  netbox/
    terraform/  NetBox-only Terraform root
    ansible/    NetBox-only Ansible playbook and SDN bootstrap
  traefik/
    terraform/  Traefik-only Terraform root
    ansible/    Traefik-only Ansible playbook
  dockerhub/
    terraform/  Docker Hub cache-only Terraform root
    ansible/    Docker Hub cache-only Ansible playbook
  etcd/
    terraform/  etcd quorum Terraform root
    ansible/    etcd quorum Ansible playbook
  postgresql/
    terraform/  PostgreSQL cluster Terraform root
    ansible/    PostgreSQL cluster Ansible playbook
  redis/
    terraform/  Redis HA cluster Terraform root
    ansible/    Redis HA cluster Ansible playbook
  openbao/
    terraform/  OpenBao-only Terraform root
    ansible/    OpenBao-only Ansible playbook
  freeipa/
    terraform/  FreeIPA-only Terraform root
    ansible/    FreeIPA-only Ansible playbook
  keycloak/
    terraform/  Keycloak-only Terraform root
    ansible/    Keycloak-only Ansible playbook
  oauth2_proxy/
    terraform/  OAuth2 Proxy-only Terraform root
    ansible/    OAuth2 Proxy-only Ansible playbook
  forgejo_runner/
    terraform/  Forgejo Runner-only Terraform root
    ansible/    Forgejo Runner-only Ansible playbook
  jenkins/
    terraform/  Jenkins controller Terraform root
    ansible/    Jenkins controller Ansible playbook
  jenkins_agent/
    terraform/  Jenkins agent Terraform root
    ansible/    Jenkins agent Ansible playbook
  llm/
    terraform/  GPU-passthrough LLM VM Terraform root
    ansible/    Debian/vLLM guest playbook
  headscale/
    terraform/  Headscale control-server Terraform root
    ansible/    Headscale Compose deployment and OIDC configuration

Run Terraform/OpenTofu from a service directory. Run Ansible from the repository root.

NetBox is the IPAM and dynamic inventory source for service LXC containers. Service Terraform roots register all VM interfaces and IP addresses in NetBox; NetBox-managed addresses are allocated from NetBox prefixes, while legacy/static addresses are recorded for audit and drift detection.

Service Terraform roots use the PostgreSQL pg backend. Legacy states that still exist in GitLab managed state can be migrated with the manually triggered terraform_state_migration child pipeline. They no longer generate services/*/ansible/ansible.env; Ansible reads settings from environment variables through shared/ansible/inventory/netbox.yml.

Application variables for the NetBox dynamic inventory live next to it:

• shared/ansible/inventory/group_vars/all.yml contains common SSH/Python settings.
• shared/ansible/inventory/host_vars/*.yml contains per-service settings.

Those files read values from environment variables and keep local defaults for non-secret settings. GitLab CI/CD and Jenkins pass those values from protected CI secrets.

CI/CD migration map

Jenkins is the CI orchestrator, while Dagger executes validation, plans, and applies in reproducible containers. Forgejo remains the Git host and pull-request review system. GitLab remains available for the current coexistence period. The repository-owned pipeline responsibilities map as follows:

GitLab pipeline or component	Forgejo replacement
.gitlab-ci.yml validation jobs	Jenkinsfile validation stage
generate-pipeline	ci/jenkins/native-plan.groovy loaded by Jenkinsfile
services-pipeline child pipeline	Jenkins per-service parallel plan and gated deployment stages
shared/templates/gitlab-ci-template.yml	shared/scripts/run_pipeline.py and Dagger
GitLab Runner / gitlab_runner_app	services/jenkins_agent local Dagger agents
Forgejo Actions OIDC OpenBao auth	Jenkins OIDC plus short-lived OpenBao workload credentials
Forgejo infrastructure deployment workflow	Jenkins multibranch pipeline with per-service approval gates

Jenkins constructs its runtime job plan in Groovy from service configuration and changed files; Python no longer generates a Jenkins pipeline manifest. The plan is passed to the shared Dagger executor for infrastructure operations. The multibranch pipeline validates the repository and plans affected services in parallel under per-service locks. Pull requests and ordinary branches are plan-only. A labelled custom deployment creates a temporary custom-deploy-N branch; only that validated branch can reach per-target approval gates and apply changes. Rollout contours and dependency ordering are computed by the native Groovy planner.

Dagger runs the repository commands in a pinned debian:13-slim execution container on a local agent in the target contour. Its cache volume carries the reviewed tfplan from plan to apply. The Dagger module is in ci/dagger/src; it can also be run locally with the Dagger CLI, for example dagger -m ci/dagger call validate --source=..

Inspect a Jenkins run plan

Jenkins writes the Groovy-built runtime plan as the .drg-runtime-manifest.json build artifact. Review it from a Jenkins build’s archived artifacts alongside the generated plans; it is not generated by generate_pipeline.py or committed.

Start a labelled custom deployment

The Forgejo Actions workflow named Run Custom starts the native Jenkins custom-deployment job. The Jenkins Groovy helper creates a new custom-deploy-N branch from drunk and cherry-picks only open pull requests labelled custom-deploy (drafts are eligible when labelled). The branch carries a seven-day expiry configured in services/jenkins/config.yaml. Expired branches are removed by the scheduled Jenkins cleanup job.

Configure the Forgejo repository Actions secret JENKINS_CUSTOM_DEPLOY_TRIGGER_TOKEN to match the Jenkins JENKINS_CUSTOM_DEPLOY_TRIGGER_TOKEN secret, and the Actions repository variable JENKINS_URL to the Jenkins base URL.

Start an infrastructure deployment

Configure the Jenkins Forgejo webhook and multibranch source for the Apoj/drg repository. Jenkins agents use the jenkins-drg label and run the repository-owned Jenkinsfile. Controller candidates use the dedicated DRBD-backed data disk for JENKINS_HOME; only the etcd lease holder promotes and mounts it, so the standby does not write Jenkins state.

Jenkins applies only on protected custom-deploy-N branches created by the Run Custom workflow. Ordinary branches, pull requests, and the protected drunk branch run validation and plans but never apply. Each affected service has its own input gate and deployment lock.

GitLab CI/CD (coexistence)

Pipelines run automatically on the default branch for changed services. Terraform jobs use one PostgreSQL schema per deployment state:

dockerhub
etcd
openbao
freeipa
keycloak
oauth2_proxy
gitlab
gitlab_runner
forgejo_runner
netbox
network
postgresql
powerdns
redis
traefik

The pipeline now has a three-stage process for deployments to ensure manual approval before applying changes:

1. Plan Stage: Generates and displays Terraform plans for all affected services
2. Approve Stage: Manual approval required (via GitLab UI) before deployment
3. Deploy Stage: Applies changes to infrastructure

GitLab CI/CD jobs retrieve secrets from OpenBao using a short-lived GitLab ID token. Jenkins jobs use its configured short-lived OpenBao workload identity. The secret declarations live in shared/templates/pipeline-services.yml (shared infrastructure credentials) and services/*/config.yaml (service credentials). Terraform receives its declared values as TF_VAR_*. Ansible receives only its native uppercase variable names and never imports TF_VAR_*. Its non-secret NETBOX_API value is generated from the compiled environment configuration. All YAML files under environments/homelab/ are loaded in lexical order; duplicate top-level sections are rejected. The compiled file is written to .runtime/homelab/platform.yaml and is not committed.

Before enabling pipelines, initialize and unseal OpenBao, then run the OpenBao playbook once from a trusted administrator workstation with OPENBAO_ROOT_TOKEN set. This creates the kv KV-v2 mount, the GitLab JWT auth mount, the Forgejo Actions OIDC auth mount, and the ci policy/role. Store common values at kv/<cluster name> and service values at the paths declared in each service config. Do not add the root token or unseal keys to GitLab CI/CD or Forgejo secrets.

Common required variables:

ANSIBLE_PRIVATE_KEY_FILE
TF_VAR_proxmox_api_token
TF_VAR_root_password
TF_VAR_ssh_public_keys
TF_VAR_netbox_token
TF_VAR_netbox_url
TF_VAR_netbox_validate_certs

ANSIBLE_PRIVATE_KEY_FILE must be a GitLab CI/CD File variable during coexistence. Forgejo obtains the same file-valued secret from OpenBao. Scope the GitLab variable to the deployment environment, for example Pushkin/*, when the same key is shared by all services in that environment. TF_VAR_ssh_public_keys is a Terraform list expression, for example ["ssh-ed25519 AAAA..."]; it must contain the public half of ANSIBLE_PRIVATE_KEY_FILE.

The ANSIBLE_PRIVATE_KEY_FILE value must be the full multiline unencrypted OpenSSH private key content, for example beginning with -----BEGIN OPENSSH PRIVATE KEY-----. Do not store a file path, public key, PuTTY .ppk, or a string with escaped \n characters.

Service-specific variables are the corresponding TF_VAR_* names from each services/<service>/terraform/variables.tf. Commonly required secret or externally supplied values include:

TF_VAR_gitlab_external_url
TF_VAR_gitlab_postgresql_password
TF_VAR_authentik_issuer_url
TF_VAR_authentik_client_id
TF_VAR_authentik_client_secret
TF_VAR_netbox_external_url
TF_VAR_netbox_allowed_hosts
TF_VAR_netbox_secret_key
TF_VAR_netbox_postgres_password
TF_VAR_netbox_redis_password
TF_VAR_netbox_redis_cache_password
TF_VAR_netbox_superuser_email
TF_VAR_netbox_superuser_password
TF_VAR_netbox_superuser_api_token
TF_VAR_powerdns_api_key
TF_VAR_powerdns_postgresql_password
TF_VAR_postgresql_admin_password
TF_VAR_postgresql_replication_password
TF_VAR_postgresql_gitlab_password
TF_VAR_postgresql_gitlab_registry_password
TF_VAR_postgresql_authentik_password
TF_VAR_postgresql_netbox_password
TF_VAR_postgresql_netbird_password
TF_VAR_postgresql_powerdns_password
OPENBAO_POSTGRESQL_PASSWORD
KEYCLOAK_POSTGRESQL_PASSWORD
KEYCLOAK_ADMIN_PASSWORD
KEYCLOAK_FREEIPA_BIND_PASSWORD
KEYCLOAK_OAUTH2_PROXY_CLIENT_SECRET
KEYCLOAK_GITLAB_CLIENT_SECRET
KEYCLOAK_NETBOX_CLIENT_SECRET
KEYCLOAK_OPENBAO_CLIENT_SECRET
OAUTH2_PROXY_COOKIE_SECRET
FREEIPA_ADMIN_PASSWORD
FREEIPA_DIRECTORY_MANAGER_PASSWORD
TF_VAR_selectelv2_account_id
TF_VAR_selectelv2_project_id
TF_VAR_selectelv2_username
TF_VAR_selectelv2_password
TF_VAR_traefik_acme_email
TF_VAR_traefik_trusted_proxy

Some Ansible-only settings can be supplied directly without a TF_VAR_ prefix when needed, for example ANSIBLE_USER, NETBOX_API, NETBOX_VALIDATE_CERTS, GITLAB_RUNNER_TOKEN, and service image tags. Defaults in shared/ansible/inventory/host_vars/*.yml cover non-secret values.

Jenkins provisioning reads static fields from OpenBao at kv/services/jenkins:

jenkins_oidc_client_secret
jenkins_drbd_shared_secret

Jenkins reuses the common ansible_private_key_file OpenBao secret for SSH connections from the controller to its agents; the agent playbook derives and installs the matching public key automatically.

The generated Forgejo SCM token is written by the Jenkins OpenBao client to automation/services/jenkins and is not provisioned as a static secret.

Create the Forgejo OAuth/OIDC application for Jenkins with the callback URL configured by the Jenkins OIDC plugin. The controller and agents are installed by services/jenkins and services/jenkins_agent. Jenkins controller candidates use the dedicated DRBD-backed data disk mounted at /srv/jenkins-home; only the etcd lease holder mounts it and starts Jenkins. Do not add a static deploy token: apply jobs obtain short-lived OpenBao credentials through the controller workload identity.

The Local backend has no isolation and is intended only for this trusted, private repository. Dagger adds container isolation for pipeline commands, but the agent still needs Docker access to start the Dagger engine.

Forgejo state migration additionally requires repository variables GITLAB_API_URL and GITLAB_PROJECT_ID, plus masked secrets GITLAB_TERRAFORM_STATE_USERNAME and GITLAB_TERRAFORM_STATE_PASSWORD.

Local Environment

For local Terraform/OpenTofu runs, either export variables with TF_VAR_* or put them in a local ignored proxmox.tfvars file. For local Ansible runs, export the same variables used by CI, plus ANSIBLE_PRIVATE_KEY_FILE pointing at a local SSH key file:

$env:NETBOX_API = "http://192.168.3.11:80"
$env:NETBOX_TOKEN = "<netbox-api-token>"
$env:NETBOX_VALIDATE_CERTS = "false"
$env:TF_VAR_netbox_token = $env:NETBOX_TOKEN
$env:TF_VAR_netbox_url = $env:NETBOX_API
$env:ANSIBLE_PRIVATE_KEY_FILE = "~/.ssh/id_ed25519"

Terraform roots declare an empty PostgreSQL pg backend. CI fills the matching PostgreSQL connection settings automatically. The TF_HTTP_* variables below are only needed when reading a legacy GitLab state during migration or for a temporary rollback:

The migration pipeline reads legacy GitLab managed state through TF_HTTP_* environment variables and writes the destination through PG_CONN_STR, PGPASSWORD, and PG_SCHEMA_NAME. Normal deployment jobs use PostgreSQL state directly. The GitLab source uses CI_JOB_TOKEN by default; if the instance rejects job-token state reads, provide masked/protected TF_HTTP_USERNAME and TF_HTTP_PASSWORD variables with permission to read Terraform state.

export SERVICE=gitlab
export TF_HTTP_ADDRESS="https://git.example.com/api/v4/projects/<project-id>/terraform/state/${SERVICE}"
export TF_HTTP_LOCK_ADDRESS="${TF_HTTP_ADDRESS}/lock"
export TF_HTTP_UNLOCK_ADDRESS="${TF_HTTP_ADDRESS}/lock"
export TF_HTTP_USERNAME="<gitlab-username>"
export TF_HTTP_PASSWORD="<personal-access-token>"
export TF_HTTP_LOCK_METHOD="POST"
export TF_HTTP_UNLOCK_METHOD="DELETE"
export TF_HTTP_RETRY_WAIT_MIN="5"

Migrating legacy GitLab state

The root pipeline exposes a manual terraform_state_migration trigger. Click its play button to open a child pipeline with two manual commands:

• migrate_explicit_states — enter comma-separated state names. It defaults to the DockerHub state.
• migrate_services — enter service names and an environment.

Each command provides a mode dropdown (plan or apply) and a boolean confirm_apply control. Enable confirm_apply for an apply. DockerHub is the default target, so the explicit-state command can be started without entering values. State names and service selections remain comma-separated text fields because they support multiple arbitrary values.

MIGRATION_STATE_NAMES=production-dockerhub,production-gitlab
MIGRATION_MODE=plan

For service selection, the tool derives the same names used by generated deployment jobs. Non-rollout services use <environment>-<service>. Contour rollouts use <environment>-<service>-<cluster>-<contour>.

After reviewing the non-sensitive migration report, run the job again with:

MIGRATION_SERVICES=dockerhub,gitlab
MIGRATION_ENVIRONMENT=production
MIGRATION_MODE=apply
MIGRATION_CONFIRM=MIGRATE

The tool preflights every source and destination, refuses an existing PostgreSQL destination by default, and leaves the GitLab source unchanged. To replace an existing destination intentionally, also set MIGRATION_ALLOW_OVERWRITE=true. Do not run Terraform jobs for the selected states during migration. The source GitLab state remains available for rollback; restore the old HTTP backend configuration only if a rollback is required.

Example service-specific local settings:

$env:TRAEFIK_ACME_EMAIL = "admin@example.com"
$env:SELECTELV2_ACCOUNT_ID = "<secret>"
$env:SELECTELV2_PROJECT_ID = "<secret>"
$env:SELECTELV2_USERNAME = "<secret>"
$env:SELECTELV2_PASSWORD = "<secret>"
$env:GITLAB_EXTERNAL_URL = "https://git.example.com"
$env:AUTHENTIK_ISSUER_URL = "https://auth.example.com/application/o/gitlab/"
$env:AUTHENTIK_CLIENT_ID = "<secret>"
$env:AUTHENTIK_CLIENT_SECRET = "<secret>"

Initial bootstrap

Bootstrap the platform services that establish NetBox and SDN first. The clustered PostgreSQL deployment uses the NetBox/service_config flow and should be deployed after NetBox inventory, DNS, Traefik, and the prod-app SDN network are available.

1. Bootstrap NetBox

cd services/netbox/terraform
tofu init
tofu plan -var-file=proxmox.tfvars
tofu apply -var-file=proxmox.tfvars

ansible-playbook -i 'netbox,' services/netbox/ansible/netbox.yml \
  -e @shared/ansible/inventory/group_vars/all.yml \
  -e @shared/ansible/inventory/host_vars/netbox.yml \
  -e "ansible_host=192.168.2.21"

ansible-playbook -i 'netbox,' services/netbox/ansible/netbox-sdn-bootstrap.yml \
  -e @shared/ansible/inventory/group_vars/all.yml \
  -e @shared/ansible/inventory/host_vars/netbox.yml \
  -e "ansible_host=192.168.2.21" \
  -e netbox_sdn_bootstrap_confirm=true

2. Reconcile Proxmox SDN from NetBox

After NetBox is running and the SDN objects exist, reconcile Proxmox SDN from NetBox:

cd services/network/terraform
tofu init
tofu plan -var-file=proxmox.tfvars
tofu apply -var-file=proxmox.tfvars

3. Deploy etcd and PostgreSQL

After the SDN bridge exists in Proxmox and NetBox inventory is available, deploy the etcd quorum first, then PostgreSQL:

$env:NETBOX_API = "https://netbox.michshch.ru"
$env:NETBOX_TOKEN = "<netbox-api-token>"
$env:NETBOX_VALIDATE_CERTS = "true"
$env:TF_VAR_netbox_token = $env:NETBOX_TOKEN

cd services/etcd/terraform
tofu init
tofu plan
tofu apply

ansible-playbook -i shared/ansible/inventory/netbox.yml \
  services/etcd/ansible/etcd.yml \
  --limit 'etcd-pve-cluster-a-*'

cd ../../postgresql/terraform
tofu init
tofu plan
tofu apply

ansible-playbook -i shared/ansible/inventory/netbox.yml \
  services/postgresql/ansible/postgresql.yml \
  --limit 'postgresql-pve-cluster-a-*'

After Traefik routes and ingress DNS reconcile, applications can use postgresql.michshch.ru:5432 for reads and writes. Only the current Patroni primary accepts application traffic; the other two PostgreSQL instances remain hot standby failover backups.

Services

cd services/gitlab/terraform
tofu init
tofu plan -var-file=proxmox.tfvars
tofu apply -var-file=proxmox.tfvars

cd services/traefik/terraform
tofu init
tofu plan -var-file=proxmox.tfvars
tofu apply -var-file=proxmox.tfvars

cd services/dockerhub/terraform
tofu init
tofu plan -var-file=proxmox.tfvars
tofu apply -var-file=proxmox.tfvars

Run Ansible against the NetBox dynamic inventory:

ansible-playbook -i shared/ansible/inventory/netbox.yml services/gitlab/ansible/gitlab.yml --limit gitlab

ansible-playbook -i shared/ansible/inventory/netbox.yml services/traefik/ansible/traefik.yml --limit traefik

ansible-inventory -i shared/ansible/inventory/netbox.yml --graph
ansible-playbook -i shared/ansible/inventory/netbox.yml services/dockerhub/ansible/dockerhub.yml --limit dockerhub

ansible-inventory -i shared/ansible/inventory/netbox.yml --graph
ansible-playbook -i shared/ansible/inventory/netbox.yml services/postgresql/ansible/postgresql.yml --limit 'postgresql-pve-cluster-a-*'

OpenBao is initially deployed alongside the existing secret configuration. It uses the legacy vmbr1 address 192.168.2.26, starts with persistent file storage, and is not yet used by any other service. See services/openbao/README.md for bootstrap and initialization instructions.

OpenTofu provider mirror

Example ~/.tofurc:

provider_installation {
  network_mirror {
    url = "https://terraform-mirror.yandexcloud.net/"
    include = ["registry.opentofu.org/*/*"]
  }
  direct {
    exclude = ["registry.opentofu.org/*/*"]
  }
}