Sunday, 1 November 2026

Running Oracle AI Database 26ai on OKE: A Developer Experience Blueprint

Running Oracle AI Database 26ai on OKE

How one GitLab pipeline takes an empty OCI compartment to a running app on OKE, backed by Autonomous AI Database 26ai, using the Oracle AI Database Operator for Kubernetes v2.2.0.

Why I built this

One GitLab pipeline now takes an empty OCI tenancy to a running app on OKE, backed by an Autonomous AI Database 26ai, in about 4 minutes for a re-run. I wrote zero lines of code that call the OCI Database API to get there.

The question I ask of any tool is simple: does it remove tickets and hand-offs? For the database tier, the answer used to be no. Developers raised a request, waited for a DBA, and got an environment that drifted from the console, the scripts and reality.

The Oracle AI Database Operator for Kubernetes (OraOperator) changes that. It extends the Kubernetes API with custom resources, so a database becomes a YAML file in Git. Version 2.2.0 (August 2026) tightened reconciliation between Kubernetes and OCI, which is what made a fully declarative pipeline practical. This post walks through the project I built to prove it: oke-adb26ai-devx.

Architecture at a glance

The pipeline never talks to the database API itself. It applies one custom resource; the operator watches it, provisions the database through the OCI Database API, and writes the mTLS wallet into a Secret that the app mounts.

One YAML file becomes a database: the operator does the OCI work GITLAB CI PIPELINE · RUN PIPELINE 1 · infra Terraform: VCN, OKE and node pool 2 · platform cert-manager + OraOperator 2.2.0 3 · database apply the CR once wait for AVAILABLE 4 · deploy image from build-app init + 2 app pods ORACLE CLOUD (OCI) COMPARTMENT OKE BASIC cluster · 2 × VM.Standard.E5.Flex nodes namespace: operator system OraOperator 2.2.0 2 replicas, ns-scoped webhooks via cert-manager namespace: devx AutonomousDatabase CR k8s/adb.yaml.tpl · hardLink: true Secrets + ConfigMap OCI API key + 3 passwords devx-adb-wallet mTLS wallet Secret (operator-written) devx-app × 2 FastAPI + python-oracledb thin OCI Load Balancer flexible, 10 Mbps users → HTTP :80 Autonomous AI Database 26ai · DEVXADB26 2 ECPU · 1 TB · mTLS required · native JSON table kubectl apply watches OCI Database API writes mTLS via wallet

Four pipeline stages build the cluster; everything in the devx namespace is declared in Git, while the database and its wallet are produced by the operator.

What's in the repository

The repo splits cleanly by owner: Terraform owns infrastructure, the operator owns the database lifecycle, and plain Kubernetes manifests own the app. Each pipeline stage is one shell script, so every step can also run from a laptop.

PathOwnsNotes
.gitlab-ci.ymlThe pipelineSix stages: infra, platform, build, database, deploy, teardown
infra/VCN, OKE BASIC cluster, node poolGitLab-managed Terraform state over the http backend
ci/common.shShared helpersOCI config, kubeconfig, node waits, retrying apply, template rendering
ci/preflight.shCredential checksValidates OCI variables before Terraform calls OCI
ci/install-operator.shcert-manager + OraOperator 2.2.0Namespace-scoped install from Oracle's repo at the 2.2.0 tag
ci/deploy-adb.shAutonomous AI Database 26aiSecrets, idempotent Create, wait for AVAILABLE and the wallet
ci/deploy-app.shApp rolloutPull secret, Deployment, LoadBalancer Service, prints the URL
ci/teardown-k8s.shOrdered cleanupService, then database, then namespace
k8s/adb.yaml.tplThe databaseOne AutonomousDatabase (database.oracle.com/v4) resource
k8s/app.yaml.tplThe appInit container for schema, two app replicas, OCI flexible LB
app/DevX Feedback BoardFastAPI + python-oracledb 26.1 thin mode, non-root image

What comes from Oracle is used unmodified: the operator's scripts/generate-namespace-install.sh, the generated CRD, RBAC and webhook manifests, and the v4 AutonomousDatabase API. The template merges two upstream samples, autonomousdatabase_create.yaml and autonomousdatabase_download_wallet.yaml, into one resource.

Building it, stage by stage

Setup takes about 15 minutes: an OCI API key, a handful of IAM policies and eleven GitLab CI/CD variables. After that, one click runs the pipeline.

0. Prerequisites

The pipeline user needs these policies in the target compartment:

OCI IAM
Allow group <group> to manage cluster-family             in compartment <c>
Allow group <group> to manage virtual-network-family     in compartment <c>
Allow group <group> to manage instance-family            in compartment <c>
Allow group <group> to manage autonomous-database-family in compartment <c>
Allow group <group> to manage load-balancers             in compartment <c>
Allow group <group> to read   work-requests              in compartment <c>
The last policy is easy to miss. Operator 2.2.0 tracks OCI work requests to know when an operation has finished. Credentials go into GitLab as masked variables; the API private key is a File variable, never pasted into the repo.

1. Infra: Terraform builds VCN and OKE

infra/ creates a VCN with three subnets (public /28 API endpoint, private /24 workers behind NAT and a Service Gateway, public /24 for load balancers), an OKE BASIC cluster with flannel overlay, and a node pool of 2 × VM.Standard.E5.Flex (2 OCPU, 16 GB). Terraform picks the newest Kubernetes version OKE offers, and the matching OKE-optimised Oracle Linux image, unless you pin one. State lives in GitLab's managed Terraform state.

Before Terraform touches OCI, ci/preflight.sh checks every credential variable. It computes the key's fingerprint with openssl and compares it to OCI_FINGERPRINT, so a mismatched key fails in seconds with a clear message instead of a 401 deep in a Terraform plan.

2. Platform: cert-manager and the operator

ci/install-operator.sh installs cert-manager v1.20.2 (the operator's admission webhooks need it), clones oracle/oracle-database-operator at tag 2.2.0, and generates a namespace-scoped install that watches only the operator namespace and devx. Three adjustments sit on top:

  • Pin the operator image (upstream ships :latest) through OPERATOR_IMAGE
  • Drop replicas from 3 to 2 to fit a two-node cluster
  • Apply with --server-side, because the CRDs are too large for client-side apply's annotation limit

The script then confirms the operator's service account can list autonomousdatabases in devx, and waits until the validating webhook has a CA bundle before any custom resource is applied.

3. Database: one YAML file

This is the core of the project. The database is this resource:

YAML
apiVersion: database.oracle.com/v4
kind: AutonomousDatabase
metadata:
  name: devx-adb
  namespace: devx
spec:
  action: Create
  hardLink: true          # deleting the resource terminates the DB in OCI
  details:
    compartmentId: ${OCI_COMPARTMENT_OCID}
    dbName: DEVXADB26
    dbVersion: "26ai"
    dbWorkload: OLTP
    computeModel: ECPU
    computeCount: 2
    dataStorageSizeInTBs: 1
    isMtlsConnectionRequired: true
    adminPassword:
      k8sSecret: { name: adb-admin-password }
  wallet:
    name: devx-adb-wallet
    password:
      k8sSecret: { name: adb-wallet-password }
  ociConfig:
    configMapName: oci-cred
    secretName: oci-privatekey

ci/deploy-adb.sh writes the OCI credentials into ConfigMap oci-cred and Secret oci-privatekey, stores three passwords as Secrets, applies the resource and polls .status.lifecycleState. The operator calls the OCI Database API, reconciles PROVISIONING to AVAILABLE (about 4 minutes in my runs), then downloads the mTLS wallet into Secret devx-adb-wallet. No console clicks, no OCI CLI database calls, no manual wallet download.

4. Build: in parallel

build-app has needs: [], so the Docker image builds and pushes to the GitLab Container Registry while OKE is still provisioning. It uses the previous :latest image as a layer cache.

5. Deploy: the app mounts the wallet

k8s/app.yaml.tpl deploys two replicas behind an OCI flexible load balancer (10 Mbps). An init container connects as ADMIN, creates a least-privilege DEVX_APP user and the feedback table, and exits. The app itself only ever connects as DEVX_APP. Both containers mount the wallet Secret read-only and use python-oracledb thin mode, so the image needs no Oracle Client. The job ends by printing App is live: http://<ip> and wiring that URL to the GitLab environment devx-demo.

6. Teardown: manual and ordered

Two manual jobs, Kubernetes first, then terraform destroy. Order matters, as covered in the lessons below.

ModeWhat runsTypical time
SKIP_INFRA=falseEverything from an empty compartment25–35 min (mostly OKE)
SKIP_INFRA=trueOperator, database and app on the existing cluster8–12 min
Re-run, database existsOperator check, build, deploy4 min 19 s (my latest run)

What the developer experience looks like

For a developer, the database is now a file they can read, review and diff. Three design choices make the workflow safe to hand to anyone on the team.

No surprise cloud spend. The pipeline's workflow:rules accept only web, API and trigger sources. A git push never creates cloud resources by accident. resource_group locks keep two runs from fighting over the same cluster or Terraform state.

Secrets never leave Kubernetes. OCI credentials, the admin, wallet and app passwords all live as masked GitLab variables, and the pipeline turns them into Kubernetes Secrets. The operator reads them by name (k8sSecret), and the app mounts the wallet the operator writes. Nobody downloads a wallet zip.

Failures explain themselves. When the database lands in FAILED, or a rollout stalls, the job prints kubectl describe, the last 80 lines of the operator log, and the init-container log. Most failures are readable without opening a shell.

The demo app: native JSON in 26ai

The DevX Feedback Board is a ~200-line FastAPI app. It stores each submission as a native JSON column and queries it with JSON_VALUE:

SQL
CREATE TABLE IF NOT EXISTS feedback (
  id         NUMBER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
  doc        JSON NOT NULL
);

The schema script uses two 26ai features that make init containers simpler. CREATE USER IF NOT EXISTS and CREATE TABLE IF NOT EXISTS make the DDL re-runnable without PL/SQL exception handling, so every pod restart runs it safely.

The page shows chips for the database banner, the ADB name, and the pod and node serving the request. Refresh a few times and you watch the two replicas take turns.

Hardened by default

  • Pods run as UID 10001, non-root, with a read-only root filesystem, all capabilities dropped and the RuntimeDefault seccomp profile
  • The app connects as DEVX_APP with a 100 MB quota, never as ADMIN
  • isMtlsConnectionRequired: true, so the public endpoint accepts only wallet-based connections
  • Readiness probe runs SELECT 1 FROM dual through the pool; liveness stays database-independent

Live commands worth knowing

BASH
kubectl get autonomousdatabase -n devx -w
kubectl get secret devx-adb-wallet -n devx
kubectl logs -n oracle-database-operator-system \
  deploy/oracle-database-operator-controller-manager -c manager --tail=30

Lessons learned the hard way

The pipeline went from red to green in one morning, and each red run taught one fix. None of them were operator bugs.

The biggest one: action: Create is not idempotent

The operator treats spec.action: Create as "provision a new database" every time it sees it. Re-applying Oracle's sample on a second pipeline run would create a second database. The fix in ci/deploy-adb.sh is a simple guard:

BASH
if kubectl -n "$APP_NAMESPACE" get autonomousdatabase "$ADB_CR_NAME" >/dev/null 2>&1; then
  log "AutonomousDatabase/${ADB_CR_NAME} already exists - not re-applying Create"
else
  apply_with_retry .adb.yaml
fi
Secrets are still reconciled on every run, so rotating a password in GitLab flows through. This is the one thing I'd most like the operator to make easier for pipelines.

Teardown order matters

  1. Delete the LoadBalancer Service first, so the cloud controller removes the OCI load balancer. Otherwise terraform destroy is blocked by an LB still sitting in the subnet.
  2. Delete the AutonomousDatabase while the operator is still running. Its finalizer terminates the database in OCI (because hardLink: true) and then releases the resource.
  3. Only then delete the namespace and run terraform destroy. Deleting the namespace first can leave finalizers stuck.

Everything else

SymptomCauseFix
OCI returns 401 during TerraformAPI key and fingerprint did not matchci/preflight.sh computes the fingerprint and fails fast with a named variable
Job exits with code 141kubectl … | head under set -o pipefail raises SIGPIPEAvoid piping into head in strict-mode scripts
CRD apply fails on annotation sizeOperator CRDs exceed the client-side apply limitkubectl apply --server-side --force-conflicts
First CR rejected right after installAdmission webhook not answering yetWait for the webhook CA bundle, plus a 12-attempt retrying apply
Init container fails on passwordAutonomous Database password complexity and reuse rules12–30 chars, mixed case + digit, no ", must not contain admin
Table with identity column failsApp user lacked the privilege to create the identity's sequenceGrant CREATE SEQUENCE alongside CREATE TABLE
JSON query fails to parseCOMMENT is a reserved word in JSON dot notationUse JSON_VALUE(doc, '$.comment') and alias it comment_text
Terraform node pool didn't register overnightNode registration issue, still under investigationAdded the node pool by hand for the demo; on the fix list

One documentation catch to share with the product team: the README's Verify Installation section names validating-webhook-configuration and mutating-webhook-configuration, but 2.2.0 creates oracle-database-operator-validating-webhook-configuration and its mutating twin.

Takeaways and what's next

Treat the database like the rest of your platform: as code, reviewed in Git, reconciled by a controller. Operator 2.2.0 is mature enough to make that the default for dev and test today.

  • Start small. Ephemeral Autonomous Databases for dev and test, or one Autonomous Container Database with ADBs on top for dedicated infrastructure.
  • Least privilege first. Namespace-scoped operator install, a scoped IAM group, an app user that is never ADMIN.
  • Use the pipeline you already run. The operator is the piece that lets the database layer fit into it.
  • Watch the cost lever. While running, the stack is 2 E5 Flex nodes, a 10 Mbps LB and a 2-ECPU database with 1 TB storage; the OKE BASIC control plane is free. Databases that disappear with their environment stop paying for idle.

What I'd like to see next from the operator: an official Helm chart for GitOps, an idempotent create mode that pipelines can apply on every run, Traffic Manager (CMAN) moving from preview to GA, and deeper docs for v2.2.0 and Private AI.

The full code is at gitlab.com/jay_suraparaju/oke-adb26ai-devx. The operator itself is open source under UPL-1.0 at github.com/oracle/oracle-database-operator. Questions and pull requests welcome.

No comments:

Post a Comment

Note: only a member of this blog may post a comment.