CI/CD and Environments¶

Status: implemented 2026-08-31. Supersedes the placeholder deploy step described in M3-13.

The model in one table¶

	Development	UAT
Branch	`develop`	`main`
Instance	t3.medium	t3.large
App	`dev-curorec.duckdns.org`	`curorec.duckdns.org`
Docs	`devdocs-curorec.duckdns.org`	`docs-curorec.duckdns.org`
Path on box	`/opt/curorec`	`/opt/curorec`
Compose file	`docker-compose.server.yml`	`docker-compose.server.yml`
`.env` template	`deploy/env.development.example`	`deploy/env.server.example`
Schema sync on deploy	automatic	never — a person runs it
Trigger	push to `develop`	push to `main`

Push to develop, dev updates. Merge develop → main, UAT updates. Both are fully automatic, with no approval gate.

The two boxes are deliberately identical in every respect except the contents of .env: same path, same compose file, same SSH key, same service names. The only per-environment editing anyone does by hand is the Auth0 block in .env.

What actually happens on a deploy¶

The workflow (.github/workflows/deploy.yml) does very little itself. It typechecks, then SSHes to the box, moves the checkout, and hands off to scripts/deploy.sh. That script is the single definition of a deploy — CI runs exactly it, and so can you:

cd /opt/curorec
git fetch --prune origin && git reset --hard origin/develop
ENVIRONMENT=development RUN_SCHEMA_SYNC=true ./scripts/deploy.sh

The script, in order:

Takes a lock, so two deploys can't build over each other.
Preflight. Confirms .env, app/backend/.env.server and app/backend/config/velocity-keys.json exist; checks free disk; warns if RAM + swap is under 8 GB.
docker compose up -d --build.
On dev only, runs schema-sync and restarts the backend.
Verifies. Waits for postgres, backend and frontend to report healthy, then probes https://<domain>/ and /api/docs through Caddy on the real hostname — which exercises TLS, the certificate, and the /api/v1 vs /api/auth path split. Finally it greps the built bundle for the domain, confirming NEXT_PUBLIC_API_BASE was really inlined.
Rolls back on failure to the commit the box was on before, rebuilds, and re-verifies.
Prunes untagged images.

Why there is no image registry¶

Deliberate, per the release-simplicity requirement: each box builds from its own checkout. The costs are real and worth stating plainly — a deploy takes 5–10 minutes rather than seconds, the build competes for RAM with the running stack, and a rollback is another full rebuild because the previous image tag was overwritten. If deploys ever become frequent enough that this hurts, publishing to GHCR (which deploy.yml already had scaffolding for) is the fix.

Why the frontend bundle check matters¶

NEXT_PUBLIC_* values are compiled into the JavaScript at build time. If a build arg fails to reach the build stage you get a clean, warning-free build that ships a bundle pointing at localhost:8000 — every user's browser then calls their own machine and the server looks broken for reasons that have nothing to do with the server. This has happened on this codebase before. It is the one check worth never skipping, so deploy.sh runs it on every deploy.

One-time setup¶

GitHub¶

Repository secret:

Secret	Value
`EC2_SSH_PRIVATE_KEY`	Full contents of `curoreckeypair.pem`, including the BEGIN/END lines
`SSH_KNOWN_HOSTS`	Optional but recommended. Output of `ssh-keyscan` for both hosts. May be a repository variable instead — host public keys are not confidential. Generate it only after DNS exists, using the exact names in `DEPLOY_HOST`: an entry is keyed by hostname, so a key scanned from `ec2-18-…compute.amazonaws.com` will not match a connection to `dev-curorec.duckdns.org`.

Then create two Environments (Settings → Environments), each with one variable:

Environment	Variable	Value
`development`	`DEPLOY_HOST`	`dev-curorec.duckdns.org`
`uat`	`DEPLOY_HOST`	`curorec.duckdns.org`

SSH_USER (default ec2-user) and APP_DIR (default /opt/curorec) can be overridden the same way if either box ever differs.

On the SSH key. Both boxes share one key pair for maintenance simplicity, which means the GitHub secret is a credential for both environments, and port 22 must accept connections from GitHub's runner IP ranges. Two consequences worth accepting knowingly: anyone who can push a workflow change to this repository can run commands on UAT, and the key cannot be rotated for one environment alone. If that becomes uncomfortable, AWS SSM Run Command with GitHub OIDC removes both problems — no inbound port 22 and no stored key.

Repository access from the box¶

origin is https://github.com/NeuralRaysAI/Curo-REC.git, a private repository, so the box needs its own read credential before git fetch can work unattended — and every deploy is a git fetch. Use a deploy key, which is read-only and scoped to this one repository:

# on the box
ssh-keygen -t ed25519 -C "curorec-dev-box" -f ~/.ssh/github_deploy -N ""
cat ~/.ssh/github_deploy.pub

Add that public key in GitHub under Settings → Deploy keys → Add deploy key (leave "Allow write access" unchecked), then point the clone at SSH rather than HTTPS:

cat >> ~/.ssh/config <<'EOF'
Host github.com
  IdentityFile ~/.ssh/github_deploy
  IdentitiesOnly yes
EOF
chmod 600 ~/.ssh/config
git clone git@github.com:NeuralRaysAI/Curo-REC.git /opt/curorec

A personal access token embedded in the remote URL also works, but it carries that user's access to every repository they can see and it expires — the deploy key does neither. Give each box its own key so one can be revoked without touching the other.

Development box (t3.medium)¶

scp -i curoreckeypair.pem scripts/server-bootstrap.sh ec2-user@<dev-host>:~
ssh -i curoreckeypair.pem ec2-user@<dev-host>

# 6G swap, not the 2G default. The t3.medium reports ~3.7 GB usable RAM and
# ships with zero swap; `next build` will not complete on that.
SWAP_SIZE=6G DUCKDNS_DOMAINS=dev-curorec,devdocs-curorec ./server-bootstrap.sh
# log out and back in so the docker group applies

git clone git@github.com:NeuralRaysAI/Curo-REC.git /opt/curorec && cd /opt/curorec
git checkout develop
cp deploy/env.development.example .env      # fill every CHANGE_ME
cp app/backend/.env.example app/backend/.env.server

velocity-keys.json is no longer needed. DEPLOY-NOTES.md §7 still tells you to scp it up; that instruction is stale. VNF_KEYS_FILE_PATH and the fallback it powered were removed on 2026-08-16 (Task 31 §5.2) — every organization, the bootstrap one included, now uses per-org keys stored in the database. UAT has been running without the file. deploy.sh prints a note if it is absent and carries on.

First deploy, run by hand so you see it work before CI does it:

ssh -i curoreckeypair.pem ec2-user@<dev-host>
cd /opt/curorec && ENVIRONMENT=development RUN_SCHEMA_SYNC=true ./scripts/deploy.sh

IAM instance role¶

Both instances share one role. It grants the machines permissions — it is not a policy on your own IAM user, which would only let you manage instances and would do nothing for them.

IAM → Roles → Create role → trusted entity AWS service → EC2 → attach AmazonSSMManagedInstanceCore → name it e.g. CuroRecEc2Role. Then, per instance: EC2 → Actions → Security → Modify IAM role. No reboot needed; the SSM agent picks it up within a couple of minutes.

Confirm it took, on each box:

TOK=$(curl -s -X PUT http://169.254.169.254/latest/api/token \
      -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
curl -s -H "X-aws-ec2-metadata-token: $TOK" \
     http://169.254.169.254/latest/meta-data/iam/security-credentials/

That should print the role name. A 404 means the role is not attached. Then, from anywhere with the AWS CLI, confirm the instances actually registered with SSM:

aws ssm describe-instance-information --region eu-west-2 \
  --query 'InstanceInformationList[].{Id:InstanceId,Ping:PingStatus,Name:ComputerName}' --output table

Both should show Online. This role is a prerequisite for three separate things: SSM Session Manager (browser shell, no SSH key), the backup profile's s3:PutObject, and eventually retiring the AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY pair currently sitting in app/backend/.env.server.

DNS and Auth0¶

Create dev-curorec and devdocs-curorec in the DuckDNS console pointed at the instance IP. With UAT's two names that is 4 of the free tier's 5.

Attach an Elastic IP to both instances. Without one, any stop/start hands the box a new address, all four DuckDNS names go stale, and DEPLOY_HOST stops resolving — so deploys fail alongside the sites. The bootstrap script's optional DUCKDNS_TOKEN updater is the free workaround, but the Elastic IP is the right answer.

Add the dev URLs to the Auth0 Application's callback, logout and web-origin lists, alongside the existing UAT and localhost entries rather than replacing them.

Before the first automated UAT deploy¶

Two things about the UAT box as it stands on 2026-08-31, both of which the first pipeline run will change silently if nobody looks first:

UAT is checked out on develop, at caafc0e. Under the new model it tracks main. The first deploy will git reset --hard origin/main and move it. main is ahead of what UAT is running, so this is a step forward rather than a rollback — but confirm that is what you expect before you merge.
deploy/env.server.example has an uncommitted local edit on the box. git reset --hard discards it. Check what it is and port it into the repo if it matters:

ssh -i curoreckeypair.pem ec2-user@curorec.duckdns.org \
  "cd /opt/curorec && git diff deploy/env.server.example"

Neither is a defect in the pipeline. Both are the kind of thing that is obvious now and baffling in three weeks.

Operating it¶

Watch a deploy: the Actions run streams deploy.sh output live. On the box, ~/curorec-deploy.log records one line per deploy with the result and commit.

Deploy a specific environment manually: Actions → Deploy → Run workflow → pick the environment.

Roll back deliberately:

ssh -i curoreckeypair.pem ec2-user@<host>
cd /opt/curorec
git reset --hard <good-sha>
ENVIRONMENT=uat ./scripts/deploy.sh

Change the schema on UAT (never automatic):

cd /opt/curorec
docker compose -f docker-compose.server.yml --profile tools run --rm schema-sync
docker compose -f docker-compose.server.yml restart backend

schema:sync ALTERs the database to match the entities, so it will drop a column the entities no longer declare. On dev that is an acceptable trade for iteration speed. On UAT, take a dump first.

Known gaps¶

These are not blockers for the pipeline, but they are the things most likely to bite next:

No migration path. The 12 TypeORM migrations are broken and diverged from the entities (DEPLOY-NOTES.md §4b), which is why both environments build the schema from entities instead. Automated deploys make this more pressing, not less: nothing here can be promoted to a production environment until a squashed baseline migration exists.
No backups on UAT. The backup profile is written and switched off. It needs BACKUP_S3_URI and an instance role with s3:PutObject. UAT now has a promotion path pointing at it, so its data is worth more than it was.
Secrets live as files on the boxes. Fine for these two environments; not acceptable for production, where Task 31 §7.4 requires Secrets Manager or Parameter Store.
The backend's redis dependency is unused and is why every install needs --legacy-peer-deps. redis@^4.6.7 is declared in app/backend/package.json but imported nowhere in src/ — no createClient, no import of the package at all. BullMQ talks to Redis through ioredis, which is what the lockfile actually resolves. Because redis is still declared, bullmq's optional peer range (>=5.0.0) conflicts with it and a strict npm ci fails with ERESOLVE, so the Dockerfile and both workflows all carry the same --legacy-peer-deps workaround. Removing the dependency and regenerating the lockfile would let all three drop the flag. Small change, wants one careful test of the queue paths.
No alerting. A box that fills its disk or dies outside a deploy is discovered by someone opening the site. CloudWatch alarms on the status check and disk >80% are the cheap fix.