Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
39b0a9f475 | ||
|
|
71d96839dc | ||
|
|
04ccbecbd7 | ||
|
|
d9d2dc246a |
@@ -0,0 +1,36 @@
|
|||||||
|
name: Sync runner checkout
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches: [master]
|
||||||
|
pull_request:
|
||||||
|
types: [closed]
|
||||||
|
branches: [master]
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
sync:
|
||||||
|
if: gitea.event_name == 'push' || gitea.event.pull_request.merged == true
|
||||||
|
runs-on: self-hosted
|
||||||
|
steps:
|
||||||
|
- name: Pull latest server-infra
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
REPO="/home/westfarn/Documents/repos/server-infra"
|
||||||
|
|
||||||
|
if [[ ! -d "${REPO}/.git" ]]; then
|
||||||
|
echo "Missing git checkout at ${REPO}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
cd "${REPO}"
|
||||||
|
|
||||||
|
if ! git diff --quiet || ! git diff --cached --quiet; then
|
||||||
|
echo "Working tree is dirty; refusing to pull"
|
||||||
|
git status --short
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
git fetch origin master
|
||||||
|
git checkout master
|
||||||
|
git pull --ff-only origin master
|
||||||
|
git rev-parse --short HEAD
|
||||||
+54
-18
@@ -187,15 +187,18 @@ After Docker install, re-SSH so the `docker` group membership takes effect.
|
|||||||
|
|
||||||
### Apps
|
### Apps
|
||||||
|
|
||||||
| App | Type | Hosts | Notes |
|
| App | Type | Hosts | Envs | Notes |
|
||||||
|-----|------|-------|-------|
|
|-----|------|-------|------|-------|
|
||||||
| `company_site` | django (docker) | adama + roslin | active/active behind NPM |
|
| `company_site` | django (docker) | adama + roslin (+ ai-server-4080) | prod | active/active behind NPM; beta port reserved |
|
||||||
| `dta_service` | django (docker) | adama + roslin + ai-server-4080 | active/active behind NPM |
|
| `dta_service` | django (docker) | adama + roslin + ai-server-4080 | beta + prod | active/active behind NPM |
|
||||||
| `dta_webapp` | node/vite static | adama + roslin | active/active; built to `/var/www/<env>_dta_webapp`, served by web-static nginx |
|
| `dta_webapp` | node/vite static | adama + roslin (+ ai-server-4080) | beta + prod | active/active; built to `/var/www/<env>.app.ditchtheagent/html`, served by web-static nginx |
|
||||||
|
| `scha` | django (docker) | adama + roslin + ai-server-4080 | prod | active/active behind NPM; beta port reserved |
|
||||||
|
| `chat_web_app` | node-static (CRA) | adama + roslin + ai-server-4080 | prod | active/active; built to `/var/www/<env>.chat.aimloperations/html`, served by web-static nginx; beta port reserved |
|
||||||
|
| `chat_backend` | django (docker) | adama + roslin + ai-server-4080 | prod | active/active behind NPM; Ollama via `OLLAMA_BASE_URL=http://10.0.0.128:11434`; beta port reserved |
|
||||||
|
|
||||||
Both environments (`beta`, `prod`) are deployed. Django apps use a **shared external
|
Django apps use a **shared external Postgres** (via `DATABASE_URL` in each host's
|
||||||
Postgres** (via `DATABASE_URL` in each host's env file) so active/active replicas
|
env file) so active/active replicas share one database. Beta and prod never share
|
||||||
share one database.
|
a DB.
|
||||||
|
|
||||||
### Data model
|
### Data model
|
||||||
|
|
||||||
@@ -206,11 +209,18 @@ share one database.
|
|||||||
|
|
||||||
### Ports
|
### Ports
|
||||||
|
|
||||||
| App | beta | prod |
|
Reserved host ports for NPM upstreams. Ports must match across every host that
|
||||||
|-----|------|------|
|
serves the same app+env. Rows marked *not deployed* keep the port free for a
|
||||||
| company_site | 8010 | 8000 |
|
future beta replica.
|
||||||
| dta_service | 8011 | 8001 |
|
|
||||||
| dta_webapp (nginx) | 8081 | 8080 |
|
| App | beta | prod | Deployed on |
|
||||||
|
|-----|------|------|-------------|
|
||||||
|
| company_site | 8010 (*not deployed*) | 8000 | adama, roslin, ai-server-4080 |
|
||||||
|
| dta_service | 8011 | 8001 | adama, roslin, ai-server-4080 |
|
||||||
|
| scha | 8012 (*not deployed*) | 8002 | adama, roslin, ai-server-4080 |
|
||||||
|
| chat_backend | 8013 (*not deployed*) | 8003 | adama, roslin, ai-server-4080 |
|
||||||
|
| dta_webapp (nginx) | 8081 | 8080 | adama, roslin, ai-server-4080 |
|
||||||
|
| chat_web_app (nginx) | 8083 (*not deployed*) | 8082 | adama, roslin, ai-server-4080 |
|
||||||
|
|
||||||
### Flow
|
### Flow
|
||||||
|
|
||||||
@@ -229,9 +239,10 @@ share one database.
|
|||||||
to host `{{ apps_env_dir }}` → git checkout at ref → copy `.env` into checkout →
|
to host `{{ apps_env_dir }}` → git checkout at ref → copy `.env` into checkout →
|
||||||
`docker compose build` → `up -d` → migrate (run once, shared DB).
|
`docker compose build` → `up -d` → migrate (run once, shared DB).
|
||||||
- **node-static**: git checkout at ref → `npm ci` → `npm run build:<env>`
|
- **node-static**: git checkout at ref → `npm ci` → `npm run build:<env>`
|
||||||
(writes to `/var/www/<env>_dta_webapp`).
|
(writes to the app's `webroot_pattern`, e.g. `/var/www/{env}.app.ditchtheagent/html`
|
||||||
- **web-static** role: one nginx container per app host (adama + roslin) serving
|
or `/var/www/{env}.chat.aimloperations/html`).
|
||||||
the static roots on their ports; NPM balances across both.
|
- **web-static** role: one nginx container per app host serving the static roots
|
||||||
|
on their ports (from `host_apps`); NPM balances across hosts.
|
||||||
|
|
||||||
### Reverse proxy / load balancing (NPM at 10.0.0.230)
|
### Reverse proxy / load balancing (NPM at 10.0.0.230)
|
||||||
|
|
||||||
@@ -243,6 +254,15 @@ point each domain at the backend(s):
|
|||||||
adama+roslin you need the **Advanced** tab with a custom `upstream {}` block
|
adama+roslin you need the **Advanced** tab with a custom `upstream {}` block
|
||||||
(or a real LB). Confirm this before relying on active/active.
|
(or a real LB). Confirm this before relying on active/active.
|
||||||
|
|
||||||
|
| App | Domains | Backends |
|
||||||
|
|-----|---------|----------|
|
||||||
|
| company_site | aimloperations.com (+ www) | `adama:8000` + `roslin:8000` |
|
||||||
|
| dta_service | (see DTA NPM hosts) | `adama:8001` / `8011` + same on roslin / ai-server-4080 |
|
||||||
|
| dta_webapp | (see DTA NPM hosts) | `adama:8080` / `8081` + same on roslin |
|
||||||
|
| scha | `schawheaton.aimloperations.com`, `schawheaton.com` (+ www) | `adama:8002` + `roslin:8002` (+ `ai-server-4080:8002`) |
|
||||||
|
| chat_web_app | `chat.aimloperations.com` (+ www) | `adama:8082` + `roslin:8082` (+ `ai-server-4080:8082`) |
|
||||||
|
| chat_backend | `chatbackend.aimloperations.com` | `adama:8003` + `roslin:8003` + `ai-server-4080:8003` |
|
||||||
|
|
||||||
### Required changes IN each app repo (owned separately)
|
### Required changes IN each app repo (owned separately)
|
||||||
|
|
||||||
- [ ] `docker-compose.prod.yml`: drop the bundled `db` service; `web` reads
|
- [ ] `docker-compose.prod.yml`: drop the bundled `db` service; `web` reads
|
||||||
@@ -252,7 +272,11 @@ point each domain at the backend(s):
|
|||||||
with a call to `server-infra/scripts/deploy.sh --app <name> --env <env> --ref <sha>`
|
with a call to `server-infra/scripts/deploy.sh --app <name> --env <env> --ref <sha>`
|
||||||
(keep the test/docker jobs).
|
(keep the test/docker jobs).
|
||||||
- [ ] `dta_webapp`: `npm run build:beta` / `build:prod` output to
|
- [ ] `dta_webapp`: `npm run build:beta` / `build:prod` output to
|
||||||
`/var/www/beta_dta_webapp` / `/var/www/prod_dta_webapp`.
|
`/var/www/beta.app.ditchtheagent/html` / `/var/www/prod.app.ditchtheagent/html`.
|
||||||
|
- [ ] `chat_web_app`: `npm run build:beta` / `build:prod` output to
|
||||||
|
`/var/www/beta.chat.aimloperations/html` / `/var/www/prod.chat.aimloperations/html`.
|
||||||
|
|
||||||
|
Companion `chat_web_app` frontend is already registered in this infrastructure repo.
|
||||||
|
|
||||||
### Shared Postgres (10.0.0.230, same box as NPM)
|
### Shared Postgres (10.0.0.230, same box as NPM)
|
||||||
|
|
||||||
@@ -266,8 +290,12 @@ do not).
|
|||||||
| company_site | beta | `company_site_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/company_site_beta` |
|
| company_site | beta | `company_site_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/company_site_beta` |
|
||||||
| dta_service | prod | `dta_service` | `postgres://westfarn:<pw>@10.0.0.230:5432/dta_service` |
|
| dta_service | prod | `dta_service` | `postgres://westfarn:<pw>@10.0.0.230:5432/dta_service` |
|
||||||
| dta_service | beta | `dta_service_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/dta_service_beta` |
|
| dta_service | beta | `dta_service_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/dta_service_beta` |
|
||||||
|
| scha | prod | `scha` | `postgres://westfarn:<pw>@10.0.0.230:5432/scha` |
|
||||||
|
| scha | beta | `scha_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/scha_beta` |
|
||||||
|
| chat_backend | prod | `chat_backend` | `postgres://westfarn:<pw>@10.0.0.230:5432/chat_backend` |
|
||||||
|
| chat_backend | beta | `chat_backend_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/chat_backend_beta` |
|
||||||
|
|
||||||
Server prereqs on 10.0.0.230: create the 4 DBs + grant `westfarn`;
|
Server prereqs on 10.0.0.230: create each DB + grant `westfarn`;
|
||||||
`listen_addresses` covers LAN; `pg_hba.conf` allows `10.0.0.0/24`; firewall opens
|
`listen_addresses` covers LAN; `pg_hba.conf` allows `10.0.0.0/24`; firewall opens
|
||||||
5432 to `10.0.0.0/24` only.
|
5432 to `10.0.0.0/24` only.
|
||||||
|
|
||||||
@@ -301,6 +329,10 @@ Server prereqs on 10.0.0.230: create the 4 DBs + grant `westfarn`;
|
|||||||
| `server-infra` checkout | Playbooks + inventory |
|
| `server-infra` checkout | Playbooks + inventory |
|
||||||
| SSH key to adama + roslin | Deploy fan-out |
|
| SSH key to adama + roslin | Deploy fan-out |
|
||||||
|
|
||||||
|
On every push or merged PR to `master`, `.gitea/workflows/sync-checkout.yml`
|
||||||
|
fast-forward pulls this repo at `~/Documents/repos/server-infra` on the Act
|
||||||
|
runner so playbooks and inventory stay current without a manual `git pull`.
|
||||||
|
|
||||||
## SSH Keys for CI Deploy
|
## SSH Keys for CI Deploy
|
||||||
|
|
||||||
| Key | Used by | Purpose |
|
| Key | Used by | Purpose |
|
||||||
@@ -333,8 +365,12 @@ Store vault password for CI in a file readable only by the Act runner (e.g. `~/.
|
|||||||
| 6 | Test on single server: `./scripts/provision.sh adama` | Manual |
|
| 6 | Test on single server: `./scripts/provision.sh adama` | Manual |
|
||||||
| 7 | Provision all: `./scripts/provision.sh` | Manual |
|
| 7 | Provision all: `./scripts/provision.sh` | Manual |
|
||||||
| 8 | Deploy SSH key for Act runner | Future |
|
| 8 | Deploy SSH key for Act runner | Future |
|
||||||
|
| 8a | Auto-sync runner checkout on `master` (`.gitea/workflows/sync-checkout.yml`) | Done |
|
||||||
| 9 | Stub `deploy-apps.yml` + update `company_site` workflow | Future |
|
| 9 | Stub `deploy-apps.yml` + update `company_site` workflow | Future |
|
||||||
| 10 | Dockerize `company_site` | Future (separate ticket) |
|
| 10 | Dockerize `company_site` | Future (separate ticket) |
|
||||||
|
| 10a | Register + deploy `scha` (all webservers, port 8002) | In progress ([scha#19](https://git.aimloperations.com/ai_ml_operations/scha/issues/19)) |
|
||||||
|
| 10b | Register + deploy `chat_web_app` (node-static, port 8082) | In progress ([chat_web_app#13](https://git.aimloperations.com/ai_ml_operations/chat_web_app/issues/13)) |
|
||||||
|
| 10c | Register + deploy `chat_backend` (django, port 8003) | In progress ([chat_backend#6](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/6)) |
|
||||||
| 11 | Gitea container registry (optional) | Future |
|
| 11 | Gitea container registry (optional) | Future |
|
||||||
|
|
||||||
## Open Decisions
|
## Open Decisions
|
||||||
|
|||||||
@@ -80,6 +80,29 @@ app_catalog:
|
|||||||
webroot_pattern: "/var/www/{env}.app.ditchtheagent/html"
|
webroot_pattern: "/var/www/{env}.app.ditchtheagent/html"
|
||||||
# deploy runs `npm ci` then `npm run build:<env>`; that script writes to
|
# deploy runs `npm ci` then `npm run build:<env>`; that script writes to
|
||||||
# {{ web_static_root }}/<env>_dta_webapp (beta/prod), served by web-static.
|
# {{ web_static_root }}/<env>_dta_webapp (beta/prod), served by web-static.
|
||||||
|
scha:
|
||||||
|
type: django
|
||||||
|
repo: "{{ git_base_url }}/ai_ml_operations/scha.git"
|
||||||
|
default_branch: master
|
||||||
|
compose_file: docker-compose.prod.yml
|
||||||
|
web_service: web
|
||||||
|
migrate_cmd: "uv run python manage.py migrate --noinput"
|
||||||
|
chat_web_app:
|
||||||
|
type: node-static
|
||||||
|
repo: "{{ git_base_url }}/ai_ml_operations/chat_web_app.git"
|
||||||
|
default_branch: master
|
||||||
|
# package.json lives in this subdir of the checkout (npm runs here).
|
||||||
|
subdir: llm-fe
|
||||||
|
# Document root served by nginx and written by `npm run build:<env>`.
|
||||||
|
# {env} is replaced with the entry's env (beta/prod).
|
||||||
|
webroot_pattern: "/var/www/{env}.chat.aimloperations/html"
|
||||||
|
chat_backend:
|
||||||
|
type: django
|
||||||
|
repo: "{{ git_base_url }}/ai_ml_operations/chat_backend.git"
|
||||||
|
default_branch: master
|
||||||
|
compose_file: docker-compose.prod.yml
|
||||||
|
web_service: web
|
||||||
|
migrate_cmd: "uv run python manage.py migrate --noinput"
|
||||||
|
|
||||||
# Deploy filter vars. CI passes these; manual runs may leave them undefined
|
# Deploy filter vars. CI passes these; manual runs may leave them undefined
|
||||||
# to (re)deploy every app listed in the host's host_apps.
|
# to (re)deploy every app listed in the host's host_apps.
|
||||||
|
|||||||
@@ -10,3 +10,9 @@ host_apps:
|
|||||||
- { name: dta_service, env: beta, port: 8011 }
|
- { name: dta_service, env: beta, port: 8011 }
|
||||||
- { name: dta_webapp, env: prod, port: 8080 }
|
- { name: dta_webapp, env: prod, port: 8080 }
|
||||||
- { name: dta_webapp, env: beta, port: 8081 }
|
- { name: dta_webapp, env: beta, port: 8081 }
|
||||||
|
- { name: scha, env: prod, port: 8002 }
|
||||||
|
# optional: - { name: scha, env: beta, port: 8012 }
|
||||||
|
- { name: chat_web_app, env: prod, port: 8082 }
|
||||||
|
# optional: - { name: chat_web_app, env: beta, port: 8083 }
|
||||||
|
- { name: chat_backend, env: prod, port: 8003 }
|
||||||
|
# optional: - { name: chat_backend, env: beta, port: 8013 }
|
||||||
|
|||||||
@@ -24,3 +24,8 @@ host_apps:
|
|||||||
- { name: dta_webapp, env: beta, port: 8081 }
|
- { name: dta_webapp, env: beta, port: 8081 }
|
||||||
- { name: dta_service, env: prod, port: 8001 }
|
- { name: dta_service, env: prod, port: 8001 }
|
||||||
- { name: dta_service, env: beta, port: 8011 }
|
- { name: dta_service, env: beta, port: 8011 }
|
||||||
|
- { name: scha, env: prod, port: 8002 }
|
||||||
|
- { name: chat_web_app, env: prod, port: 8082 }
|
||||||
|
# optional: - { name: chat_web_app, env: beta, port: 8083 }
|
||||||
|
- { name: chat_backend, env: prod, port: 8003 }
|
||||||
|
# optional: - { name: chat_backend, env: beta, port: 8013 }
|
||||||
@@ -7,3 +7,9 @@ host_apps:
|
|||||||
- { name: dta_service, env: beta, port: 8011 }
|
- { name: dta_service, env: beta, port: 8011 }
|
||||||
- { name: dta_webapp, env: prod, port: 8080 }
|
- { name: dta_webapp, env: prod, port: 8080 }
|
||||||
- { name: dta_webapp, env: beta, port: 8081 }
|
- { name: dta_webapp, env: beta, port: 8081 }
|
||||||
|
- { name: scha, env: prod, port: 8002 }
|
||||||
|
# optional: - { name: scha, env: beta, port: 8012 }
|
||||||
|
- { name: chat_web_app, env: prod, port: 8082 }
|
||||||
|
# optional: - { name: chat_web_app, env: beta, port: 8083 }
|
||||||
|
- { name: chat_backend, env: prod, port: 8003 }
|
||||||
|
# optional: - { name: chat_backend, env: beta, port: 8013 }
|
||||||
|
|||||||
+1
-1
@@ -17,7 +17,7 @@ Deploy applications with deploy-apps.yml.
|
|||||||
HOST Optional. Limit to one host: adama, roslin, or ai-server-4080.
|
HOST Optional. Limit to one host: adama, roslin, or ai-server-4080.
|
||||||
|
|
||||||
Options:
|
Options:
|
||||||
--app NAME App to deploy (company_site, dta_service, dta_webapp)
|
--app NAME App to deploy (company_site, dta_service, dta_webapp, scha, chat_web_app, chat_backend)
|
||||||
--env ENV Environment: beta or prod
|
--env ENV Environment: beta or prod
|
||||||
--ref REF Git ref/sha to deploy (default: master)
|
--ref REF Git ref/sha to deploy (default: master)
|
||||||
--check Dry run
|
--check Dry run
|
||||||
|
|||||||
Reference in New Issue
Block a user