d79cb6223e
Because kubernetes_asyncio watcher have correctness issues, as they seem to re-emit events that pre-date the initial load. Although I ended up having to implement an infinite watcher, to handle socket timeout errors...
127 lines
3.5 KiB
Markdown
127 lines
3.5 KiB
Markdown
# runboat ☸️
|
|
|
|
A simple Odoo runbot lookalike on kubernetes. Main goal is replacing the OCA runbot.
|
|
|
|
## Requirements
|
|
|
|
For running the builds:
|
|
|
|
- A namespace in a kubernetes cluster.
|
|
- A wildcard DNS domain that points to the kubernetes ingress.
|
|
- A postgres database, accessible from within the cluster namespace with a user with
|
|
permissions to create database.
|
|
|
|
For running the controller:
|
|
|
|
- Python 3.10
|
|
- `kubectl`
|
|
- A `KUBECONFIG` that provides access to the namespace where the builds are deployed,
|
|
with permissions to create and delete Service, Job, Deployment, Ingress, Secret and
|
|
ConfigMap resources.
|
|
|
|
## Developing
|
|
|
|
- setup environment variables (start from `.env.sample`)
|
|
- create a virtualenv, make sure to have pip>=21.3.1 and `pip install -e .`
|
|
- run with `uvicorn runboat.app:app --log-config=log-config-dev.yaml`
|
|
|
|
## Running in production
|
|
|
|
`gunicorn -w 1 -k runboat.uvicorn.RunboatUvicornWorker runboat.app:app`.
|
|
|
|
One and only one worker process !
|
|
|
|
Gunicorn also necessary so SIGINT/SIGTERM shutdowns after a few seconds. Since we use
|
|
`run_in_executor`, SIGINT/SIGTERM handling does not work very well in python, and
|
|
gunicorn makes it more robust. https://bugs.python.org/issue29309
|
|
|
|
## Author and contributors
|
|
|
|
Authored by Stéphane Bidoul (@sbidoul).
|
|
|
|
Contributions welcome.
|
|
|
|
## TODO
|
|
|
|
Prototype (min required to do load testing):
|
|
|
|
- plug it on a bunch of OCA and shopinvader repos to test load
|
|
- configuring many repos in a .env file may be difficult, switch to a toml file ?
|
|
|
|
MVP:
|
|
|
|
- deployment and more load testing
|
|
- build/log and build/init-log api endpoints
|
|
- report build status to github
|
|
- secure github webhooks
|
|
- k8s init container timeout
|
|
- better error handling in API (return 400 on user errors)
|
|
- basic tests
|
|
- build and publish runboat container image
|
|
- look at other TODO in code to see if anything important remains
|
|
- basic UI (single page with a combo box to select repo and show builds by branch/pr,
|
|
with start/stop buttons)
|
|
|
|
More:
|
|
|
|
- shiny UI
|
|
- websocket stream of build changes, for a dynamic UI
|
|
- handle PR close (delete all builds for PR)
|
|
- handle branch delete (delete all builds for branch)
|
|
- create builds for all supported repos on startup (goes with sticky branches)
|
|
- never undeploy last build of sticky branches
|
|
- make build images configurable (see `build_images.py`)
|
|
|
|
|
|
## Kubefiles
|
|
|
|
Kustomize template with 3 modes (deploy, initialize, cleanup).
|
|
|
|
## Synchronous actions on builds (fast)
|
|
|
|
- deploy
|
|
|
|
- create deployment with 0 replicas and runboat/init-status="todo"
|
|
|
|
- start:
|
|
|
|
- if runboat/init-status=="ready", scale to 1
|
|
- elif runboat/init-status in ("todo", "initializing"), do nothing
|
|
- elif runboat/init-status=="failed", set runboat/init-status="todo"
|
|
|
|
- stop:
|
|
|
|
- scale deployment to 0
|
|
|
|
- undeploy:
|
|
|
|
- scale deployment to 0
|
|
- set runboat/init-status to "dropping"
|
|
- start dropdb job (restart=Never, backoffLimit=6)
|
|
|
|
## Workers
|
|
|
|
- initializer (works on deployments with runboat/init-status="todo", ordered by
|
|
runboat/init-status-timestamp), obeying max_initializing:
|
|
|
|
- set runboat/init-status to "initializing"
|
|
- (re)create init job which will drop and init db (restart=Never, backoffLimit=0)
|
|
|
|
- job-watcher:
|
|
|
|
- on successful termination of initdb job: set runboat/init-status to "ready", scale
|
|
deployment to 1
|
|
- on failure of initdb job: set runboat/init-status to "failed"
|
|
- on success of dropdb job: delete all resources
|
|
|
|
- deployment-watcher:
|
|
|
|
- maintains an in-memory db of deployments
|
|
|
|
- stopper:
|
|
|
|
- stop old started, to reach max_running
|
|
|
|
- undeployer:
|
|
|
|
- undeploy old stopped, to reach max_deployed
|