diff --git a/PHASE3_DEPLOYMENT_CHECKLIST.md b/PHASE3_DEPLOYMENT_CHECKLIST.md new file mode 100644 index 0000000..af038a0 --- /dev/null +++ b/PHASE3_DEPLOYMENT_CHECKLIST.md @@ -0,0 +1,298 @@ +# Phase 3: Runboat E2E & Performance Testing — Deployment Checklist + +**Status**: Ready for deployment +**Last Updated**: 2026-05-30 + +## Pre-Deployment Checklist + +### 1. Code Readiness + +- [x] E2E test infrastructure complete (conftest.py) +- [x] 19 E2E scenarios implemented +- [x] 7 performance benchmark tests implemented +- [x] PHASE3_ROADMAP.md documented +- [x] .gitlab-ci.yml updated with preview + e2e + performance stages +- [x] CLAUDE.md updated with SLO targets and patterns +- [x] All code committed and pushed to main + +### 2. CI/CD Variables Setup (Action Required) + +Before merging Phase 3 code, set these variables in **GitLab Project Settings → CI/CD Variables**: + +#### Variable 1: RUNBOAT_API_URL +- **Type**: Secret +- **Value**: Runboat API endpoint (e.g., `https://api.runboat.dev`) +- **Protected**: Yes +- **Masked**: Yes +- **Source**: Request from Acsone/infrastructure team + +#### Variable 2: RUNBOAT_TOKEN +- **Type**: Secret +- **Value**: Bearer token for Runboat API authentication +- **Protected**: Yes +- **Masked**: Yes +- **Source**: Request from Acsone/infrastructure team +- **Format**: `rbk_...` (Runboat Bearer Key) + +#### Variable 3: GITLAB_BOT_TOKEN +- **Type**: Secret +- **Value**: GitLab personal/bot token for posting MR comments +- **Protected**: Yes +- **Masked**: Yes +- **Source**: **Create locally** via GitLab Settings → Access Tokens + - Token name: `itsulu-blog-publisher-bot` + - Scopes: `api`, `read_api`, `read_repository` + - Expiration: 1 year + - Save and copy the value + +### 3. Runboat Account Verification + +Before deploying, verify Runboat is accessible: + +```bash +# Test Runboat API (after variables are set) +curl -fsSL -X POST "${RUNBOAT_API_URL}/builds" \ + -H "Authorization: Bearer ${RUNBOAT_TOKEN}" \ + -H "Content-Type: application/json" \ + -d '{"repo":"example/repo","sha":"abc123"}' + +# Should return JSON with "url" field (success) +# Or 401 error (invalid token — fix variables) +``` + +### 4. E2E Test Dry-Run (Local) + +Test E2E suite against local dev instance: + +```bash +# Install dependencies +pip install -r e2e/requirements.txt + +# Run against local Odoo (default http://localhost:8069) +pytest e2e/ -v + +# Or specify custom URL +ODOO_BASE_URL=http://staging:8069 pytest e2e/ -v +``` + +Expected result: +- All 19 scenarios should execute +- Some may fail if features not in local DB (normal for dry-run) +- Key is to verify Playwright fixtures work + +### 5. Performance Test Dry-Run (Local) + +Test performance tests against local instance: + +```bash +# Run performance tests +pytest addons/itsulu_blog_publisher/tests/test_performance.py \ + -m performance -v + +# Should output: +# - test_generation_latency_under_30_seconds PASSED +# - test_query_count tests PASSED +# - test_token_usage_baseline PASSED +# - test_concurrent_generation PASSED +``` + +## Deployment Steps + +### Step 1: Create/Verify GitLab Bot Token + +```bash +# In GitLab web UI: +1. Click your avatar → Settings → Access Tokens +2. Click "Add New Token" +3. Fill: + - Token name: "itsulu-blog-publisher-bot" + - Scopes: Check api, read_api, read_repository + - Expiration: 1 year from now +4. Click "Create Personal Access Token" +5. Copy the token value (only shown once) +``` + +### Step 2: Set CI/CD Variables + +```bash +# In GitLab web UI: +1. Project → Settings → CI/CD → Variables +2. Add RUNBOAT_API_URL (get from infrastructure team) +3. Add RUNBOAT_TOKEN (get from infrastructure team) +4. Add GITLAB_BOT_TOKEN (paste from step 1) +5. Check all are: Protected=Yes, Masked=Yes +6. Click Save +``` + +### Step 3: Push Phase 3 Code to Feature Branch + +```bash +# Verify all commits are in git log +git log --oneline -5 + +# Should see: +# d122b77 feat: integrate Runboat E2E testing into CI/CD pipeline +# acfa1d9 feat: establish Phase 3 E2E testing infrastructure +# 7ee393a feat: add performance benchmark tests +# ... Phase 2 commits ... +``` + +### Step 4: Create Merge Request + +```bash +# Push to feature branch +git push origin main:refs/heads/phase-3-e2e-testing + +# Create MR in GitLab UI +# Title: "Phase 3: Runboat E2E Testing & Performance Benchmarks" +# Description: +# - E2E infrastructure with Runboat polling and auth fixtures +# - 19 E2E test scenarios (generation, scheduling, error recovery) +# - 7 performance benchmark tests (latency, queries, tokens, concurrency) +# - CI/CD pipeline integration (preview + e2e + performance stages) +# - SLO targets established for 7 performance metrics +``` + +### Step 5: Verify Pipeline Runs + +When MR is created, GitLab CI/CD should automatically start: + +1. **Lint stage** (2 min) + - black --check + - pylint-odoo + - Status: ✅ Should pass + +2. **Test stage** (10 min) — runs in parallel: + - unit_tests: TDD + BDD tests + - performance_tests: Performance benchmarks + - Status: ✅ Should pass + +3. **Build stage** (3 min) + - Docker image build & push + - Status: ✅ Should pass + +4. **Preview stage** (5 min) + - runboat_preview: Requests Runboat build + - Status: ✅ Should create preview URL (or ⚠️ if Runboat unavailable) + +5. **E2E stage** (15 min) — waits for preview + - e2e_tests: Runs Playwright scenarios + - Status: ✅ Should pass or ⚠️ if preview not ready + +### Step 6: Review Pipeline Artifacts + +After pipeline completes, download/review artifacts: + +- **report-unit.html** — Unit test coverage report +- **report-performance.html** — Performance test results +- **report-e2e.html** — E2E test results (if ran) +- **e2e/traces/** — Playwright trace files for debugging + +## Troubleshooting + +### Runboat Preview Fails + +**Error**: `Authorization failed` or `401 Unauthorized` + +**Fix**: +1. Verify RUNBOAT_TOKEN is set in CI/CD variables +2. Check token is not expired (request new from infrastructure team) +3. Verify RUNBOAT_API_URL is correct + +**Error**: `Cannot reach Runboat API` + +**Fix**: +1. Verify RUNBOAT_API_URL is not behind firewall +2. Check infrastructure team has enabled preview access for this project + +### E2E Tests Timeout + +**Error**: `Runboat instance not ready after 180s` + +**Fix**: +1. Runboat cold-start can take 60s, normal behavior +2. If > 180s, Runboat infrastructure may be overloaded +3. Retry pipeline after 30 minutes + +### E2E Tests Fail (Selector Not Found) + +**Error**: `locator.fill('value') → TimeoutError: locator did not resolve` + +**Fix**: +1. Form field names may differ in template DB +2. Update selector in test file to match actual field names +3. Use `page.pause()` to inspect live page: `pytest e2e/ --pdb-trace` + +### Performance Tests Fail + +**Error**: `AssertionError: Generation took 35s, target <30s` + +**Fix**: +1. Mock LLM response is artificially fast +2. Real API calls will add network time +3. If local test > 30s, investigate N+1 queries or missing indexes + +## Post-Deployment + +### Ongoing Maintenance + +After Phase 3 is live: + +1. **Monitor E2E Results** (Weekly) + - Check MR pipeline E2E reports + - Flag flaky scenarios (run 3× to verify) + - Update fixtures if Odoo UI changes + +2. **Track Performance Metrics** (Weekly) + - Monitor `ir_logging` table for generation latency + - Alert if P99 latency > 60s (bottleneck detected) + - Adjust SLOs if production baseline differs + +3. **Update Runboat Token** (Annually) + - RUNBOAT_TOKEN expires yearly + - Request renewal 1 month before expiration + - Update CI/CD variable before it expires + +4. **Runboat Account Maintenance** + - Monitor preview instance quota + - Cleanup old instances if quota fills + - Contact Acsone if persistent issues + +### Success Criteria + +✅ **Phase 3 is successful when**: + +- [ ] All MRs include E2E test results +- [ ] No E2E flakiness (< 2% failure rate) +- [ ] Performance baselines recorded in first full month +- [ ] Runboat previews available for all MRs +- [ ] Zero issues with Runboat integration after 1 week +- [ ] Performance SLOs consistently met +- [ ] Team trained on reading E2E/performance reports + +## Reference Links + +- [Runboat Documentation](https://docs.acsone.eu/runboat/) +- [Playwright Python API](https://playwright.dev/python/) +- [.gitlab-ci.yml Pipeline](https://gitlab.com/YOUR_PROJECT/-/blob/main/.gitlab-ci.yml) +- [PHASE3_ROADMAP.md](./PHASE3_ROADMAP.md) +- [CLAUDE.md Testing Framework](./CLAUDE.md) + +## Approval Sign-Off + +This Phase 3 deployment is ready when: + +- [ ] Code review approved +- [ ] All CI/CD variables set (RUNBOAT_API_URL, RUNBOAT_TOKEN, GITLAB_BOT_TOKEN) +- [ ] Runboat account verified operational +- [ ] Local dry-runs pass (E2E + performance tests) +- [ ] Infrastructure team confirms Runboat access +- [ ] At least one team member trained on results interpretation + +**Deploy**: Create MR and monitor first pipeline run + +--- + +**Deployment Date**: (To be filled on merge) +**Deployed By**: (To be filled on merge) +**Notes**: (To be filled with any issues encountered)