docs: add Phase 3 deployment checklist with CI/CD variable setup
Created comprehensive deployment guide for Phase 3 Runboat E2E testing: Pre-Deployment Checklist: - Code readiness verification (all Phase 3 artifacts listed) - CI/CD variables setup with exact requirements: * RUNBOAT_API_URL (from infrastructure team) * RUNBOAT_TOKEN (from infrastructure team) * GITLAB_BOT_TOKEN (create locally via GitLab access tokens) - Runboat account verification script - E2E test dry-run instructions - Performance test dry-run instructions Deployment Steps (Step-by-step): 1. Create GitLab bot token (copy/paste walkthrough) 2. Set CI/CD variables in GitLab UI (protected, masked) 3. Push Phase 3 code to feature branch 4. Create MR with standard description 5. Verify pipeline runs (expected stages + timing) 6. Review pipeline artifacts (reports + traces) Troubleshooting Guide: - Runboat preview fails: Authorization, timeout, connectivity - E2E selector failures: Form field mismatches - Performance test failures: Network time vs mocked - With fixes and verification steps for each Post-Deployment: - Weekly monitoring procedures - Annual token renewal reminder - Success criteria checklist - Approval sign-off section Expected Pipeline Timeline: - Lint: 2 min - Test (parallel): 10 min - Build: 3 min - Preview: 5 min - E2E: 15 min - Total: ~35 minutes Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
This commit is contained in:
parent
d122b773d4
commit
124d9308c7
1 changed files with 298 additions and 0 deletions
298
PHASE3_DEPLOYMENT_CHECKLIST.md
Normal file
298
PHASE3_DEPLOYMENT_CHECKLIST.md
Normal file
|
|
@ -0,0 +1,298 @@
|
|||
# Phase 3: Runboat E2E & Performance Testing — Deployment Checklist
|
||||
|
||||
**Status**: Ready for deployment
|
||||
**Last Updated**: 2026-05-30
|
||||
|
||||
## Pre-Deployment Checklist
|
||||
|
||||
### 1. Code Readiness
|
||||
|
||||
- [x] E2E test infrastructure complete (conftest.py)
|
||||
- [x] 19 E2E scenarios implemented
|
||||
- [x] 7 performance benchmark tests implemented
|
||||
- [x] PHASE3_ROADMAP.md documented
|
||||
- [x] .gitlab-ci.yml updated with preview + e2e + performance stages
|
||||
- [x] CLAUDE.md updated with SLO targets and patterns
|
||||
- [x] All code committed and pushed to main
|
||||
|
||||
### 2. CI/CD Variables Setup (Action Required)
|
||||
|
||||
Before merging Phase 3 code, set these variables in **GitLab Project Settings → CI/CD Variables**:
|
||||
|
||||
#### Variable 1: RUNBOAT_API_URL
|
||||
- **Type**: Secret
|
||||
- **Value**: Runboat API endpoint (e.g., `https://api.runboat.dev`)
|
||||
- **Protected**: Yes
|
||||
- **Masked**: Yes
|
||||
- **Source**: Request from Acsone/infrastructure team
|
||||
|
||||
#### Variable 2: RUNBOAT_TOKEN
|
||||
- **Type**: Secret
|
||||
- **Value**: Bearer token for Runboat API authentication
|
||||
- **Protected**: Yes
|
||||
- **Masked**: Yes
|
||||
- **Source**: Request from Acsone/infrastructure team
|
||||
- **Format**: `rbk_...` (Runboat Bearer Key)
|
||||
|
||||
#### Variable 3: GITLAB_BOT_TOKEN
|
||||
- **Type**: Secret
|
||||
- **Value**: GitLab personal/bot token for posting MR comments
|
||||
- **Protected**: Yes
|
||||
- **Masked**: Yes
|
||||
- **Source**: **Create locally** via GitLab Settings → Access Tokens
|
||||
- Token name: `itsulu-blog-publisher-bot`
|
||||
- Scopes: `api`, `read_api`, `read_repository`
|
||||
- Expiration: 1 year
|
||||
- Save and copy the value
|
||||
|
||||
### 3. Runboat Account Verification
|
||||
|
||||
Before deploying, verify Runboat is accessible:
|
||||
|
||||
```bash
|
||||
# Test Runboat API (after variables are set)
|
||||
curl -fsSL -X POST "${RUNBOAT_API_URL}/builds" \
|
||||
-H "Authorization: Bearer ${RUNBOAT_TOKEN}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"repo":"example/repo","sha":"abc123"}'
|
||||
|
||||
# Should return JSON with "url" field (success)
|
||||
# Or 401 error (invalid token — fix variables)
|
||||
```
|
||||
|
||||
### 4. E2E Test Dry-Run (Local)
|
||||
|
||||
Test E2E suite against local dev instance:
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
pip install -r e2e/requirements.txt
|
||||
|
||||
# Run against local Odoo (default http://localhost:8069)
|
||||
pytest e2e/ -v
|
||||
|
||||
# Or specify custom URL
|
||||
ODOO_BASE_URL=http://staging:8069 pytest e2e/ -v
|
||||
```
|
||||
|
||||
Expected result:
|
||||
- All 19 scenarios should execute
|
||||
- Some may fail if features not in local DB (normal for dry-run)
|
||||
- Key is to verify Playwright fixtures work
|
||||
|
||||
### 5. Performance Test Dry-Run (Local)
|
||||
|
||||
Test performance tests against local instance:
|
||||
|
||||
```bash
|
||||
# Run performance tests
|
||||
pytest addons/itsulu_blog_publisher/tests/test_performance.py \
|
||||
-m performance -v
|
||||
|
||||
# Should output:
|
||||
# - test_generation_latency_under_30_seconds PASSED
|
||||
# - test_query_count tests PASSED
|
||||
# - test_token_usage_baseline PASSED
|
||||
# - test_concurrent_generation PASSED
|
||||
```
|
||||
|
||||
## Deployment Steps
|
||||
|
||||
### Step 1: Create/Verify GitLab Bot Token
|
||||
|
||||
```bash
|
||||
# In GitLab web UI:
|
||||
1. Click your avatar → Settings → Access Tokens
|
||||
2. Click "Add New Token"
|
||||
3. Fill:
|
||||
- Token name: "itsulu-blog-publisher-bot"
|
||||
- Scopes: Check api, read_api, read_repository
|
||||
- Expiration: 1 year from now
|
||||
4. Click "Create Personal Access Token"
|
||||
5. Copy the token value (only shown once)
|
||||
```
|
||||
|
||||
### Step 2: Set CI/CD Variables
|
||||
|
||||
```bash
|
||||
# In GitLab web UI:
|
||||
1. Project → Settings → CI/CD → Variables
|
||||
2. Add RUNBOAT_API_URL (get from infrastructure team)
|
||||
3. Add RUNBOAT_TOKEN (get from infrastructure team)
|
||||
4. Add GITLAB_BOT_TOKEN (paste from step 1)
|
||||
5. Check all are: Protected=Yes, Masked=Yes
|
||||
6. Click Save
|
||||
```
|
||||
|
||||
### Step 3: Push Phase 3 Code to Feature Branch
|
||||
|
||||
```bash
|
||||
# Verify all commits are in git log
|
||||
git log --oneline -5
|
||||
|
||||
# Should see:
|
||||
# d122b77 feat: integrate Runboat E2E testing into CI/CD pipeline
|
||||
# acfa1d9 feat: establish Phase 3 E2E testing infrastructure
|
||||
# 7ee393a feat: add performance benchmark tests
|
||||
# ... Phase 2 commits ...
|
||||
```
|
||||
|
||||
### Step 4: Create Merge Request
|
||||
|
||||
```bash
|
||||
# Push to feature branch
|
||||
git push origin main:refs/heads/phase-3-e2e-testing
|
||||
|
||||
# Create MR in GitLab UI
|
||||
# Title: "Phase 3: Runboat E2E Testing & Performance Benchmarks"
|
||||
# Description:
|
||||
# - E2E infrastructure with Runboat polling and auth fixtures
|
||||
# - 19 E2E test scenarios (generation, scheduling, error recovery)
|
||||
# - 7 performance benchmark tests (latency, queries, tokens, concurrency)
|
||||
# - CI/CD pipeline integration (preview + e2e + performance stages)
|
||||
# - SLO targets established for 7 performance metrics
|
||||
```
|
||||
|
||||
### Step 5: Verify Pipeline Runs
|
||||
|
||||
When MR is created, GitLab CI/CD should automatically start:
|
||||
|
||||
1. **Lint stage** (2 min)
|
||||
- black --check
|
||||
- pylint-odoo
|
||||
- Status: ✅ Should pass
|
||||
|
||||
2. **Test stage** (10 min) — runs in parallel:
|
||||
- unit_tests: TDD + BDD tests
|
||||
- performance_tests: Performance benchmarks
|
||||
- Status: ✅ Should pass
|
||||
|
||||
3. **Build stage** (3 min)
|
||||
- Docker image build & push
|
||||
- Status: ✅ Should pass
|
||||
|
||||
4. **Preview stage** (5 min)
|
||||
- runboat_preview: Requests Runboat build
|
||||
- Status: ✅ Should create preview URL (or ⚠️ if Runboat unavailable)
|
||||
|
||||
5. **E2E stage** (15 min) — waits for preview
|
||||
- e2e_tests: Runs Playwright scenarios
|
||||
- Status: ✅ Should pass or ⚠️ if preview not ready
|
||||
|
||||
### Step 6: Review Pipeline Artifacts
|
||||
|
||||
After pipeline completes, download/review artifacts:
|
||||
|
||||
- **report-unit.html** — Unit test coverage report
|
||||
- **report-performance.html** — Performance test results
|
||||
- **report-e2e.html** — E2E test results (if ran)
|
||||
- **e2e/traces/** — Playwright trace files for debugging
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Runboat Preview Fails
|
||||
|
||||
**Error**: `Authorization failed` or `401 Unauthorized`
|
||||
|
||||
**Fix**:
|
||||
1. Verify RUNBOAT_TOKEN is set in CI/CD variables
|
||||
2. Check token is not expired (request new from infrastructure team)
|
||||
3. Verify RUNBOAT_API_URL is correct
|
||||
|
||||
**Error**: `Cannot reach Runboat API`
|
||||
|
||||
**Fix**:
|
||||
1. Verify RUNBOAT_API_URL is not behind firewall
|
||||
2. Check infrastructure team has enabled preview access for this project
|
||||
|
||||
### E2E Tests Timeout
|
||||
|
||||
**Error**: `Runboat instance not ready after 180s`
|
||||
|
||||
**Fix**:
|
||||
1. Runboat cold-start can take 60s, normal behavior
|
||||
2. If > 180s, Runboat infrastructure may be overloaded
|
||||
3. Retry pipeline after 30 minutes
|
||||
|
||||
### E2E Tests Fail (Selector Not Found)
|
||||
|
||||
**Error**: `locator.fill('value') → TimeoutError: locator did not resolve`
|
||||
|
||||
**Fix**:
|
||||
1. Form field names may differ in template DB
|
||||
2. Update selector in test file to match actual field names
|
||||
3. Use `page.pause()` to inspect live page: `pytest e2e/ --pdb-trace`
|
||||
|
||||
### Performance Tests Fail
|
||||
|
||||
**Error**: `AssertionError: Generation took 35s, target <30s`
|
||||
|
||||
**Fix**:
|
||||
1. Mock LLM response is artificially fast
|
||||
2. Real API calls will add network time
|
||||
3. If local test > 30s, investigate N+1 queries or missing indexes
|
||||
|
||||
## Post-Deployment
|
||||
|
||||
### Ongoing Maintenance
|
||||
|
||||
After Phase 3 is live:
|
||||
|
||||
1. **Monitor E2E Results** (Weekly)
|
||||
- Check MR pipeline E2E reports
|
||||
- Flag flaky scenarios (run 3× to verify)
|
||||
- Update fixtures if Odoo UI changes
|
||||
|
||||
2. **Track Performance Metrics** (Weekly)
|
||||
- Monitor `ir_logging` table for generation latency
|
||||
- Alert if P99 latency > 60s (bottleneck detected)
|
||||
- Adjust SLOs if production baseline differs
|
||||
|
||||
3. **Update Runboat Token** (Annually)
|
||||
- RUNBOAT_TOKEN expires yearly
|
||||
- Request renewal 1 month before expiration
|
||||
- Update CI/CD variable before it expires
|
||||
|
||||
4. **Runboat Account Maintenance**
|
||||
- Monitor preview instance quota
|
||||
- Cleanup old instances if quota fills
|
||||
- Contact Acsone if persistent issues
|
||||
|
||||
### Success Criteria
|
||||
|
||||
✅ **Phase 3 is successful when**:
|
||||
|
||||
- [ ] All MRs include E2E test results
|
||||
- [ ] No E2E flakiness (< 2% failure rate)
|
||||
- [ ] Performance baselines recorded in first full month
|
||||
- [ ] Runboat previews available for all MRs
|
||||
- [ ] Zero issues with Runboat integration after 1 week
|
||||
- [ ] Performance SLOs consistently met
|
||||
- [ ] Team trained on reading E2E/performance reports
|
||||
|
||||
## Reference Links
|
||||
|
||||
- [Runboat Documentation](https://docs.acsone.eu/runboat/)
|
||||
- [Playwright Python API](https://playwright.dev/python/)
|
||||
- [.gitlab-ci.yml Pipeline](https://gitlab.com/YOUR_PROJECT/-/blob/main/.gitlab-ci.yml)
|
||||
- [PHASE3_ROADMAP.md](./PHASE3_ROADMAP.md)
|
||||
- [CLAUDE.md Testing Framework](./CLAUDE.md)
|
||||
|
||||
## Approval Sign-Off
|
||||
|
||||
This Phase 3 deployment is ready when:
|
||||
|
||||
- [ ] Code review approved
|
||||
- [ ] All CI/CD variables set (RUNBOAT_API_URL, RUNBOAT_TOKEN, GITLAB_BOT_TOKEN)
|
||||
- [ ] Runboat account verified operational
|
||||
- [ ] Local dry-runs pass (E2E + performance tests)
|
||||
- [ ] Infrastructure team confirms Runboat access
|
||||
- [ ] At least one team member trained on results interpretation
|
||||
|
||||
**Deploy**: Create MR and monitor first pipeline run
|
||||
|
||||
---
|
||||
|
||||
**Deployment Date**: (To be filled on merge)
|
||||
**Deployed By**: (To be filled on merge)
|
||||
**Notes**: (To be filled with any issues encountered)
|
||||
Loading…
Reference in a new issue