itsulu-blog-publisher/PHASE3_DEPLOYMENT_CHECKLIST.md

Phase 3: K8s-Based E2E & Performance Testing — Deployment Checklist

Status: Ready for deployment on ITSulu K8s cluster
Environment: itsulu-testing namespace
Last Updated: 2026-05-30

Pre-Deployment Checklist

1. Code Readiness

• E2E test infrastructure complete (conftest.py)
• 19 E2E scenarios implemented
• 7 performance benchmark tests implemented
• PHASE3_ROADMAP.md documented
• .gitlab-ci.yml updated with preview + e2e + performance stages
• CLAUDE.md updated with SLO targets and patterns
• All code committed and pushed to main

2. CI/CD Variable Setup (Action Required)

Before merging Phase 3 code, set this variable in GitLab Project Settings → CI/CD Variables:

Variable: KUBE_CONFIG

• Type: Secret
• Value: Base64-encoded kubeconfig for ITSulu K8s cluster access
• Protected: Yes
• Masked: Yes
• Source: From infrastructure team
• Requirements:
  • Must have access to itsulu-testing namespace
  • Must have permissions to create Jobs
  • Must have permissions to get pods and copy files

How to obtain:

# From infrastructure team, get the kubeconfig for itsulu-testing namespace
# Then encode it:
cat ~/.kube/config | base64 | pbcopy  # macOS
# or
cat ~/.kube/config | base64 -w0       # Linux

# Paste into GitLab CI/CD variable KUBE_CONFIG

3. K8s Cluster Verification

Before deploying, verify K8s cluster access:

# Verify kubeconfig is valid (after KUBE_CONFIG variable is set)
mkdir -p ~/.kube
echo "$KUBE_CONFIG" | base64 -d > ~/.kube/config
kubectl config use-context itsulu-testing

# Check cluster connectivity
kubectl cluster-info

# Verify namespace and services
kubectl get namespace itsulu-testing
kubectl get svc -n itsulu-testing

# Should see: test-db-svc (PostgreSQL)
# Should see: Secrets: test-db-info, gitlab-docker-creds
# Should see: ServiceAccount: test-runner

4. E2E Test Dry-Run (Local)

Test E2E suite against local dev instance:

# Install dependencies
pip install -r e2e/requirements.txt

# Run against local Odoo (default http://localhost:8069)
pytest e2e/ -v

# Or specify custom URL
ODOO_BASE_URL=http://staging:8069 pytest e2e/ -v

Expected result:

• All 19 scenarios should execute
• Some may fail if features not in local DB (normal for dry-run)
• Key is to verify Playwright fixtures work

5. Performance Test Dry-Run (Local)

Test performance tests against local instance:

# Run performance tests
pytest addons/itsulu_blog_publisher/tests/test_performance.py \
  -m performance -v

# Should output:
# - test_generation_latency_under_30_seconds PASSED
# - test_query_count tests PASSED
# - test_token_usage_baseline PASSED
# - test_concurrent_generation PASSED

Deployment Steps

Step 1: Get K8s Configuration

# Contact infrastructure team to:
# 1. Get kubeconfig for itsulu-testing namespace
# 2. Verify you have Job create permissions
# 3. Verify pod access and file copy permissions

# Base64 encode the kubeconfig:
cat kubeconfig.yaml | base64 | pbcopy  # macOS
# or
cat kubeconfig.yaml | base64 -w0       # Linux

Step 2: Set CI/CD Variable

# In GitLab web UI:
1. Project → Settings → CI/CD → Variables
2. Click "Add Variable"
3. Fill:
   - Key: KUBE_CONFIG
   - Value: (paste base64-encoded kubeconfig from step 1)
   - Type: Secret
   - Protected: ✓ Yes
   - Masked: ✓ Yes
4. Click "Add variable"

Step 3: Push Phase 3 Code to Feature Branch

# Verify all commits are in git log
git log --oneline -5

# Should see:
# d122b77 feat: integrate Runboat E2E testing into CI/CD pipeline
# acfa1d9 feat: establish Phase 3 E2E testing infrastructure
# 7ee393a feat: add performance benchmark tests
# ... Phase 2 commits ...

Step 4: Create Merge Request

# Push to feature branch
git push origin main:refs/heads/phase-3-e2e-testing

# Create MR in GitLab UI
# Title: "Phase 3: Runboat E2E Testing & Performance Benchmarks"
# Description:
# - E2E infrastructure with Runboat polling and auth fixtures
# - 19 E2E test scenarios (generation, scheduling, error recovery)
# - 7 performance benchmark tests (latency, queries, tokens, concurrency)
# - CI/CD pipeline integration (preview + e2e + performance stages)
# - SLO targets established for 7 performance metrics

Step 5: Verify Pipeline Runs

When MR is created, GitLab CI/CD should automatically start:

1. Lint stage (2 min)

   • black --check
   • pylint-odoo
   • Status: ✅ Should pass

2. Test stage (10 min)

   • unit_tests: TDD + BDD + performance tests
   • Status: ✅ Should pass

3. Build stage (3 min)

   • Docker image build & push to registry
   • Status: ✅ Should pass

4. E2E stage (on K8s)

   • e2e_tests: Creates job on itsulu-testing namespace
   • Installs Odoo, addon, runs Playwright tests
   • Expected: 30-45 min (K8s job setup + Odoo init + tests)
   • Status: ✅ Should pass or ⚠️ if allowed_failure=true

Step 6: Review Pipeline Artifacts

After pipeline completes, download/review artifacts:

• report-unit.html — Unit test coverage report
• report-performance.html — Performance test results
• report-e2e.html — E2E test results (if ran)
• e2e/traces/ — Playwright trace files for debugging

Troubleshooting

Runboat Preview Fails

Error: Authorization failed or 401 Unauthorized

Fix:

1. Verify RUNBOAT_TOKEN is set in CI/CD variables
2. Check token is not expired (request new from infrastructure team)
3. Verify RUNBOAT_API_URL is correct

Error: Cannot reach Runboat API

Fix:

1. Verify RUNBOAT_API_URL is not behind firewall
2. Check infrastructure team has enabled preview access for this project

E2E Tests Timeout

Error: Runboat instance not ready after 180s

Fix:

1. Runboat cold-start can take 60s, normal behavior
2. If > 180s, Runboat infrastructure may be overloaded
3. Retry pipeline after 30 minutes

E2E Tests Fail (Selector Not Found)

Error: locator.fill('value') → TimeoutError: locator did not resolve

Fix:

1. Form field names may differ in template DB
2. Update selector in test file to match actual field names
3. Use page.pause() to inspect live page: pytest e2e/ --pdb-trace

Performance Tests Fail

Error: AssertionError: Generation took 35s, target <30s

Fix:

1. Mock LLM response is artificially fast
2. Real API calls will add network time
3. If local test > 30s, investigate N+1 queries or missing indexes

Post-Deployment

Ongoing Maintenance

After Phase 3 is live:

1. Monitor E2E Results (Weekly)

   • Check MR pipeline E2E reports
   • Flag flaky scenarios (run 3× to verify)
   • Update fixtures if Odoo UI changes

2. Track Performance Metrics (Weekly)

   • Monitor ir_logging table for generation latency
   • Alert if P99 latency > 60s (bottleneck detected)
   • Adjust SLOs if production baseline differs

3. Update Runboat Token (Annually)

   • RUNBOAT_TOKEN expires yearly
   • Request renewal 1 month before expiration
   • Update CI/CD variable before it expires

4. Runboat Account Maintenance

   • Monitor preview instance quota
   • Cleanup old instances if quota fills
   • Contact Acsone if persistent issues

Success Criteria

✅ Phase 3 is successful when:

• All MRs include E2E test results
• No E2E flakiness (< 2% failure rate)
• Performance baselines recorded in first full month
• Runboat previews available for all MRs
• Zero issues with Runboat integration after 1 week
• Performance SLOs consistently met
• Team trained on reading E2E/performance reports

Reference Links

• Runboat Documentation
• Playwright Python API
• .gitlab-ci.yml Pipeline
• PHASE3_ROADMAP.md
• CLAUDE.md Testing Framework

Approval Sign-Off

This Phase 3 deployment is ready when:

• Code review approved
• All CI/CD variables set (RUNBOAT_API_URL, RUNBOAT_TOKEN, GITLAB_BOT_TOKEN)
• Runboat account verified operational
• Local dry-runs pass (E2E + performance tests)
• Infrastructure team confirms Runboat access
• At least one team member trained on results interpretation

Deploy: Create MR and monitor first pipeline run

---

Deployment Date: (To be filled on merge)
Deployed By: (To be filled on merge)
Notes: (To be filled with any issues encountered)