# Phase 3: Runboat E2E & Performance Testing — Deployment Checklist

**Status**: Ready for deployment  
**Last Updated**: 2026-05-30

## Pre-Deployment Checklist

### 1. Code Readiness

- [x] E2E test infrastructure complete (conftest.py)
- [x] 19 E2E scenarios implemented
- [x] 7 performance benchmark tests implemented
- [x] PHASE3_ROADMAP.md documented
- [x] .gitlab-ci.yml updated with preview + e2e + performance stages
- [x] CLAUDE.md updated with SLO targets and patterns
- [x] All code committed and pushed to main

### 2. CI/CD Variables Setup (Action Required)

Before merging Phase 3 code, set these variables in **GitLab Project Settings → CI/CD Variables**:

#### Variable 1: RUNBOAT_API_URL
- **Type**: Secret
- **Value**: Runboat API endpoint (e.g., `https://api.runboat.dev`)
- **Protected**: Yes
- **Masked**: Yes
- **Source**: Request from Acsone/infrastructure team

#### Variable 2: RUNBOAT_TOKEN
- **Type**: Secret
- **Value**: Bearer token for Runboat API authentication
- **Protected**: Yes
- **Masked**: Yes
- **Source**: Request from Acsone/infrastructure team
- **Format**: `rbk_...` (Runboat Bearer Key)

#### Variable 3: GITLAB_BOT_TOKEN
- **Type**: Secret
- **Value**: GitLab personal/bot token for posting MR comments
- **Protected**: Yes
- **Masked**: Yes
- **Source**: **Create locally** via GitLab Settings → Access Tokens
  - Token name: `itsulu-blog-publisher-bot`
  - Scopes: `api`, `read_api`, `read_repository`
  - Expiration: 1 year
  - Save and copy the value

### 3. Runboat Account Verification

Before deploying, verify Runboat is accessible:

```bash
# Test Runboat API (after variables are set)
curl -fsSL -X POST "${RUNBOAT_API_URL}/builds" \
  -H "Authorization: Bearer ${RUNBOAT_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"repo":"example/repo","sha":"abc123"}'

# Should return JSON with "url" field (success)
# Or 401 error (invalid token — fix variables)
```

### 4. E2E Test Dry-Run (Local)

Test E2E suite against local dev instance:

```bash
# Install dependencies
pip install -r e2e/requirements.txt

# Run against local Odoo (default http://localhost:8069)
pytest e2e/ -v

# Or specify custom URL
ODOO_BASE_URL=http://staging:8069 pytest e2e/ -v
```

Expected result:
- All 19 scenarios should execute
- Some may fail if features not in local DB (normal for dry-run)
- Key is to verify Playwright fixtures work

### 5. Performance Test Dry-Run (Local)

Test performance tests against local instance:

```bash
# Run performance tests
pytest addons/itsulu_blog_publisher/tests/test_performance.py \
  -m performance -v

# Should output:
# - test_generation_latency_under_30_seconds PASSED
# - test_query_count tests PASSED
# - test_token_usage_baseline PASSED
# - test_concurrent_generation PASSED
```

## Deployment Steps

### Step 1: Create/Verify GitLab Bot Token

```bash
# In GitLab web UI:
1. Click your avatar → Settings → Access Tokens
2. Click "Add New Token"
3. Fill:
   - Token name: "itsulu-blog-publisher-bot"
   - Scopes: Check api, read_api, read_repository
   - Expiration: 1 year from now
4. Click "Create Personal Access Token"
5. Copy the token value (only shown once)
```

### Step 2: Set CI/CD Variables

```bash
# In GitLab web UI:
1. Project → Settings → CI/CD → Variables
2. Add RUNBOAT_API_URL (get from infrastructure team)
3. Add RUNBOAT_TOKEN (get from infrastructure team)
4. Add GITLAB_BOT_TOKEN (paste from step 1)
5. Check all are: Protected=Yes, Masked=Yes
6. Click Save
```

### Step 3: Push Phase 3 Code to Feature Branch

```bash
# Verify all commits are in git log
git log --oneline -5

# Should see:
# d122b77 feat: integrate Runboat E2E testing into CI/CD pipeline
# acfa1d9 feat: establish Phase 3 E2E testing infrastructure
# 7ee393a feat: add performance benchmark tests
# ... Phase 2 commits ...
```

### Step 4: Create Merge Request

```bash
# Push to feature branch
git push origin main:refs/heads/phase-3-e2e-testing

# Create MR in GitLab UI
# Title: "Phase 3: Runboat E2E Testing & Performance Benchmarks"
# Description:
# - E2E infrastructure with Runboat polling and auth fixtures
# - 19 E2E test scenarios (generation, scheduling, error recovery)
# - 7 performance benchmark tests (latency, queries, tokens, concurrency)
# - CI/CD pipeline integration (preview + e2e + performance stages)
# - SLO targets established for 7 performance metrics
```

### Step 5: Verify Pipeline Runs

When MR is created, GitLab CI/CD should automatically start:

1. **Lint stage** (2 min)
   - black --check
   - pylint-odoo
   - Status: ✅ Should pass

2. **Test stage** (10 min) — runs in parallel:
   - unit_tests: TDD + BDD tests
   - performance_tests: Performance benchmarks
   - Status: ✅ Should pass

3. **Build stage** (3 min)
   - Docker image build & push
   - Status: ✅ Should pass

4. **Preview stage** (5 min)
   - runboat_preview: Requests Runboat build
   - Status: ✅ Should create preview URL (or ⚠️ if Runboat unavailable)

5. **E2E stage** (15 min) — waits for preview
   - e2e_tests: Runs Playwright scenarios
   - Status: ✅ Should pass or ⚠️ if preview not ready

### Step 6: Review Pipeline Artifacts

After pipeline completes, download/review artifacts:

- **report-unit.html** — Unit test coverage report
- **report-performance.html** — Performance test results
- **report-e2e.html** — E2E test results (if ran)
- **e2e/traces/** — Playwright trace files for debugging

## Troubleshooting

### Runboat Preview Fails

**Error**: `Authorization failed` or `401 Unauthorized`

**Fix**:
1. Verify RUNBOAT_TOKEN is set in CI/CD variables
2. Check token is not expired (request new from infrastructure team)
3. Verify RUNBOAT_API_URL is correct

**Error**: `Cannot reach Runboat API`

**Fix**:
1. Verify RUNBOAT_API_URL is not behind firewall
2. Check infrastructure team has enabled preview access for this project

### E2E Tests Timeout

**Error**: `Runboat instance not ready after 180s`

**Fix**:
1. Runboat cold-start can take 60s, normal behavior
2. If > 180s, Runboat infrastructure may be overloaded
3. Retry pipeline after 30 minutes

### E2E Tests Fail (Selector Not Found)

**Error**: `locator.fill('value') → TimeoutError: locator did not resolve`

**Fix**:
1. Form field names may differ in template DB
2. Update selector in test file to match actual field names
3. Use `page.pause()` to inspect live page: `pytest e2e/ --pdb-trace`

### Performance Tests Fail

**Error**: `AssertionError: Generation took 35s, target <30s`

**Fix**:
1. Mock LLM response is artificially fast
2. Real API calls will add network time
3. If local test > 30s, investigate N+1 queries or missing indexes

## Post-Deployment

### Ongoing Maintenance

After Phase 3 is live:

1. **Monitor E2E Results** (Weekly)
   - Check MR pipeline E2E reports
   - Flag flaky scenarios (run 3× to verify)
   - Update fixtures if Odoo UI changes

2. **Track Performance Metrics** (Weekly)
   - Monitor `ir_logging` table for generation latency
   - Alert if P99 latency > 60s (bottleneck detected)
   - Adjust SLOs if production baseline differs

3. **Update Runboat Token** (Annually)
   - RUNBOAT_TOKEN expires yearly
   - Request renewal 1 month before expiration
   - Update CI/CD variable before it expires

4. **Runboat Account Maintenance**
   - Monitor preview instance quota
   - Cleanup old instances if quota fills
   - Contact Acsone if persistent issues

### Success Criteria

✅ **Phase 3 is successful when**:

- [ ] All MRs include E2E test results
- [ ] No E2E flakiness (< 2% failure rate)
- [ ] Performance baselines recorded in first full month
- [ ] Runboat previews available for all MRs
- [ ] Zero issues with Runboat integration after 1 week
- [ ] Performance SLOs consistently met
- [ ] Team trained on reading E2E/performance reports

## Reference Links

- [Runboat Documentation](https://docs.acsone.eu/runboat/)
- [Playwright Python API](https://playwright.dev/python/)
- [.gitlab-ci.yml Pipeline](https://gitlab.com/YOUR_PROJECT/-/blob/main/.gitlab-ci.yml)
- [PHASE3_ROADMAP.md](./PHASE3_ROADMAP.md)
- [CLAUDE.md Testing Framework](./CLAUDE.md)

## Approval Sign-Off

This Phase 3 deployment is ready when:

- [ ] Code review approved
- [ ] All CI/CD variables set (RUNBOAT_API_URL, RUNBOAT_TOKEN, GITLAB_BOT_TOKEN)
- [ ] Runboat account verified operational
- [ ] Local dry-runs pass (E2E + performance tests)
- [ ] Infrastructure team confirms Runboat access
- [ ] At least one team member trained on results interpretation

**Deploy**: Create MR and monitor first pipeline run

---

**Deployment Date**: (To be filled on merge)  
**Deployed By**: (To be filled on merge)  
**Notes**: (To be filled with any issues encountered)