docs: add E2E testing and performance SLO documentation to CLAUDE.md

Added comprehensive sections documenting Phase 3 testing:

Section 7.4 - Runboat Integration:
- Explanation of Runboat (ephemeral preview instances)
- Cold-start polling pattern with 180s timeout
- CI/CD integration example with buildenv artifact
- E2E test invocation against Runboat URL

Section 13 - Performance SLO Targets:
- Test infrastructure SLOs (pipeline times, test coverage, flakiness)
- Generation performance targets:
  * Latency P50: <30s, P99: <60s
  * Token efficiency: 800-1200 per post
  * Query count: <50 per generation
  * Concurrent posts: 5+
  * Email latency: <5s
  * Template DB prime: <60s
- Measurement tools and patterns:
  * time.monotonic() for latency profiling
  * assertQueryCount() for N+1 detection
  * Token usage logging and assertions

These targets are verified by test_performance.py and E2E tests.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
This commit is contained in:
Nicholas Riegel 2026-05-30 00:52:37 -04:00
parent 7ee393afc7
commit 11918da2ea

102
CLAUDE.md
View file

@ -406,18 +406,63 @@ def page(browser, browser_context_args, auth_state):
### 7.3 Example scenario ### 7.3 Example scenario
```python ```python
# e2e/test_sale_order.py # e2e/test_generation.py
from playwright.sync_api import expect from playwright.sync_api import expect
def test_gold_customer_discount_shows_in_ui(page): def test_user_generates_blog_post_on_demand(page):
page.goto('/odoo/sales/new') """Navigate to Blog Publisher, fill form, generate post, verify published."""
page.get_by_label('Customer').fill('Acme Corp') page.goto('/odoo/blog/generate-now')
page.get_by_role('option', name='Acme Corp').click() page.wait_for_load_state('networkidle')
page.get_by_role('button', name='Add a product').click()
page.get_by_label('Product').last.fill('Widget') page.get_by_label('Topic').fill('Kubernetes Cost Optimization')
page.get_by_role('option', name='Widget').click() page.get_by_label('LLM Provider').select_option('anthropic')
page.get_by_label('Auto-publish').check()
page.get_by_role('button', name='Generate').click()
page.wait_for_url('**/blog/**', timeout=60_000)
expect(page.locator('h1')).to_contain_text('Kubernetes')
```
expect(page.locator('[data-test-id="line-discount"]').first).to_have_text('10.00') ### 7.4 Runboat Integration
**Runboat** provides ephemeral preview instances of Odoo per CI commit:
- **Auto-deployment**: Fresh Odoo instance with addon pre-installed
- **Live URL**: For E2E tests (no local bootstrapping)
- **Auto-cleanup**: Instance removed 5 minutes after test completion
- **Template DB**: Primed once, cloned for each test
**Cold-start Handling**:
```python
def wait_for_odoo(url, timeout=180):
"""Poll until Odoo responds (instance startup takes 30-60s)."""
deadline = time.time() + timeout
while time.time() < deadline:
try:
if requests.get(f"{url}/web/login", timeout=5).status_code == 200:
return
except Exception:
pass
time.sleep(2)
raise TimeoutError(f"Odoo not ready after {timeout}s")
```
**CI/CD Integration**:
```yaml
# .gitlab-ci.yml
runboat_preview:
stage: preview
script: |
RESP=$(curl -fsSL -X POST $RUNBOAT_URL/builds \
-H "Authorization: Bearer $RUNBOAT_TOKEN" \
-d "{\"repo\":\"$CI_PROJECT_PATH\",\"sha\":\"$CI_COMMIT_SHA\"}")
BUILD_URL=$(echo "$RESP" | jq -r '.url')
echo "BUILD_URL=$BUILD_URL" >> build.env
e2e_tests:
stage: e2e
needs: [runboat_preview]
script: pytest e2e/ --base-url=$BUILD_URL -v
``` ```
--- ---
@ -813,7 +858,9 @@ post = self.env['blog.post'].sudo().create({
--- ---
## 13. SLO Targets ## 13. Performance SLO Targets
### Test Infrastructure
| Metric | Target | Why | | Metric | Target | Why |
|---|---|---| |---|---|---|
@ -823,7 +870,40 @@ post = self.env['blog.post'].sudo().create({
| Flaky test rate | < 2%/week | Maintains trust in suite | | Flaky test rate | < 2%/week | Maintains trust in suite |
| Coverage on new code | ≥ 80% | Enforces test-first habit | | Coverage on new code | ≥ 80% | Enforces test-first habit |
| Runboat cold-start P95 | < 120 s | E2E does not time out | | Runboat cold-start P95 | < 120 s | E2E does not time out |
| Production deploy MTTR | < 15 min | Git revert + ArgoCD sync |
### Generation Performance (Phase 3)
| Metric | Target | Measurement |
|---|---|---|
| Generation latency P50 | < 30 seconds | RED POST created |
| Generation latency P99 | < 60 seconds | 99th percentile |
| Tokens per post | 8001200 | Cost baseline |
| Queries per generation | < 50 | N+1 detection |
| Concurrent posts | 5+ | Peak throughput |
| Email send latency | < 5 seconds | Notification speed |
| Template DB prime | < 60 seconds | CI/CD overhead |
**Measurement Tools**:
```python
# Latency profiling
import time
start = time.monotonic()
post = schedule.run_generation()
elapsed = time.monotonic() - start
assert elapsed < 30 # P50 target
# Query count assertion
with self.assertQueryCount(50):
schedule.run_generation() # Must use < 50 queries
# Token usage logging
log.tokens_used # Recorded in generation log
assert 800 <= log.tokens_used <= 1200
```
---
## 14. Production Deploy
--- ---