docs: add E2E testing and performance SLO documentation to CLAUDE.md
Added comprehensive sections documenting Phase 3 testing: Section 7.4 - Runboat Integration: - Explanation of Runboat (ephemeral preview instances) - Cold-start polling pattern with 180s timeout - CI/CD integration example with buildenv artifact - E2E test invocation against Runboat URL Section 13 - Performance SLO Targets: - Test infrastructure SLOs (pipeline times, test coverage, flakiness) - Generation performance targets: * Latency P50: <30s, P99: <60s * Token efficiency: 800-1200 per post * Query count: <50 per generation * Concurrent posts: 5+ * Email latency: <5s * Template DB prime: <60s - Measurement tools and patterns: * time.monotonic() for latency profiling * assertQueryCount() for N+1 detection * Token usage logging and assertions These targets are verified by test_performance.py and E2E tests. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
This commit is contained in:
parent
7ee393afc7
commit
11918da2ea
1 changed files with 91 additions and 11 deletions
102
CLAUDE.md
102
CLAUDE.md
|
|
@ -406,18 +406,63 @@ def page(browser, browser_context_args, auth_state):
|
|||
### 7.3 Example scenario
|
||||
|
||||
```python
|
||||
# e2e/test_sale_order.py
|
||||
# e2e/test_generation.py
|
||||
from playwright.sync_api import expect
|
||||
|
||||
def test_gold_customer_discount_shows_in_ui(page):
|
||||
page.goto('/odoo/sales/new')
|
||||
page.get_by_label('Customer').fill('Acme Corp')
|
||||
page.get_by_role('option', name='Acme Corp').click()
|
||||
page.get_by_role('button', name='Add a product').click()
|
||||
page.get_by_label('Product').last.fill('Widget')
|
||||
page.get_by_role('option', name='Widget').click()
|
||||
def test_user_generates_blog_post_on_demand(page):
|
||||
"""Navigate to Blog Publisher, fill form, generate post, verify published."""
|
||||
page.goto('/odoo/blog/generate-now')
|
||||
page.wait_for_load_state('networkidle')
|
||||
|
||||
expect(page.locator('[data-test-id="line-discount"]').first).to_have_text('10.00')
|
||||
page.get_by_label('Topic').fill('Kubernetes Cost Optimization')
|
||||
page.get_by_label('LLM Provider').select_option('anthropic')
|
||||
page.get_by_label('Auto-publish').check()
|
||||
|
||||
page.get_by_role('button', name='Generate').click()
|
||||
page.wait_for_url('**/blog/**', timeout=60_000)
|
||||
|
||||
expect(page.locator('h1')).to_contain_text('Kubernetes')
|
||||
```
|
||||
|
||||
### 7.4 Runboat Integration
|
||||
|
||||
**Runboat** provides ephemeral preview instances of Odoo per CI commit:
|
||||
- **Auto-deployment**: Fresh Odoo instance with addon pre-installed
|
||||
- **Live URL**: For E2E tests (no local bootstrapping)
|
||||
- **Auto-cleanup**: Instance removed 5 minutes after test completion
|
||||
- **Template DB**: Primed once, cloned for each test
|
||||
|
||||
**Cold-start Handling**:
|
||||
```python
|
||||
def wait_for_odoo(url, timeout=180):
|
||||
"""Poll until Odoo responds (instance startup takes 30-60s)."""
|
||||
deadline = time.time() + timeout
|
||||
while time.time() < deadline:
|
||||
try:
|
||||
if requests.get(f"{url}/web/login", timeout=5).status_code == 200:
|
||||
return
|
||||
except Exception:
|
||||
pass
|
||||
time.sleep(2)
|
||||
raise TimeoutError(f"Odoo not ready after {timeout}s")
|
||||
```
|
||||
|
||||
**CI/CD Integration**:
|
||||
```yaml
|
||||
# .gitlab-ci.yml
|
||||
runboat_preview:
|
||||
stage: preview
|
||||
script: |
|
||||
RESP=$(curl -fsSL -X POST $RUNBOAT_URL/builds \
|
||||
-H "Authorization: Bearer $RUNBOAT_TOKEN" \
|
||||
-d "{\"repo\":\"$CI_PROJECT_PATH\",\"sha\":\"$CI_COMMIT_SHA\"}")
|
||||
BUILD_URL=$(echo "$RESP" | jq -r '.url')
|
||||
echo "BUILD_URL=$BUILD_URL" >> build.env
|
||||
|
||||
e2e_tests:
|
||||
stage: e2e
|
||||
needs: [runboat_preview]
|
||||
script: pytest e2e/ --base-url=$BUILD_URL -v
|
||||
```
|
||||
|
||||
---
|
||||
|
|
@ -813,7 +858,9 @@ post = self.env['blog.post'].sudo().create({
|
|||
|
||||
---
|
||||
|
||||
## 13. SLO Targets
|
||||
## 13. Performance SLO Targets
|
||||
|
||||
### Test Infrastructure
|
||||
|
||||
| Metric | Target | Why |
|
||||
|---|---|---|
|
||||
|
|
@ -823,7 +870,40 @@ post = self.env['blog.post'].sudo().create({
|
|||
| Flaky test rate | < 2%/week | Maintains trust in suite |
|
||||
| Coverage on new code | ≥ 80% | Enforces test-first habit |
|
||||
| Runboat cold-start P95 | < 120 s | E2E does not time out |
|
||||
| Production deploy MTTR | < 15 min | Git revert + ArgoCD sync |
|
||||
|
||||
### Generation Performance (Phase 3)
|
||||
|
||||
| Metric | Target | Measurement |
|
||||
|---|---|---|
|
||||
| Generation latency P50 | < 30 seconds | RED → POST created |
|
||||
| Generation latency P99 | < 60 seconds | 99th percentile |
|
||||
| Tokens per post | 800–1200 | Cost baseline |
|
||||
| Queries per generation | < 50 | N+1 detection |
|
||||
| Concurrent posts | 5+ | Peak throughput |
|
||||
| Email send latency | < 5 seconds | Notification speed |
|
||||
| Template DB prime | < 60 seconds | CI/CD overhead |
|
||||
|
||||
**Measurement Tools**:
|
||||
```python
|
||||
# Latency profiling
|
||||
import time
|
||||
start = time.monotonic()
|
||||
post = schedule.run_generation()
|
||||
elapsed = time.monotonic() - start
|
||||
assert elapsed < 30 # P50 target
|
||||
|
||||
# Query count assertion
|
||||
with self.assertQueryCount(50):
|
||||
schedule.run_generation() # Must use < 50 queries
|
||||
|
||||
# Token usage logging
|
||||
log.tokens_used # Recorded in generation log
|
||||
assert 800 <= log.tokens_used <= 1200
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 14. Production Deploy
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue