loop-benchmarking

SPEC.md (18082B)

---

 # Gameplay Bot Master Spec
 
 ## Overview
 
 The gameplay bot is a Playwright-based automated tester that evaluates Tetris games
 built by AI agents. Every game is different: different DOM structures, different
 controls, different start mechanisms, different rendering approaches (canvas, DOM,
 SVG, WebGL). The bot must work with all of them.
 
 The bot does NOT use Claude or any LLM for grading. All evaluation is deterministic
 code. The bot plays the game using a known-good AI algorithm (4-heuristic genetic
 optimization, reference implementation: LeeYiyuan/tetrisai, MIT License) and records what happens.
 
 ## Architecture
 
 Two main components:
 1. **Grid reader**: Reads the game state by sampling pixels (canvas) or DOM colors.
    Produces a 10x20 boolean grid. Works regardless of rendering approach.
 2. **AI player**: Given a grid state and current piece type, computes the optimal
    placement using aggregate height, lines cleared, holes, and bumpiness heuristics.
 
 The bot requires GPU access for reliable pixel readback from canvas. Without GPU,
 headless Chromium's canvas `getImageData()` returns all zeros. The host must expose
 `/dev/dri/card0` and `/dev/dri/renderD128` to the container.
 
 ## Test Structure: Conditional Phases
 
 Tests are organized as conditional phases. Each phase depends on the previous one
 succeeding. If a phase fails, all downstream tests are marked "skipped: [phase] failed"
 instead of producing false positives.
 
 ### Pre-test Survey (not scored, just data collection)
 
 Before any tests run, survey the page:
 - Is there a visible element with tetris grid proportions (~2:1 height:width)?
 - Is there a full-screen overlay (high z-index element covering >80% of viewport)?
 - Are there clickable elements (buttons, links, divs with click handlers)?
 - Is there a canvas element? Multiple canvases?
 - Is there a DOM-based grid (table, grid of divs)?
 - What text is visible? ("Press Enter", "Start", "Play", etc.)
 
 Store all survey data. This informs the start mechanism detection but is not a test.
 
 ### Phase 1: Page Load (test 1)
 
 **Test: `game_loads`**
 - Navigate to the game URL
 - Wait 3 seconds for scripts to execute
 - Check for console errors
 - Pass: page loaded, no critical errors
 - Fail: page failed to load or has critical JS errors
 
 ### Phase 2: Game Start Detection (tests 2-3)
 
 This phase determines if and how the game starts. It uses a cascading strategy:
 
 **Step 2a: Auto-start check**
 - Take 10 screenshots at 100ms intervals (1 second total)
 - Look for a colored cluster of pixels (~4 cells, roughly square-ish bounding box)
   that moved downward between frames
 - This is the "falling piece detector"
 - If found: game auto-starts, no button needed
 - Store: `start_mechanism: "auto"`
 
 **Step 2b: Button/overlay detection (if 2a failed)**
 - Check for full-screen overlay (element covering >80% viewport with high z-index)
 - If overlay found:
   - Try pressing Enter
   - Wait 500ms, run falling piece detector (10 screenshots, 100ms each)
   - If piece found: `start_mechanism: "enter"`, overlay was a start screen
   - If not, try pressing Space, same check
   - If not, try clicking the overlay center
   - If not, try clicking any visible buttons in the overlay
 - If no overlay:
   - Find all clickable elements (buttons, elements with onclick, role="button")
   - Try clicking each one, run falling piece detector after each
   - If piece found: `start_mechanism: "button"`, remember which element worked
   - The start button might change state or disappear after clicking -- that's fine
 
 **Step 2c: Keyboard fallback (if 2b failed)**
 - Try pressing: Enter, Space, ArrowDown, Z, P, any key
 - After each, run falling piece detector
 - If piece found: store which key started the game
 
 **Step 2d: Canvas click (if 2c failed)**
 - Click the center of the canvas/grid element
 - Run falling piece detector
 - Some games render their start button on the canvas (no DOM element to find)
 
 **Falling piece detector algorithm:**
 - Take 10 screenshots at 100ms intervals
 - For each consecutive pair, diff the pixels
 - Look for a cluster of changed pixels that:
   - Is roughly rectangular (bounding box aspect ratio between 1:1 and 4:1)
   - Is in the upper portion of the game area
   - Moved downward between frames (centroid Y increased)
 - A "cluster" = contiguous region of non-background-color pixels that appeared
 - Size: roughly 1-4 cells worth of pixels (each cell is typically 15-40px)
 - The piece may have rounded corners, glow effects, shadows -- look at bounding box
 - Must see movement in at least 2 frame pairs to confirm it's falling
 
 **Tests derived from Phase 2:**
 - `game_starts`: pass if falling piece detected by any mechanism
 - `auto_drop`: pass if piece falls on its own without any key input (only valid if
   game auto-started or we confirmed start mechanism worked)
 
 ### Phase 3: Mechanics Tests (tests 4-9)
 
 Only runs if Phase 2 succeeded (game started, piece detected).
 
 Reload the page. Start the game using the mechanism discovered in Phase 2.
 Wait for a piece to appear (falling piece detector).
 
 For each control test:
 1. Read the grid state (grid reader)
 2. Press the key
 3. Wait 60ms
 4. Read the grid state again
 5. Compare: did the relevant change happen?
 
 **Test: `move_left`** -- ArrowLeft, piece column decreased
 **Test: `move_right`** -- ArrowRight, piece column increased
 **Test: `move_down`** -- ArrowDown, piece row increased (soft drop)
 **Test: `rotate`** -- ArrowUp (or Z), piece shape changed (bounding box dimensions swapped for non-O pieces)
 **Test: `hard_drop`** -- Space, piece instantly at bottom (filled cells appear in bottom rows)
 **Test: `all_pieces_rotate`** -- Track piece types seen during play, confirm rotation works for non-O pieces
 
 If the grid reader cannot read the grid (no GPU, bad calibration), fall back to
 screenshot comparison for these tests. Mark as "(screenshot-verified)" not
 "(grid-verified)" in the detail string.
 
 ### Phase 4: Piece Lifecycle Tests (tests 10-12)
 
 Only runs if Phase 3 mechanics worked.
 
 Continue from Phase 3 state or reload + start.
 
 **Test: `piece_locks`**
 - Hard drop a piece
 - Wait 300ms
 - Read grid: are there filled cells at the bottom that persist?
 - Must see cells that don't move for 2 consecutive reads 500ms apart
 
 **Test: `new_piece_spawns`**
 - After a piece locks, check top 4 rows of grid
 - A new piece should appear (filled cells in top rows)
 - Track: `piecesSpawned` counter
 
 **Test: `multiple_pieces`**
 - Play 10+ pieces (hard drop repeatedly)
 - Must detect at least 3 distinct piece placements
 - Track piece types seen (I, O, T, S, Z, J, L)
 
 ### Phase 5: Gameplay Tests (tests 13-14)
 
 Only runs if Phase 4 piece lifecycle works.
 
 Reload the page. Start game. Play using the AI player for an extended session:
 - **60 pieces max, 45 seconds max**
 - **60ms polling** between grid reads
 - Read score element on every 5th poll cycle (integrated score tracking)
 
 **Test: `line_clear`**
 - During AI play, watch for complete rows (all cells filled)
 - After complete row detected, wait 200-500ms for clear animation
 - Read grid again: did the complete row disappear?
 - If AI play doesn't clear a line, try brute force: drop pieces at each column
 - If brute force fails, check if total filled cells decreased (indirect clear detection)
 - Pass: at least 1 line cleared (grid-verified)
 
 **Test: `score_changes`**
 - Read score element before play begins (record initial value)
 - During play, read score on every 5th poll cycle
 - After play, read final score
 - Pass: score increased from initial value
 - If no score element found, try scanning page text for changing numbers
 - Record: all score values observed, deltas between readings
 
 ### Phase 6: Game Over Test (test 15)
 
 Only runs if Phase 5 gameplay works (pieces can be placed and lines can clear,
 or at minimum pieces can be placed).
 
 Reload the page. Start game.
 
 **Test: `game_over`**
 - Stack pieces to trigger game over: hard drop in the same column repeatedly
   to build a tall tower
 - After each drop, check grid reader: are there filled cells in the top 2 rows?
 - Once top rows are filled, check:
   1. Does the game stop accepting input? (press keys, check if grid changes)
   2. Does "Game Over" or similar text appear in DOM?
   3. Does the page become static? (2 screenshots 1s apart are identical)
 - Pass: game stopped after filling to top (at least 1 of the 3 checks)
 - Do NOT use screenshot comparison alone (false positives on static start screens)
 - Must have evidence that pieces WERE being placed before the game stopped
 
 ### Phase 7: Endurance Test (test 16)
 
 Only runs if Phase 5 gameplay works.
 
 Reload the page. Start game.
 
 **Test: `playable_30s`**
 - Play using AI player for 30 seconds
 - Track: pieces placed, lines cleared, console errors, play errors
 - Pass: played for 30+ seconds without crashing, placed 5+ pieces, no critical errors
 
 ### Phase 8: Competitive Play (not pass/fail, produces metrics + 8 additional tests)
 
 Only runs if Phase 5 gameplay works.
 
 Reload the page. Start game. Play competitively for 60 seconds using AI player.
 
 **Purpose**: Find bugs that the basic tests miss. A game might start, move pieces,
 and clear single lines, but fail on multi-line clears, score scaling, level
 progression, etc.
 
 **Data recorded:**
 ```json
 {
   "duration_seconds": 45,
   "pieces_placed": 62,
   "total_lines_cleared": 18,
   "single_clears": 12,
   "double_clears": 2,
   "triple_clears": 1,
   "tetris_clears": 0,
   "max_combo": 3,
   "score_readings": [0, 100, 200, 500, ...],
   "score_final": 4200,
   "score_increases": [100, 100, 300, ...],
   "level_readings": [1, 1, 1, 2, 2, 3],
   "level_final": 3,
   "game_over_reached": true,
   "game_over_text_found": "Game Over",
   "restart_available": true,
   "next_piece_visible": true,
   "speed_increased": true,
   "bugs_detected": ["score_does_not_scale_with_simultaneous_clears"]
 }
 ```
 
 **8 additional tests (tests 17-24):**
 
 Each has three outcomes: pass (tested, works), fail (tested, broken), skip (no opportunity to test).
 
 **Test 17: `multi_line_clear`**
 - During play, detect when 2+ rows are complete simultaneously
 - Wait for clear animation (200-500ms)
 - Check: did all complete rows disappear?
 - Bug: `multi_line_clear_only_removes_one_row` -- only 1 row cleared when 2+ were complete
 - Skip: if no multi-line clear opportunity occurred during 60s play
 
 **Test 18: `score_scaling`**
 - Track score delta for each clear event
 - Compare: single clear delta vs multi-line clear delta
 - Bug: `score_does_not_scale_with_simultaneous_clears` -- multi-line gives same points as single
 - Standard Tetris scoring: single=100, double=300, triple=500, tetris=800 (x level)
 - Don't enforce exact formula, just check that multi > single
 - Skip: if no multi-line clear occurred
 
 **Test 19: `level_progression`**
 - Track lines cleared and level display throughout the session
 - After 10+ lines cleared, level should have increased from initial value
 - Bug: `level_does_not_increase`
 - Skip: if fewer than 10 lines cleared
 
 **Test 20: `speed_progression`**
 - Measure auto-drop interval at game start (time between automatic downward moves)
 - After level increases, measure again
 - Bug: `speed_does_not_increase` -- interval didn't decrease
 - Skip: if level didn't increase
 
 **Test 21: `next_piece_preview`**
 - Look for a secondary display area showing the next piece
 - Check: small canvas/div near the main grid with a single piece shape
 - Pass: found a next piece display
 - Fail: no next piece preview found
 
 **Test 22: `game_over_display`**
 - When game over is triggered (from Phase 6 or competitive play):
 - Check for "Game Over" or similar text
 - Check for restart button/prompt
 - Pass: both message and restart option present
 - Fail: missing either
 - Skip: game over not reached during competitive play
 
 **Test 23: `counter_clockwise_rotation`**
 - During play, occasionally press Z key instead of Up arrow
 - Compare piece shape before and after
 - Pass: Z rotates opposite direction from Up
 - Fail: Z does same as Up, or doesn't rotate
 - Skip: could not detect rotation direction
 
 **Test 24: `soft_drop_distinct`**
 - Press Down arrow: piece should move one row
 - Press Space: piece should drop to bottom
 - Compare: Down should NOT behave like Space
 - Bug: `soft_drop_acts_as_hard_drop`
 - Pass: Down moves 1 row, Space drops to bottom
 - Fail: Down drops to bottom like Space
 
 ## Polling and Timing
 
 - **Grid polling**: 60ms between reads during play
 - **Post-lock wait**: 100ms after piece locks before reading settled state
 - **Falling piece detector**: 10 screenshots at 100ms intervals
 - **Post-trigger settling**: 500ms after pressing start key/button before detection
 - **Score reading**: every 5th poll cycle during play (every 300ms)
 - **Clear animation wait**: 200-500ms between detecting complete row and checking if it cleared
 
 ## Grid Reader
 
 The grid reader samples pixels to determine cell state (filled/empty).
 
 **Canvas games**: Use `getImageData()` to sample 5 points per cell (center + corners).
 Requires GPU access for reliable readback.
 
 **DOM games**: Read `backgroundColor` or `style.background` of each cell element.
 
 **Grid detection**: Find the game grid by looking for a rectangular region with
 approximately 2:1 height:width ratio containing a 10-column x 20-row grid of
 uniformly-sized cells.
 
 **Validation**: Reject grids where >60% of cells are filled (likely reading UI chrome,
 not game state). Validate aspect ratio (height ~= 2 * width).
 
 ## AI Player
 
 Pierre Dellacherie's 4-heuristic evaluation (2003) with Colin Fahey's GA-optimized weights, reference implementation: LeeYiyuan/tetrisai (MIT License):
 - Aggregate height: sum of column heights (weight: -0.510066)
 - Lines cleared: number of complete rows (weight: 0.760666)
 - Holes: empty cells below filled cells (weight: -0.35663)
 - Bumpiness: sum of absolute height differences between adjacent columns (weight: -0.184483)
 
 For each possible (rotation, column) placement:
 1. Simulate piece drop
 2. Score the resulting board
 3. Pick the highest-scoring placement
 
 Execute placement: rotate N times, move to target column, hard drop.
 
 ## Report Structure
 
 ```json
 {
   "implementation": {
     "renderer": "canvas|dom|svg|webgl|unknown",
     "grid_detected": true,
     "grid_bounds": { "x": 0, "y": 0, "width": 320, "height": 640 },
     "controls": { "left": "ArrowLeft", "right": "ArrowRight", ... },
     "start_mechanism": "auto|enter|space|button|click_canvas|unknown",
     "score_element_found": true,
     "grid_confidence": 0.95,
     "survey": {
       "has_overlay": false,
       "has_canvas": true,
       "has_dom_grid": false,
       "visible_text": ["TETRIS", "Score: 0", "Level: 1"],
       "clickable_elements": 2
     }
   },
   "tests": [
     { "name": "game_loads", "pass": true, "detail": "..." },
     { "name": "game_starts", "pass": true, "detail": "started via enter" },
     ...
   ],
   "summary": {
     "total": 24,
     "passed": 20,
     "failed": 2,
     "skipped": 2,
     "score": 0.83
   },
   "gameplay": {
     "pieces_placed": 45,
     "lines_cleared": 12,
     "max_score_observed": 4200,
     "play_duration_seconds": 30,
     "errors_during_play": 0
   },
   "competitive_play": {
     "duration_seconds": 55,
     "pieces_placed": 62,
     "total_lines_cleared": 18,
     "single_clears": 12,
     "double_clears": 2,
     "triple_clears": 1,
     "tetris_clears": 0,
     "score_readings": [0, 100, 200, ...],
     "score_final": 4200,
     "level_final": 3,
     "bugs_detected": []
   },
   "session": {
     "frames": 500,
     "pieces_spawned": 45,
     "pieces_locked": 44,
     "lines_cleared": 12,
     "piece_types_seen": ["I", "O", "T", "S", "Z", "J", "L"],
     "grid_read_success_rate": 0.98
   },
   "performance": {
     "load_time_ms": 150
   },
   "accessibility": {
     "issues": ["canvas without aria-label"],
     "issue_count": 1,
     "pass": false
   }
 }
 ```
 
 ## Score Calculation
 
 The bot score is: `passed / total` (excluding skipped tests from both numerator
 and denominator). So if 20/22 non-skipped tests pass, score = 0.91.
 
 Skipped tests don't penalize -- they indicate the bot couldn't test that feature
 because a prerequisite failed. The game may still be good; we just can't verify.
 
 ## Files
 
 - `types.ts` -- TypeScript interfaces for all data structures
 - `calibrate.ts` -- Grid detection, control detection, start mechanism, survey
 - `grid-reader.ts` -- Pixel sampling, grid state reading, piece detection
 - `player.ts` -- AI player, placement execution, heuristic evaluation
 - `tests.ts` -- Phase execution, test derivation, falling piece detector
 - `index.ts` -- Playwright test entry point, HTTP server, report output
 
 ## Known Limitations
 
 The bot does NOT test:
 - Wall kicks (piece sliding along collision line)
 - Lock delay (brief window to slide before piece locks)
 - T-spins
 - Hold piece functionality
 - Ghost piece (shadow showing where piece will land)
 - Next piece preview accuracy (only checks if it exists)
 - Level/speed progression exact values (only checks direction)
 - DAS (delayed auto-shift for held keys)
 - Piece randomizer fairness (bag system vs pure random)
 
 The bot CAN be fooled by:
 - Games that render pieces identically to UI chrome (rare)
 - Games with unusual grid sizes (not 10x20)
 - Games where the grid is not visible (3D Tetris, first-person Tetris)
 - Games requiring mouse input for gameplay (not just start)
 - Games with very fast initial drop speed (piece may lock before bot reads it)
 
 ## GPU Requirement
 
 Without GPU access in the container, canvas `getImageData()` returns all zeros
 in headless Chromium. The bot falls back to DOM-based grid reading for DOM-rendered
 games, but canvas games will fail grid reading entirely.
 
 To enable GPU in Proxmox LXC:
 ```
 lxc.cgroup2.devices.allow: c 226:* rwm
 lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
 lxc.hook.autodev: sh -c "chmod 666 ${LXC_ROOTFS_MOUNT}/dev/dri/card0 ${LXC_ROOTFS_MOUNT}/dev/dri/renderD128 2>/dev/null || true"
 ```
 
 The bot should detect GPU availability at startup and log a warning if
 `/dev/dri/renderD128` is not accessible. Canvas-based tests will report
 "grid reader unavailable (no GPU)" instead of false results.