Skip to content

Beta Test Plan

Overview

This document provides manual test scenarios for Inquisitor 1.2.0. The goal is to verify that all core features work correctly in a live IRC environment, for example after installing a release or before tagging a new one.

Version under test: the version shown in the bot's start banner (Inquisitor v<version> "Plato"), for example v1.2.0+20261029

Purpose: Verify gameplay correctness, permission enforcement, configuration management, question management, import workflows, and edge case handling across real IRC sessions with real MariaDB data.

Reporting issues: Submit bug reports to the Forgejo issue tracker at https://git.ptp-trivia.me/Inquisitor/Inquisitor/issues. Include the version shown in the start banner, the channel and network, the exact commands run, the expected behavior, and the actual behavior.

Prerequisites

Before beginning beta testing, ensure the following:

  • Working Inquisitor installation (plugin loaded in Limnoria, @load Inquisitor)
  • MariaDB 10.5+ running and accessible
  • Database credentials configured via @config plugins.Inquisitor.database.*
  • At least one question set imported (standard questions and at least one KAOS question set)
  • At least 2 IRC clients available for multiplayer testing scenarios
  • Global setup completed (@setup) and at least one network activated (@activate)
  • Test user accounts with Gazelle-style hostmasks (e.g., [email protected])

Test Scenarios

Game Loop (Standard Questions)

  • [ ] Start a game with @trivia — bot announces game start with version banner and question set name
  • [ ] Question is delivered with: question ID (#N), category, question text, and point value, followed immediately by the first hint (fully obfuscated)
  • [ ] Type a correct answer — bot awards points, shows canonical answer, answer speed, standings, and period totals in one message
  • [ ] Observe hint progression: second hint appears after game.hintDelay seconds with partial letter reveal, showing current points and remaining time
  • [ ] Observe second hint: more letters revealed, point value halved
  • [ ] Observe third hint: maximum reveal, point value halved again
  • [ ] Let the question time out — bot reveals the correct answer
  • [ ] Observe point decay: verify 3-hint points are approximately half of 2-hint, etc.
  • [ ] Stop the game with @trivia — bot announces game stopped and no more questions delivered
  • [ ] Idle stop: @settings set game.idleWarnQuestions=2 game.idleStopQuestions=3 (both in one command, or lower the warning first), start trivia and stay silent — after the second unanswered question the bot says No answers for 2 questions. Trivia will stop automatically after 1 more unanswered question., and after the third No answers for 3 questions, stopping trivia. followed by the normal stop summary; the master log channel shows [<network>] Game stopped in #<channel> (idle: 3 unanswered questions), with no nick
  • [ ] Idle stop: answer one question mid-run — the count starts again, and the warning comes only after two more unanswered questions
  • [ ] Idle stop: in a KAOS question, claim one answer and let it time out — it does not count as unanswered
  • [ ] Idle stop: @skip a question during a run — the count starts again
  • [ ] @settings set game.idleStopQuestions=0 — no warning and no stop, however many questions go unanswered
  • [ ] @settings set game.idleWarnQuestions=10 while game.idleStopQuestions is 5 — rejected with a constraint violation
  • [ ] Verify pool reshuffle: after exhausting all questions, bot announces pool reset and continues
  • [ ] Verify @recall during active question: shows question, current hint level, category, remaining points, approximate time remaining
  • [ ] Verify @recall when no game is active: bot reports no active game

KAOS Questions

  • [ ] Trigger a KAOS question (set kaos.frequency=1 to force frequent KAOS)
  • [ ] Verify KAOS display: shows topic, answer count, and point value; 1st hint shows answer count and per-answer points; 2nd+ hints show bracketed slots
  • [ ] Two players each claim different answers — both receive full base points for their claimed answers
  • [ ] Observe progressive KAOS hints: slots already claimed are shown filled, remaining show progressive letter reveal
  • [ ] One player claims all answers alone — verify NO completion bonus (bonus requires multi-player)
  • [ ] Multi-player: all answers found collectively — verify completion bonus is awarded and displayed
  • [ ] KAOS answer does NOT increment player streak
  • [ ] Non-KAOS correct answer after KAOS question properly increments streak
  • [ ] Duplicate answer claim (same answer twice by different players) — bot silently ignores, no double award
  • [ ] KAOS question with two similar answers (e.g. question 2353's Tee Back / Vee Back) — typing either exactly claims that answer, not its neighbour
  • [ ] Then claim the other one — both halves of the pair are claimable
  • [ ] Retype a claimed answer with different spacing (veeback after Vee Back) — silently ignored, does not consume the neighbour
  • [ ] Set kaos.frequency=-1 (KAOS-only mode) — verify only KAOS questions are delivered
  • [ ] Set kaos.frequency=0 (disabled) — verify no KAOS questions appear in game session
  • [ ] @recall during KAOS: shows claimed answers and remaining unclaimed slots

Scramble Questions

  • [ ] Trigger a scramble question (requires scramble type questions in the question set)
  • [ ] Verify scramble display: #N Unscramble: <question text> [Category] [pts] followed by HINT: DRSECBMAL (scrambled answer in uppercase)
  • [ ] Type the exact unscrambled word — bot awards points
  • [ ] Attempt fuzzy answer (close but not exact) — bot does NOT accept it
  • [ ] Observe progressive hints: first hint is the scrambled word shown immediately, then timed hints with partial letter reveal, points, and remaining time
  • [ ] Verify scramble.hintDelay setting works independently from game.hintDelay
  • [ ] Verify scramble.questionTimeout setting works independently from game.questionTimeout
  • [ ] Multi-word scramble: each word scrambled independently, spaces preserved

Scoring and Streaks

  • [ ] Answer multiple consecutive questions correctly — streak counter increments and is displayed
  • [ ] Another player answers a question — your streak resets to zero, no streak displayed on your next correct answer
  • [ ] Streak is NOT shown when it equals 1 (only shown when > 1)
  • [ ] Verify period totals in @stats: daily, weekly, monthly, all-time totals all update after each correct answer
  • [ ] Verify leaderboard @topday shows correct ranking with comma-formatted point values
  • [ ] Verify @topweek, @topmonth, @topever/@topall all function
  • [ ] Verify paging: @topday20 shows ranks 11-20 (second page)
  • [ ] Verify @stats <nick> shows stats for another player
  • [ ] Verify @mystats shows your own stats without typing your nick
  • [ ] Period totals reset correctly at the configured timezone boundary (daily reset at midnight per scoring.timezone)
  • [ ] Runner-up detection: second correct answer within game.runnerUpWindow (default 2s) of the first receives runner-up recognition (no points awarded)
  • [ ] @trivia strict — start banner names STRICT mode; stop summary marks it too
  • [ ] @lockout on maintenance — confirms with the blocked list and the reason
  • [ ] During lockout: @trivia refuses to start with a warning naming the reason
  • [ ] During lockout: a running game keeps going, and @trivia still stops it
  • [ ] During lockout: @question add, @points add, @importpreview, @confirm, @activate all warn and do nothing
  • [ ] During lockout: @question show, @questionset list, @stats and the leaderboards still work
  • [ ] During lockout as ADMIN: still blocked — admins are not exempt
  • [ ] @lockout with no argument reports who set it, when and why
  • [ ] Reload the bot while locked out — it comes back still locked
  • [ ] @lockout off — everything works again
  • [ ] In a strict game: a near miss is rejected, and The X does not answer X
  • [ ] In a strict game: case and punctuation are still forgiven (dont stop answers Don't Stop)
  • [ ] In a strict game: a runner-up near miss is rejected as well
  • [ ] @trivia strict while a game runs — refused, the running game keeps going
  • [ ] @trivia nonsense — usage reply, no game started
  • [ ] @settings set game.runnerUpWindow=0 — no runner-up line is announced at all
  • [ ] @trivia to stop, then @trivia again straight away — refused with you can restart in Ns
  • [ ] Same, as an ADMIN — starts immediately, bypass recorded in the master log
  • [ ] Wait out game.restartCooldown, then @trivia — starts normally
  • [ ] @settings set game.restartCooldown=0 — restart is never refused
  • [ ] @settings set game.runnerUpWindow=10 with game.delayBetweenQuestions at 5 — rejected with a constraint violation

Configuration

  • [ ] @settings list — displays all settings organized by namespace with current values and scope indicators
  • [ ] @settings get game.hintDelay — returns current value and scope (channel/network/global)
  • [ ] @settings set game.hintDelay 20 — updates the setting, verify shorter/longer hint delay in next game
  • [ ] @settings reset game.hintDelay — resets to next-level default
  • [ ] @settings set-default scoring.minPoints 5 — changes global default, verify inherited by channels with no override
  • [ ] Constraint validation: @settings set game.hintDelay 999 where 999 >= game.questionTimeout — bot rejects with constraint violation message
  • [ ] Type validation: @settings set game.hintDelay abc — bot rejects with type error
  • [ ] Unknown setting: @settings get nonexistent.setting — bot reports unknown setting and suggests @settings list
  • [ ] @settings export — returns JSON string of current settings
  • [ ] @settings import with exported JSON — re-applies same settings, no changes reported

Permissions

  • [ ] @grant <nick> PLAY — user can answer questions and view stats
  • [ ] @grant <nick> START — user can @trivia to start/stop games
  • [ ] @grant <nick> MODERATE — user can @poolreset, @setquestionset, and manage config settings
  • [ ] @grant <nick> MANAGE — user can @question add/edit/delete
  • [ ] @grant <nick> ADMIN — user has full control including @grant/@revoke
  • [ ] @revoke <nick> START — user can no longer start games; verify denial message on @trivia
  • [ ] @deny <nick> PLAY — user is blocked even from default PLAY permission; verify denial on answer
  • [ ] @revoke <nick> PLAY — denial removed; user can answer again
  • [ ] @mypermissions — shows your effective permissions with source attribution (explicit vs default)
  • [ ] @permissions <nick> — shows another user's effective permissions (requires ADMIN)
  • [ ] @listperms — lists all explicit permission grants on current network (ADMIN-only, via private notice)
  • [ ] Hierarchy: user with ADMIN can use MANAGE/MODERATE/START/PLAY commands without needing individual grants
  • [ ] Default permission: set permissions.defaultStart=true, verify users without explicit grants can @trivia
  • [ ] Mode map +a/+q: with permissions.modeMapEnabled=true on a network that supports +q and +a, a +q user can run an ADMIN command, a +a user can run a MANAGE command but not an ADMIN one; after -q the owner falls back to the op mapping; after @reload Inquisitor the tiers return once the NAMES reply arrives
  • [ ] @settings set permissions.modeMapOwner=NONE — the owner gets only the op mapping (permissions.modeMapOwner removes just the +q tier)

Question Management

  • [ ] @question add "Question text" "answer" --set SetName --category Cat — adds question, verify with @question show <id>
  • [ ] @question add with KAOS type: @question add "Topic" "answer1" --set SetName --type kaos --alternatives "answer2,answer3"
  • [ ] @question edit <id> --answer "new_answer" — changes answer, shows diff output answer: 'old' -> 'new'
  • [ ] @question delete <id> — shows preview requiring @confirm within 5 minutes
  • [ ] @confirm within time window — question deleted
  • [ ] @question list --set SetName — lists questions with truncated display, paginated
  • [ ] @question search "term" — returns results sorted by relevance score percentage
  • [ ] @question move <id> --to-set <set> — moves question to different set
  • [ ] @questionset add <name> — creates a new empty question set
  • [ ] @questionset rename <old> <new> — renames question set
  • [ ] @questionset list — lists all sets with question counts and active/inactive status
  • [ ] @questionset stats <name> — shows detailed stats on one line (total, types, categories, asked, answered)
  • [ ] @questionset delete <name> — preview and confirm required; verify cascade delete of all questions
  • [ ] @category list --set SetName — lists categories with question counts
  • [ ] Duplicate detection: add a question very similar to an existing one — bot warns about match percentage
  • [ ] @question add --force — bypasses duplicate check for intentional near-duplicate

Import

  • [ ] IRC import: @importpreview /path/to/file.ques General — shows preview with question count, duplicates found
  • [ ] @importconfirm /path/to/file.ques General — commits the import; verify questions appear in @questionset stats
  • [ ] Import with duplicate questions: preview shows duplicate count, confirmed import skips duplicates
  • [ ] @importpointspreview /path/to/points.txt — previews player score import
  • [ ] @importpointsconfirm /path/to/points.txt — commits points import
  • [ ] Hostmask import: a BogusTrivia file whose player lines carry ident@host — @importpointspreview shows Hostmasks to attach: N and writes nothing; after @importpointsconfirm the player is recognised from the identifier under another nick
  • [ ] Hostmask conflict: a file whose identifier already belongs to another account — preview and confirm list <id> (file nick <nick>) already belongs to <owner> and the points still import
  • [ ] Backfill: @importpointspreview --hostmasks-only /path/to/players.pts then @importpointsconfirm --hostmasks-only /path/to/players.pts — identifiers are attached, totals are unchanged, unknown nicks are listed as no account: <nick>; confirming without the flag is refused
  • [ ] CLI tool: python tools/import_questions.py --config supybot.conf path/to/file.ques — imports without IRC
  • [ ] CLI --dry-run flag: reports what would be imported without writing to database
  • [ ] CLI wildcard: python tools/import_questions.py *.ques — imports multiple files
  • [ ] Resume after interruption: interrupt a large import, restart, verify checkpoint resumes from last saved position

Multi-Network

  • [ ] Activate on a second network with a different question set — verify scores are isolated (no cross-contamination)
  • [ ] Game on network A does not affect scores or state on network B
  • [ ] @deactivate on one network — verify game stops and network is deactivated; other network unaffected
  • [ ] @reactivate — verify network is reactivated and game can start again
  • [ ] Master log channel receives events from all networks (game start/stop, answer events, admin actions)
  • [ ] Per-network log channel receives only events from its own network
  • [ ] @listnetworks — shows all configured networks with their status and question sets (ADMIN)
  • [ ] @crossstats — shows combined statistics across all networks (ADMIN)
  • [ ] Per-network timezone: set different scoring.timezone on each network; verify daily reset at correct local midnight

Edge Cases

  • [ ] Multi-channel: verify two channels on the same network run independent games (separate state, separate pools)
  • [ ] Special characters in answers: answers with apostrophes, hyphens, accented characters all matched correctly
  • [ ] Long question answer: answer longer than 400 chars handled without IRC truncation errors
  • [ ] Empty question set: @trivia with a set containing no questions — bot reports no questions available
  • [ ] Rate-limited denial: trigger a permission denial, trigger again within 60 seconds — only one denial message is sent to IRC (both are logged)
  • [ ] Game active guard: attempt to change question set with @setquestionset while game is running — bot refuses with clear message
  • [ ] @question bulkdelete --set "SetName" — requires ADMIN, shows preview with count, @confirm required
  • [ ] @question bulkmove --from-set "Set1" --to-set "Set2" — bulk move requires ADMIN, preview and confirm required
  • [ ] Invalid hostmask: attempt to @grant a user whose nick is not in IRC state — bot falls back to nick!*@* pattern

Known Limitations

  • See SECURITY.md at the repository root for a full list of known security limitations, including:
    • SQL injection defenses and audit scope
    • Input validation rules and what is not validated
    • Rate limiting scope and bypass scenarios
    • File path validation rules for import commands

Additional known limitations for beta:

  • Multi-channel support on a single network is not guaranteed to be stable; single-channel per network is the tested configuration
  • Very large question sets (>100,000 questions) have not been load-tested; checkpoint imports should be used
  • Questions with extremely long text may be truncated by IRC protocol limits on some clients
  • debug.verbose generates high log volume; disable on production systems

Reporting Issues

Submit issues to the Forgejo tracker:

URL: https://git.ptp-trivia.me/Inquisitor/Inquisitor/issues

When reporting, include:

  1. Version: the version shown in the start banner (e.g., v1.2.0+20261029)
  2. Network and channel: e.g., EFNet #trivia
  3. Steps to reproduce: The exact sequence of IRC commands
  4. Expected behavior: What you expected to happen
  5. Actual behavior: What actually happened (include bot output if possible)
  6. Log excerpt: Relevant lines from the Limnoria bot log (if available)

Label issues with the appropriate label from the issue tracker's label list to help route them to the right area.