Replace bot tuning guesses with enforced calibration

This commit is contained in:
Jay
2026-07-26 05:13:54 -04:00
parent fffd60ad9d
commit 0ac69b101a
11 changed files with 690 additions and 162 deletions
+27 -10
View File
@@ -12,6 +12,9 @@ export JAVA_HOME="/Applications/Android Studio.app/Contents/jbr/Contents/Home"
./gradlew :engine:jvmTest # engine tests (JVM) ./gradlew :engine:jvmTest # engine tests (JVM)
./gradlew :engine:testAndroidHostTest # same suite, Android variant ./gradlew :engine:testAndroidHostTest # same suite, Android variant
./gradlew :sim:run --args="50000" # simulate 50k hands, print bot stats ./gradlew :sim:run --args="50000" # simulate 50k hands, print bot stats
./gradlew :sim:test # fast calibration-policy tests
./gradlew :sim:run --args="styles" # enforced controlled style experiment
./gradlew :sim:run --args="calibrate" # enforced 4×100k fixed-pool skill + style calibration
./gradlew :app:assembleDebug # build the APK ./gradlew :app:assembleDebug # build the APK
``` ```
@@ -44,12 +47,21 @@ export JAVA_HOME="/Applications/Android Studio.app/Contents/jbr/Contents/Home"
169 starting hands so `looseness` means "plays the top N%". 169 starting hands so `looseness` means "plays the top N%".
4. **The simulator is how bots get tuned.** Run it after any bot change, across 4. **The simulator is how bots get tuned.** Run it after any bot change, across
several seeds — a single seed will happily agree with a wrong conclusion. several seeds — a single seed will happily agree with a wrong conclusion.
The quick 50k run is diagnostic; only `calibrate` is an enforced result.
5. **A skill parameter must not smuggle in a style change.** Several bugs came 5. **A skill parameter must not smuggle in a style change.** Several bugs came
from exactly this: `positionAwareness` silently reduced hands played, from exactly this: `positionAwareness` silently reduced hands played,
`potOddsRespect` systematically loosened weak players (which is a *winning* `potOddsRespect` systematically loosened weak players (which is a *winning*
adjustment, so it inverted the gradient), and error direction overwrote style adjustment, so it inverted the gradient), and error direction overwrote style
entirely. Skill should change how *well* a decision is made, not how loose or entirely. Skill should change how *well* a decision is made, not how loose or
tight the player is. tight the player is.
6. **Pot odds are the post-flop baseline.** Never apply a blanket implied-odds
discount: it is categorically wrong on the river, and future value on earlier
streets must account for future costs and reverse implied odds before it is
called an advantage.
7. **DecisionTrace is the coach contract.** It records raw pot odds, the actual
adjusted threshold, every adjustment, intended and chosen actions, and whether
a skill error changed the decision. The coach explains these values; it does
not reconstruct hidden bot logic.
## Testing notes ## Testing notes
@@ -59,8 +71,9 @@ export JAVA_HOME="/Applications/Android Studio.app/Contents/jbr/Contents/Home"
share one: bots consume RNG proportional to their `equityIterations`, so a share one: bots consume RNG proportional to their `equityIterations`, so a
shared stream means changing a profile silently changes the cards dealt. shared stream means changing a profile silently changes the cards dealt.
- `./gradlew :sim:run --args="chart"` dumps the starting-hand ranking. - `./gradlew :sim:run --args="chart"` dumps the starting-hand ranking.
- Small samples lie. 1,000 hands is not enough to rank profiles — use 50,000+ - Small samples lie. 1,000 hands is not enough to rank profiles. Use the paired
before believing a gradient. 4×100k fixed-opponent calibration before accepting a skill change; it computes
confidence bounds and exits nonzero when the contract fails.
## Status ## Status
@@ -70,14 +83,18 @@ export JAVA_HOME="/Applications/Android Studio.app/Contents/jbr/Contents/Home"
action order, malformed agent output, **TDA Rule 47** (incomplete raises do not action order, malformed agent output, **TDA Rule 47** (incomplete raises do not
reopen betting, but several that cumulatively reach a full raise do), and reopen betting, but several that cumulatively reach a full raise do), and
**TDA Rule 20** (odd chip to the first winner left of the button). **TDA Rule 20** (odd chip to the first winner left of the button).
- Bots: profiles play like their labels (Rock 14.5% VPIP against a 12% setting; - Bots: controlled style calibration holds style constant against the same five
`ProfileBehaviourTest` asserts this). Win rates are in a plausible range — opponents and deal seed. Rock is 10.3% VPIP, Maniac 67.5%; looseness ordering,
roughly +20 bb/100 for a strong seat rather than the earlier +113. Calling Station passivity (9.8% PFR, 0.27 AF), Maniac aggression, and PFR
- **Adjacent top tiers are not separable.** Advanced and Expert sit inside relationships all pass.
seed-to-seed noise of each other over 100k hands. The simulator therefore - Skill calibration pairs four 100k-hand seeds. Every candidate occupies the same
asserts each level beats the one *two* tiers below it, which holds on every seat against the same fixed opponent pool and deal seed. Advanced and Expert
seed tried; claiming strict adjacent ordering from one seed would be reading are allowed to overlap, but both must beat Intermediate and Intermediate must
noise as signal. beat Beginner with a positive 95% lower confidence bound. Current lower bounds
are +26.63, +16.19, and +24.71 bb/100 respectively.
- Expert differs by mechanism: it alone maintains opponent reads. The aggression
prior is measured by the controlled neutral TAG experiment (0.229 observed,
0.22 configured), not selected because it looks plausible.
### Rules invariants that are easy to get wrong ### Rules invariants that are easy to get wrong
@@ -19,11 +19,20 @@ import kotlin.random.Random
data class DecisionTrace( data class DecisionTrace(
val equity: Double, val equity: Double,
val breakEvenEquity: Double, val breakEvenEquity: Double,
/** The final threshold actually used after every adjustment. */
val decisionThreshold: Double,
val potOdds: String, val potOdds: String,
/** What the strategy selected before a skill error was applied. */
val intended: Action,
val chosen: Action, val chosen: Action,
val mistakeApplied: Boolean,
/** Multipliers applied to the raw threshold, in evaluation order. */
val adjustments: List<ThresholdAdjustment>,
val reason: String, val reason: String,
) )
data class ThresholdAdjustment(val name: String, val factor: Double)
/** /**
* A bot that decides from equity and pot odds, then distorts that decision through * A bot that decides from equity and pot odds, then distorts that decision through
* its [SkillLevel] and [PlayStyle]. * its [SkillLevel] and [PlayStyle].
@@ -66,27 +75,27 @@ class MathBot(
val breakEven = Equity.potOdds(ctx.pot, ctx.toCall) val breakEven = Equity.potOdds(ctx.pot, ctx.toCall)
// Discipline is ACCURACY, not strictness. // Pot odds are the honest baseline. In particular, the river has no
// // future street from which to earn "implied" chips, so a blanket discount
// This used to shift weak players systematically toward calling, which is // is mathematically wrong. Flop/turn future value can be added later only
// not a weakness at all — calling wider than break-even against bad // as an explicit model that also accounts for future costs and reverse
// opponents is a winning adjustment, so it handed low-skill bots a real // implied odds; until then raw pot odds are the defensible threshold.
// edge and inverted the difficulty gradient. A weak player misjudges the val adjustments = ArrayList<ThresholdAdjustment>()
// threshold in *either* direction; being wrong is what costs money.
// Raw pot odds OVERSTATE the threshold. Calling also buys the chance to
// win more on later streets, so the genuinely correct bar sits below
// break-even. Anchoring a perfect player at raw break-even made them fold
// profitable hands, and Advanced — whose imprecision sometimes dipped
// below it — beat Expert consistently across seeds.
val trueBar = breakEven * IMPLIED_ODDS_DISCOUNT
val discipline = skill.potOddsRespect val discipline = skill.potOddsRespect
val misjudgement = (1.0 - discipline) * 0.50 val misjudgement = (1.0 - discipline) * 0.50
var bar = trueBar * (1.0 + (random.nextDouble() * 2 - 1) * misjudgement) val accuracyFactor = 1.0 + (random.nextDouble() * 2 - 1) * misjudgement
bar *= (1.0 - looseness * 0.35) var bar = breakEven * accuracyFactor
adjustments += ThresholdAdjustment("skill estimate", accuracyFactor)
val styleFactor = 1.0 - looseness * 0.35
bar *= styleFactor
adjustments += ThresholdAdjustment("style risk tolerance", styleFactor)
// Position is worth real equity, and better players know it. // Position is worth real equity, and better players know it.
bar *= if (ctx.inPosition) 1.0 - 0.12 * skill.positionAwareness val positionFactor = if (ctx.inPosition) 1.0 - 0.12 * skill.positionAwareness
else 1.0 + 0.10 * skill.positionAwareness else 1.0 + 0.10 * skill.positionAwareness
bar *= positionFactor
adjustments += ThresholdAdjustment("position", positionFactor)
// Exploitation: a habitual bettor's bet means less, so call wider against // Exploitation: a habitual bettor's bet means less, so call wider against
// them; a passive player's bet means strength, so fold more. // them; a passive player's bet means strength, so fold more.
@@ -96,27 +105,37 @@ class MathBot(
}?.seat }?.seat
if (bettor != null && bettor != ctx.seat.index && reads.actionsObserved(bettor) >= 25) { if (bettor != null && bettor != ctx.seat.index && reads.actionsObserved(bettor) >= 25) {
val edge = reads.aggressionRate(bettor) - OpponentModel.NEUTRAL_AGGRESSION val edge = reads.aggressionRate(bettor) - OpponentModel.NEUTRAL_AGGRESSION
bar *= (1.0 - edge * 0.50).coerceIn(0.7, 1.3) val readFactor = (1.0 - edge * 0.50).coerceIn(0.7, 1.3)
bar *= readFactor
adjustments += ThresholdAdjustment("opponent read", readFactor)
} }
} }
val raiseBar = (0.62 - aggression * 0.22).coerceIn(0.30, 0.75) val raiseBar = (0.62 - aggression * 0.22).coerceIn(0.30, 0.75)
var decision = decide(ctx, equity, bar, raiseBar, aggression, style, opponents) val intended = decide(ctx, equity, bar, raiseBar, aggression, style, opponents)
// Outright mistakes, on top of misjudgement. // Skill errors degrade the selected action; style has already chosen how
if (random.nextDouble() < skill.errorRate) { // this player prefers to express a close decision.
decision = blunder(ctx, decision) val chosen = if (random.nextDouble() < skill.errorRate) {
postflopMistake(ctx, intended, equity, breakEven)
} else {
intended
} }
val mistakeApplied = chosen != intended
lastTrace = DecisionTrace( lastTrace = DecisionTrace(
equity = equity, equity = equity,
breakEvenEquity = breakEven, breakEvenEquity = breakEven,
decisionThreshold = bar,
potOdds = if (ctx.toCall > 0) "${ctx.pot}:${ctx.toCall}" else "no bet to call", potOdds = if (ctx.toCall > 0) "${ctx.pot}:${ctx.toCall}" else "no bet to call",
chosen = decision, intended = intended,
reason = traceReason(equity, breakEven, ctx), chosen = chosen,
mistakeApplied = mistakeApplied,
adjustments = adjustments,
reason = traceReason(equity, breakEven, bar, intended, chosen, mistakeApplied, ctx),
) )
return decision return chosen
} }
/** /**
@@ -132,26 +151,21 @@ class MathBot(
val looseness = (style.looseness + mood.loosenessBonus()).coerceIn(0.02, 1.0) val looseness = (style.looseness + mood.loosenessBonus()).coerceIn(0.02, 1.0)
val aggression = (style.aggression + mood.aggressionBonus()).coerceIn(0.0, 1.0) val aggression = (style.aggression + mood.aggressionBonus()).coerceIn(0.0, 1.0)
// Weak players enter a few more pots than their style says — but the // Position awareness shifts where a range is played without changing its
// widening has to saturate. Multiplying overshot badly at the loose end // average width. One of N players is last to act; the out-of-position
// (a 0.75 maniac became 0.93) while still drowning out the tight end. // reduction is derived from that share rather than tuned independently.
// Interpolating toward "play everything" scales with the room left. val positional = preflopPositionFactor(ctx, skill.positionAwareness)
val slop = (1.0 - skill.potOddsRespect) * 0.08
// Position awareness must shift WHERE hands are played, not how many.
// Widening 45% in position while narrowing 25% out of it is not neutral:
// a seat is last to act only about a quarter of the time, so the tighter
// branch dominated and higher awareness silently became lower volume.
// Expert then played fewer pots than Advanced and collected less from the
// weak seats, which is why it kept losing to a strictly worse profile.
val positional = if (ctx.inPosition) 1.0 + 0.45 * skill.positionAwareness
else 1.0 - 0.12 * skill.positionAwareness
val facingRaise = ctx.toCall > ctx.bigBlind val facingRaise = ctx.toCall > ctx.bigBlind
var gate = (looseness + (1.0 - looseness) * slop) * positional var gate = looseness * positional
if (facingRaise) gate *= 0.45 if (facingRaise) gate *= 0.45
gate = gate.coerceIn(0.01, 1.0) gate = gate.coerceIn(0.01, 1.0)
val raiseGate = gate * (0.30 + aggression * 0.45) // Aggression controls what share of the playable range is raised. The old
// 30% floor made even a zero-aggression "Calling Station" raise nearly a
// third of every entered range, contradicting the profile name.
val raiseShare = (0.05 + aggression * 0.75).coerceIn(0.05, 0.85)
val raiseGate = gate * raiseShare
val action = when { val action = when {
pct <= raiseGate && ctx.canRaise -> pct <= raiseGate && ctx.canRaise ->
@@ -162,14 +176,31 @@ class MathBot(
if (ctx.canCheck) Action(ActionType.CHECK) else Action(ActionType.FOLD) if (ctx.canCheck) Action(ActionType.CHECK) else Action(ActionType.FOLD)
} }
val final = if (random.nextDouble() < skill.errorRate) blunder(ctx, action) else action val final = if (random.nextDouble() < skill.errorRate) {
preflopMistake(ctx, action, pct, gate)
} else {
action
}
val mistakeApplied = final != action
val equityScore = 1.0 - pct
val threshold = 1.0 - gate
lastTrace = DecisionTrace( lastTrace = DecisionTrace(
equity = 1.0 - pct, equity = equityScore,
breakEvenEquity = Equity.potOdds(ctx.pot, ctx.toCall), breakEvenEquity = Equity.potOdds(ctx.pot, ctx.toCall),
decisionThreshold = threshold,
potOdds = if (ctx.toCall > 0) "${ctx.pot}:${ctx.toCall}" else "no bet to call", potOdds = if (ctx.toCall > 0) "${ctx.pot}:${ctx.toCall}" else "no bet to call",
intended = action,
chosen = final, chosen = final,
reason = "Starting hand is in the top ${(pct * 100).roundToInt()}% " + mistakeApplied = mistakeApplied,
"and this profile plays about the top ${(gate * 100).roundToInt()}%.", adjustments = listOf(
ThresholdAdjustment("position", positional),
ThresholdAdjustment("facing raise", if (facingRaise) 0.45 else 1.0),
),
reason = buildString {
append("Starting hand is in the top ${(pct * 100).roundToInt()}%; ")
append("this profile's range here is ${(gate * 100).roundToInt()}%.")
if (mistakeApplied) append(" Skill error changed $action to $final.")
},
) )
return final return final
} }
@@ -256,85 +287,93 @@ class MathBot(
} }
/** /**
* A mistake, in the direction this particular player tends to make them. * Pre-flop errors stay local to the style's range boundary.
* *
* The previous model pushed every error the same way — FOLD became CALL, * A global FOLD→CALL rewrite gave every beginner a 30% VPIP floor. Here a
* bets became checks, and only half of CALL ever tightened — so a 30% error * marginal hand can cross the boundary, and a marginal call can be folded,
* rate put a "Rock" near a 30% VPIP floor no matter how tight its style said * while trash remains trash and a missed raise remains inside the same range.
* it was. Errors degraded skill by overwriting character.
*
* Real players have characteristic leaks. A nit's mistake is folding a hand
* they should have played or flat-calling a hand they should have raised; a
* maniac's is firing at nothing. So the *direction* is drawn from the
* player's own looseness, and only passivity — missing value with a hand
* worth betting, the most common beginner leak of all — is style-neutral.
*/ */
private fun blunder(ctx: DecisionContext, intended: Action): Action = private fun preflopMistake(
if (ctx.street == Street.PREFLOP) preflopBlunder(ctx, intended) ctx: DecisionContext,
else postflopBlunder(ctx, intended) intended: Action,
percentile: Double,
/** gate: Double,
* Pre-flop mistakes follow the player's character. ): Action = when (intended.type) {
* ActionType.FOLD ->
* A nit's pre-flop error is folding a hand they should have played, not if (ctx.toCall > 0 && percentile <= gate + PREFLOP_ERROR_BAND) {
* suddenly playing like a maniac. Letting the error direction ignore style is Action(ActionType.CALL, ctx.toCall)
* what put a 12% Rock at 41% VPIP: it overwrote the label rather than } else {
* degrading the skill behind it. intended
*/
private fun preflopBlunder(ctx: DecisionContext, intended: Action): Action {
val loose = random.nextDouble() < profile.style.looseness
return when (intended.type) {
ActionType.FOLD ->
if (loose && ctx.toCall > 0) Action(ActionType.CALL, ctx.toCall) else intended
ActionType.CHECK ->
if (loose && ctx.canRaise) Action(ActionType.BET, ctx.minRaiseTo) else intended
ActionType.CALL -> when {
loose && ctx.canRaise -> Action(ActionType.RAISE, ctx.minRaiseTo)
!loose && ctx.toCall > 0 -> Action(ActionType.FOLD)
else -> intended
} }
ActionType.BET, ActionType.RAISE -> ActionType.CALL ->
if (ctx.canCheck) Action(ActionType.CHECK) else Action(ActionType.CALL, ctx.toCall) if (percentile >= (gate - PREFLOP_ERROR_BAND).coerceAtLeast(0.0)) {
} Action(ActionType.FOLD)
} else {
intended
}
ActionType.BET, ActionType.RAISE ->
if (ctx.canCheck) Action(ActionType.CHECK) else Action(ActionType.CALL, ctx.toCall)
ActionType.CHECK -> intended
} }
/** /**
* Post-flop mistakes cost money, whatever the player's style. * Post-flop errors choose an action with worse immediate value whenever that
* * comparison is available. Style has already selected the intended action.
* This is where weak players actually lose: paying off when beaten and
* checking back hands that should have bet. Style must NOT soften these, or
* a high error rate becomes free — a tight player would simply fold more,
* which costs almost nothing, and the difficulty gradient collapses. That
* inversion showed up immediately in the simulator when errors were made
* style-directed everywhere.
*/ */
private fun postflopBlunder(ctx: DecisionContext, intended: Action): Action = when (intended.type) { private fun postflopMistake(
// Curiosity call: the classic way a beginner donates. ctx: DecisionContext,
intended: Action,
equity: Double,
rawBreakEven: Double,
): Action = when (intended.type) {
ActionType.FOLD -> ActionType.FOLD ->
if (ctx.toCall > 0) Action(ActionType.CALL, ctx.toCall) else Action(ActionType.CHECK) if (ctx.toCall > 0) Action(ActionType.CALL, ctx.toCall) else intended
ActionType.CHECK -> intended ActionType.CALL -> when {
ActionType.CALL -> equity >= rawBreakEven -> Action(ActionType.FOLD)
if (random.nextDouble() < 0.35 && ctx.toCall > 0) Action(ActionType.FOLD) else intended ctx.canRaise -> Action(ActionType.RAISE, ctx.minRaiseTo)
// Missing value with a hand worth betting. else -> intended
}
ActionType.BET, ActionType.RAISE -> ActionType.BET, ActionType.RAISE ->
if (ctx.canCheck) Action(ActionType.CHECK) else Action(ActionType.CALL, ctx.toCall) if (ctx.canCheck) Action(ActionType.CHECK) else Action(ActionType.CALL, ctx.toCall)
ActionType.CHECK ->
if (equity < 0.25 && ctx.canRaise) Action(ActionType.BET, ctx.minRaiseTo) else intended
}
private fun preflopPositionFactor(ctx: DecisionContext, awareness: Double): Double {
val players = (ctx.activeOpponents + 1).coerceAtLeast(2)
val inPositionShare = 1.0 / players
val widening = 0.45 * awareness
val narrowing = widening * inPositionShare / (1.0 - inPositionShare)
return if (ctx.inPosition) 1.0 + widening else 1.0 - narrowing
} }
private companion object { private companion object {
/** const val PREFLOP_ERROR_BAND = 0.08
* How far below raw pot odds the correct calling threshold sits, allowing
* for the value a call captures on later streets.
*/
const val IMPLIED_ODDS_DISCOUNT = 0.88
} }
private fun traceReason(equity: Double, breakEven: Double, ctx: DecisionContext): String { private fun traceReason(
equity: Double,
breakEven: Double,
threshold: Double,
intended: Action,
chosen: Action,
mistakeApplied: Boolean,
ctx: DecisionContext,
): String {
val pct = (equity * 100).roundToInt() val pct = (equity * 100).roundToInt()
return if (ctx.toCall > 0) { return if (ctx.toCall > 0) {
val need = (breakEven * 100).roundToInt() val need = (breakEven * 100).roundToInt()
"About $pct% equity against ${ctx.activeOpponents} opponent(s); needed $need% to call profitably." val used = (threshold * 100).roundToInt()
buildString {
append("About $pct% equity against ${ctx.activeOpponents} opponent(s); ")
append("raw pot odds require $need%, adjusted decision threshold $used%.")
if (mistakeApplied) append(" Skill error changed $intended to $chosen.")
}
} else { } else {
"About $pct% equity against ${ctx.activeOpponents} opponent(s), no bet to face." buildString {
append("About $pct% equity against ${ctx.activeOpponents} opponent(s), no bet to face.")
if (mistakeApplied) append(" Skill error changed $intended to $chosen.")
}
} }
} }
} }
@@ -56,14 +56,13 @@ class OpponentModel {
private const val MIN_SAMPLE = 25 private const val MIN_SAMPLE = 25
/** /**
* What an ordinary player's bet-and-raise share of actions looks like. * Prior for an ordinary player's bet-and-raise share of actions.
* *
* NOT 0.5. Actions counted include folds, checks and calls, so even an * Actions counted include folds, checks and calls. The controlled style
* aggressive player only bets or raises about a third of the time. * calibration measures a neutral Tight-Aggressive profile at about 0.22;
* Treating 0.5 as neutral made every opponent read as passive, so the two * this prior is guarded there rather than justified by an assertion that
* skill levels that consult this model tightened against the whole table * merely checks whether a constant looks plausible.
* and lost money for it — the exploitation feature was a handicap.
*/ */
const val NEUTRAL_AGGRESSION = 0.32 const val NEUTRAL_AGGRESSION = 0.22
} }
} }
@@ -22,7 +22,7 @@ enum class SkillLevel(
// hands, which is not two difficulty levels — it is one, labelled twice. // hands, which is not two difficulty levels — it is one, labelled twice.
BEGINNER("Beginner", equityIterations = 120, errorRate = 0.30, potOddsRespect = 0.20, positionAwareness = 0.10, readsOpponents = false), BEGINNER("Beginner", equityIterations = 120, errorRate = 0.30, potOddsRespect = 0.20, positionAwareness = 0.10, readsOpponents = false),
INTERMEDIATE("Intermediate", equityIterations = 500, errorRate = 0.16, potOddsRespect = 0.52, positionAwareness = 0.40, readsOpponents = false), INTERMEDIATE("Intermediate", equityIterations = 500, errorRate = 0.16, potOddsRespect = 0.52, positionAwareness = 0.40, readsOpponents = false),
ADVANCED("Advanced", equityIterations = 1200, errorRate = 0.07, potOddsRespect = 0.76, positionAwareness = 0.70, readsOpponents = true), ADVANCED("Advanced", equityIterations = 1200, errorRate = 0.07, potOddsRespect = 0.76, positionAwareness = 0.70, readsOpponents = false),
EXPERT("Expert", equityIterations = 3000, errorRate = 0.010, potOddsRespect = 1.00, positionAwareness = 1.00, readsOpponents = true), EXPERT("Expert", equityIterations = 3000, errorRate = 0.010, potOddsRespect = 1.00, positionAwareness = 1.00, readsOpponents = true),
} }
@@ -0,0 +1,72 @@
package com.jsjdesigns.poker.bot
import com.jsjdesigns.poker.core.cardsOf
import com.jsjdesigns.poker.game.DecisionContext
import com.jsjdesigns.poker.game.Seat
import com.jsjdesigns.poker.game.Street
import kotlinx.coroutines.test.runTest
import kotlin.random.Random
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFalse
import kotlin.test.assertTrue
class DecisionTraceTest {
private fun context(bot: MathBot, street: Street, board: String): DecisionContext {
val seat = Seat(0, "Expert", 200, bot)
seat.hole = cardsOf("Ah Qh")
return DecisionContext(
street = street,
seat = seat,
board = cardsOf(board),
pot = 80,
toCall = 20,
minRaiseTo = 60,
maxRaiseTo = 200,
activeOpponents = 1,
seatsActingAfter = 1,
bigBlind = 2,
history = emptyList(),
handNumber = 1,
decisionToken = 1,
bettingReopened = true,
)
}
@Test
fun `river does not receive a fictional implied-odds discount`() = runTest {
val profile = BotProfile("E", SkillLevel.EXPERT, PlayStyle.TIGHT_AGGRESSIVE)
val flopBot = MathBot(profile, Random(77))
val riverBot = MathBot(profile, Random(77))
flopBot.act(context(flopBot, Street.FLOP, "2c 7d 9s"))
riverBot.act(context(riverBot, Street.RIVER, "2c 7d 9s Jc 3h"))
val flop = flopBot.lastTrace!!
val river = riverBot.lastTrace!!
assertEquals(
flop.decisionThreshold,
river.decisionThreshold,
1e-12,
"street alone must not apply a blanket discount to pot odds",
)
assertFalse(river.adjustments.any { "implied" in it.name.lowercase() })
}
@Test
fun `trace records the rule that actually produced the action`() = runTest {
val bot = MathBot(
BotProfile("B", SkillLevel.BEGINNER, PlayStyle.TIGHT_AGGRESSIVE),
Random(11),
)
bot.act(context(bot, Street.RIVER, "2c 7d 9s Jc 3h"))
val trace = bot.lastTrace!!
assertEquals(trace.intended != trace.chosen, trace.mistakeApplied)
assertTrue(trace.decisionThreshold >= 0.0)
assertTrue(trace.adjustments.isNotEmpty())
assertTrue("raw pot odds require" in trace.reason)
assertTrue("adjusted decision threshold" in trace.reason)
}
}
@@ -83,15 +83,13 @@ class OpponentModelTest {
} }
/** /**
* The neutral point is NOT 0.5. Folds, checks and calls are counted too, so * This is only a broad sanity bound. The actual prior is validated by the
* even an aggressive player bets or raises only about a third of the time. * controlled simulator style calibration against a neutral TAG profile.
* Assuming 0.5 made every opponent read as passive, and the skill levels that
* consult this model tightened against the whole table and lost money for it.
*/ */
@Test @Test
fun `the neutral baseline reflects a realistic bet-raise share`() { fun `the neutral baseline reflects a realistic bet-raise share`() {
assertTrue( assertTrue(
OpponentModel.NEUTRAL_AGGRESSION in 0.2..0.45, OpponentModel.NEUTRAL_AGGRESSION in 0.15..0.35,
"a plausible neutral bet/raise share, was ${OpponentModel.NEUTRAL_AGGRESSION}", "a plausible neutral bet/raise share, was ${OpponentModel.NEUTRAL_AGGRESSION}",
) )
} }
@@ -43,8 +43,9 @@ class ProfileBehaviourTest {
minRaiseTo = 4, minRaiseTo = 4,
maxRaiseTo = 200, maxRaiseTo = 200,
activeOpponents = 5, activeOpponents = 5,
// Alternate position so the measurement is not biased either way. // Six-handed: one seat per orbit is last to act. A 50/50 split
seatsActingAfter = if (it % 2 == 0) 0 else 2, // materially over-weighted the in-position widening branch.
seatsActingAfter = if (it % 6 == 0) 0 else 2,
bigBlind = 2, bigBlind = 2,
history = emptyList(), history = emptyList(),
handNumber = it + 1, handNumber = it + 1,
@@ -57,7 +58,7 @@ class ProfileBehaviourTest {
return entered.toDouble() / hands return entered.toDouble() / hands
} }
private suspend fun assertPlaysLikeLabel(style: PlayStyle, skill: SkillLevel, tolerance: Double = 0.14) { private suspend fun assertPlaysLikeLabel(style: PlayStyle, skill: SkillLevel, tolerance: Double = 0.06) {
val vpip = measureVpip(BotProfile(style.label, skill, style)) val vpip = measureVpip(BotProfile(style.label, skill, style))
assertTrue( assertTrue(
abs(vpip - style.looseness) <= tolerance, abs(vpip - style.looseness) <= tolerance,
@@ -108,7 +109,7 @@ class ProfileBehaviourTest {
val beginner = measureVpip(BotProfile("R", SkillLevel.BEGINNER, PlayStyle.ROCK)) val beginner = measureVpip(BotProfile("R", SkillLevel.BEGINNER, PlayStyle.ROCK))
val expert = measureVpip(BotProfile("R", SkillLevel.EXPERT, PlayStyle.ROCK)) val expert = measureVpip(BotProfile("R", SkillLevel.EXPERT, PlayStyle.ROCK))
assertTrue( assertTrue(
abs(beginner - expert) < 0.12, abs(beginner - expert) < 0.06,
"a beginner rock (${"%.1f".format(beginner * 100)}%) and an expert rock " + "a beginner rock (${"%.1f".format(beginner * 100)}%) and an expert rock " +
"(${"%.1f".format(expert * 100)}%) should both be recognisably tight", "(${"%.1f".format(expert * 100)}%) should both be recognisably tight",
) )
@@ -0,0 +1,20 @@
package com.jsjdesigns.poker.bot
import kotlin.test.Test
import kotlin.test.assertFalse
import kotlin.test.assertTrue
class SkillMechanismTest {
@Test
fun `expert has a distinct opponent-reading mechanism`() {
assertFalse(
SkillLevel.ADVANCED.readsOpponents,
"Advanced should play strong card-and-position poker without exploitation reads",
)
assertTrue(
SkillLevel.EXPERT.readsOpponents,
"Expert should differ by mechanism, not merely a nearby constant",
)
}
}
+1
View File
@@ -6,6 +6,7 @@ plugins {
dependencies { dependencies {
implementation(project(":engine")) implementation(project(":engine"))
implementation(libs.kotlinx.coroutines.core) implementation(libs.kotlinx.coroutines.core)
testImplementation(kotlin("test"))
} }
application { application {
@@ -2,6 +2,7 @@ package com.jsjdesigns.poker.sim
import com.jsjdesigns.poker.bot.BotProfile import com.jsjdesigns.poker.bot.BotProfile
import com.jsjdesigns.poker.bot.MathBot import com.jsjdesigns.poker.bot.MathBot
import com.jsjdesigns.poker.bot.OpponentModel
import com.jsjdesigns.poker.bot.PlayStyle import com.jsjdesigns.poker.bot.PlayStyle
import com.jsjdesigns.poker.bot.SkillLevel import com.jsjdesigns.poker.bot.SkillLevel
import com.jsjdesigns.poker.core.Card import com.jsjdesigns.poker.core.Card
@@ -12,6 +13,7 @@ import com.jsjdesigns.poker.game.Seat
import com.jsjdesigns.poker.game.Street import com.jsjdesigns.poker.game.Street
import com.jsjdesigns.poker.game.Table import com.jsjdesigns.poker.game.Table
import kotlin.math.abs import kotlin.math.abs
import kotlin.math.sqrt
import kotlinx.coroutines.runBlocking import kotlinx.coroutines.runBlocking
import kotlin.random.Random import kotlin.random.Random
@@ -19,7 +21,7 @@ private const val SMALL_BLIND = 1
private const val BIG_BLIND = 2 private const val BIG_BLIND = 2
private const val STARTING_STACK = 200 // 100 big blinds private const val STARTING_STACK = 200 // 100 big blinds
private class Stats(val name: String, val profile: BotProfile) { internal class Stats(val name: String, val profile: BotProfile) {
var net = 0L var net = 0L
var hands = 0 var hands = 0
var vpip = 0 var vpip = 0
@@ -27,14 +29,24 @@ private class Stats(val name: String, val profile: BotProfile) {
var wins = 0 var wins = 0
var postflopBets = 0 var postflopBets = 0
var postflopCalls = 0 var postflopCalls = 0
var actions = 0
var aggressiveActions = 0
fun bbPer100(): Double = if (hands == 0) 0.0 else (net.toDouble() / BIG_BLIND) / hands * 100.0 fun bbPer100(): Double = if (hands == 0) 0.0 else (net.toDouble() / BIG_BLIND) / hands * 100.0
fun pct(n: Int): Double = if (hands == 0) 0.0 else n * 100.0 / hands fun pct(n: Int): Double = if (hands == 0) 0.0 else n * 100.0 / hands
fun aggressionFactor(): Double = fun aggressionFactor(): Double =
if (postflopCalls == 0) postflopBets.toDouble() else postflopBets.toDouble() / postflopCalls if (postflopCalls == 0) postflopBets.toDouble() else postflopBets.toDouble() / postflopCalls
fun aggressiveActionShare(): Double =
if (actions == 0) 0.0 else aggressiveActions.toDouble() / actions
} }
private fun runTable(label: String, roster: List<BotProfile>, hands: Int, seed: Long): List<Stats> = runBlocking { private fun runTable(
label: String,
roster: List<BotProfile>,
hands: Int,
seed: Long,
verbose: Boolean = true,
): List<Stats> = runBlocking {
// The deck gets its own RNG. If bots drew from the same stream, the number of // The deck gets its own RNG. If bots drew from the same stream, the number of
// Monte Carlo rollouts a bot performs — which varies by skill level — would // Monte Carlo rollouts a bot performs — which varies by skill level — would
// shift every subsequent deal, so changing a profile would silently change the // shift every subsequent deal, so changing a profile would silently change the
@@ -45,8 +57,10 @@ private fun runTable(label: String, roster: List<BotProfile>, hands: Int, seed:
val seats = roster.mapIndexed { i, p -> Seat(i, p.name, STARTING_STACK, bots[i]) } val seats = roster.mapIndexed { i, p -> Seat(i, p.name, STARTING_STACK, bots[i]) }
val table = Table(seats, SMALL_BLIND, BIG_BLIND, deckRandom) val table = Table(seats, SMALL_BLIND, BIG_BLIND, deckRandom)
println("\n=== $label ===") if (verbose) {
println("$hands hands, ${roster.size}-handed, ${STARTING_STACK / BIG_BLIND}bb stacks, seed=$seed") println("\n=== $label ===")
println("$hands hands, ${roster.size}-handed, ${STARTING_STACK / BIG_BLIND}bb stacks, seed=$seed")
}
val started = System.nanoTime() val started = System.nanoTime()
// VPIP/PFR are per-hand booleans, not action counts: a player who calls and // VPIP/PFR are per-hand booleans, not action counts: a player who calls and
@@ -64,6 +78,10 @@ private fun runTable(label: String, roster: List<BotProfile>, hands: Int, seed:
for (e in result.events) { for (e in result.events) {
val st = stats[e.seat] val st = stats[e.seat]
st.actions++
if (e.action.type == ActionType.BET || e.action.type == ActionType.RAISE) {
st.aggressiveActions++
}
if (e.street == Street.PREFLOP) { if (e.street == Street.PREFLOP) {
when (e.action.type) { when (e.action.type) {
ActionType.CALL -> enteredPot[e.seat] = true ActionType.CALL -> enteredPot[e.seat] = true
@@ -97,20 +115,272 @@ private fun runTable(label: String, roster: List<BotProfile>, hands: Int, seed:
} }
val elapsed = (System.nanoTime() - started) / 1_000_000.0 val elapsed = (System.nanoTime() - started) / 1_000_000.0
println("%.1f ms (%.0f hands/sec)\n".format(elapsed, hands / (elapsed / 1000.0))) if (verbose) {
println("%.1f ms (%.0f hands/sec)\n".format(elapsed, hands / (elapsed / 1000.0)))
println("%-7s %-28s %9s %7s %7s %6s %7s".format("Player", "Profile", "bb/100", "VPIP%", "PFR%", "AF", "Won%")) println("%-7s %-28s %9s %7s %7s %6s %7s".format("Player", "Profile", "bb/100", "VPIP%", "PFR%", "AF", "Won%"))
println("-".repeat(78)) println("-".repeat(78))
for (s in stats.sortedByDescending { it.bbPer100() }) { for (s in stats.sortedByDescending { it.bbPer100() }) {
println(
"%-7s %-28s %9.2f %7.1f %7.1f %6.2f %7.1f".format(
s.name, s.profile.description, s.bbPer100(),
s.pct(s.vpip), s.pct(s.pfr), s.aggressionFactor(), s.pct(s.wins),
)
)
}
println("chip conservation: %d (must be 0)".format(stats.sumOf { it.net }))
}
check(stats.sumOf { it.net } == 0L) { "$label seed=$seed did not conserve chips" }
stats
}
internal data class LadderSeedResult(
val seed: Long,
val bbPer100: Map<SkillLevel, Double>,
)
internal data class SeparationResult(
val label: String,
val meanDifference: Double,
val lower95: Double,
) {
val passes: Boolean get() = lower95 > 0.0
}
internal data class CalibrationReport(
val seeds: List<LadderSeedResult>,
val separations: List<SeparationResult>,
) {
val passes: Boolean get() = separations.all { it.passes }
}
internal data class StyleMetrics(
val label: String,
val loosenessSetting: Double,
val vpip: Double,
val pfr: Double,
val aggressionFactor: Double,
val aggressiveActionShare: Double,
)
internal data class ContractResult(val label: String, val passes: Boolean, val detail: String)
internal fun evaluateStyleCalibration(metrics: List<StyleMetrics>): List<ContractResult> {
val byName = metrics.associateBy { it.label }
fun style(style: PlayStyle): StyleMetrics =
requireNotNull(byName[style.label]) { "missing ${style.label} style metrics" }
val rock = style(PlayStyle.ROCK)
val maniac = style(PlayStyle.MANIAC)
val station = style(PlayStyle.CALLING_STATION)
val lag = style(PlayStyle.LOOSE_AGGRESSIVE)
val tag = style(PlayStyle.TIGHT_AGGRESSIVE)
val expectedOrder = metrics.sortedBy { it.loosenessSetting }.map { it.label }
val actualOrder = metrics.sortedBy { it.vpip }.map { it.label }
return listOf(
ContractResult(
"Rock remains tight",
rock.vpip in 7.0..20.0,
"VPIP ${"%.1f".format(rock.vpip)}% for ${"%.1f".format(rock.loosenessSetting * 100)}% setting",
),
ContractResult(
"Maniac remains loose",
maniac.vpip in 55.0..90.0,
"VPIP ${"%.1f".format(maniac.vpip)}% for ${"%.1f".format(maniac.loosenessSetting * 100)}% setting",
),
ContractResult(
"Styles preserve looseness order",
expectedOrder == actualOrder,
"expected $expectedOrder, observed $actualOrder",
),
ContractResult(
"Calling Station stays passive",
station.aggressionFactor < 0.8,
"AF ${"%.2f".format(station.aggressionFactor)}",
),
ContractResult(
"Calling Station rarely raises pre-flop",
station.pfr < 10.0,
"PFR ${"%.1f".format(station.pfr)}%",
),
ContractResult(
"Maniac stays aggressive",
maniac.aggressionFactor > 1.5,
"AF ${"%.2f".format(maniac.aggressionFactor)}",
),
ContractResult(
"LAG raises more often than Calling Station",
lag.pfr > station.pfr,
"LAG PFR ${"%.1f".format(lag.pfr)}%, station ${"%.1f".format(station.pfr)}%",
),
ContractResult(
"Opponent-model prior matches neutral TAG",
abs(tag.aggressiveActionShare - OpponentModel.NEUTRAL_AGGRESSION) <= 0.05,
"observed ${"%.3f".format(tag.aggressiveActionShare)}, prior ${OpponentModel.NEUTRAL_AGGRESSION}",
),
)
}
/**
* Evaluates the skill contract without pretending Advanced and Expert have a
* reliable order. Both top tiers must beat Intermediate, and Intermediate must
* beat Beginner. Each comparison is paired by seed so card/run variance cancels
* as much as this simulator permits.
*/
internal fun evaluateCalibration(results: List<LadderSeedResult>): CalibrationReport {
require(results.size >= 2) { "calibration needs at least two independent seeds" }
fun separation(label: String, stronger: SkillLevel, weaker: SkillLevel): SeparationResult {
val differences = results.map { result ->
val high = requireNotNull(result.bbPer100[stronger]) { "$stronger missing for seed ${result.seed}" }
val low = requireNotNull(result.bbPer100[weaker]) { "$weaker missing for seed ${result.seed}" }
high - low
}
val mean = differences.average()
val variance = differences.sumOf { (it - mean) * (it - mean) } / (differences.size - 1)
val standardError = sqrt(variance / differences.size)
val lower95 = mean - studentTCritical95(differences.size - 1) * standardError
return SeparationResult(label, mean, lower95)
}
return CalibrationReport(
seeds = results,
separations = listOf(
separation("Expert > Intermediate", SkillLevel.EXPERT, SkillLevel.INTERMEDIATE),
separation("Advanced > Intermediate", SkillLevel.ADVANCED, SkillLevel.INTERMEDIATE),
separation("Intermediate > Beginner", SkillLevel.INTERMEDIATE, SkillLevel.BEGINNER),
),
)
}
/** Two-sided 95% Student-t critical value, used for a one-sided conservative lower bound. */
private fun studentTCritical95(degreesOfFreedom: Int): Double = when (degreesOfFreedom) {
1 -> 12.706
2 -> 4.303
3 -> 3.182
4 -> 2.776
5 -> 2.571
6 -> 2.447
7 -> 2.365
8 -> 2.306
9 -> 2.262
10 -> 2.228
in 11..14 -> 2.201
in 15..19 -> 2.120
in 20..29 -> 2.086
else -> 1.960
}
private fun skillCandidateRoster(candidate: SkillLevel): List<BotProfile> = listOf(
BotProfile("Target", candidate, PlayStyle.TIGHT_AGGRESSIVE),
BotProfile("ControlTAG", SkillLevel.INTERMEDIATE, PlayStyle.TIGHT_AGGRESSIVE),
BotProfile("ControlLAG", SkillLevel.INTERMEDIATE, PlayStyle.LOOSE_AGGRESSIVE),
BotProfile("ControlStation", SkillLevel.INTERMEDIATE, PlayStyle.CALLING_STATION),
)
private fun styleRoster(target: PlayStyle): List<BotProfile> = buildList {
add(BotProfile("Target", SkillLevel.INTERMEDIATE, target))
repeat(5) { index ->
add(BotProfile("Control${index + 1}", SkillLevel.INTERMEDIATE, PlayStyle.TIGHT_AGGRESSIVE))
}
}
private fun runStyleCalibration(hands: Int, seed: Long): List<ContractResult> {
// One target at the same seat against the same neutral roster and deck seed.
// Putting all styles in one table changes each player's matchup, so a style
// comparison would be confounded by who happened to sit around it.
val metrics = PlayStyle.ALL.map { style ->
val target = runTable(
label = "Style calibration: ${style.label}",
roster = styleRoster(style),
hands = hands,
seed = seed,
verbose = false,
).first()
StyleMetrics(
label = style.label,
loosenessSetting = style.looseness,
vpip = target.pct(target.vpip),
pfr = target.pct(target.pfr),
aggressionFactor = target.aggressionFactor(),
aggressiveActionShare = target.aggressiveActionShare(),
)
}
println("\nStyle behavior (${hands} hands, seed=$seed):")
for (metric in metrics.sortedBy { it.loosenessSetting }) {
println( println(
"%-7s %-28s %9.2f %7.1f %7.1f %6.2f %7.1f".format( " %-18s VPIP=%5.1f PFR=%5.1f AF=%4.2f AggShare=%5.3f".format(
s.name, s.profile.description, s.bbPer100(), metric.label,
s.pct(s.vpip), s.pct(s.pfr), s.aggressionFactor(), s.pct(s.wins), metric.vpip,
metric.pfr,
metric.aggressionFactor,
metric.aggressiveActionShare,
) )
) )
} }
println("chip conservation: %d (must be 0)".format(stats.sumOf { it.net })) return evaluateStyleCalibration(metrics)
stats }
private fun runCalibration(handsPerSeed: Int, seedCount: Int, baseSeed: Long) {
require(handsPerSeed >= 50_000) { "calibration requires at least 50,000 hands per seed" }
require(seedCount >= 2) { "calibration requires at least two seeds" }
println("=== Skill calibration ===")
println("$seedCount seeds × $handsPerSeed hands; top pair unordered by contract")
val results = (0 until seedCount).map { offset ->
val seed = baseSeed + offset
// Each candidate occupies the same seat against the same opponents and
// deal seed. Putting all candidates in one table makes every score depend
// on the other candidates' strengths and non-transitive matchups.
val rates = SkillLevel.entries.associateWith { candidate ->
runTable(
label = "Skill candidate: ${candidate.label}",
roster = skillCandidateRoster(candidate),
hands = handsPerSeed,
seed = seed,
verbose = false,
).first().bbPer100()
}
println(
"seed=$seed E=%7.2f A=%7.2f I=%7.2f B=%7.2f".format(
rates.getValue(SkillLevel.EXPERT),
rates.getValue(SkillLevel.ADVANCED),
rates.getValue(SkillLevel.INTERMEDIATE),
rates.getValue(SkillLevel.BEGINNER),
)
)
LadderSeedResult(seed, rates)
}
val report = evaluateCalibration(results)
println("\nPaired 95% lower bounds:")
for (result in report.separations) {
println(
" %-26s mean=%7.2f lower95=%7.2f %s".format(
result.label,
result.meanDifference,
result.lower95,
if (result.passes) "PASS" else "FAIL",
)
)
}
val styleContracts = runStyleCalibration(
hands = maxOf(20_000, handsPerSeed / 5),
seed = baseSeed + seedCount,
)
println("\nStyle contracts:")
for (contract in styleContracts) {
println(" %-42s %s %s".format(contract.label, if (contract.passes) "PASS" else "FAIL", contract.detail))
}
check(report.passes && styleContracts.all { it.passes }) {
"bot calibration failed; do not tune constants against a single seed or one aggregate number"
}
} }
/** Dumps the starting-hand ranking so the ordering can be eyeballed against a real chart. */ /** Dumps the starting-hand ranking so the ordering can be eyeballed against a real chart. */
@@ -137,6 +407,24 @@ private fun printChart() {
fun main(args: Array<String>) { fun main(args: Array<String>) {
if (args.firstOrNull() == "chart") { printChart(); return } if (args.firstOrNull() == "chart") { printChart(); return }
if (args.firstOrNull() == "styles") {
val hands = args.getOrNull(1)?.toIntOrNull() ?: 20_000
val seed = args.getOrNull(2)?.toLongOrNull() ?: 20_260_732L
val contracts = runStyleCalibration(hands, seed)
println("\nStyle contracts:")
for (contract in contracts) {
println(" %-42s %s %s".format(contract.label, if (contract.passes) "PASS" else "FAIL", contract.detail))
}
check(contracts.all { it.passes }) { "style calibration failed" }
return
}
if (args.firstOrNull() == "calibrate") {
val hands = args.getOrNull(1)?.toIntOrNull() ?: 100_000
val seeds = args.getOrNull(2)?.toIntOrNull() ?: 4
val baseSeed = args.getOrNull(3)?.toLongOrNull() ?: 20_260_724L
runCalibration(hands, seeds, baseSeed)
return
}
val hands = args.getOrNull(0)?.toIntOrNull() ?: 50_000 val hands = args.getOrNull(0)?.toIntOrNull() ?: 50_000
val seed = args.getOrNull(1)?.toLongOrNull() ?: 20_260_724L val seed = args.getOrNull(1)?.toLongOrNull() ?: 20_260_724L
@@ -152,15 +440,16 @@ fun main(args: Array<String>) {
), hands, seed ), hands, seed
) )
// Controlled: identical style, only skill varies. This is the experiment that // Quick diagnostic only. The enforced calibration evaluates each candidate
// actually tests whether the difficulty axis produces a real skill gradient. // separately against a fixed pool; this mixed ladder is intentionally not
// accepted as evidence because matchups are non-transitive.
val ladder = runTable( val ladder = runTable(
"Skill ladder (style held constant at Tight-Aggressive)", listOf( "Skill ladder (style held constant at Tight-Aggressive)",
BotProfile("Expert", SkillLevel.EXPERT, PlayStyle.TIGHT_AGGRESSIVE), SkillLevel.entries.reversed().map { level ->
BotProfile("Advncd", SkillLevel.ADVANCED, PlayStyle.TIGHT_AGGRESSIVE), BotProfile(level.label, level, PlayStyle.TIGHT_AGGRESSIVE)
BotProfile("Interm", SkillLevel.INTERMEDIATE, PlayStyle.TIGHT_AGGRESSIVE), },
BotProfile("Begin", SkillLevel.BEGINNER, PlayStyle.TIGHT_AGGRESSIVE), hands,
), hands, seed + 1 seed + 1,
) )
println("\nDifficulty gradient:") println("\nDifficulty gradient:")
@@ -172,19 +461,9 @@ fun main(args: Array<String>) {
var strictlyMonotonic = true var strictlyMonotonic = true
for (i in 1 until byLevel.size) if (byLevel[i].second > byLevel[i - 1].second) strictlyMonotonic = false for (i in 1 until byLevel.size) if (byLevel[i].second > byLevel[i - 1].second) strictlyMonotonic = false
// Adjacent tiers can sit within seed-to-seed noise of each other, so the
// invariant actually worth asserting is separation across a two-step gap.
// Claiming strict adjacent ordering from a single seed would be reading noise
// as signal — see the gap check below for what is genuinely verified.
var twoStepOk = true
for (i in 2 until byLevel.size) if (byLevel[i].second >= byLevel[i - 2].second) twoStepOk = false
println( println(
if (strictlyMonotonic) " -> strictly monotonic this run." if (strictlyMonotonic) " -> strictly monotonic this run."
else " -> adjacent tiers overlap this run (expected; they are close by design)." else " -> adjacent tiers overlap this run (expected; they are close by design)."
) )
println( println(" -> diagnostic only; run `:sim:run --args=\"calibrate\"` for an enforced multi-seed result.")
if (twoStepOk) " -> PASS: every level beats the one two tiers below it."
else " -> FAIL: the skill axis is not separating levels at all."
)
} }
@@ -0,0 +1,102 @@
package com.jsjdesigns.poker.sim
import com.jsjdesigns.poker.bot.SkillLevel
import kotlin.test.Test
import kotlin.test.assertFalse
import kotlin.test.assertTrue
class CalibrationPolicyTest {
private fun seed(
id: Long,
expert: Double,
advanced: Double,
intermediate: Double,
beginner: Double,
) = LadderSeedResult(
seed = id,
bbPer100 = mapOf(
SkillLevel.EXPERT to expert,
SkillLevel.ADVANCED to advanced,
SkillLevel.INTERMEDIATE to intermediate,
SkillLevel.BEGINNER to beginner,
),
)
@Test
fun `advanced and expert may overlap while both clear intermediate`() {
val report = evaluateCalibration(
listOf(
seed(1, expert = 18.0, advanced = 22.0, intermediate = -5.0, beginner = -30.0),
seed(2, expert = 24.0, advanced = 17.0, intermediate = -8.0, beginner = -27.0),
seed(3, expert = 16.0, advanced = 25.0, intermediate = -10.0, beginner = -35.0),
seed(4, expert = 21.0, advanced = 19.0, intermediate = -6.0, beginner = -31.0),
)
)
assertTrue(report.passes)
}
@Test
fun `intermediate beating advanced is not hidden by a two-tier comparison`() {
val report = evaluateCalibration(
listOf(
seed(1, expert = 30.0, advanced = 2.0, intermediate = 8.0, beginner = -30.0),
seed(2, expert = 28.0, advanced = 3.0, intermediate = 9.0, beginner = -28.0),
seed(3, expert = 32.0, advanced = 1.0, intermediate = 7.0, beginner = -32.0),
seed(4, expert = 29.0, advanced = 2.0, intermediate = 8.0, beginner = -29.0),
)
)
assertFalse(report.passes)
assertFalse(report.separations.first { it.label == "Advanced > Intermediate" }.passes)
}
@Test
fun `beginner beating intermediate fails calibration`() {
val report = evaluateCalibration(
listOf(
seed(1, expert = 30.0, advanced = 28.0, intermediate = -8.0, beginner = 2.0),
seed(2, expert = 31.0, advanced = 27.0, intermediate = -7.0, beginner = 3.0),
seed(3, expert = 29.0, advanced = 26.0, intermediate = -9.0, beginner = 1.0),
seed(4, expert = 32.0, advanced = 29.0, intermediate = -6.0, beginner = 4.0),
)
)
assertFalse(report.passes)
assertFalse(report.separations.first { it.label == "Intermediate > Beginner" }.passes)
}
@Test
fun `style contracts reject a misnamed rock and passive maniac`() {
val normal = listOf(
StyleMetrics("Rock", 0.12, vpip = 12.0, pfr = 4.0, aggressionFactor = 0.4, aggressiveActionShare = 0.18),
StyleMetrics("Tight-Aggressive", 0.22, vpip = 20.0, pfr = 12.0, aggressionFactor = 2.0, aggressiveActionShare = 0.22),
StyleMetrics("Trapper", 0.28, vpip = 26.0, pfr = 9.0, aggressionFactor = 0.7, aggressiveActionShare = 0.20),
StyleMetrics("Loose-Aggressive", 0.45, vpip = 42.0, pfr = 28.0, aggressionFactor = 2.8, aggressiveActionShare = 0.39),
StyleMetrics("Calling Station", 0.62, vpip = 58.0, pfr = 8.0, aggressionFactor = 0.3, aggressiveActionShare = 0.12),
StyleMetrics("Maniac", 0.75, vpip = 70.0, pfr = 45.0, aggressionFactor = 3.5, aggressiveActionShare = 0.46),
)
assertTrue(evaluateStyleCalibration(normal).all { it.passes })
val broken = normal.map {
when (it.label) {
"Rock" -> it.copy(vpip = 31.0)
"Maniac" -> it.copy(aggressionFactor = 0.5)
else -> it
}
}
val results = evaluateStyleCalibration(broken)
assertFalse(results.first { it.label == "Rock remains tight" }.passes)
assertFalse(results.first { it.label == "Maniac stays aggressive" }.passes)
val overAggressiveStation = normal.map {
if (it.label == "Calling Station") it.copy(pfr = 19.0) else it
}
assertFalse(
evaluateStyleCalibration(overAggressiveStation)
.first { it.label == "Calling Station rarely raises pre-flop" }
.passes
)
}
}