diff --git a/reports/draft-report.html b/reports/draft-report.html index fe715bf..ef69ed7 100644 --- a/reports/draft-report.html +++ b/reports/draft-report.html @@ -10,7 +10,7 @@ - + Early vs delayed PVI for claudication: known-truth simulation, draft results @@ -1482,7 +1482,7 @@ border-radius: 0px; known-truth simulation, draft results

Draft scale: n = 50,000 patients, 100 replicates per world x scenario

-

2026-10-28

+

2026-10-30

@@ -1497,10 +1497,10 @@ physician variants, 3 negative controls, an oracle run, and 2 simulated RCTs. 2,400 replicate files, 0 errors. 16 of 288,000 estimates are missing (sequential trials with physician fixed effects, 4 replicates), and these are dropped. -
  • Outcome shown. The headline outcome is the -disease composite (CLTI or major amputation), whose -true effect under the null is exactly 0 (D34). MALE is shown where it -adds something.
    • +
  • Outcome shown. The headline outcome is limb +MALE (CLTI, major amputation or open bypass; D56), whose true +effect under the null is exactly 0 (D34). MALE is shown where it adds +something.
  • Two errors, kept separate (D27). Estimation bias = mean estimate − the method’s own truth. Question gap = the method’s own truth − the reference truth (CCW A vs B1). @@ -1520,14 +1520,14 @@ so treat those parts as provisional.
    • percentage points, world W0. - - + + - - + + @@ -1549,10 +1549,10 @@ percentage points, world W0. 0.0 0.0 0.0 --0.3 --0.33 (0.26) -2.6 -85% +-0.4 +-0.35 (0.27) +2.7 +91% null @@ -1560,219 +1560,252 @@ percentage points, world W0. 0.0 0.0 0.0 --0.3 --0.33 (0.26) -2.6 -85% - - -null -Sequential trials, ITT -0.0 -0.0 -0.0 --0.0 --0.02 (0.10) -1.0 -94% - - -null -Landmark at 6 months -0.0 --0.3 --0.3 --0.2 -0.14 (0.06) -1.0 -97% - - -null -Naive ‘ever early PVI’ -0.0 -1.3 -1.3 -1.1 --0.20 (0.06) -1.2 -95% - - -null -Realistic RCT (n = 240), ITT -0.0 -0.0 -0.0 --0.1 --0.07 (0.39) -3.9 -94% - - -null -Ideal pragmatic RCT, ITT -0.0 -0.0 -0.0 --0.1 --0.05 (0.03) -0.3 -95% - - -harm -CCW, A vs B1 (reference) -5.1 -5.1 -0.0 -5.3 -0.17 (0.28) -2.8 -96% - - -harm -CCW, A vs B2 -5.1 -5.2 -0.0 -5.3 -0.17 (0.28) -2.8 -96% - - -harm -Sequential trials, ITT -5.1 -5.7 -0.6 -5.7 --0.01 (0.15) -1.5 +-0.4 +-0.35 (0.27) +2.7 91% +null +Sequential trials, ITT +0.0 +0.0 +0.0 +0.1 +0.05 (0.12) +1.2 +94% + + +null +Landmark at 6 months +0.0 +-0.2 +-0.2 +-0.0 +0.17 (0.06) +1.2 +95% + + +null +Landmark, + month-5 severity (suppl.) +0.0 +-0.3 +-0.3 +-0.1 +0.18 (0.06) +1.2 +95% + + +null +Naive ‘ever early PVI’ +0.0 +1.2 +1.2 +1.0 +-0.15 (0.07) +1.3 +91% + + +null +Realistic RCT (n = 240), ITT +0.0 +0.0 +0.0 +-0.2 +-0.25 (0.37) +3.7 +98% + + +null +Ideal pragmatic RCT, ITT +0.0 +0.0 +0.0 +-0.1 +-0.06 (0.03) +0.3 +95% + + +harm +CCW, A vs B1 (reference) +6.2 +6.2 +0.0 +6.4 +0.21 (0.30) +3.0 +94% + + +harm +CCW, A vs B2 +6.2 +6.2 +0.1 +6.4 +0.21 (0.30) +3.0 +94% + + +harm +Sequential trials, ITT +6.2 +6.9 +0.7 +6.9 +-0.01 (0.15) +1.5 +95% + + harm Landmark at 6 months -5.1 -7.9 -2.8 -7.8 --0.10 (0.16) -3.6 -92% +6.2 +7.1 +0.9 +7.1 +0.02 (0.07) +1.6 +93% + + +harm +Landmark, + month-5 severity (suppl.) +6.2 +9.4 +3.3 +9.4 +-0.06 (0.17) +4.1 +91% harm Naive ‘ever early PVI’ -5.1 -7.2 +6.2 +8.2 2.1 -6.9 --0.29 (0.06) +8.0 +-0.26 (0.07) 1.6 -88% +92% harm Realistic RCT (n = 240), ITT -5.1 -5.3 -0.2 -5.5 -0.21 (0.43) -4.3 -97% +6.2 +6.6 +0.4 +7.3 +0.73 (0.52) +5.2 +95% harm Ideal pragmatic RCT, ITT -5.1 -5.2 -0.0 -5.2 -0.04 (0.03) +6.2 +6.3 +0.1 +6.3 +0.03 (0.03) 0.3 -95% +96% benefit_harm CCW, A vs B1 (reference) -3.1 -3.1 +3.6 +3.6 0.0 -3.3 -0.28 (0.29) -2.9 -89% +3.9 +0.33 (0.31) +3.1 +91% benefit_harm CCW, A vs B2 -3.1 -3.0 +3.6 +3.6 -0.0 -3.3 -0.28 (0.29) -2.9 -89% +3.9 +0.33 (0.31) +3.1 +91% benefit_harm Sequential trials, ITT -3.1 -4.0 -0.9 -4.2 -0.26 (0.11) +3.6 +4.7 1.1 -99% +5.0 +0.32 (0.12) +1.2 +98% benefit_harm Landmark at 6 months -3.1 +3.6 +5.2 +1.6 5.3 -2.3 -5.4 -0.03 (0.19) -3.5 -95% +0.13 (0.07) +1.2 +98% benefit_harm -Naive ‘ever early PVI’ -3.1 -5.7 +Landmark, + month-5 severity (suppl.) +3.6 +6.2 2.6 -5.4 --0.26 (0.07) -1.2 -97% +6.4 +0.21 (0.21) +4.0 +94% benefit_harm -Realistic RCT (n = 240), ITT -3.1 -3.9 -0.8 -3.8 --0.08 (0.43) -4.3 -94% +Naive ‘ever early PVI’ +3.6 +6.3 +2.7 +6.1 +-0.19 (0.08) +1.3 +97% benefit_harm +Realistic RCT (n = 240), ITT +3.6 +4.8 +1.2 +4.4 +-0.36 (0.43) +4.3 +94% + + +benefit_harm Ideal pragmatic RCT, ITT -3.1 -3.9 -0.8 -3.9 -0.01 (0.03) +3.6 +4.6 +1.0 +4.6 +-0.02 (0.03) 0.3 -92% +95% @@ -1794,25 +1827,26 @@ Under the null its 95% CI excludes 0 in 15% of replicates (Section ITT, but its SE is about 4.2 pp. It detects the 5-point harm in only 25% of replicates. -

    -

    +

    +

    2. Aim 2: surgery-specific threats

    - +
    +truth, percentage points. Harm scenario, limb MALE.+---++++-++-- @@ -1825,6 +1859,7 @@ truth, percentage points. Harm scenario, disease composite. + @@ -1833,162 +1868,188 @@ truth, percentage points. Harm scenario, disease composite. - - + + - - - + + + + - - - - - - - - - - - - - + + + - - - - - + + + + + + + + + + + + + + + + + - - + + - + - - - + + + + - - + + + - - - - + + + + - - - - - - - - - - - - - - - - - - - - - - - + - - - - + + + + + + + + + + - - + + + + + + + + + + + + + + + + + + + + + - - - - - - - - + + + + + + + + + - + - - - - - - - - + + + + + + + + + - + + + + + + - - - - - - - + + + + + + + + + + + + + + + + - - - - + + + + - - + + +
    Table 3. Estimation bias (MCSE) against each method’s own -truth, percentage points. Harm scenario, disease composite.
    W3 Style -> outcome W4 Deferral W5 Unmeasured severityWE EHR/registry-like WR Realistic
    CCW, A vs B1 (reference) Ignore physician (cluster SE) 0.2 (0.3)1.1 (0.3)0.2 (0.3)1.2 (0.3)0.3 (0.4) 0.2 (0.3) 0.7 (0.3)-0.0 (0.3)4.4 (0.2)7.1 (0.4)0.1 (0.3)5.5 (0.3)1.6 (0.3)7.9 (0.4)
    CCW, A vs B1 (reference) Random intercept0.2 (0.3)0.9 (0.3)0.4 (0.3)0.1 (0.4)0.1 (0.4)0.0 (0.3)4.4 (0.3)6.4 (0.4)
    CCW, A vs B1 (reference)Within-physician (FE)0.1 (0.4)NA1.0 (0.3)0.3 (0.4) 0.3 (0.4) 0.2 (0.4)-0.2 (0.5)-0.3 (0.4)-0.1 (0.3)4.4 (0.3)6.1 (0.4)NA5.5 (0.3)1.6 (0.3)7.3 (0.4)
    CCW, A vs B1 (reference)Within-physician (FE)0.4 (0.4)0.4 (0.4)0.1 (0.4)0.0 (0.5)-0.3 (0.5)0.2 (0.4)5.6 (0.3)1.6 (0.3)7.0 (0.4)
    Sequential trials, ITT Ignore physician (cluster SE)-0.0 (0.1)1.2 (0.1)-0.0 (0.2)1.3 (0.2) 0.1 (0.1)0.2 (0.1)0.3 (0.1) 1.4 (0.1)-0.1 (0.1)4.9 (0.1)9.3 (0.2)-0.0 (0.1)6.0 (0.1)1.7 (0.1)10.2 (0.2)
    Sequential trials, ITT Random intercept-0.0 (0.2)0.9 (0.1)NA1.0 (0.2) 0.1 (0.1)0.1 (0.2) 0.0 (0.2)0.2 (0.2)-0.2 (0.1)5.0 (0.1)7.4 (0.2)NA6.0 (0.1)1.7 (0.1)8.3 (0.2)
    Sequential trials, ITT Within-physician (FE) -0.3 (0.2)0.2 (0.2)-0.1 (0.2)-0.3 (0.2)-0.5 (0.2)-0.5 (0.1)4.9 (0.1)6.7 (0.2)
    Landmark at 6 monthsIgnore physician (cluster SE)-0.1 (0.2)3.9 (0.4)1.1 (0.3)0.1 (0.8)1.5 (0.7)-0.0 (0.2)2.5 (0.4)1.2 (1.1)
    Landmark at 6 monthsRandom intercept0.3 (0.2) -0.2 (0.2)2.7 (0.3)0.8 (0.3)-0.3 (0.7)0.3 (0.5)-0.3 (0.2)-0.7 (0.2)-0.5 (0.2)6.0 (0.2)1.6 (0.2)7.6 (0.2)
    Landmark at 6 monthsIgnore physician (cluster SE) 0.0 (0.1)2.6 (0.4)-0.5 (1.1)0.9 (0.1)-0.1 (0.1)1.0 (0.4)2.0 (0.4)0.1 (0.1)7.4 (0.1)2.9 (0.1)11.4 (0.5)
    Landmark at 6 monthsRandom interceptNA0.8 (0.1)0.1 (0.1)0.7 (0.4)0.7 (0.4)NA7.5 (0.1)3.0 (0.1)9.8 (0.5)
    Landmark at 6 months Within-physician (FE)-0.7 (0.3)1.5 (0.5)0.8 (0.5)-1.1 (0.7)-1.0 (0.6)-0.4 (0.3)2.5 (0.4)-1.3 (1.1)-0.2 (0.1)0.1 (0.1)-0.2 (0.1)-0.0 (0.4)-0.4 (0.4)-0.3 (0.1)7.5 (0.1)2.9 (0.1)9.2 (0.5)
    Naive ‘ever early PVI’Landmark, + month-5 severity (suppl.) Ignore physician (cluster SE)-0.3 (0.1)0.5 (0.1)-0.3 (0.1)-10.5 (1.0)-6.4 (0.9)-0.2 (0.1)5.5 (0.1)1.5 (1.0)-0.1 (0.2)3.7 (0.4)1.0 (0.3)-0.3 (0.8)2.0 (0.7)0.0 (0.2)3.2 (0.4)-0.3 (0.5)2.1 (1.0)
    Naive ‘ever early PVI’Random interceptIgnore physician (cluster SE)-0.3 (0.1)0.6 (0.1)-0.3 (0.1)-9.8 (1.0)-5.5 (0.9) -0.2 (0.1)0.4 (0.1)-0.2 (0.1)-9.0 (0.9)-6.1 (0.9)-0.1 (0.1)5.6 (0.1)0.6 (1.0)6.7 (0.1)2.3 (0.1)3.2 (0.9)
    Naive ‘ever early PVI’Random interceptNA0.4 (0.1)-0.1 (0.1)-8.5 (0.9)-5.5 (0.8)NA6.8 (0.1)2.4 (0.1)2.2 (0.9)
    Naive ‘ever early PVI’ Within-physician (FE)-0.9 (0.1)-0.7 (0.1) -0.8 (0.1)-11.5 (1.0)-8.9 (0.9)-0.6 (0.2)-0.8 (0.1)-11.0 (1.0)-8.3 (0.9) -1.0 (0.1)5.3 (0.1)-1.0 (1.0)6.6 (0.1)2.0 (0.2)0.6 (1.0)
    +B1), percentage points. Harm scenario, limb MALE.+----+++++---++ @@ -2001,6 +2062,7 @@ B1), percentage points. Harm scenario, disease composite. + @@ -2009,145 +2071,170 @@ B1), percentage points. Harm scenario, disease composite. - - + + - - - + + + + - - - - - - - - - - - - - + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + - + - - + + + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + + + + + + + + + + + + + + + + + + + - - - - - - + - - + + + + + + + + + + + + + + + + + + + + + - - + + - + - - + + +
    Table 3b. Total error against the reference truth (CCW A vs -B1), percentage points. Harm scenario, disease composite.
    W3 Style -> outcome W4 Deferral W5 Unmeasured severityWE EHR/registry-like WR Realistic
    CCW, A vs B1 (reference) Ignore physician (cluster SE) 0.21.10.21.20.3 0.2 0.7-0.04.47.10.15.51.67.9
    CCW, A vs B1 (reference) Random intercept0.20.90.40.10.10.04.46.4
    CCW, A vs B1 (reference)Within-physician (FE)0.1NA1.00.3 0.3 0.2-0.2-0.3-0.14.46.1
    Sequential trials, ITTIgnore physician (cluster SE)0.62.00.90.82.00.55.510.0
    Sequential trials, ITTRandom intercept0.61.70.90.60.80.55.68.2
    Sequential trials, ITTWithin-physician (FE)0.31.00.70.30.10.15.57.4
    Landmark at 6 monthsIgnore physician (cluster SE)2.716.514.13.23.52.45.69.8
    Landmark at 6 monthsRandom intercept2.715.413.72.92.22.45.68.1
    Landmark at 6 monthsWithin-physician (FE)2.214.213.72.01.02.0NA 5.51.6 7.3
    CCW, A vs B1 (reference)Within-physician (FE)0.40.40.10.0-0.30.25.61.67.0
    Naive ‘ever early PVI’Sequential trials, ITT Ignore physician (cluster SE)1.83.10.72.21.01.1 2.12.13.20.76.72.411.1
    Sequential trials, ITTRandom interceptNA1.91.00.90.8NA6.82.49.2
    Sequential trials, ITTWithin-physician (FE)0.51.30.80.50.00.36.82.38.4
    Landmark at 6 monthsIgnore physician (cluster SE)0.92.51.31.32.30.98.63.612.6
    Landmark at 6 monthsRandom interceptNA2.41.41.11.0NA8.73.711.0
    Landmark at 6 monthsWithin-physician (FE)0.7 1.77.611.81.20.3-0.10.58.73.510.3
    Landmark, + month-5 severity (suppl.)Ignore physician (cluster SE)3.217.014.53.94.23.36.73.010.9
    Naive ‘ever early PVI’Random intercept1.93.02.23.63.5Ignore physician (cluster SE) 1.87.710.93.22.02.33.21.88.84.212.9
    Naive ‘ever early PVI’Random interceptNA3.12.23.63.3NA9.04.311.9
    Naive ‘ever early PVI’ Within-physician (FE) 1.21.91.62.01.5 1.10.70.4 0.97.49.38.83.810.4
    @@ -2177,8 +2264,8 @@ severity dominates.

    3. False alarms under the null (headline for clinicians)

    -

    -

    +

    +

    What it shows.

    4. Aim 3: do negative controls detect bias?

    - +
    -+--+++++-- @@ -2229,6 +2317,7 @@ in that world. + @@ -2237,54 +2326,58 @@ in that world. - - - - - - - - + + + + + + + + + - - - - - - - - + + + + + + + + + - - - + + + - - + + + - - - - - - - - + + + + + + + + +
    Table 4. Sensitivity / specificity of each negative control for MALE bias > 1 pp. A cell (world × scenario × method × variant) counts as biased when its mean MALE bias exceeds 1 pp; a replicate is flagged when the control’s 95% CI excludes 0. ‘–’: no biased (or no clean) cells in that world.
    W3 Style -> outcome W4 Deferral W5 Unmeasured severityWE EHR/registry-like WR Realistic All worlds
    NCO-1 (hip fracture / pneumonia) –11% / 94%19% / 89%13% / 87%13% / 89%12% / 92%10% / –12% / 88%12% / 90%8% / 91%–8% / 89%7% / 88%2% / 93%26% / –6% / 86%15% / 82%14% / 91%
    NCO-2 (vein ablation) –25% / 78%7% / 75%54% / 45%52% / 26%11% / 74%42% / –67% / 92%48% / 70%28% / 74%–71% / 46%82% / 67%6% / 67%43% / –33% / 87%65% / 55%51% / 67%
    NCE (early low-value procedure) –4% / 94%6% / 96% 5% / 95%9% / 88%–3% / 95% 7% / 94%– 6% / –9% / 93%7% / 94%6% / 92%7% / –6% / 94%
    Any of the three –37% / 72%30% / 65%63% / 36%61% / 20%25% / 65%50% / –73% / 78%57% / 61%37% / 67%–74% / 39%84% / 56%8% / 59%55% / –39% / 67%70% / 46%58% / 59%
    -

    +

    What it shows (provisional, see Q2).