diff --git a/docs/analysis-plan-v1.0.docx b/docs/analysis-plan-v1.0.docx index a20b0a4..fc2b5a5 100644 Binary files a/docs/analysis-plan-v1.0.docx and b/docs/analysis-plan-v1.0.docx differ diff --git a/docs/analysis-plan-v1.0.md b/docs/analysis-plan-v1.0.md index 9aaf2db..b15e415 100644 --- a/docs/analysis-plan-v1.0.md +++ b/docs/analysis-plan-v1.0.md @@ -152,7 +152,7 @@ W1 is run at both ρ = −0.3 and ρ = 0 (D22). WR is calibrated to the Medicare | **Scenario** | **Mechanism** | **Target** | |---|---|---| -| **Null** | See open decision 1 (§16) | Clinically relevant RD(A − B1) at 60 months = 0 | +| **Null (sharp disease null, D34)** | Δ = 0, κ = 0: PVI changes nothing about severity, CLTI, amputation, bypass or death | RD(A − B1) = 0 **exactly** for the disease composite (CLTI or major amputation) and every disease component. MALE is still > 0, because only patients who have had a PVI can have a repeat revascularization: this is the mechanical effect of how the outcome is defined, and it is reported as a finding | | **Harm** | Δ small, κ moderate | RD(A − B1) ≈ +5 percentage points | | **Benefit then harm** | Δ large, κ moderate | Cumulative incidence curves cross: A lower before about month 18, higher at 60 | @@ -166,7 +166,7 @@ W1 is run at both ρ = −0.3 and ρ = 0 (D22). WR is calibrated to the Medicare | Late PVI (>6 months, ~4.4 years) | 22.5% after early PVI vs 3.6% without | Sorber 2023 | | CLTI (~4.4 years) | 16.4% vs 7.8% | Sorber 2023 | | Mean age | about 76 | Dun 2025 (40782724) | -| 5-year mortality | *to be sourced from Medicare claudication cohorts* | — | +| All-cause mortality | Interim target about 35% at 5 years (Swedish population cohort: 10-year mortality in IC 63% vs 27% reference). **Replace with the 5-year mortality of the authors' own Medicare claudication cohort** (Sorber 2023 / VESS 2026 data) | Sigvant 2018 *EJVES* (29478910); cardiovascular-only 5-year mortality 14–16%: Baubeta Fridh 2018 *BJS* (29116654) | | Amputation | about 0.4%/year after revascularization for IC | Baubeta Fridh 2017 (28797662) | Calibration solves for the intercepts and the key coefficients (α₀, α_S, σ_S, β_def, the outcome intercepts) by stochastic approximation on 10⁵ patients, as in `tte-papers/simulations/01_calibrate.R`. @@ -222,7 +222,7 @@ Truths are stored in `simulations/truths/` as small committed CSVs, and each has | V4 Within-physician | Physician fixed effects in the weight models. Physicians with no variation in early PVI are dropped, and the change to the estimand that this causes is reported. Mundlak (physician means) is the fallback if FE is too slow. Cluster-robust SE | | V5 Preference IV (supplement) | Instrument = the physician's early-PVI rate among the physician's *other* patients; two-stage residual inclusion | -**Negative controls (D32).** NCO-1 and NCO-2 replace MALE as the outcome in every method × variant. The NCE replaces early PVI as the exposure in the CCW and landmark designs. The NCE models condition on early-PVI status, because otherwise the path NCE ← style → PVI → MALE would make the NCE "detect" an effect that is real (see §16, decision 3). +**Negative controls (D32).** NCO-1 and NCO-2 replace MALE as the outcome in every method × variant. The NCE replaces early PVI as the exposure in the CCW and landmark designs. The NCE models adjust for early-PVI status (D36). Without that adjustment, the path NCE ← style → PVI → MALE would make the NCE "detect" a real effect of PVI. The unadjusted NCE analysis is in the supplement. # 8. Simulated RCTs (D18) @@ -250,7 +250,7 @@ Running replicates of the 240-patient trial also gives its power and CI width, w | Model SE | Mean of the estimated SEs | | Coverage | Coverage of θ_m by the 95% CI | | RMSE | Against θ_m and against θ* | -| **False-alarm rate** (headline) | In the null scenario, the proportion of replicates whose 95% CI excludes 0 | +| **False-alarm rate** (headline) | In the null scenario, the proportion of replicates whose 95% CI excludes 0 **for the disease composite (CLTI or major amputation)**, whose true effect is exactly 0 (D34) | MCSE is reported for every measure (Morris 2019). @@ -283,7 +283,7 @@ For each world × scenario × method × variant × replicate, record: - **Fig 1. Estimand ladder (D30).** One row per method. Each row shows the method's own truth (hollow dot), its mean estimate and 95% interval across replicates (filled dot, coloured by physician variant), and the clinically relevant truth (vertical line). Panels: W0, W1–W5, WR. Two versions: null and harm. -- **Fig 2. False-alarm rate in the null scenario.** Heat map of method × variant (rows) against world (columns). The 5% nominal level is marked. +- **Fig 2. False-alarm rate in the null scenario** (disease composite). Heat map of method × variant (rows) against world (columns). The 5% nominal level is marked. An inset shows the MALE estimate in the same null world: "harm" produced purely by how the outcome is defined. - **Fig 3. What a clean negative control is worth.** For each method, |bias| plotted for flagged and clean replicates, by world. @@ -305,7 +305,7 @@ For each world × scenario × method × variant × replicate, record: - PS matching and doubly robust estimators (Paper 1 territory). - Data-quality problems (Paper 2 territory): coding error beyond factor 3, missing data, disenrollment. -- Patient–physician sorting, where sicker patients see particular physicians. Patients are assigned to physicians at random, conditional on nothing (see §16, decision 4). +- Patient–physician sorting, where sicker patients see particular physicians. Patients are assigned to physicians at random (D37); sorting is a possible single sensitivity world. - Cost-effectiveness. # 15. Practicalities @@ -319,21 +319,14 @@ For each world × scenario × method × variant × replicate, record: | Reporting | ADEMP (Morris 2019); TARGET 2025 items for the emulated trial; code and seeds public | | Target journal | JAMA Surgery (Original Investigation). The mathematics and the full grid go in the supplement | -# 16. Decisions still open +# 16. Decisions resolved after the first draft (D34–D37) -1. **How to define "null" when MALE includes repeat revascularization.** Repeat revascularization can only happen after an index PVI. Strategy A therefore has more months "at risk" of it, so **a sharp null (PVI changes nothing about the disease) still gives RD(A − B1) > 0 for MALE**. Options: - - (a) **Balanced null.** Tune the relief Δ so that fewer CLTI events exactly offset the extra reinterventions, making RD(MALE) = 0. The components are then non-null, which is itself a lesson about composites. - - (b) **Sharp disease null.** PVI does not affect severity, CLTI, amputation, bypass or death. The headline false-alarm rate is then reported for the disease-only composite (CLTI or amputation), and MALE in this world shows the mechanical effect of the definition. - - (c) Both, as two null scenarios. That adds a fourth scenario. - *Recommendation:* (b). The false alarm is then a pure test of bias in a world where early PVI does nothing to the disease. The fact that MALE still shows "harm" there is a key teaching point about outcomes surgeons generate. +1. **Null (D34).** Sharp disease null. The headline false-alarm rate is reported for the disease composite (CLTI or major amputation). MALE in the null world shows "harm" that comes purely from the outcome definition. +2. **Strength of each threat when switched on (D35).** Use the empirical quantity where one exists (physician-style spread from the Hicks 2020 outlier distribution; deferred PVI from Sorber's late-PVI rates; early-PVI rate). Where none exists (skill, coding intensity), set the threat so that on its own it moves the naive or landmark estimate by about 2–4 percentage points in the harm scenario. +3. **NCE (D36).** The NCE analysis is adjusted for early-PVI status. The unadjusted version goes in the supplement as a warning. +4. **Patient–physician sorting (D37).** Not in the main grid. One sensitivity world, if time allows. -2. **Strength of the "on" level of each threat.** Proposed: each threat on its own moves the naive or landmark estimate by about 2–4 percentage points in the harm scenario, and is set during calibration. An alternative is to tie each one to an empirical quantity (e.g., the Hicks outlier share). *Recommendation:* use the empirical quantity where one exists (style, deferral, early-PVI rate) and the 2–4 pp rule where none does (skill, coding). - -3. **How the NCE models handle early PVI.** Physician style drives both the NCE and PVI, and PVI really affects MALE. So an NCE analysis that ignores PVI will flag even when the PVI analysis is unbiased. *Recommendation:* adjust the NCE analysis for early-PVI status, and show the unadjusted version in the supplement as a warning. - -4. **Patient–physician sorting** (sicker patients referred to particular physicians). This is realistic, and it creates between-physician patient confounding that FE would remove. *Recommendation:* leave it out of the main grid (§14), and add it as a single sensitivity world if time allows. - -5. **5-year mortality source** for calibration: a published Medicare claudication cohort is needed. +Still to do: replace the interim mortality target (§5.4) with the 5-year mortality of the authors' own Medicare cohort. --- diff --git a/docs/decisions.md b/docs/decisions.md index b3f2d38..34323ad 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -64,3 +64,13 @@ Each decision is listed with the date and the reason for it. Settled decisions a |---|---|---| | 32 | **Three negative controls.** **NCO-1:** hip fracture / pneumonia admission. Shares the latent patient frailty, severity and smoking, so it targets factor 5. **NCO-2:** varicose vein ablation by the same physician. Shares physician procedural style, so it targets factors 2–3. **NCE:** early low-value procedure (e.g. vein ablation) within 6 months, which shares both. All three are generated only from latent traits and never depend on PVI, so their true effect is exactly 0. They go through the same method × physician-variant pipeline as MALE. | Negative controls are the practical bias check available in real claims. CCW and landmark condition on survival, so the controls also need testing under each design. | | 33 | **Main text (Figure 3 / Table 3):** how accurately the negative controls detect bias. A control is "flagged" if its CI excludes 0. Bias in the MALE risk difference counts if |bias| > 1 percentage point. Report sensitivity and specificity, plus how much bias can sit behind a clean negative control. **Supplement:** each control's own false-positive rate. | Answers a new, practical question: what does a clean negative control actually tell a surgeon? | + +## Round 7: issues raised by drafting the plan (2026-10-27) + +| # | Decision | Why | +|---|---|---| +| 34 | **Null scenario = sharp disease null** (Δ = κ = 0). The headline false-alarm rate is reported for the disease composite (CLTI or major amputation), whose true effect is exactly 0. MALE in the null world is reported as "harm" that comes from the outcome definition alone. | Repeat revascularization can only happen after an index PVI, so MALE cannot be exactly null. Showing that is a teaching point. | +| 35 | **Strength of each threat:** use an empirical anchor where one exists (style spread from Hicks 2020; deferred PVI from Sorber late-PVI rates; the early-PVI rate). Where none exists (skill, coding), set the threat so that on its own it moves the naive or landmark estimate by about 2–4 pp in the harm scenario. | Credible to surgeons while keeping each threat's effect visible. | +| 36 | **The NCE analysis is adjusted for early-PVI status.** The unadjusted version goes in the supplement. | Otherwise NCE ← style → PVI → MALE flags a real effect of PVI. | +| 37 | **Patient–physician sorting is not in the main grid.** It becomes one sensitivity world if time allows. | Scope. | +| — | **Fact (not a decision):** interim calibration target for all-cause mortality is about 35% at 5 years (Sigvant 2018 *EJVES*, PMID 29478910: 10-year mortality in IC 63%). To be replaced by the 5-year mortality in the authors' own Medicare cohort. | No published 5-year all-cause figure for Medicare claudicants was found. |