v2 run (D53-D57): recomputed truths and RCT truths, re-rendered draft report; one thread per worker

Main draft grid 2,700 jobs and physician-cluster bootstrap 80 jobs, 0 errors.
Report prose in Section 1+ still quotes first-run numbers; to be rewritten after the Aim 1 review.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

docs/decisions.md

@@ -102,3 +102,20 @@ Each decision is listed with the date and the reason for it. Settled decisions a
 | 48 | **The PVI model is fitted separately inside the window (month indicators) and in the deferred period (spline in month).** | A pooled model misspecified the window probabilities that the CCW and sequential weights depend on. |
 | 49 | **V2 is the V1 fit with a physician-cluster-robust SE.** Under V4, physicians without an estimable fixed effect in a phase get probability 0, and the V4 population is restricted to physicians with variation in early PVI. | Same point estimate by construction (D15). |
 | 50 | **Validation (W0, harm, 40 replicates of 50,000):** CCW A−B1 is +5.12 vs a truth of +5.11 on the disease composite (z = 0.0); sequential ITT is +5.83 vs +5.71 (z = 0.6); MALE and NCO-1 have |z| ≤ 0.7. The model SE is 0–20% below the empirical SE for CCW, so the bootstrap check in D28 still needs doing. | Plan §15 validation (a). |
+| 51 | **Speed and scale.** The PVI model is fitted once per replicate and variant, and shared by CCW and the sequential trials. CCW clones and weights are built once for all outcomes. The deferred-period PVI model uses every PVI plus a fixed 10% sample of the other person-months, with offset log(10). On W0 validation the estimates are identical to 4 decimals. The main designs take about 39 s per replicate instead of about 130 s. **Scale:** n = 50,000 stays. Draft is 100 replicates; final is 200 replicates (replacing 500 in D29), and the final run extends the draft's replicates 1–100. Total about 12 hours on 11 cores. | User asked for a faster, smaller run. The sample size is not reduced, because CCW's strategy A already depends on about 250 patients treated in month 5 at n = 50,000. With 200 replicates, MCSE of bias ≈ empirical SE / 14 (CCW about 0.2 pp) and MCSE of coverage ≈ 1.5%. |
+
+## Draft run (2026-10-28)
+
+| # | Decision | Why |
+|---|---|---|
+| 52 | **Draft run done:** 8 worlds × 3 scenarios × 100 replicates of 50,000 patients; 9.4 h on 11 cores; 0 errors. 16 of 288,000 estimates are NA (sequential trials, V4, 4 replicates) and are dropped. Summary: `simulations/04_summarise.R` writes `results/summary/`. Report: `reports/draft-report.Rmd`. Two open questions: (Q1) the landmark propensity model adjusts for month-5 severity, a post-PVI mediator, which drives the landmark question gap (W1 own truth about 20 pp vs a reference of 7.2 pp); with baseline covariates only, the 20-replicate oracle mean is 8.1. (Q2) NCO-1 has no severity term in the DGP, contrary to D32, so it cannot detect W5/WR bias. The bootstrap check (D28) waits on Q1/Q2. | Surfaced by the draft results. |
+| 53 | **Answers to D52 (user, 2026-10-28).** (Q1) **The primary landmark propensity score uses baseline covariates only** (diagnosis covariates and month-0 severity). The version that also adjusts for month-5 severity becomes `landmark_m5` (supplement). It conditions on a mediator, because a PVI in months 0–5 has already changed that severity. Both versions get their own oracle. (Q2) **NCO-1 now depends on severity (coefficient 0.3)**, using the patient's **PVI-free severity path** (the same baseline, rate and monthly noise, with no relief and no kappa). That way NCO-1 shares the confounder but still cannot be affected by PVI (only through death as a competing risk, D46). Intercept −6.75 keeps 5-year NCO-1 incidence at about 9.3%. This is the best case for a patient-level negative control, and the paper says so. **Bootstrap (D28, reduced):** W0 and WR × null and harm, replicates 1–20 of the main run, 50 physician-cluster resamples each, V1 point estimates, about 8 s per resample. Compare the bootstrap SD with the model SE (patient and cluster) and the empirical SE. **The rerun goes to `results/main/n50k_v2`;** the first draft stays in `n50k`. Truths are recomputed (NCO-1 changed). | Landmark analyses in practice adjust for baseline covariates. D32 intended NCO-1 to share severity. |
+
+## Paper framing (2026-10-28)
+
+| # | Decision | Why |
+|---|---|---|
+| 54 | **The paper's question: "Can target trial emulation overcome the problems that make randomized trials hard in surgery?"** Early PVI for claudication is the worked example. Medicare only supplies realistic simulation inputs, and the paper makes no claims about Medicare findings or the authors' earlier papers. The structure is a scorecard: each surgical-RCT barrier (small/stopped early, short follow-up/soft endpoints, crossover, selected surgeons, practice style, unmeasured severity) against whether TTE solves it, checked against the known truth. Aims 1–3 stay. Aim 1 = "TTE is not one method", Aim 2 = the scorecard rows, Aim 3 = whether the one unfixable problem can be detected. The MALE/repeat-revascularization point is secondary, general outcome-definition guidance. The Medicare "dial" idea is dropped. **Novelty (quick PubMed check, 2026-10-28):** there are TTE guides for surgeons (e.g. JBJS Rev 2026) and immortal-time design comparisons (Duchesneau 2022 *CEBP*), but no search hit for a known-truth simulation comparing TTE with a feasible surgical RCT under surgeon-level threats. | User: the aim is to test whether TTE can solve RCT challenges in surgery, without arguing against the authors' earlier results. |
+| 55 | **Three additions from literature review 3 (user, 2026-10-28), for the overnight run.** (1) **New world WE, "EHR/registry-like":** the same DGP and calibration as W0, but the analyst sees severity with measurement error (monthly Z_t + N(0, 0.75²), correlation with true severity about 0.8), a frailty index with reliability 0.7, and true smoking. It sits between W0 (exact) and W5 (claims proxy) and answers the claims-vs-EHR row of Table 1. (2) **Non-random consent in the realistic trial:** mean 30%, OR 0.7 per 10 years of age and per SD of frailty (Wasmann 2019, King 2005). The trial's ITT truth is weighted by consent probability. (3) **Trial horizons 12, 24 and 60 months:** ITT and as-treated at each. Truths `rct_<design>_itt_h12/_h24`. The reference at the short horizons is RD(A − B1) at that month, from the risk curves. **Learning curve:** not simulated; limitation (user). | Completes Table 1 rows 3, 5 and 17. |
+| 56 | **Primary outcome changed to limb MALE: the first of coded CLTI, major amputation or open bypass** (user, 2026-10-28). D7's MALE (which also counts any revascularization after the first) moves to the supplement as "MALE including repeat revascularization". Repeat revascularization is reported on its own as a physician-driven outcome. Limb MALE also replaces the disease composite (CLTI or amputation) as the headline (D34); the disease composite stays as a secondary outcome. Negative-control bias (Aim 3) is now measured on limb MALE. Clean-world truths: 0 under the null (exactly); +6.2 under harm; −0.7 at 12 months, −1.7 at 24 and +3.6 at 60 under benefit-then-harm. Acute limb ischemia is not simulated (rare in claudication; limitation). | Standard MALE definitions (Conte 2009 *JVS* 19897335; Nordanstig 2024 *EJVES* 39121905) are built for follow-up **after** a revascularization: reintervention exists only for treated patients. D7 argued that excluding the index PVI in both arms made the outcome symmetric. It does not: under A everyone is at risk of a repeat for about 55 months, under B1 about 3.6% are. In the clean world under the null, repeat revascularization is 18.6% vs 0.7%, giving +14 pp of MALE from the definition alone. |
+| 57 | **Run-time trims** (user, 2026-10-28). One replicate is about 77 s of compute, half of it in the random-intercept variant (V3). Trims: `landmark_m5` runs under V1/V2 only (supplement); the NCE runs under V1/V2 and V4, not V3; V3 is skipped in W0 and W4, where the physician plays no role by design. V3 stays in W1, W1_rho0, W2, W3, W5, WE and WR. About 20–25% less run time. The run is started manually when the laptop is plugged in (`nohup caffeinate -i -s zsh results/run_v2.sh &`), at nice 10. | Faster, with no loss for the main text. |

docs/literature/03-rct-challenges-and-tte.md

@@ -0,0 +1,61 @@
+# Literature review 3: challenges of surgical RCTs, and whether TTE solves them
+
+*Merged from three parallel searches (2026-10-28). The evidence, with verified PMIDs and numbers taken from abstracts, is in `drafts/A-feasibility.md`, `drafts/B-surgeon-intervention.md` and `drafts/C-pathway-and-tte.md`, together with files 01 and 02. In total, 220 PMIDs were checked against PubMed; all exist, and a random sample of 15 titles matched the claims made about them. Framing: D54. TTE here means emulation on any observational data: claims, EHR or clinical registry (VQI; VQI linked to Medicare, VISION).*
+
+## How to read Table 1
+
+- **Can TTE solve it?** asks whether target trial emulation removes the problem *in principle*, separately for claims and for EHR/registries, because the answer often depends on the data source.
+- **Tested in our simulation** names the simulated world or analysis that measures it against a known truth. *Candidate* means not yet simulated.
+
+## Draft Table 1. Why surgical RCTs struggle, and whether target trial emulation solves it
+
+| # | Challenge | How common / how big | Vascular / claudication example | TTE on claims | TTE on EHR / registry | Tested in our simulation |
+|---|---|---|---|---|---|---|
+| **A. Running the trial** | | | | | | |
+| 1 | Recruitment failure, early stopping, non-publication | Only 14.6% of surgical RCTs finish on time and at full size (Shadbolt 2023). 21% stopped early, mostly for recruitment (Chapman 2014). In vascular surgery, 22% are discontinued and 80% of those are never published (Binko 2023). | CLEVER planned 252 and randomized 111. SUPER planned 400, randomized 240, and stopped early. OBACT randomized 56 of 434. | **Yes**: no recruitment step | **Yes**, but registries hold only treated patients, so a no-PVI arm needs linkage | Realistic trial (n = 240) vs TTE on the full population |
+| 2 | Equipoise is absent, and surgeons have strong preferences | Vascular surgeons' equipoise is often absent, and preference strength predicts refusal to randomize (Young 2004). Clinicians' "hunches" undermine trials (Donovan 2014). | Physician early-PVI rates range from 0% to 58% (Hicks 2020) | **Partly**: preference becomes the source of treatment variation, and an unmeasured confounder if it tracks prognosis | **Partly**: drivers such as Rutherford class and ABI are recorded | Practice-style worlds (W2, W3) |
+| 3 | Who consents is selective | Refusal >50% in most preference trials, related to age, sex and education. Outcomes rarely differ (Wasmann 2019; King 2005; Vist 2005). Preference may modify effects, ES about 0.16 (PCRG 2008). | Whyman: 62 of 600 screened claudicants randomized | **Yes for selection** (no consent step); patient preference is still unrecorded | Same | *Candidate*: make consent in the simulated trial depend on patient characteristics (currently random) |
+| 4 | Too small to detect hard outcomes | Vascular RCT conclusions hinge on 3–7 events (Javidan 2024; Li 2023). Median negative surgical RCT has about 80 patients (Ahmed Ali 2018). | Procedure-related amputation within 1 year after revascularization for claudication is 0.2% (Djerf 2020) | **Yes**: tens of thousands of patients. A precise estimate can still be biased. | **Partly**: smaller samples | Realistic trial: power 25% for a true 5-point harm. TTE spread is 1.5–2.8 points. |
+| 5 | Short follow-up and soft endpoints | Claudication trials use 541 different outcomes and few PROs (Shwan 2024). 88% of device endpoints are surrogates (Dhruva 2009). Benefit fades by 2–5 years (IRONIC, ERASE). | No claudication RCT was powered for CLTI, amputation or death. Only the Swedish SFA trial reports them by arm (about 2% amputation per arm at 5 years, n = 100). | **Yes for hard long-term outcomes; no** for walking distance and QoL | **Partly**: walking distance, ABI and Rutherford class recorded at visits, but follow-up is short and fragmented. VQI–Medicare linkage reaches 74–82% (Hoel 2017). | Benefit-then-harm scenario (benefit at 2 years, harm at 5). *Candidate*: add a 12-month trial analysis. |
+| 6 | Cost, funding, industry, ethics of withholding or sham | IRONIC revascularization cost about 2× exercise. Trials save money when they use routine data (Griessbach 2024). | No sham-controlled PVI trial exists; ORBITA is the nearest analogue | **Yes** for cost and withholding; **no** for placebo questions | Same; symptom outcomes are prone to placebo effects | Not testable (Discussion) |
+| **B. The surgeon and the procedure** | | | | | | |
+| 7 | Learning curve | Plateau at about 250 cases (Vickers 2007). 1 of 247 surgical RCTs analysed learning effects (Conroy 2019). | In Medicare claims, carotid stenting by low-experience operators had aOR 1.7 (Nallamothu 2011) | **Partly**: cumulative operator volume is computable | **Partly**: plus technical success | *Candidate*: new learning-curve world |
+| 8 | Surgeon skill varies | Video-rated skill: bottom vs top quartile complications 14.5% vs 5.2% (Birkmeyer 2013). Skill explains 25.8% of complication variation (Stulberg). | PVI operator volume and MALE, OR 0.73 (Chunawala) | **Partly**: proxies only (volume) | **Partly** | Surgeon-skill worlds (W1, W1 ρ = 0) |
+| 9 | Trials recruit selected surgeons and centres | CEA mortality 1.4% at trial hospitals vs up to 2.5% elsewhere (Wennberg 1998). Trial patients are younger than registry patients (Cutteridge 2025). | Carotid and EVAR trials vs practice | **Yes**: all operators are included | **Partly**: centres choose to join registries | Realistic trial (selected surgeons) vs ideal trial vs TTE |
+| 10 | Procedure not standardized; device heterogeneity | 29.4% of trial interventions standardized (Blencowe) | Tibial PVI rate varies by physician, IQR 9–38% (Bose 2023) | **Partly**: modality and territory only | **Mostly**: lesion and device data | Not simulated (Discussion: "versions of treatment") |
+| 11 | Outcomes cluster by surgeon | 4% of surgical RCTs adjusted for provider (Biau 2008). Ignoring clustering overstates precision (Lee & Thompson 2005). | Hundreds of PVI operators per registry | **Yes** for SEs; confounding by physician remains | **Yes** | Physician variants V1–V4 in every world |
+| 12 | No blinding; outcome detection bias | Non-blinded assessors inflate subjective effects by 36–68%; objective outcomes are not affected (Hróbjartsson 2012; Wood 2008) | Claims codes separate claudication from CLTI with 81% agreement against VQI (Bose 2025) | **No blinding**, but amputation and death are objective. CLTI coding is at risk. | **No**: clinician-reported | Style affects CLTI coding (W3) |
+| **C. The treatment pathway** | | | | | | |
+| 13 | Crossover: "conservative" becomes delayed surgery | 27–43% crossover within 1–2 years across spine, knee and appendicitis trials, up to two-thirds by 10 years in hernia trials. Effects shrink toward the null (Gaudino 2019); STICH ITT was not significant at 5 years. | SUPER 29% at 1 year; Swedish SFA 25% by 5 years; ERASE later revascularization HR 2.50 | **Yes**: explicit strategies with CCW and sequential trials, **if** the triggers (worsening) are measured. Claims lack symptoms. | **Yes, better**: symptoms partly recorded | Deferral world (W4) |
+| 14 | Patients refuse or don't adhere | ACST: only 50% were operated on by 1 month. SET: 24.2% recruited, US completion 9.8–49.9%. | SUPER SET compliance 50% | **Partly**: refusal of an offered operation is invisible | **Partly**: sometimes noted | Trial arm A has 10% refusal |
+| 15 | Co-interventions and usual care differ | Usual care poorly reported (TIDieR median 1/12). Only 38.1% of VQI claudicants on complete OMT before revascularization. | "Conservative" = advice vs SET + BMT vs cilostazol across trials | **Partly**: drugs and Medicare SET claims are visible; home exercise and smoking are not | **Partly**: VQI has OMT only at the procedure | *Candidate*: unmeasured supervised exercise (supplement) |
+| 16 | The comparator is ill-defined | Most "conservative" arms are really "surgery if needed". CCW and sequential trials give different numbers (Garcia-Albeniz 2026). | "No PVI for 6 months" vs "PVI only for CLTI" vs "never PVI" | **Yes, TTE's main gain**: several named strategies from one dataset | Same | Aim 1: A vs B1, A vs B2, sequential trials, landmark |
+| **D. The price TTE pays** | | | | | | |
+| 17 | Confounding by indication (what randomization fixes) | TTE agrees with RCTs when the emulation is close (r = 0.93, RCT-DUPLICATE; 0.83, Wang 2026), poorly otherwise (0.53). Only 30.8% of published TTEs address unmeasured confounding (Ren 2026). 19.5% of CCW studies still contain immortal time (Manzanilla 2026). | No claudication or PVI TTE found in PubMed | **No**: severity poorly captured | **Partly**: ABI and Rutherford class recorded | Unmeasured-severity world (W5) and realistic world (WR). *Candidate*: EHR-like world with noisy severity. |
+| 18 | Misclassified eligibility | About half of registry "claudication" amputations were in misclassified CLTI patients (Djerf 2020) | Claudication vs CLTI codes 81% agreement (Bose 2025) | **Risk** | **Risk** | Not simulated (out of scope, D38) |
+
+## What the review means for the simulation
+
+**Already covered (no change needed).** Rows 1, 2, 4, 8, 9, 11, 13, 16 and 17 map onto existing worlds or analyses, and rows 12 and 14 are partly covered.
+
+**Candidate additions, in order of value for cost:**
+
+| Candidate | Table 1 row | What it adds | Cost | Recommendation |
+|---|---|---|---|---|
+| **EHR/registry-like world**: severity measured with noise (e.g., ABI and symptom class) instead of exact or only visit counts | 17 | Answers "does better data close the gap?", which is the claims-vs-EHR question | Parameters only; same DGP as W0, about 1–1.5 h of running | **Add** |
+| **Non-random consent in the simulated trial**: consent depends on age and frailty, moderately (Wasmann 2019, King 2005) | 3 | More realistic trial; the trial's own truth must then be weighted by consent probability | Small code change in `rct.R` plus RCT truths | **Add** |
+| **12-month trial analysis** in the benefit-then-harm scenario | 5 | Shows a short trial reaching the opposite conclusion from 5-year TTE | Small: ITT at 12 and 24 months plus truths at those horizons | **Add** |
+| **Learning-curve world**: skill rises with each operator's cumulative volume | 7 | TTE-specific issues: interference, delayed PVIs done by more experienced operators, physician confounding that changes over time | Needs calendar-time simulation across patients; a real build (about a day) | **Later / supplement** |
+| **Unmeasured supervised exercise** | 15 | Co-intervention that differs by arm | One more confounder | Supplement (already planned) |
+
+## Points for the Introduction and Discussion
+
+1. **No PubMed hit for a claudication or PVI target trial emulation**, no systematic review of TTE in surgery or vascular surgery, and only scattered benchmarks of TTE against procedural RCTs. TTE guides for surgeons exist (e.g., *JBJS Rev* 2026), but no known-truth tests.
+2. **No data source wins everywhere.** Claims give long, complete follow-up of hard outcomes but miss severity and symptoms. Registries and EHR record severity but have short, fragmented follow-up and no untreated arm unless linked. Linkage (VISION) is incomplete.
+3. **TTE complements trials rather than replacing them.** It covers long-term hard outcomes trials cannot afford, while trials remain needed for walking distance and quality of life.
+
+## Open items
+
+- EXITPAD could not be found in PubMed; it is left out of the trial table.
+- Some trial details are not in the abstracts and need full texts: planned n for MIMIC, IRONIC and ERASE; crossover in IRONIC, CLEVER and MIMIC; ERASE's long-term CLI result.
+- **Authors' own papers.** Bose 2023 (36058433) and Bose 2025 (40439664, plus a second 2025 paper) are cited neutrally, for practice variation and code validation. The authors should decide whether to keep them.

docs/literature/drafts/A-feasibility.md

@@ -0,0 +1,254 @@
+# Draft A: trial feasibility and design challenges in surgical RCTs
+
+Draft input for Table 1. It covers six challenges: recruitment, equipoise, consent selection, sample size, endpoints and follow-up, and cost, funding and ethics.
+
+**Method.** Every PMID below was checked with NCBI esummary, and the abstract was read with efetch, on 2026-10-28. Numbers come from the PubMed abstracts only; no full texts were read. Papers already listed in `01-surgical-rct-limitations.md` or `02-claudication-evidence-and-methods.md` are not listed again. They are cited by PMID where they help the argument and are marked "(file 01)" or "(file 02)".
+
+**Data sources for TTE.** We assume three kinds of data, and they differ in what they can do:
+- **Claims** (Medicare fee-for-service): population-based, with long and nearly complete follow-up for death, procedures, amputation and diagnoses. They contain no walking distance, ABI, Rutherford class, or patient-reported outcomes (PROs).
+- **EHR and clinical registries** (VQI): severity at the time of the procedure (Rutherford class, ABI, anatomy) and some PROs. Follow-up is short or fragmented, and VQI captures patients only when they have a procedure, so patients managed without one are mostly missing.
+- **Linked registry–claims data** (VQI–Medicare, VISION): registry severity plus claims follow-up. Linkage is incomplete, and the cohort is still defined by having a procedure.
+
+---
+
+## 1. Recruitment, slow accrual, early discontinuation, non-publication
+
+**What it is.** Surgical trials often fail to recruit the planned number of patients, run years late, or stop early. Trials that stop early are rarely published, and so are many completed trials. The evidence base is therefore smaller and more biased than the registered trial activity suggests.
+
+**How common / how big.**
+- Completion on time and at full size is rare. Of 2,542 surgical RCTs registered in 2010–2014, only **14.6%** finished both on time and at full enrollment. **21.5%** were discontinued, and when they stopped they were a median **92.7%** short of their enrollment target (36648945).
+- Surgical trials are discontinued more often than nonsurgical ones. Among 88,498 US trials, discontinuation was **15.97% vs 11.07%**, and poor recruitment was cited more often (**44.65% vs 34.74%**) (29804847).
+- Industry funding predicted discontinuation (OR 1.25 in surgical trials) (29804847).
+- These figures build on Chapman 2014 (25491195), Rosenthal 2015 (24979608) and Kasenda 2014 (24618966) (file 01).
+
+**Vascular / claudication example.**
+- Vascular surgery RCTs: of 108 registered in 2010–2019, **22.2%** were discontinued, most often for poor enrollment (**45.8%**). Discontinued trials reached only **28.4%** of planned enrollment, and **80%** of them were never published. Of completed trials, **25%** were unpublished; industry funding cut the odds of publication (OR 0.18). In total, **4,788 enrollees** have no public results (37311508).
+- PAD exercise trials: one recruitment effort spent $1,330 on community outreach and got two referrals and no enrollments. Health-system referral produced 44 referrals and one enrollment (30139448).
+- Cardiac surgery is a neighbouring field: **22.6%** of trials were terminated early, and aortic surgery trials had higher odds of termination (OR 2.86) (39133169).
+- Under-representation of women and minorities is a related problem. Across 110 North American PAD trials (57,451 patients), **28%** of patients were women and **5.0%** were Black (42051894).
+- BASIL-3 reaching 481 of 861 planned patients is noted in file 01.
+
+| Paper | PMID | Takeaway |
+|---|---|---|
+| Shadbolt C et al. 2023, *JAMA Netw Open* | 36648945 | 2,542 surgical RCTs. 14.6% finished on time and at full size, 20.4% on time, and 45.9% met their enrollment target. Median delay was 12.2 months (66.7% longer than planned). 21.5% were discontinued, with a median shortfall of 92.7%. |
+| Mouw TJ et al. 2018, *J Surg Res* | 29804847 | 5,779 surgical vs 82,719 nonsurgical US trials. Discontinued: 15.97% vs 11.07%. Poor recruitment was cited in 44.65% vs 34.74%. Industry funding predicted discontinuation. |
+| **Binko MA et al. 2023, *Ann Vasc Surg*** | 37311508 | Vascular surgery RCTs: 22.2% discontinued, 45.8% of them for poor enrollment. Of trials stopped after enrollment began, 80% were unpublished. Of completed trials, 25% were unpublished (industry funding OR 0.18 for publication). |
+| Cancelli G et al. 2024, *Eur J Cardiothorac Surg* | 39133169 | 746 cardiac surgery trials: 22.6% terminated early, 29.6% of them for slow recruitment. Odds of termination were higher for valve (OR 4.30), LVAD (OR 3.65) and aortic (OR 2.86) trials. |
+| Caldieraro-Bentley AJ et al. 2018, *J Vasc Nurs* | 30139448 | PAD exercise trials. Indirect recruitment ($1,330): 2 referrals, 0 enrollments. Direct recruitment: 44 referrals, 1 enrollment. Barriers were comorbidity, transport, and strict eligibility criteria. |
+| Shah T et al. 2026, *JSCAI* | 42051894 | 110 North American PAD trials (2000–2023): 28% women, 5.0% Black, 1.3% Hispanic. 80% were industry-funded. |
+
+**Can TTE solve it? Yes for feasibility; partly for representativeness, depending on the data source.**
+- TTE has no recruitment, consent or discontinuation step, so all eligible patients in the data enter the emulation.
+- **Claims:** Medicare covers nearly every older US claudicant, including women, Black patients and rural patients whom trials under-enroll. Patients under 65 and those in Medicare Advantage are missing.
+- **Registries (VQI):** these include only patients who had a procedure, at participating centres. A registry cannot supply patients managed without a procedure (the "delayed / never" arm) unless it is linked to claims or an EHR.
+- Non-publication is a separate problem. TTE analyses can also go unpublished selectively, so a protocol should be registered in advance (TARGET, 40899949, file 01).
+
+**Testable in a known-truth simulation?** Partly. Simulate a trial that recruits a small, under-powered and selected subset (see §3) and compare it with a TTE run on the full simulated population. The selective-publication part is not testable.
+
+---
+
+## 2. Equipoise: surgeon and patient treatment preferences
+
+**What it is.**
+- Randomization is ethical only under equipoise, which is genuine uncertainty about which option is better. Freedman's "clinical equipoise" places that uncertainty in the expert community rather than in each individual clinician.
+- Surgeons often hold strong individual beliefs and tolerate uncertainty poorly, so they may not offer the trial or may selectively enrol patients.
+- Patients also bring strong preferences, especially when one arm is "an operation" and the other is "no operation".
+
+**How common / how big.**
+- Surgeons were significantly more intolerant of uncertainty than physicians (P = 0.007), but their attitudes to RCTs did not differ (16078288).
+- 62.6% of Australian colorectal surgeons named difficulty convincing patients to accept random allocation as a barrier (11598468).
+- In a US national survey, there was consensus (≥60% agreement) on whether to operate in 19 of 30 cases. Variation was largest when the modal answer was "do not operate" or "equipoise" (38328985).
+- In spine surgery, surgeons judged 12 of 20 cases eligible for randomization (17304133).
+- File 01 covers Donovan 2014 (24811157) on clinicians' "hunches" and Rooshenas 2016 (27755555) on how equipoise is presented in consultations.
+
+**Vascular / claudication example.**
+- In a survey of vascular surgeons in Australia and New Zealand (77% response), **24%** were in individual equipoise on endarterectomy vs stenting for high-risk carotid patients. **89%** favoured endarterectomy for average-risk patients, so there was no community equipoise. Strength of preference predicted unwillingness to join 4 of 6 hypothetical trials (15523403).
+- Patients with severe claudication were **risk-seeking** about invasive treatment, and patients who could walk only short distances accepted more risk (18455642).
+- In Medicare, the physician-level rate of tibial PVI for claudication varied widely: median **20.0%**, IQR **9.1–37.5%** (36058433). This is practice variation that no trial would capture.
+
+| Paper | PMID | Takeaway |
+|---|---|---|
+| **Freedman B 1987, *NEJM*** | 3600702 | Defines "clinical equipoise": uncertainty in the expert community. Requiring individual equipoise "presents nearly insuperable obstacles" to starting a trial and can cause recruitment failure. |
+| McCulloch P et al. 2005, *Br J Surg* | 16078288 | 2,000 consultants surveyed (36.5% responded). Surgeons were more intolerant of uncertainty (P = 0.007) but not more hostile to RCTs. Intolerance may block decisions to enrol individual patients. |
+| **Young J et al. 2004, *Surgery*** | 15523403 | Vascular surgeons: 24% in equipoise on carotid stenting in high-risk patients, 89% preferred endarterectomy in average-risk patients, and 88% favoured a minimally invasive approach for high-risk AAA. Preference strength predicted unwillingness to randomize. |
+| Gattellari M et al. 2001, *Dis Colon Rectum* | 11598468 | 89% response. 66.7% said RCTs should be the design of choice. Barriers: funding (74.4%), convincing patients to accept random allocation (62.6%), lack of wider surgical support (55.9%). |
+| Zychowski KL et al. 2024, *Ann Surg* | 38328985 | 580 US surgeons. Consensus on 19 of 30 cases, with most variation where the mode was against surgery or at equipoise. Surgeons often "let patients decide". |
+| Ghogawala Z et al. 2007, *Spine* | 17304133 | Spine surgeons split 51% / 38% / 11% between approaches. Only 12 of 20 real cases were judged eligible for randomization. |
+| Letterstål A et al. 2008, *J Vasc Surg* | 18455642 | 50 patients with severe claudication: EQ-5D 0.51, standard gamble 0.88. Patients were risk-seeking about surgery, more so if their walking distance was short. |
+| Bose S et al. 2023, *J Vasc Surg* | 36058433 | Medicare, 59,930 PVIs for claudication: 27.7% tibial. Physician-level rate median 20.0% (IQR 9.1–37.5%). Practice setting (office-based lab aOR 2.37) and specialty predicted use. |
+
+**Can TTE solve it? Partly.**
+- TTE needs no one to be in equipoise. It instead relies on preference-driven variation between physicians and regions, which is exactly what makes trials hard to run. Physician preference (for example, a physician's past PVI rate) can serve as an instrument or as a source of positivity.
+- Patient preference, and the clinical judgement behind a surgeon's choice for a particular patient, are not recorded. If they predict outcomes, they are unmeasured confounders.
+- **EHR and registries** partly measure what drives the choice (Rutherford class, ABI, anatomy), which narrows the problem. **Claims** do not.
+- Wherever community consensus is near-total, TTE suffers positivity violations, just as a trial would struggle to recruit.
+
+**Testable in a known-truth simulation? Yes.** Generate a latent physician "intervention propensity" (already in the data-generating process, or DGP) and a latent patient preference that affects both treatment and outcome. Vary how strongly each confounds the outcome, and check how much TTE bias each produces.
+
+---
+
+## 3. Patient preference and consent selection: who agrees to be randomized
+
+**What it is.** Eligible patients who decline randomization usually do so because they prefer one treatment. Trial participants can therefore differ from the eligible population, which limits external validity. If preference also changes how patients respond, it can affect the outcome. Comprehensive-cohort and partially randomized preference designs follow patients who decline, so the two groups can be compared directly.
+
+**How common / how big.**
+- Partially randomized preference trials (44 trials, 24,873 patients): refusal of randomization was **over 50% in 26 of 44 trials** (range 19–99%). Decliners differed by education, sex, age, race and prior experience of one treatment. Primary outcomes were similar in the randomized and preference cohorts (mean difference 0.093, 95% CI −0.178 to 0.364) (31619428).
+- King 2005 (32 comprehensive cohorts and two-stage trials): preferences led "a substantial proportion" to refuse randomization, but there was little evidence of bias in who agreed, or in outcomes (15741531).
+- Musculoskeletal trials, patient-level data (n = 1,594): patients randomized to their preferred treatment did better than indifferent patients (effect size **0.162**, 95% CI 0.011–0.314) (18977792).
+- Trial participants vs comparable non-participants (31,140 vs 20,380 patients): no overall difference in outcomes (SMD 0.01), but heterogeneity was high (I² = 89%) for dichotomous outcomes (15905256).
+
+**Surgical examples (no vascular example with a non-participant cohort was found).**
+- SPORT disc herniation: the 743 patients who declined randomization formed an observational cohort. Their self-selected surgery-vs-nonoperative difference was large (bodily pain 14.8 points at 3 months), whereas the randomized ITT difference was not significant (17119141; ITT trial 17119140, file 01).
+- PREOPANC (pancreatic cancer): only **33%** of 455 eligible patients were randomized, and 55% of the rest refused. Survival (15.2 vs 15.5 months) and resection rates were the same in the two groups (33273349).
+- ProtecT (prostate cancer): men who chose their treatment were similar to randomized men, apart from occupation (29288137).
+- Meniscal tear: willingness to join a planned surgery-vs-nonoperative RCT fell from **62% to 46%** after a course of physiotherapy (41367623).
+
+| Paper | PMID | Takeaway |
+|---|---|---|
+| **King M et al. 2005, *JAMA*** | 15741531 | 32 preference-design studies. Preference drives refusal, but there was little evidence that it biases who enrols or the trial's internal validity. |
+| **Preference Collaborative Review Group 2008, *BMJ*** | 18977792 | Patient-level data, n = 1,594. Randomized to preferred treatment vs indifferent: ES 0.162 (0.011–0.314). Allocated to an unwanted treatment: OR 1.70 for loss to first follow-up compared with indifferent patients. |
+| **Wasmann KA et al. 2019, *BMJ Open*** | 31619428 | 44 partially randomized preference trials (n = 24,873). Refusal over 50% in 26 trials. Decliners differed by education, sex, age, race and prior treatment. Outcomes similar between cohorts. Loss to follow-up and crossover were higher in the randomized cohort. |
+| Vist GE et al. 2005, *BMJ* | 15905256 | Trial participants vs non-participants: no consistent effect of participation (SMD 0.01), but large heterogeneity (I² = 89%). |
+| **Weinstein JN et al. 2006, *JAMA* (SPORT observational cohort)** | 17119141 | Patients who declined randomization and chose surgery improved more (treatment effect 14.8 to 15.4 points at 3 months). The authors warn that this comparison is confounded. |
+| Versteijne E et al. 2022, *Ann Surg* (PREOPANC) | 33273349 | 33% of eligible patients randomized. Non-randomized patients had the same survival and resection rates. |
+| Donovan JL et al. 2018, *J Clin Epidemiol* (ProtecT) | 29288137 | Extended comprehensive cohort: 1,643 randomized, 997 chose treatment, and 557 were excluded. Those who chose were similar to the randomized men; those excluded were sicker. |
+| Fox KB et al. 2025, *Osteoarthr Cartil Open* | 41367623 | Advance preference assessment: willingness to enrol in a surgery-vs-nonoperative RCT fell from 62% to 46%. The strongest preference was to avoid surgery (47%). |
+
+**Can TTE solve it? Mostly for selection into the study; no for preference as a confounder.**
+- TTE does not need consent, so it estimates the effect in everyone eligible rather than in those willing to be randomized.
+- The same preference that drives refusal in a trial drives treatment choice in observational data, as SPORT's observational cohort shows. Neither claims nor EHR record patient preference.
+- If preference also modifies the treatment effect, as PCRG 2008 suggests, the trial and the TTE estimate different estimands even when both are unbiased.
+
+**Testable in a known-truth simulation? Yes, and this is directly relevant to our design.** The evidence supports making consent to the simulated trial **non-random but only moderately selective**:
+- consent depends on covariates (age, sex, prior treatment) and on a latent preference;
+- refusal rates are well over 50% (Wasmann 2019);
+- consent has only a small effect on outcomes (King 2005, Vist 2005);
+- add an optional scenario with preference × treatment effect modification of about 0.15 SD (PCRG 2008).
+
+The trial's ITT estimate for consenters can then be compared with the TTE estimate for the whole population, against the known truth for each.
+
+---
+
+## 4. Sample size, underpowering, imprecision
+
+**What it is.** Surgical trials are small because recruitment is hard, funding is limited, and the evidence is spread across many devices and techniques. Small trials miss real effects, over-estimate the effects they do find, and depend on a handful of events. Hard outcomes in claudication, such as amputation and progression to CLTI, are rare, so trials powered for them would need thousands of patients.
+
+**How common / how big.**
+- Only **53%** of surgical RCTs in leading journals reported their sample-size calculation adequately, falling to **12%** against the full SPIRIT items (31787318).
+- File 01 already covers Ahmed Ali 2018 (29907196): the median size of negative surgical RCTs is about 76–80.
+
+**Vascular / claudication example.**
+- Fragility: in positive open-vs-endovascular RCTs, a median of **3** event changes would reverse significance, and **70%** of outcomes had more patients lost to follow-up than that (37315910). In negative trials, a median of **7** changes would make them significant (7 for PAD), and **47%** had more patients lost than that (36572321).
+- Cochrane (file 02): **10 RCTs, 1,087 participants** in total for endovascular revascularisation in claudication. Evidence was downgraded for "small sample size", and the endovascular vs no-specific-treatment comparison rested on 3 studies with **125 participants** (29518253).
+- Device dispersal: 112 PAD endovascular RCTs tested **228 different devices or techniques** (37678660).
+- Rare hard outcomes: major amputation within 1 year of revascularisation for claudication was **0.2%** (9 of 5,860; Swedvasc) (31866238).
+- Progression to CLTI under conservative management ranged from **1.1% to 36.7%** across studies, with I² = 99%. Rates that uncertain make power calculations unreliable (39694186).
+
+| Paper | PMID | Takeaway |
+|---|---|---|
+| **Javidan A et al. 2024, *J Vasc Surg*** | 37315910 | 21 positive open-vs-endovascular RCTs. Median fragility index 3 (3, 20). In 70% of outcomes, loss to follow-up exceeded the fragility index. |
+| **Li A et al. 2023, *J Vasc Surg*** | 36572321 | 49 negative vascular RCTs. Median reverse fragility index 7 (PAD 7). In 47%, loss to follow-up exceeded it. |
+| Speich B et al. 2020, *Surgery* | 31787318 | 85 surgical RCTs (2017). Sample-size reporting adequate in 53% by CONSORT and 12% with SPIRIT items. |
+| Zywicka EM et al. 2024, *EJVES* | 37678660 | 112 PAD endovascular RCTs covering 228 devices or techniques. Interventions were described well enough to replicate in only 21%. |
+| Fakhry F et al. 2018, *Cochrane* (file 02) | 29518253 | 10 RCTs, n = 1,087, downgraded for small size and heterogeneity. Long-term data on walking distance came from only 2–3 studies (103–184 patients). |
+| Djerf H et al. 2020, *EJVES* | 31866238 | Procedure-related major amputation 0.2% at 1 year after revascularisation for claudication. Of 109 registered amputations, 51 were in patients with CLTI misclassified as claudication. |
+| Froud JLJ et al. 2025, *Ann Vasc Surg* | 39694186 | 9 cohorts (n = 4,115): progression to CLTI 1.1–36.7%, I² = 99%. Women made up 22.7% of patients. |
+
+**Can TTE solve it? Yes for size; partly for precision on what matters.**
+- **Claims** offer very large samples (tens of thousands of claudicants), which makes rare hard outcomes (amputation, CLTI, death) estimable with narrow confidence intervals.
+- Large samples do not remove bias. A precise but confounded estimate can mislead more than a wide trial estimate does.
+- **Registries** are smaller and centre-limited but record severity. Djerf 2020 shows that registry indication codes can be wrong: about half of the "claudication" amputations were in misclassified CLTI patients. Misclassification of eligibility is therefore a real risk in claims and in registries.
+- Effective sample size can be much smaller than the raw count after cloning, censoring and weighting, especially with extreme weights.
+
+**Testable in a known-truth simulation? Yes.** Compare the sampling variability and coverage of a small trial (n ≈ 100–200) with a population-scale TTE across replicates. Show the bias–variance trade-off: unbiased but noisy trial vs precise TTE with possible bias.
+
+---
+
+## 5. Short follow-up and soft or surrogate endpoints
+
+**What it is.**
+- Claudication trials mostly measure treadmill walking distance, ABI, patency or quality of life at 6–12 months. Patients and payers care about durability, reintervention, CLTI, amputation and death over years.
+- Device trials rely heavily on surrogate endpoints such as patency.
+- Outcome definitions vary so much that trials cannot be pooled.
+- Long follow-up raises costs and loss to follow-up, especially for patient-reported outcomes.
+
+**How common / how big.**
+- Of 213 primary endpoints in FDA premarket approvals of cardiovascular devices, **88%** were surrogates. Only **27%** of the supporting studies were randomized (20040556).
+- In surgical RCTs, loss to follow-up was far higher when outcomes were patient-reported (RR **14.21**) and when follow-up lasted 3 months or more (OR **10.09**) (38795671).
+
+**Vascular / claudication example.**
+- **408 claudication studies** used **541 unique outcomes**. 386 of these were used only once, ABI was the most common, PROs appeared in only **36.2%** of studies, and definitions of MALE and primary patency varied (39485886).
+- A new core outcome set for chronic symptomatic PAD has 11 outcomes, a mix of hard and patient-centred ones: death, MI, major amputation, acute limb ischaemia, pain-free walking distance, quality of life, and others (40992592).
+- Trials in file 02: CLEVER (6-month treadmill, n = 111; 22090168), ERASE (12 months, n = 212; 26547465) and IRONIC (1 year; benefit lost by 5 years; 25095886, 31937137).
+- The paclitaxel mortality signal appeared only once long-term data were pooled (30561254, file 02).
+- SWEDEPAD 2 is the counter-example: a registry-based RCT with **98.3%** follow-up and a median of **7.1 years** for mortality (40902614, file 02).
+
+| Paper | PMID | Takeaway |
+|---|---|---|
+| **Shwan A et al. 2024, *BJS Open*** | 39485886 | 408 claudication studies with 541 unique outcomes; 386 were used only once. PROs in 36.2%. MALE and patency defined inconsistently. Only 38.9% of studies included claudicants alone. |
+| Zwetsloot SLM et al. 2026, *EJVES* | 40992592 | Delphi core outcome set for chronic symptomatic PAD: 11 outcomes, including pain-free walking distance, physical activity, quality of life, amputation, and death. |
+| **Dhruva SS et al. 2009, *JAMA*** | 20040556 | 78 cardiovascular device premarket approvals: 27% of studies randomized, 14% blinded, 65% based on a single study, 88% of primary endpoints surrogate. |
+| Walker RJB et al. 2024, *J Surg Res* | 38795671 | 200 surgical RCTs. Loss to follow-up 3.10 per 100 patient-years. PROs RR 14.21; follow-up ≥3 months OR 10.09; standard-of-care assessments RR 0.17. |
+| Hoel AW et al. 2017, *J Vasc Surg* | 28222989 | VQI matched to Medicare: 82.4% of EVAR and 74.4% of open repair patients matched. Agreement was good for mortality (κ 0.64–0.82) but poor for reintervention (κ 0.16–0.26). |
+
+**Can TTE solve it? Partly, and the answer depends on the data source.**
+- **Claims:** strong on long, complete follow-up of hard outcomes (death, amputation, reintervention, CLTI diagnosis codes) over 5–10 years. No walking distance, ABI, Rutherford class or quality of life, so the trials' main efficacy endpoints cannot be emulated.
+- **EHR and registries (VQI):** record walking distance, ABI, Rutherford class, and sometimes PROs, so soft and functional endpoints and baseline severity are partly available. Follow-up is short and fragmented, and functional outcomes are measured only at clinical visits, so they are missing not at random.
+- **Linked VQI–Medicare (VISION):** combines registry severity with claims follow-up. Linkage is incomplete (74–82% in Hoel 2017), and reintervention agrees poorly between sources.
+- No source captures the main claudication benefit (walking and quality of life) with long follow-up. TTE complements trials by covering the long-term hard outcomes they cannot afford to measure, rather than replacing them for symptom endpoints.
+
+**Testable in a known-truth simulation? Yes, for hard outcomes and time-varying effects.** The DGP already has an early benefit and late harm on CLTI and amputation. Show that a 12-month trial and a long-follow-up TTE reach different conclusions. PRO endpoints could be added as a latent outcome that is observed only at visits, to show missingness in EHR data, but this is optional.
+
+---
+
+## 6. Cost, funding, industry sponsorship, and ethics (sham surgery, randomizing to no treatment)
+
+**What it is.**
+- Surgical RCTs are expensive and are rarely funded by public bodies. Device makers fund many vascular trials, which affects which questions get asked, which endpoints are used, and whether and how results are published.
+- Placebo (sham) procedures are needed to separate the specific effect of a symptom-relieving procedure from the placebo effect, but they raise ethical and recruitment problems.
+- Randomizing patients to "no procedure" can feel unethical to surgeons and patients when practice favours intervention.
+
+**How common / how big.**
+- Cost: the median investigator-initiated RCT cost **$645,824**, and **69.7%** exceeded their budget by more than 50%. Using routinely collected data for the primary outcome was associated with lower costs (39307405).
+- Public funding in vascular surgery: only **1%** of US vascular surgeons (41 of 4,037) hold NIH funding, and only **25%** of their projects are clinical or health-services research (37327950).
+- Industry outcomes across fields: industry-sponsored studies more often have favourable efficacy results (RR **1.27**) and favourable conclusions (RR **1.34**), with no difference between drug and device studies (28207928).
+- Industry outcomes in surgery: industry funding was associated with pro-industry significant results in surgical trials (OR **8.0**, 95% CI 1.1–53.2) (14970094).
+- Device publication: only **49%** of 177 premarket approval studies for cardiovascular devices were published, and when published, primary endpoints were often relabelled or missing (26063311).
+- Sham surgery: placebo-controlled surgical trials recruit as well as non-operative-control trials, but have about twice the attrition (IRR **1.8**) (38637129).
+- Willingness for sham trials: **31%** of surgeons and **40%** of patients would not take part in any placebo-controlled knee trial design offered (39615522).
+
+**Vascular / claudication example.**
+- Funding: **80%** of PAD trials were industry-funded (42051894, §1). Industry-sponsored vascular RCTs rose from **6.7% to 28%** over a decade (26298221). Industry funding predicted non-publication of completed vascular RCTs (OR 0.18; 37311508, §1). Jones 2026 found 39% industry-funded (42526742, file 01).
+- Sham procedures: ORBITA showed that a placebo-controlled trial of a symptom-relieving revascularization is possible. PCI did not improve exercise time beyond placebo (**16.6 s**, 95% CI −8.9 to 42.0) (29103656). No sham-controlled trial of PVI for claudication was found; this is an inference, since the search was not exhaustive.
+- Ethics of "no procedure": in claudication, randomizing to exercise alone is ethically easy, because exercise is guideline first-line therapy.
+
+| Paper | PMID | Takeaway |
+|---|---|---|
+| Griessbach A et al. 2024, *J Clin Epidemiol* | 39307405 | 93 investigator-initiated RCTs. Median cost $645,824. 69.7% overran their budget by more than 50%. Routinely collected outcome data were linked to lower cost. |
+| **Mirzaie AA et al. 2023, *J Vasc Surg*** | 37327950 | 1% of US vascular surgeons hold NIH funding (41 of 4,037). 75% of projects are basic or translational. |
+| **Lundh A et al. 2017, *Cochrane*** | 28207928 | 75 studies. Industry sponsorship: favourable results RR 1.27, favourable conclusions RR 1.34, same for devices as for drugs. Bias not explained by standard risk-of-bias tools. |
+| Bhandari M et al. 2004, *CMAJ* | 14970094 | 332 RCTs. Industry funding linked to pro-industry results (adjusted OR 1.8). In surgical trials, OR 8.0 (1.1–53.2). |
+| Dhruva 2009 (see §5); Chang L et al. 2015, *BMJ* | 26063311 | 177 premarket approval studies for cardiovascular devices: 49% published. Of 152 FDA primary endpoints, 28% were unlabelled and 10% missing in publications. 11% of results were substantially different. |
+| Hajibandeh S et al. 2015, *EJVES* | 26298221 | Vascular RCTs: share industry-sponsored rose from 6.67% to 28%. Methodological quality did not improve. |
+| **Beard DJ et al. 2020, *Lancet* (ASPIRE)** | 32145797 | Guidance: surgical placebo controls are justified when evidence of efficacy is poor and there is a real risk of bias from placebo effects. Feasibility work is recommended. |
+| **Al-Lamee R et al. 2018, *Lancet* (ORBITA)** | 29103656 | 200 patients randomized to PCI or placebo procedure. Exercise time difference 16.6 s (−8.9 to 42.0), not significant. Shows that a sham-controlled revascularization trial is feasible. |
+| Natarajan P et al. 2024, *BMJ Open* | 38637129 | 61 placebo-control vs 38 non-operative-control surgical RCTs. Recruitment similar; attrition IRR 1.8 (1.1–3.0) for placebo trials. |
+| Wall L et al. 2024, *Bone Joint J* | 39615522 | Discrete choice experiment. 31% of surgeons and 40% of patients would join no sham-controlled knee trial design. |
+
+**Can TTE solve it? Yes for cost and for withholding treatment; partly for industry influence; no for sham and placebo questions.**
+- **Cost:** TTE on existing claims or registry data costs a small fraction of an RCT. Griessbach 2024 shows that even trials save money by using routinely collected data.
+- **Ethics of withholding treatment:** no one is assigned anything, so randomizing to "no procedure" is not an issue.
+- **Industry influence:** independent investigators can run TTE on population data. However, claims do not identify devices well (file 01, limitation #10), so device-specific questions such as paclitaxel need registries or linked device identifiers.
+- **Sham and placebo:** TTE cannot separate the specific effect of a procedure from the placebo effect. Hard outcomes in claims (amputation, death) are not prone to placebo effects. Symptom outcomes in EHR or registries are prone to them, and observational data cannot blind.
+
+**Testable in a known-truth simulation?** Only partly. Cost and ethics are not testable. A placebo component could be added to a simulated PRO outcome that only a blinded trial removes, but it is outside the current DGP; this is optional.
+
+---
+
+## Summary: the most important new papers
+
+1. **Shadbolt 2023 *JAMA Netw Open* (36648945) and Binko 2023 *Ann Vasc Surg* (37311508).** Only 14.6% of surgical RCTs finish on time and at full size. In vascular surgery, 22% are discontinued, mostly for poor enrollment, and 80% of those are never published.
+2. **Wasmann 2019 *BMJ Open* (31619428), King 2005 *JAMA* (15741531) and PCRG 2008 *BMJ* (18977792).** Most eligible patients often refuse randomization, and those who refuse differ by covariates. Outcomes rarely differ, but preference may modify effects (ES about 0.16). These set the parameters for non-random consent in our simulated trial.
+3. **Young 2004 *Surgery* (15523403) and Freedman 1987 *NEJM* (3600702).** Vascular surgeons' equipoise can be measured and is often absent, and preference strength predicts refusal to randomize. Clinical equipoise is the ethical standard.
+4. **Javidan 2024 / Li 2023 *JVS* (37315910, 36572321) and Djerf 2020 *EJVES* (31866238).** Vascular RCT results hinge on 3–7 events, and procedure-related amputation within 1 year of revascularisation for claudication is 0.2%. Hard outcomes need population-scale data.
+5. **Shwan 2024 *BJS Open* (39485886), Dhruva 2009 *JAMA* (20040556) and Hoel 2017 *JVS* (28222989).** Claudication trials use 541 different outcomes and few PROs, and 88% of device endpoints are surrogates. Linking VQI to Medicare restores long-term hard outcomes but not PROs.

docs/literature/drafts/B-surgeon-intervention.md

@@ -0,0 +1,271 @@
+# Draft B: surgeon and intervention challenges of surgical RCTs, and what TTE can do about them
+
+Scope: learning curve; skill and expertise variation; selection of surgeons and centres into trials; standardization and device heterogeneity; clustering by surgeon; blinding, sham and outcome assessment.
+
+All PMIDs were checked with NCBI E-utilities esummary on 2026-10-28. All numbers come from the PubMed abstracts fetched with efetch; full texts were not read. Papers already listed in `01-surgical-rct-limitations.md` are cited by PMID only and are not repeated in the tables.
+
+**Data sources for "TTE" in this draft.** TTE can run on three kinds of data, and they differ on exactly the problems in this chapter:
+- **Claims** (Medicare FFS): the operator NPI, CPT/HCPCS codes (vessel territory; angioplasty vs stent vs atherectomy), place of service, and objective outcomes (amputation, death, repeat procedures). There is no lesion anatomy, no device brand, no technical success, and no patient-reported outcomes.
+- **Clinical registries** (e.g., the Vascular Quality Initiative, VQI): de-identified physician and centre identifiers, lesion characteristics, devices, and technical details. Some outcomes are clinician-reported. Centres choose to join.
+- **Registry–claims linkage** (VQI–Medicare, "VISION"): registry procedure detail plus long-term claims follow-up. Mao 2022 (PMID 35181518) matched 93.0% of VQI patients to Medicare with 99.4% accuracy.
+
+---
+
+## 1. Learning curve
+
+**What it is.** Surgeons get better at a procedure the more often they do it, so outcomes depend on where on their learning curve the operator is. If a trial starts while a new procedure is still early on its curve, the new procedure looks worse than it will be in practice. If the trial waits until surgeons are proficient, equipoise may already be gone (Papachristofi 2016). Learning curves are measured and reported inconsistently.
+
+**How common / how big.**
+- A review of 592 learning-curve studies in minimally invasive abdominal surgery found that 86% used operative time as the proxy and only 6% used patient-oriented outcomes (Harrysson 2014).
+- An umbrella review of 62 reviews (1,990 studies) gave a definition of the learning curve in only 63% (so 37% gave none, by subtraction), 712 different data items were used, and 60 of the 62 reviews were of low or critically low quality (Ramirez 2026).
+- For radical prostatectomy, the learning curve did not plateau until about 250 prior operations. Predicted 5-year recurrence was 17.9% after 10 prior cases and 10.7% after 250 (Vickers 2007).
+- TAVR in the TVT registry: from the 1st to the 400th case, risk-adjusted mortality fell from 3.57% to 2.15% and bleeding from 9.56% to 5.08% (Carroll 2017).
+- Only 1 of 247 surgical RCTs analysed learning, and 2% used an expertise-based design (Conroy 2019; see Section 5).
+
+**Vascular example.**
+- **Carotid stenting in Medicare** (Nallamothu 2011): 1,792 new operators started after the CMS coverage decision. 30-day mortality was 2.3% for a new operator's 1st–11th procedures vs 1.4% from the 12th on (aOR 1.7, 1.2–2.4). This is a learning curve measured in claims through the operator identifier.
+- **Endovascular device trials** handle the learning curve with "roll-in" patients who are excluded from the randomized analysis. The IntraCoil femoropopliteal trial used 93 roll-in patients across 22 sites (Ansel 2002).
+
+| Paper | PMID | Takeaway (numbers from abstract) |
+|---|---|---|
+| Harrysson IJ et al. 2014, *Ann Surg* | 24670849 | 592 learning-curve studies. Operative time was the proxy in 86%, postoperative outcomes in 52%, patient-oriented outcomes in 6%. No increase over time in the complexity of analysis. |
+| Ramirez J et al. 2026, *Surg Innov* (umbrella review) | 42527319 | 62 reviews / 1,990 studies. A definition was given in 63%, surgeon expertise reported in 63%, 712 unique data items, and 60/62 reviews were low or critically low quality. |
+| Papachristofi O et al. 2016, *Trials* | 27245050 | Two-phase model that estimates initial skill, learning rate, final skill, and **duration of learning**. Frames the "when to run the definitive RCT" dilemma. |
+| Biau DJ et al. 2008, *Br J Surg* (LC-CUSUM) | 18498126 | Individual learning-curve monitoring. ERCP success rose from 82.0% (first 100) to 96.1% (last 129). Competence signalled at case 79. |
+| Vickers AJ et al. 2007, *JNCI* | 17652279 | 7,765 prostatectomies by 72 surgeons. Plateau at about 250 cases. 5-year recurrence 17.9% vs 10.7% (10 vs 250 prior cases). |
+| **Nallamothu BK et al. 2011, *JAMA*** | 21954477 | Medicare CAS, 24,701 procedures, 2,339 operators. Median operator volume was 3.0/year. Early vs late experience: 30-day mortality 2.3% vs 1.4% (aOR 1.7). Very-low vs high volume: aOR 1.9. |
+| Carroll JD et al. 2017, *JACC* (TVT Registry) | 28662805 | 42,988 TAVRs at 395 hospitals. Risk-adjusted mortality 3.57% → 2.15% from case 1 to 400. Vascular and bleeding risk higher in the first 100 cases. |
+| Ansel GM et al. 2002, *Catheter Cardiovasc Interv* | 12124949 | Femoropopliteal stent trial. 93 roll-in ("learning phase") patients across 22 sites, reported separately from the randomized comparison. |
+| Luft HS et al. 1987, *Health Serv Res* | 3112042 | Volume–outcome links reflect both "practice makes perfect" and "selective referral" (better surgeons attract more patients). The relative importance varies by procedure. |
+
+Already in 01: Cook 2004 (16279280), Ramsay 2000/2001 (11155830, 11319991).
+
+**Can TTE solve it? Partly.**
+- **Claims.** TTE does not remove the learning curve, but it changes which question is answered: it estimates the effect *as delivered* across the real mix of operator experience, not at a chosen point on the curve.
+- **Claims and registries.** Cumulative operator volume can be built from the operator identifier (NPI in claims, physician ID in VQI) and used as a time-varying covariate or effect modifier (Nallamothu 2011 shows this is feasible). Caveats:
+  - Medicare FFS volume is only part of an operator's total volume.
+  - Volume is left-truncated at the start of the data.
+  - Volume partly reflects selective referral (Luft 1987).
+- **Registries only.** These add technical success and procedural detail, which are closer to direct measures of proficiency.
+- **What TTE cannot do** is target "the effect once everyone is proficient" unless proficient operators can be identified and positivity holds among them.
+
+**Testable in a known-truth simulation? Yes.** Let operator skill rise with cumulative PVI volume (e.g., skill_j(t) = s_j + λ·log(1 + V_j(t))). This world shows three things that are specific to TTE:
+1. **Interference.** One patient's early PVI raises the operator's skill for later patients, which violates no-interference/SUTVA. The individual-level TTE contrast then differs from the effect of a population policy of "early PVI for all", which would also change volumes.
+2. **Strategy effects and skill are entangled.** In the delayed arm, PVIs happen later, when operators are more experienced. Part of any "benefit of waiting" is then skill gain, not timing. The simulation can split the two by comparing truths with λ = 0 and λ > 0.
+3. **Time-varying confounding by physician.** High-preference physicians build volume faster, so skill becomes correlated with preference *and* changes over time. A physician fixed effect cannot remove this; adjusting for cumulative volume might. The simulation can check whether a time-varying volume covariate recovers the truth.
+
+---
+
+## 2. Surgeon skill, expertise, and volume–outcome
+
+**What it is.** Surgeons differ in technical skill. In a conventional RCT, each surgeon does both procedures, often with unequal expertise ("differential expertise bias"; Devereaux 2005, in 01). Volume is a crude, widely used proxy for skill, and it does not separate skill from referral patterns.
+
+**How common / how big.**
+- **Video-rated skill.** In a meta-analysis of 7 studies (151 surgeons, 17,932 procedures), top- vs bottom-quartile skill was associated with less reoperation (OR 0.44) and fewer medical complications (OR 0.23) (Woods 2023). Peer-rated skill explained 25.8% of the variation in post-colectomy complications (Stulberg 2020).
+- **Volume.** A systematic review found that 71% of hospital-volume and 69% of physician-volume studies reported significant volume–outcome associations. AAA surgery was among the strongest, at a median of 3.3–13 excess deaths per 100 cases across the strongest conditions (Halm 2002).
+- **Reporting of expertise in trials.** Only 34.4% of surgical interventions in RCTs came with any data on operator expertise (Blencowe 2015, BJS; Section 4).
+
+**Vascular example.**
+- **PVI.** In the NCDR PVI registry (555 operators), high- vs low-volume operators had lower in-hospital MALE (OR 0.73) and MACE (OR 0.64), while hospital volume was not associated (Chunawala 2025).
+- **AAA.** Elective AAA repair: OR 0.66 for mortality at high-volume hospitals (Holt 2007).
+- **Carotid stenting.** In pooled trial data, operator *in-trial* annual volume mattered (≤3.2/year: aRR 2.30 vs >5.6/year) but lifetime experience did not (Calvet 2014).
+
+| Paper | PMID | Takeaway (numbers from abstract) |
+|---|---|---|
+| Stulberg JJ et al. 2020, *JAMA Surg* | 32838425 | 17 surgeons, video-rated skill 2.8–4.6 out of 5. Higher skill: complications 15.5% vs 20.6%. Skill explained 25.8% of the variation in complications. |
+| Woods MS et al. 2023, *JSLS* (meta-analysis) | 36818767 | 7 studies, 151 surgeons. Top vs bottom skill quartile: reoperation OR 0.44, any medical complication OR 0.23. Mortality not significant. Few studies exist. |
+| Halm EA et al. 2002, *Ann Intern Med* (systematic review) | 12230353 | 71% / 69% of hospital / physician volume studies were significant. Effect is larger for AAA and pancreatic surgery, and smaller for CEA and angioplasty. Clinical risk adjustment weakens associations. |
+| Holt PJ et al. 2007, *Br J Surg* (meta-analysis) | 17380547 | 421,299 elective AAA repairs. High-volume OR 0.66 (threshold 43/year). Ruptured: OR 0.78. |
+| Chunawala Z et al. 2025, *JAHA* (NCDR PVI) | 41147410 | 55,785 PVIs, 97 hospitals, 555 operators. Operator volume: MALE OR 0.73, MACE OR 0.64. Hospital volume: no significant association. |
+| Setogawa N et al. 2025, *J Endovasc Ther* (Japan) | 37675778 | Hospital EVT volume was not associated with 12-month amputation (Q4 vs Q1 OR 1.00), but was associated with death (OR 0.71). |
+| Calvet D et al. 2014, *Stroke* (CSTC pooled RCTs) | 24347422 | 1,546 CAS patients in EVA-3S/SPACE/ICSS. 30-day stroke or death 10.1% (low in-trial volume) vs 5.1% (high). aRR 2.30. Lifetime experience had no effect. |
+| Bednarska E et al. 2008, *Clin Orthop Relat Res* | 18446421 | Surgeon survey (n = 276): 53.4% willing to join an expertise-based RCT vs 18.3% a conventional RCT. |
+| Cook JA et al. 2018, *Trials* (qualitative) | 30189868 | Surgeons and methodologists: expertise-based designs help participation, but surgeons in each arm may differ systematically. Patient transfer between surgeons is a practical problem. |
+
+Already in 01: Devereaux 2005 (15637373), Cook 2015 (26025450), Birkmeyer 2013 (24106936), Birkmeyer 2003 (14645640).
+
+**Can TTE solve it? Partly.**
+- **What TTE gains.** It needs no "both-procedures" surgeon. Each operator does what they normally do, which resembles an expertise-based design: early-PVI operators and deferring operators are compared as they practise. This also means **skill is bundled with strategy and physician**.
+- **Claims.** These contain no skill measure. Volume and specialty are proxies, and the NPI enables random or fixed effects.
+- **Registries** add technical success and lesion complexity, which partly measure skill. Video-based skill is not available in any routine source.
+- **Remaining risk.** If skill is correlated with intervention preference, confounding by physician remains, unless within-physician comparisons are possible (see 02, B6).
+
+**Testable in a known-truth simulation? Yes.** Give physicians a latent skill correlated with preference. Compare three estimands: (i) the population as-practised effect (what TTE targets), (ii) the effect if every PVI were done by top-skill operators (an "expert trial"), and (iii) the effect under differential expertise.
+
+---
+
+## 3. Selection of surgeons and centres into trials (generalizability)
+
+**What it is.** Trials enrol centres and surgeons who are interested, have high volume, and are often credentialed with explicit audit thresholds. Their results can overstate the benefit, or understate the harm, of a procedure in routine practice. The classic case is carotid endarterectomy after NASCET/ACAS.
+
+**How common / how big.**
+- **Medicare CEA, 1992–93** (Wennberg 1998). Mortality was 1.4% at the 86 NASCET/ACAS trial hospitals vs 1.7%, 1.9%, and 2.5% at high-, average-, and low-volume non-trial hospitals. These were all above the trials' own 0.6% (NASCET) and 0.1% (ACAS).
+- **Hospitals that take part in trials** (acute coronary syndromes) had lower in-hospital mortality: 5.9% at non-participants vs 3.5% at high enrollers (Majumdar 2008).
+- **Counterpoint.** A Cochrane review found no consistent difference in outcomes between patients inside vs outside RCTs (63 of 73 comparisons not significant; Vist 2007). So the gap reflects *who delivers* care more than trial participation itself.
+- **Not all settings show a gap.** Population-based CEA in Olmsted County (Hallett 1998) and a statewide Kentucky registry (Mayo 1998; stroke plus death 2.8%) roughly matched trial results. However, Mayo's registry covered only 58% of the state's procedures and 10 of 17 hospitals.
+
+**Vascular example.**
+- **CEA:** Wennberg 1998 (above).
+- **Carotid stenting:** trial thresholds vs practice. Calvet 2014 recommends ≥6 CAS/year, while the median Medicare operator did 3.0/year (Nallamothu 2011).
+- **EVAR:** in Medicare (Schermerhorn 2008), the early advantage of EVAR (perioperative mortality 1.2% vs 4.8%) carried into practice, showing that population data can confirm the direction of trial results.
+
+| Paper | PMID | Takeaway (numbers from abstract) |
+|---|---|---|
+| **Wennberg DE et al. 1998, *JAMA*** | 9565008 | 113,300 Medicare CEAs. Trial hospitals 1.4% vs non-trial 1.7–2.5% mortality (trend p < .001). All were above the trials' 0.6%/0.1%. "Caution … translating efficacy … to effectiveness". |
+| Hallett JW et al. 1998, *J Vasc Surg* | 9620136 | Olmsted County, all CEAs 1970–95 (n = 297 patients). Freedom from ipsilateral stroke at 2 years was 97% vs NASCET 91%. Higher 30-day stroke rate in patients over 80. |
+| Mayo SW et al. 1998, *J Vasc Surg* | 9652463 | Kentucky statewide registry: stroke plus death 2.8%, comparable to NASCET/ACAS. Covered 58% of CEAs. Surgeon stroke rates ranged 0–10%. |
+| Vist GE et al. 2007, *Cochrane* | 17443630 | 55 studies, 31,140 trial vs 20,380 non-trial patients. 63 of 73 comparisons not significant. Participation itself is not associated with better or worse outcomes. |
+| Majumdar SR et al. 2008, *Arch Intern Med* | 18362259 | Non-ST-elevation acute coronary syndromes (non-surgical). In-hospital mortality 5.9% / 4.4% / 3.5% by trial-enrolment tertile. High-enrolling hospitals aOR 0.8. |
+| Schermerhorn ML et al. 2008, *NEJM* | 18234751 | 22,830 matched pairs in Medicare. EVAR vs open perioperative mortality 1.2% vs 4.8%. By 4 years: rupture 1.8% vs 0.5% and AAA reintervention 9.0% vs 1.7%. Practice data add the late harms that short trials miss. |
+| Mao J et al. 2022, *J Vasc Surg* | 35181518 | VQI–Medicare indirect linkage: 93.0% matched, 99.4% accuracy. Registry procedure detail plus claims follow-up. |
+
+Already in 01: Rothwell 2005/2006 (15639683, 16871331), Cutteridge 2025 (40105904), Jones 2026 (42526742).
+
+**Can TTE solve it? Yes for claims, partly for registries.**
+- **Claims.** Medicare FFS includes essentially every operator who bills Medicare, so no surgeons or centres are selected in. The TTE estimand is the effect as practised, which is what policy needs.
+- **VQI.** Centres choose to join, and they tend to be engaged in quality improvement. Registry TTEs therefore carry a milder version of the trial's centre selection. VISION linkage helps with follow-up but not with which centres take part.
+- **What TTE cannot give** is the "ideal-operator" effect, unless operators are restricted to high-volume ones (a transportability or subgroup analysis).
+
+**Testable in a known-truth simulation? Yes.** Run a simulated RCT restricted to the top skill/volume quintile of physicians and compare its estimate with the population truth and with TTE on everyone. The gap is the generalizability bias, and TTE should close it.
+
+---
+
+## 4. Standardization, quality control, technique variation, device heterogeneity
+
+**What it is.** A surgical "treatment" is a bundle of steps, devices, and peri-operative care. Trials must decide how much to standardize. Too little, and nobody knows what was tested; too much, and the result may not transfer to practice. In endovascular trials, "PVI" can mean plain balloon, stent, drug-coated device, or atherectomy, in different vessel territories, and these may not share one effect.
+
+**How common / how big.**
+- Of 160 interventions in surgical RCTs, standardization was mentioned for only 29.4%, and only 28% of articles measured adherence (Blencowe 2015).
+- In placebo-controlled surgical trials, only 7% attempted to standardize and 4% monitored delivery (Cousins 2020; Section 6).
+- In arteriovenous-graft RCTs, quality assurance had "not improved over the past three decades", and no trial assessed technical performance (Kingsmore 2025).
+
+**Vascular / claudication example.**
+- **Tibial PVI for claudication.** In Medicare 2017–19, 27.7% of index claudication PVIs included tibial treatment. The physician-level rate had a median of 20.0% (IQR 9.1–37.5%) and was associated with early-career status (aOR 2.97) and office-based practice (aOR 2.37) (Bose 2023). Infrapopliteal vs femoropopliteal PVI had a higher risk of CLTI (aHR 1.31) and amputation (aHR 1.72) (Bose 2025). The "intervention" is therefore technically heterogeneous, and the heterogeneity varies by physician.
+- **Devices.** A network meta-analysis of femoropopliteal devices found that technical success and restenosis differed substantially by device class (Katsanos 2014).
+
+| Paper | PMID | Takeaway (numbers from abstract) |
+|---|---|---|
+| **Blencowe NS et al. 2015, *Br J Surg*** | 26041565 | 80 surgical RCTs / 160 interventions. Described 80.6%, standardization mentioned 29.4%, adherence measured in 28% of articles, operator expertise reported for 34.4%. Pragmatic and explanatory trials were similar. |
+| Blencowe NS et al. 2016, *Br J Surg* | 27462835 | Typology for standardizing surgical interventions: purpose → components and steps → monitoring of deviations. |
+| Butterworth JW et al. 2021, *Eur J Surg Oncol* | 33059943 | 34 oncology studies. Quality-assurance challenges fall into credentialing (volume is a weak criterion), standardization (variation between centres), and monitoring. |
+| Kingsmore DB et al. 2025, *J Vasc Access* | 38501338 | Vascular-access (AV graft) RCTs: quality assurance highly variable, no pre-trial monitoring, none assessed technical performance. |
+| Katsanos K et al. 2014, *J Vasc Surg* (network meta-analysis) | 24661896 | 16 femoropopliteal RCTs. Technical success OR 13.6 for covered stents vs balloon. Target lesion revascularization RR 0.36 for paclitaxel-coated balloons. Device classes are not interchangeable. |
+| **Bose S et al. 2023, *J Vasc Surg*** | 36058433 | Medicare claudication PVI (n = 59,930; 1,542 physicians). Tibial PVI 27.7%. Physician-level rate IQR 9.1–37.5%. Early career aOR 2.97, office-based aOR 2.37. |
+| Bose S et al. 2025, *Eur J Vasc Endovasc Surg* | 38906366 | Medicare claudication PVI (n = 36,147). Infrapopliteal vs femoropopliteal: CLTI aHR 1.31, repeat PVI aHR 1.12, amputation aHR 1.72. |
+| Sansosti AA et al. 2024, *J Vasc Surg* (VQI) | 38944400 | Femoropopliteal atherectomy. Embolic protection used in 40% (hospital) vs 4.4% (office). Office setting had more reintervention at 2 years. Technique varies by setting. |
+
+Already in 01: Boutron 2017 (28630973), Garas 2018 (29110271), Jones 2026 (42526742).
+
+**Can TTE solve it? Partly.**
+- **TTE cannot standardize anything.** The "early PVI" strategy is whatever PVI was actually done.
+- **Claims.** CPT codes record vessel territory and modality (angioplasty, stent, atherectomy) and the setting, so the strategy can be defined more finely, or modality treated as a version of treatment. Device brand, drug coating (only partly, via HCPCS pass-through codes), lesion length, and technical success are not recorded.
+- **Registries** (VQI) record lesion characteristics, devices, and technical details, so technique variation is largely measurable there, and "versions of treatment" can be modelled explicitly.
+- **The open question** is whether a single "early PVI" effect is meaningful when the technique mix differs by physician. This is a consistency (well-defined intervention) problem.
+
+**Testable in a known-truth simulation? Yes.** Give each physician a probability of a high-risk technique (e.g., tibial PVI) with its own effect. Then compare a TTE that ignores versions (estimating the effect of the as-practised mixture) with one stratified on modality.
+
+---
+
+## 5. Clustering of outcomes by surgeon
+
+**What it is.** Patients treated by the same surgeon or centre have correlated outcomes, even under individual randomization. Ignoring this understates the standard errors and, when provider effects are tied to volume or treatment, can bias the effect estimate. Few trials handle it.
+
+**How common / how big.**
+- **Trials rarely adjust.** Of 68 large interventional RCTs, 16% reported the number of providers, 9% stratified randomization by provider, and 4% adjusted the analysis for provider (Biau 2008, *JCE*).
+- **The picture is little better ten years later.** Of 247 surgical RCTs, 40% of trials stratified by surgeon did not adjust the analysis for surgeon (Conroy 2019).
+- **Re-analysis changes results.** Re-analysing three surgical RCTs found significant provider effects. In the hip-replacement trial, the treatment effect *became* significant after adjusting for provider and volume (Biau 2008, *Ann Surg*).
+- **Size of clustering.** Cook 2012 reported ICCs for 198 outcomes from 10 trials and concluded that clustering is "more of an issue than has been previously acknowledged" (no single pooled ICC in the abstract).
+
+**Vascular example.** Physician-level variation in claudication PVI technique is wide (tibial PVI rate IQR 9.1–37.5%; Bose 2023), and the analysis used hierarchical models. In NCDR, 55,785 PVIs were performed by 555 operators, an average of about 100 per operator (Chunawala 2025). Clustering in claims PVI data is therefore substantial in both technique and volume.
+
+| Paper | PMID | Takeaway (numbers from abstract) |
+|---|---|---|
+| **Biau DJ et al. 2008, *J Clin Epidemiol*** | 18394535 | 68 multicentre interventional RCTs (n ≥ 200). Number of providers reported 16%. Stratified by provider 9%. Analysis adjusted for provider 4% and for centre 6%. |
+| **Biau DJ et al. 2008, *Ann Surg*** | 18438129 | Patient-level re-analysis of 3 RCTs (22–88 providers each). Significant provider effects (ophthalmology p < .0001; orthopaedics p = .0037). Treatment × volume interaction. Adjustment changed the conclusion. |
+| Cook JA et al. 2012, *Trials* | 22217216 | ICCs for 198 outcomes in 10 surgical trials. Median cluster size 32 (centre) and 6 (surgeon). ICCs varied widely by outcome type. |
+| **Conroy EJ et al. 2019, *J Clin Epidemiol*** | 30439544 | 247 surgical RCTs. Expertise-based design 2%, credentials predefined 39%, only 1 trial analysed learning. One-third of centre-stratified and 40% of surgeon-stratified trials did not adjust the analysis. |
+| Conroy EJ et al. 2019, *J Clin Epidemiol* (funded applications) | 31121302 | 50 funded UK trials. 60% of surgeon-stratified trials planned to adjust. One-third of referee responses involved funder-driven changes on learning or clustering. |
+| Conroy EJ et al. 2022, *Trials* (guidance review) | 36221107 | 28 guidance documents. Clustering by centre is covered well, by surgeon poorly. Analysis methods for clustering in 5 documents and for learning in 4. No single guidance document exists. |
+
+Already in 01: Lee & Thompson 2005 (15649931, 16279138), Roberts 2005 (16279137), Walters 2010 (20492055). Analysis methods in 02 B6 (Panageas, Kahan & Morris, within-physician designs).
+
+**Can TTE solve it? Yes for inference, partly for confounding.**
+- **Claims and registries** both have operator identifiers (NPI; VQI physician ID), so cluster-robust SEs, random effects, physician fixed effects, or within-physician designs are standard.
+- **Clustering is larger in TTE than in a trial**, because physicians choose the strategy. Physician-level clustering therefore becomes **confounding by physician** (preference correlated with outcome style), not just a precision problem (see 02 B6).
+- **Within-physician comparisons** need physicians who sometimes intervene early and sometimes defer.
+
+**Testable in a known-truth simulation? Yes.** Random intercepts and random slopes by physician, with the intercept correlated with intervention preference. Check (a) the coverage of naive vs cluster-robust vs mixed-model SEs and (b) bias with and without within-physician adjustment.
+
+---
+
+## 6. Blinding, sham control, and outcome-assessment (detection) bias
+
+**What it is.**
+- Surgeons cannot be blinded, patients rarely are, and outcome assessors often are not.
+- Unblinded subjective outcomes (pain, symptom scores, self-limited walking distance) are prone to expectation and observer bias.
+- Sham-controlled trials of invasive cardiovascular procedures have repeatedly shrunk effects seen in open-label trials.
+- Observational data have their own version of the problem. Outcomes are recorded when patients are looked for (surveillance bias), and diagnoses are coded by the treating physician.
+
+**How common / how big.**
+- **Non-blinded assessors exaggerate effects.** By 36% for subjective binary outcomes (Hróbjartsson 2012) and by 68% for measurement-scale outcomes (Hróbjartsson 2013). In an updated 2025 analysis the figure was 29% overall and larger in non-drug trials (ratio of odds ratios 0.62; Salazar 2025).
+- **The bias is in subjective outcomes, not objective ones.** In a meta-epidemiological study, lack of blinding biased subjective outcomes (ROR 0.75) but not objective ones (ROR 1.01) (Wood 2008).
+- **Blinding is under-used in surgery.** In 378 surgical RCTs only 24.0% of patients were blinded. Failure to blind surgeons where it was feasible was associated with more significant results (OR 13.7; very wide CI 1.3–142.8) (Probst 2019).
+
+**Vascular / claudication example.**
+- **Sham analogue for claudication.** The closest is ORBITA, which tested revascularization for an exertional symptom against a sham procedure. PCI did not improve exercise time beyond placebo (16.6 s, −8.9 to 42.0). ORBITA-2, in patients off antianginals, did show a symptom benefit (OR 2.21).
+- **Renal denervation.** SYMPLICITY HTN-3 found that the large blood-pressure falls in unblinded studies largely disappeared against sham (difference −2.39 mmHg, not significant).
+- **Claudication trials.** Claudication revascularization trials (ERASE, CLEVER, IRONIC, SWEDEPAD 2; in 01) use unblinded walking distance or quality of life.
+- **Claims outcomes.** The usual claims endpoints (amputation, death, repeat PVI) are objective in Wood's sense. Even so:
+  - ICD-10 coding distinguishes claudication from CLTI with only 81.3% agreement against VQI (Bose 2025).
+  - About half of PVI patients get no surveillance imaging in year 1 (Brooke 2018), so restenosis and repeat PVI depend on who looks.
+
+| Paper | PMID | Takeaway (numbers from abstract) |
+|---|---|---|
+| **Hróbjartsson A et al. 2012, *BMJ*** | 22371859 | 21 trials with both blinded and non-blinded assessors. Non-blinded assessors exaggerated the OR by 36% (ROR 0.64), driven by misclassifying a median of only 3% of patients. |
+| Hróbjartsson A et al. 2013, *CMAJ* | 23359047 | Measurement-scale outcomes, 16 trials. Non-blinded assessors exaggerated effect size by 68% (95% CI 14–230%). |
+| Salazar J et al. 2025, *J Clin Epidemiol* (update) | 40258524 | 66 trials. Pooled ROR 0.71, about a 29% exaggeration. Larger in non-drug trials (ROR 0.62) and industry-funded trials (0.57). |
+| **Wood L et al. 2008, *BMJ*** | 18316340 | Lack of blinding: ROR 0.75 for subjective outcomes vs 1.01 for objective outcomes. Little bias for all-cause mortality. |
+| Probst P et al. 2019, *Langenbecks Arch Surg* | 30824993 | 378 surgical RCTs. 24.0% of patients blinded. Assessor blinding was possible in 69%. Not blinding feasible surgeons was associated with significant results (OR 13.7). |
+| Cousins S et al. 2020, *J Clin Epidemiol* | 31786153 | 96 placebo-controlled surgical RCTs. 68% randomized fewer than 100. 34% gave no rationale for the placebo. Standardization 7%, monitoring 4%. |
+| **Al-Lamee R et al. 2018, *Lancet* (ORBITA)** | 29103656 | PCI vs sham in stable angina, n = 200. Exercise-time difference 16.6 s (−8.9 to 42.0), not significant. |
+| Rajkumar CA et al. 2023, *NEJM* (ORBITA-2) | 38015442 | n = 301, off antianginals. Angina score 2.9 vs 5.6 (OR 2.21). A sham can show *real* benefit. |
+| Bhatt DL et al. 2014, *NEJM* (SYMPLICITY HTN-3) | 24678939 | n = 535. Systolic BP −14.1 vs −11.7 mmHg with sham. Difference −2.39 (−6.89 to 2.12). |
+| **Bose S et al. 2025, *Circ Cardiovasc Qual Outcomes*** | 40439664 | VQI-VISION, n = 54,180 PVIs. ICD-10 claudication vs CLTI: sensitivity 80.9%, specificity 81.9%, agreement 81.3%. Machine learning raised the AUC to 0.892. |
+| Bilimoria KY et al. 2013, *JAMA* | 24100354 | Surveillance bias: hospital VTE rates rose from 5.0 to 13.5 per 1,000 across quartiles of imaging use. More looking, more events found. |
+| Brooke BS et al. 2018, *JAMA Surg* | 29049809 | Medicare. Only 52–58% of patients had imaging follow-up within 1 year after bypass or PVI. Joining VQI did not change this (RR about 0.99). |
+| Jacob-Brassard J et al. 2021, *Clin Invest Med* (Ontario) | 34152705 | Administrative PAD revascularization codes: PPV 94.3%, or 98.1% with physician claims. Procedures are coded accurately. |
+
+Already in 01: Wartolowska 2014/2016 (24850821, 27008687), Boutron 2017 (28630973).
+
+**Can TTE solve it? No for blinding, partly for outcome bias.**
+- **No blinding.** TTE cannot blind anyone.
+- **Claims.** These have no patient-reported outcomes or walking distance, so the subjective endpoints most open to bias are simply not available. The objective claims endpoints (death, major amputation) are the ones Wood 2008 found free of blinding bias. Physician-recorded endpoints remain at risk:
+  - CLTI diagnosis codes (81% agreement with the registry).
+  - Repeat PVI, which is itself a physician decision.
+  - Restenosis, which depends on surveillance (Brooke 2018).
+- **Registries** record walking or symptom status and patency, but these are clinician-reported and unblinded. Registries add detail without solving detection bias.
+- **Practical rule for the paper:** make death and major amputation primary; treat CLTI and repeat PVI as at risk of detection or decision bias.
+
+**Testable in a known-truth simulation? Yes, optionally.** Make the probability that CLTI is *recorded* (or that repeat PVI is triggered) depend on prior PVI or physician style (differential ascertainment), and keep amputation and death recorded perfectly. TTE should be unbiased for the objective outcomes and biased for the ascertainment-dependent ones. Blinding itself has no counterpart in the simulation.
+
+---
+
+## Table 1 inputs (one line each)
+
+| Challenge | Evidence it is common | Vascular example | TTE: claims | TTE: registry (VQI/VISION) | Simulation |
+|---|---|---|---|---|---|
+| Learning curve | Plateau at about 250 cases (Vickers); 1/247 RCTs analysed learning (Conroy) | CAS Medicare early-experience aOR 1.7 (Nallamothu) | Partly: cumulative NPI volume | Partly: plus technical success | Yes. Interference, delayed arm treated by more skilled operators, time-varying physician confounding |
+| Skill and expertise | Skill explains 25.8% of complication variation (Stulberg) | PVI operator volume MALE OR 0.73 (Chunawala) | Partly: proxies only | Partly | Yes: latent skill × preference |
+| Surgeon and centre selection | CEA trial vs non-trial hospitals 1.4% vs up to 2.5% (Wennberg) | CEA, CAS volume thresholds | Yes: all operators | Partly: centres self-select | Yes: "expert-only RCT" vs population |
+| Standardization and devices | 29.4% of trial interventions standardized (Blencowe) | Tibial PVI physician IQR 9–38% (Bose 2023) | Partly: CPT modality and territory | Mostly: lesion and device data | Yes: versions of treatment |
+| Clustering | 4% of RCTs adjusted for provider (Biau) | 555 operators in NCDR PVI | Yes (inference); confounding remains | Yes | Yes: random intercepts and slopes |
+| Blinding and detection | Non-blinded assessors +36% / +68% (Hróbjartsson) | ORBITA-type sham; CLTI code agreement 81% | No blinding; objective endpoints are fine | No; clinician-reported | Optional: differential ascertainment |
+
+---
+
+## Summary: most important new papers
+1. **Wennberg 1998 *JAMA* (9565008):** the canonical trial vs non-trial centre gap for CEA in Medicare. It anchors the argument that claims-based TTE estimates the effect as practised.
+2. **Nallamothu 2011 *JAMA* (21954477):** a learning curve and volume effect measured *in Medicare claims* through operator identifiers, for an endovascular procedure. It shows that operator experience can be modelled in claims-based TTE.
+3. **Biau 2008 *JCE* / *Ann Surg* (18394535, 18438129) and Conroy 2019 *JCE* (30439544):** surgical RCTs almost never account for provider effects or learning, and doing so can change conclusions.
+4. **Hróbjartsson 2012 *BMJ* (22371859) and Wood 2008 *BMJ* (18316340):** unblinded subjective outcomes are inflated by about 30–70%, while objective outcomes are not. This justifies TTE using death and amputation and is cautious about walking distance and CLTI coding.
+5. **Bose 2023 *JVS* (36058433) and Bose 2025 *CCQO* (40439664):** in claudication PVI, technique varies widely by physician, and the claims coding of CLTI is about 81% accurate against VQI. These define the "versions of treatment" and outcome-misclassification limits for claims vs registry TTE.

docs/literature/drafts/C-pathway-and-tte.md

@@ -0,0 +1,330 @@
+# Draft C: treatment-pathway challenges, claudication trials, and evidence on TTE itself
+
+Every PMID below was checked with NCBI E-utilities (esummary) on 2026-10-28. Every number comes from the PubMed abstract fetched with efetch; full texts were not read. If a paper has no PubMed abstract, the entry says so and gives no numbers. Papers already in `01-surgical-rct-limitations.md` or `02-claudication-evidence-and-methods.md` are cited by PMID only and are not listed again as new entries, apart from short reminders in the claudication table.
+
+"TTE" here covers emulation in **claims** (e.g., Medicare), **EHR** (e.g., VA, health-system data), and **clinical registries** (e.g., Vascular Quality Initiative [VQI], and VQI linked to Medicare as VQI-VISION). Where the data source changes the answer, the "Can TTE solve it?" paragraphs say so.
+
+---
+
+# Part 1. Treatment-pathway challenges
+
+## Challenge 1. Crossover and contamination
+
+**What it is.** Patients receive the other arm's treatment. In timing trials and conservative-vs-surgery trials, crossover runs mostly one way: the conservative or "watchful waiting" arm later has the operation. The intention-to-treat (ITT) contrast then compares "operate now" with "operate later if needed", not with "never operate", and the difference shrinks toward the null. Naive per-protocol or as-treated analyses (excluding or reclassifying crossovers) remove the dilution but break randomization, because the patients who cross over are the ones who are getting worse.
+
+**How common / how big.**
+- Crossover from conservative care to surgery is the norm, not the exception: 39% by about 19 weeks for sciatica, 30% by 6 months for meniscal tear, 43% by 2 years for spinal stenosis, 27–29% within 90 days to 1 year for appendicitis, and two-thirds by 10 years for hernia.
+- In cardiac surgery RCTs the median crossover was 7.0% from experimental to control. Trials with more of this crossover had RRs nearer the null (1.01 vs 0.80).
+- STICH is the clearest case where crossover changed the conclusion. The ITT result was not significant at 5 years (HR 0.86, P = 0.12). Per-protocol and time-dependent analyses favored CABG. Benefit was significant by ITT only after 10 years of follow-up.
+- Per-protocol estimates were on average only 2% larger than ITT across 156 RCTs in general medical journals. The gap grows with non-adherence, and surgical timing trials sit at the high end of non-adherence.
+
+**Vascular / claudication example.**
+- SUPER (iliac disease): 29% of the SET arm had endovascular revascularization within 1 year, and 49% had an additional intervention by 5.5 years.
+- Swedish SFA stenting trial: 25% of the control arm had crossed over to stenting by 60 months. The authors say this "affected the power of the final analysis".
+- ERASE: the exercise arm had a higher hazard of later revascularization (HR 2.50).
+- ACST-1 (carotid): the "deferral" arm still had CEA at 4% per year.
+- Already listed elsewhere: ADAM 61.6% (12000813) and CAESAR 84.5% at 54 months (20869890).
+
+| Paper | PMID | Takeaway |
+|---|---|---|
+| Peul WC et al. 2007, *NEJM* (sciatica: early surgery vs prolonged conservative care) | 17538084 | 89% of the early-surgery arm had surgery (mean 2.2 weeks). 39% of the conservative arm had surgery (mean 18.7 weeks). Disability at 1 year was no different by ITT, but recovery was faster with early surgery (HR 1.97). |
+| Katz JN et al. 2013, *NEJM* (METEOR) | 23506518 | **30%** of the physical-therapy arm had surgery within 6 months, and 6% of the surgery arm did not have surgery. No ITT difference. |
+| Weinstein JN et al. 2008, *NEJM* (SPORT stenosis) | 18287602 | At 2 years, 67% of the surgery arm and **43% of the nonsurgical arm** had surgery. ITT was significant for only 1 of 3 primary outcomes. The as-treated analysis (adjusted, combined with the observational cohort) favored surgery on all of them. |
+| CODA Collaborative 2020, *NEJM* | 33017106 | 29% of the antibiotics arm had appendectomy by 90 days (41% of those with an appendicolith). |
+| Salminen P et al. 2015, *JAMA* (APPAC) | 26080338 | 27.3% of the antibiotics arm had appendectomy within 1 year. |
+| van de Wall BJM et al. 2017, *Lancet Gastroenterol Hepatol* (DIRECT) | 28404008 | Stopped early for poor recruitment (n = 109). 23% of the conservative arm went on to elective resection. |
+| Velazquez EJ et al. 2011, *NEJM* (STICH) | 21463150 | 17% of the medical-therapy arm had CABG by a median of 56 months, and 91% of the CABG arm had CABG. ITT on all-cause death: HR 0.86 (0.72–1.04). |
+| Velazquez EJ et al. 2016, *NEJM* (STICHES) | 27040723 | At a median of 9.8 years, ITT HR 0.84 (0.73–0.97). The benefit appeared only with long follow-up. |
+| Doenst T et al. 2013, *Circ Heart Fail* (STICH crossover) | 23515275 | Crossover in year 1: 10.8% from MED to CABG (driven by worsening symptoms) and 9.0% from CABG to MED (mainly patient or family decision). Per-protocol and time-dependent analyses favored CABG. Crossover "diminished the impact of CABG" in the ITT analysis. |
+| **Gaudino M et al. 2019, *JAHA*** | 31663420 | 47 cardiac surgery RCTs. Median crossover was 7.0% from experimental to control and 1.3% from control to experimental. Experimental-to-control crossover was associated with effect estimates nearer the null. |
+| Reistrup H et al. 2021, *Hernia* (systematic review) | 32910297 | Watchful waiting for hernia: about one-third crossed over by 3 years and more than two-thirds by 10 years, mainly for pain. |
+| Halliday A et al. 2004, *Lancet* (ACST-1) | 15135594 | Immediate-CEA arm: half had CEA by 1 month and 88% by 1 year. Deferral arm: 4% per year had CEA. The authors note that full compliance would have produced "slightly bigger differences". |
+| Hernán MA, Hernández-Díaz S 2012, *Clin Trials* | 21948059 | ITT does not guarantee an estimate of clinical effectiveness. With differential adherence, ITT can even overestimate. Adherence-adjusted per-protocol analyses need IPW or g-estimation and time-varying data. |
+| Murray EJ, Hernán MA 2016, *Clin Trials* | 26951361 | Coronary Drug Project: placebo adherers vs non-adherers differed by 9.4 percentage points in 5-year mortality. With modern adjustment (IPW for post-randomization covariates) the gap shrank to 2.5 (−2.1 to 7.0). Valid per-protocol estimation is possible. |
+| Mostazir M et al. 2021, *J Clin Epidemiol* | 34161805 | Across 156 RCTs, per-protocol estimates were 2% larger than ITT (ratio of ORs 1.02), and the gap grew with non-adherence. |
+| Peduzzi P et al. 1993, *Stat Med* (VA CABG trial) | 8210821 | Compares ITT with four naive ways of handling non-adherers (exclude, transfer at randomization, censor, transfer at switch) and describes the bias in each. |
+| Morden JP et al. 2011, *BMC Med Res Methodol* | 21223539 | Simulation study (16 scenarios, 1,000 datasets each). Simple switching methods are biased when switchers' prognosis differs. Adjusted (AFT-type) methods do better. |
+| Latimer NR et al. 2014, *Med Decis Making* | 24449433 | IPCW relies on no unmeasured confounding of switching. RPSFTM relies on a common treatment effect. Both can be biased, and ITT is sometimes the least biased. |
+
+**Can TTE solve it? Yes, in principle; this is TTE's strongest case.** TTE defines the strategies explicitly ("PVI within 6 months" vs "no PVI for 6 months, then usual care", or "no PVI unless CLTI"). Anyone who deviates is artificially censored, and IPCW corrects for the selection this causes (Hernán 2006, 16611197; Hernán 2018, 29419381). This estimates the per-protocol effect that the RCTs above could only approximate with naive as-treated analyses. The price is that the weights are valid only if the *triggers* for crossover are measured. In timing trials the main trigger is symptom progression (STICH: "progressive symptoms"; hernia: pain).
+- **Claims** do not record symptom severity, walking distance or ABI, so the weights miss the main reason patients cross over.
+- **EHR and VQI** may record ABI, Rutherford class and symptoms. But VQI records them only at the time of a procedure, so they are missing for patients who stay conservative.
+- The same IPCW-with-measured-triggers assumption sinks trial switching adjustments (Latimer 2014). TTE does not escape it.
+
+**Testable in a known-truth simulation? Yes.** Generate deferred PVI driven by a latent progression variable, then vary whether that variable is observed (EHR-like) or not (claims-like), and compare ITT-analogue, naive as-treated, and CCW estimates against the true per-protocol effect.
+
+---
+
+## Challenge 2. Non-adherence and refusal of the assigned surgery (and of SET)
+
+**What it is.** Some patients assigned to surgery never have it: they decline, become unfit, or die waiting. Some patients assigned to a conservative package never start it or drop out. In claudication the conservative package is supervised exercise, which needs about 3 sessions a week for 12 weeks, so adherence is a major problem. Refusal of the assigned operation differs from crossover in both cause and direction: it is often driven by patient preference or frailty, not by disease progression.
+
+**How common / how big.**
+- **Refusal or delay of the assigned operation:** ACST-1 managed CEA within 1 month in only half of the immediate-CEA arm. In STICH, 9.0% of the CABG arm received medical therapy only, mainly because of patient or family decision, and that group had higher 5-year mortality (59% vs 33%). SPORT disc herniation: only 50% of the surgery arm had surgery (01 list, 17119140).
+- **SET has low uptake and adherence:**
+  - Across 23 reports that gave screening numbers, only **24.2%** of screened claudicants were recruited to supervised exercise (Harwood 2016).
+  - In a health system with a strong SET program, 773 of 5,320 PAD patients were referred, 415 enrolled, and **49.9%** of those completed (Whipple 2024).
+  - Of 51 Chicago PAD patients who had started SET, **9.8% completed 12 weeks** (Cetlin 2023).
+
+**Vascular / claudication example.**
+- SUPER: compliance with SET was **50%** at 6 months, and 8% of the endovascular arm never received the intervention.
+- CLEVER: only 79 of 111 randomized patients completed the 18-month assessment (attrition, not refusal).
+- In US practice, 85.1% of surveyed PAD patients said SET had never been prescribed (Cetlin 2023), and 49% of surveyed vascular physicians had never referred a patient (Dua 2020).
+
+| Paper | PMID | Takeaway |
+|---|---|---|
+| Doenst T et al. 2013, *Circ Heart Fail* | 23515275 | CABG-to-MED crossover was mostly patient or family choice. These patients had worse prognosis (5-year mortality 59% vs 33%), so they were not a random subset. |
+| Halliday A et al. 2004, *Lancet* (ACST-1) | 15135594 | Even in the "immediate" arm, only half had CEA by 1 month. "Immediate" in practice meant a distribution of delays. |
+| Koelemay MJW et al. 2022, *EJVES* (SUPER) | 35151572 | SET compliance was 50% at 6 months. 8% of the ER arm did not get ER. |
+| **Harwood AE et al. 2016, *Ann Vasc Surg*** | 27126713 | Systematic review: 24.2% of 7,517 screened claudicants were recruited to supervised exercise, and 75.1% of 4,012 completed it. Only one article defined a minimum attendance. About 1 in 3 screened patients was suitable and willing. |
+| Whipple MO et al. 2024, *Vasc Med* | 38084864 | One US health system, 2017–2022: of 5,320 PAD patients, 773 were referred to SET, 415 enrolled, and 49.9% completed. |
+| Cetlin MD et al. 2023, *J Vasc Surg* | 36150636 | Of 489 people with PAD, 85.1% had never had SET prescribed. Of those willing to travel, 59.9% were unwilling or unable to pay the $11 copay per session. Of 51 who started, 9.8% completed 12 weeks. |
+| Dua A et al. 2020, *J Vasc Surg* | 31699514 | Survey of 135 physicians: 54% had no SET program at their facility, 49% had never referred a patient, and 26% did not know CMS covered SET. |
+| Kahan BC et al. 2024, *BMJ* (estimands primer) | 38262663 | The ICH E9(R1) estimands framework, worked through on an emergency bowel surgery trial. It names how "intercurrent events" (refusal, switching) are handled: the treatment-policy (ITT) estimand vs the hypothetical (per-protocol) estimand. |
+
+**Can TTE solve it? Partly.** TTE avoids the *consent* step, so trial refusal does not exist. But refusing an *offered* operation still happens in routine care.
+- **Claims** cannot separate "offered surgery and declined" from "never offered", so the comparator arm absorbs refusers, who may be frailer or more risk-averse. This is unmeasured confounding.
+- **EHR** notes sometimes record refusal or shared-decision documentation.
+- **Adherence to SET** is partly visible. SET has been a Medicare-covered, billable service since 2017 (Dua 2020), so claims can count sessions. Home exercise and walking advice cannot be seen in any of these data sources.
+- **Adherence to the surgery itself** is easy to observe in all three sources: a procedure either happens or it does not.
+
+**Testable in a known-truth simulation? Yes.** Add a latent "refuser" trait that lowers the chance of PVI and independently raises the risk of the outcome. A claims-like scenario leaves it unobserved; an EHR-like scenario observes it with misclassification.
+
+---
+
+## Challenge 3. Co-interventions and heterogeneous usual care in the comparator arm
+
+**What it is.** The outcome depends on everything else patients receive, not just the index procedure: statins, antiplatelets, smoking cessation, supervised or home exercise, cilostazol, and surveillance. In claudication trials the "conservative" arm ranges from "aspirin plus advice" to "12 weeks of supervised treadmill training plus best medical therapy". Usual care is undefined by design and varies between sites, so the comparator is a moving target. Co-interventions can also differ *by arm* because nobody is blinded: patients who had a procedure get more follow-up visits and more statin prescriptions.
+
+**How common / how big.**
+- Usual-care comparators are reported very poorly. In 67 knee-OA exercise RCTs, the median TIDieR score was 1 of 12 for the usual-care arm vs 4 for the exercise arm.
+- Exercise arms in claudication RCTs scored on average 10 of 19 on CERT, with adherence and fidelity the least reported items.
+- In practice, only **38.1%** of VQI claudicants undergoing elective revascularization had complete optimal medical therapy (antiplatelet, statin, non-smoking). Complete OMT was associated with lower 1-year reintervention (aOR 0.72).
+- Preoperative statin-plus-antiplatelet use in VQI held at about 62% from 2013 to 2023.
+
+**Vascular / claudication example: the comparator differs from trial to trial.**
+- Whyman: aspirin plus advice on smoking and exercise.
+- CLEVER: "optimal medical care" (claudication pharmacotherapy), with or without SE.
+- IRONIC: BMT plus a structured *non-supervised* exercise program (Djerf 2018).
+- MIMIC: SE plus BMT in both arms.
+- Cochrane 2018: "conservative" meant supervised exercise in some trials and cilostazol in others.
+- A meta-analysis therefore pools different comparators.
+
+| Paper | PMID | Takeaway |
+|---|---|---|
+| Thompson BT, Schoenfeld D 2007, *Proc Am Thorac Soc* | 17878473 | Usual care is "the full spectrum of patient care practices". Whether to use it as a control depends on the question and on how uniform usual care is. |
+| Freedland KE et al. 2011, *Psychosom Med* | 21536837 | Extrinsic, non-study care has "important but under-recognized" effects on trials with usual-care control groups. |
+| Ribeiro da Silva Neto J et al. 2026, *Musculoskeletal Care* | 42366407 | 67 RCTs: median TIDieR score 1 for the usual-care arm vs 4 for the exercise arm. No improvement over time. |
+| Ibeggazene S et al. 2023, *Vascular* | 35130092 | 73 claudication exercise RCTs: mean CERT 10/19. Adherence and fidelity were the least reported items. |
+| Fakhry F et al. 2018, *Cochrane* | 29518253 | Across 10 RCTs (1,087 patients), "conservative" meant supervised exercise or cilostazol. Effects differed by comparator: ER vs no specific treatment MWD SMD 0.70; ER vs SET −0.42 (not significant). |
+| **Jarosinski MC et al. 2025, *J Vasc Surg* (VQI)** | 39208918 | Of 39,088 claudication revascularizations: 38.1% complete OMT, 56.4% partial, 5.4% none. Complete OMT was associated with lower reintervention (aOR 0.72) and MALE (aOR 0.70). |
+| Ramkumar N et al. 2026, *J Vasc Surg* (VQI) | 42248488 | Of 257,129 patients, preop statin plus antiplatelet use held at 62%. Postop use rose from 68% to 81%. Lower in women (OR 0.88). |
+| Jansen SCP et al. 2020, *EJVES* (Dutch national claims) | 32888779 | 54,504 new IC patients. Primary SET rose from 63% to 87% (2013–2017). **Claims captured SET because it is reimbursed there.** Primary ER was associated with more secondary revascularizations (HR 1.44) and higher mortality (HR 1.38). These are observational associations. |
+| Djerf H et al. 2018, *Br J Surg* (IRONIC cost) | 30357819 | Describes the IRONIC control arm as BMT "including a structured, non-supervised exercise programme". |
+| Divakaran S et al. 2021, *Circ Cardiovasc Qual Outcomes*; 2024, *Vasc Med* | 34293930; 39129558 | Medicare SET participation 2017–2018, and "underutilization" of SET in Medicare. **No PubMed abstract (research letters), so no numbers are given here.** |
+
+**Can TTE solve it? Partly.** TTE cannot randomize co-interventions. It can either (a) build them into the strategy definition ("PVI plus statin" vs "no PVI plus statin") or (b) treat them as time-varying confounders or mediators. Both need the co-intervention to be measured.
+- **Claims.** Part D captures statin and P2Y12 fills, but not OTC aspirin. Medicare SET is billable from 2017. Smoking is captured poorly, and home exercise not at all.
+- **EHR.** May record SET referral, smoking status, ABI and symptom class.
+- **VQI.** Records OMT and smoking only *at the time of a procedure* (Jarosinski 2025). Patients who are never revascularized are not in the registry, so **VQI alone cannot form a "no PVI" comparator arm**. Linking it to claims or EHR is required.
+- **A known advantage over RCTs.** The usual-care comparator in an emulation is real-world usual care, so it is representative. It is just not standardized.
+
+**Testable in a known-truth simulation? Yes.** Add an unmeasured exercise or OMT variable that differs by arm (more OMT after PVI) and affects outcomes. Compare scenarios where it is observed (EHR/Part D) and unobserved (exercise in claims).
+
+---
+
+## Challenge 4. Defining the comparator: "conservative" vs "delayed surgery" vs "never surgery"
+
+**What it is.** "Conservative management" is a *dynamic strategy*: start without the operation, and operate if and when a trigger occurs (symptoms, patient request, progression). It is not "never operate". Trials label it "watchful waiting", "deferral", "surveillance", "initial conservative", or "prolonged conservative treatment with surgery if needed". These labels name different estimands. A TTE has the same problem in sharper form, because the analyst has to write the strategy down: the trigger, the grace period, and what happens after the grace period.
+
+**How common / how big.**
+- Nearly every trial in Challenge 1 used a "surgery-if-needed" comparator, and 23–68% of the comparator arm had the operation.
+- ISCHEMIA explicitly defined its comparator as "medical therapy alone and angiography if medical therapy failed".
+- The same data analysed as a CCW emulation vs a sequential emulation can answer different questions with different numbers. Example (mAb timing): CCW 4-year survival difference 4.3% vs sequential −2.8%. A naive design gave an implausible RR of 0.13.
+
+**Vascular / claudication example.**
+- IRONIC and the Gothenburg trial (Nordanstig 2011) allowed patients in the noninvasive arm to "request invasive treatment if they deteriorated".
+- SUPER: the authors suggest it is "reasonable to start with SET … and accept a 30% failure rate".
+- In PVI-for-claudication terms, three candidate comparators answer different questions:
+  - "no PVI within 6 months" (a deferral estimand);
+  - "no PVI unless CLTI develops" (a progression-triggered dynamic strategy);
+  - "never PVI" (an abstention estimand, hard to emulate because positivity fails once CLTI appears).
+
+| Paper | PMID | Takeaway |
+|---|---|---|
+| Maron DJ et al. 2020, *NEJM* (ISCHEMIA) | 32227755 | Comparator defined as "medical therapy alone and angiography if medical therapy failed". Early harm from the invasive arm (5.3% vs 3.4% at 6 months) crossed over to a non-significant later benefit (16.4% vs 18.2% at 5 years). The timing of effects depends on the strategy definition. |
+| Halliday A et al. 2004, *Lancet* (ACST-1) | 15135594 | Comparator was "indefinite deferral of any CEA", which still produced CEA at 4% per year. |
+| Nordanstig J et al. 2011, *EJVES* | 21397530 | n = 201, invasive vs noninvasive. Noninvasive patients could request invasive treatment if they deteriorated. 7% of the invasive arm crossed over. |
+| Hernán MA et al. 2006, *Basic Clin Pharmacol Toxicol* | 16611197 | Comparing dynamic regimes by artificial censoring plus IPW. Valid if all joint predictors of survival and of deviating from the regime are measured. |
+| Young JG et al. 2011, *Stat Biosci* | 24039638 | Parametric g-formula for "when to start" dynamic regimes, with measured time-varying confounders affected by earlier treatment. |
+| Kahan BC et al. 2024, *BMJ* | 38262663 | The estimands language (treatment-policy vs hypothetical strategies) for writing down what "conservative" means. |
+| Garcia-Albeniz X et al. 2026, *Cancer Med* | 42321955 | "When to start" emulation: aligning eligibility with time zero, CCW vs sequential trials (4-year differences 4.3% vs −2.8%). Biased designs gave a 6-month RR of 0.13. |
+| Ahn J et al. 2026, *JAMA Netw Open* (registry) | 42832200 | Surgical timing TTE: early MV surgery (≤6 months) vs watchful waiting, CCW with a 6-month grace period, n = 802. RMST gain of 66 days at 5 years; HR 0.69. Shows the grace-period comparator in a surgical setting. |
+| Cheung LC et al. 2026, *Ann Intern Med* (Kaiser EHR) | 42330497 | "Delayed" was explicitly defined as "after 6 months or no" treatment. Immediate excision gave more unnecessary excisions (36.2% vs 7.8%) and similar 3-year cancer risk. |
+| Murray J et al. 2025, *J Clin Epidemiol* (TAA cohort linked to HES) | 39929323 | Repair within 12 months vs no repair within 12 months (CCW), with competing risks. 7-year survival 57.4% vs 49.9%. |
+| Yoshiyama T et al. 2026, *Clin Orthop Relat Res* (Japanese administrative data) | 42821763 | Early (≤2 days) vs delayed (3–10 days) surgery by CCW, explicitly to remove the immortal time in "delayed" groups. |
+
+**Can TTE solve it? Yes, and this is where TTE adds most.** A trial chooses one comparator, and crossover then blurs it. A TTE can emulate *several* comparators from the same data: deferral (no PVI for 6 months), progression-triggered PVI, and, where positivity allows, abstention. It reports each as a separate, named estimand.
+- **Triggers must be measurable.** "PVI only after CLTI" is emulable in claims because CLTI has codes; ICD-10 discriminates claudication from CLTI with 81.3% agreement (Bose 2025). "PVI only if walking distance deteriorates" needs EHR or registry symptom data.
+- **"Never PVI" is hard to emulate.** It may lack positivity once CLTI develops.
+
+**Testable in a known-truth simulation? Yes, and central.** The DGP can produce the true risk under each named comparator by intervening on the simulated data, so landmark, CCW and sequential estimates can each be scored against their own truth (02, B7).
+
+---
+
+# Part 2. RCTs of revascularization vs exercise or medical therapy for intermittent claudication
+
+All rows come from abstracts. "NR" means not reported in the abstract (it may be in the full text). **EXITPAD could not be verified:** searches for "EXITPAD" and "EXIT-PAD" in PubMed returned no trial report or protocol, so it is left out.
+
+| Trial (first report) | PMID(s) | n planned → randomized | Comparison | Primary endpoint | Follow-up | Crossover / non-adherence | Stopped early? | Hard limb outcome reported? |
+|---|---|---|---|---|---|---|---|---|
+| Lundgren 1989, *Ann Surg* | 2647051 | NR → 75 | Surgery vs surgery + training vs training | Walking performance | ~13 months | NR | NR | No |
+| Creasy 1990, *Eur J Vasc Surg* (Oxford) | 2140987 | NR → 36 (20 PTA, 16 exercise) | PTA vs supervised exercise | ABPI and treadmill distances | 12 months (preliminary) | 2 PTA patients later had surgery. 1 exercise patient later had PTA. | NR | No |
+| Perkins 1996, *EJVES* (Oxford long term) | 8846172 | NR → 56 | PTA vs exercise training | ABPI, treadmill distances | 15 months and ~6 years (37 reviewed) | NR | NR | 2 amputations (4%) in the whole cohort. Not compared by arm. |
+| Whyman 1996 / 1997, *EJVES* / *JVS* (Edinburgh) | 8760978; 9357454 | 600 screened → 62 | PTA + medical (aspirin, advice) vs medical | Walking, ABPI, QoL, duplex | 6 months; 2 years | NR | NR | No. Only 10% of claudicants had lesions suitable for PTA. |
+| Gelin 2001 / Taft 2001, *EJVES* (Gothenburg) | 11472042; 11472043 | NR → 264 (HRQL paper: 253) | Invasive (open or endovascular) vs supervised exercise vs observation | Maximum walking power; HRQL | 1 year | NR. HRQL data at 1 year available for 171. | NR | No |
+| OBACT, Nylaende 2007, *EJVES* | 17055756 | 434 considered → 56 | OMT + PTA vs OMT | QoL | 2 years | NR | NR | Mortality was a secondary measure. No limb outcome. |
+| MIMIC, Greenhalgh 2008, *EJVES* | 19022184 | NR → 93 femoropopliteal + 34 aortoiliac | PTA vs no PTA, on top of SE + BMT | ICD / AWD on treadmill | 24 months | NR | NR | No (ABPI only) |
+| Spronk 2009, *Radiology*; long term Fakhry 2013, *BJS* | 19188327; 23842830 | NR → 151 | Endovascular vs hospital-based SET | Rutherford-based clinical success | 12 months; ~7 years | Secondary interventions significantly higher after SET (P = 0.001). Total interventions still higher after ER. | NR | No (secondary interventions only) |
+| Nordanstig 2011, *EJVES* (Gothenburg INV vs NON) | 21397530 | NR → 201 | Invasive (endovascular and/or surgical) vs noninvasive. NON could request invasive treatment. | Maximal walking performance | 2 years | 7% of the INV arm crossed over | NR | No |
+| CLEVER, Murphy 2012, *Circulation*; 2015, *JACC* | 22090168; 25766947 (design 18440181; update 19660897) | **252 planned (design) → 217 planned (update) → 111** | OMC vs OMC + SE vs OMC + stent (aortoiliac) | Change in peak walking time at 6 months | 6 and 18 months (79 completed at 18 months) | NR | Fewer patients than planned. Abstract does not say whether it was formally stopped. | No |
+| Mazari 2012 / 2017, *BJS* (Hull) | 22021102; 27763685 | NR → 178 | PTA vs SEP vs PTA + SEP (femoropopliteal) | Clinical and QoL outcomes | 12 months; median 5.2 years | Withdrawals at 1 year: 8 PTA, 14 SEP, 11 combined | NR | 18 (16.2%) had a major cardiovascular event. Reinterventions: PTA 29, SEP 17, combined 9. No amputation data in abstract. |
+| IRONIC, Nordanstig 2014, *Circulation*; Djerf 2020, *Circ CV Interv* | 25095886; 31937137 | NR → 158 | Revascularization + BMT + structured exercise vs BMT + structured exercise | SF-36 at 1 year | 1, 2, 5 years | NR in abstract | No | No. Benefit lost by 5 years; cost about 2× ($13,098 vs $6,965). |
+| ERASE, Fakhry 2015, *JAMA*; Klaphake 2022, *Ann Surg* | 26547465; 33378308 | NR → 212 | ER + SET vs SET | MWD at 12 months (+282 m) | 12 months; median 5.4 years | SET arm had higher hazard of later revascularization (HR 2.50). Total procedures 65 (SET) vs 149 (combined). | No | Progression to CLI was a secondary endpoint at long-term follow-up. **The result is not given in the abstract.** |
+| Swedish SFA stenting trial, Lindgren 2017, *EJVES*; Gunnarsson 2023, *EJVES* | 28372983; 36642399 (cost 34454817) | NR → 100 | Primary nitinol stent + BMT vs BMT (SFA) | HRQoL (SF-36, EQ-5D) at 12 months | 12, 24, 36, 60 months | **25% control-to-stent crossover by 60 months** | No | **Yes:** no differences in CLTI progression, amputation (2.1% vs 1.9%) or mortality (14.6% vs 15.4%). |
+| SUPER, Koelemay 2022, *EJVES* | 35151572 (design 22326696) | **400 planned → 240** | SET vs ER (iliac) | MWD and VascuQol at 1 year | 1 year; mean 5.5 years | **29% of the SET arm had ER within 1 year.** SET compliance 50%. 8% of the ER arm did not get ER. Additional interventions by 5.5 years: 49% (SET) vs 27% (ER). | **Yes, terminated prematurely** | No (interventions only) |
+| Cochrane, Fakhry 2018 | 29518253 | 10 RCTs, 1,087 participants | ER vs no treatment, SET, or cilostazol | MWD, PFWD, QoL | Short and long term | — | — | No hard limb outcomes were synthesized. Secondary invasive interventions were reported (e.g., ER + SET vs SET: OR 0.27). |
+
+**Reading across the table.**
+1. No trial was designed or powered for CLTI, amputation or death. Only the Swedish SFA trial reports CLTI or amputation by arm in its abstract, with about 2% amputation per arm at 5 years in 100 patients.
+2. Under-recruitment is common:
+   - CLEVER 252 → 111;
+   - SUPER 400 → 240 (stopped);
+   - OBACT 56 of 434;
+   - Whyman 62 of 600.
+3. Crossover from the conservative arm to revascularization is 25–49% wherever it is reported (SUPER, Swedish SFA, ERASE by hazard ratio).
+4. "Conservative" means different things across rows (see Challenge 3).
+
+---
+
+# Part 3. Evidence on TTE itself
+
+## 5. How well does TTE or observational analysis reproduce RCT results?
+
+| Paper | PMID | Data source | Agreement metrics (from abstract) |
+|---|---|---|---|
+| **Wang SV et al. 2023, *JAMA* (RCT-DUPLICATE, 32 trials)** | 37097356 (in 01) | US claims (Optum, MarketScan, Medicare) | Overall Pearson r **0.82** (0.64–0.91). 75% agreed on statistical significance, 66% on the estimate, 75% on standardized difference. **Close emulations (16): r 0.93**, with 94% / 88% / 88% agreement. **Emulations that could not closely match PICOT (16): r 0.53**, with 56% / 50% / 69%. All drug trials, deliberately selected for feasibility. |
+| Franklin JM et al. 2021, *Circulation* (first 10 of RCT-DUPLICATE) | 33327727 | US claims | Regulatory conclusion agreed in 6/10. HR inside the RCT 95% CI in 8/10. Standardized difference <2 in 9/10. Worst agreement where a sulfonylurea was used as a stand-in for placebo. |
+| **Wang C et al. 2026, *BMJ* (systematic review and meta-analysis)** | 42156120 | Mixed (claims, EHR, registries) | 107 TTE–RCT pairs: r **0.59** (0.45–0.70). Standardized difference agreement 79%. Ratio of ratios 0.96. **Closer emulations (63): r 0.83**, 87% agreement. **Claims-based emulations underestimated effects (ratio of ratios 0.90, 0.82–0.99).** Worse agreement with imbalanced baselines, treatment started in hospital, and poor outcome emulation. |
+| Dahabreh IJ et al. 2020, *Epidemiology* | 32740470 | Commentary | "Benchmarking observational methods by comparing randomized trials and their emulations." **No PubMed abstract, so no numbers.** |
+| Matthews AA et al. 2022, *Am J Epidemiol* (TASTE benchmark) | 35641151 | Swedish registries (SWEDEHEART) | Procedure (thrombus aspiration). The emulation reproduced the null 1-year trial result (RD 0.7 for death, −0.2 for MI), then extended follow-up to 3 years and to under-represented subgroups. Benchmark first, then extend. |
+| Westerberg M et al. 2024, *J Clin Epidemiol* | 39159770 | Swedish prostate cancer registry | Prostatectomy vs radiotherapy (treatment chosen by surgeon and patient). Crude HR 1.94. After Charlson adjustment 1.32. After richer comorbidity indices **1.14 (0.91–1.44), in line with the RCT.** Shows how strong confounding by indication is for procedures and how much depends on comorbidity measurement. |
+| Forbes SP et al. 2020, *J Gen Intern Med* | 32193818 | Published PS-based studies | 127 observational-vs-RCT comparisons, **49 in surgery**. Large disagreement (relative RR <0.7 or >1.43) in **54%**. These were not explicit emulations, and the authors call for target-trial-based benchmarking. |
+| Soni PD et al. 2019, *J Clin Oncol* | 30897037 | SEER, SEER-Medicare, NCDB | 350 comparisons vs 121 RCTs. **Concordance correlation 0.083**. 40% agreed. No modifiable factor (PS, IV, sensitivity analyses) improved agreement. Not TTE: conventional registry analyses. |
+| Kumar A et al. 2020, *JAMA Netw Open* | 32729921 | NCDB (cancer registry) | 141 RCTs emulated by eligibility. HRs concordant in 64% of PS models, P values concordant in 45%. No trial characteristic (including type of intervention) predicted concordance. |
+| Keyhani S et al. 2020, *JAMA Neurol* | 32478802 | VA EHR and other VA databases | **Vascular surgery vs initial medical therapy (asymptomatic carotid).** 5-year stroke risk 5.6% vs 7.8% (RD −2.3%). Accounting for competing death: RD −0.8% (not significant). The RCT-eligible subset gave similar numbers. The authors read this as a benefit smaller than in the trials of 20 years earlier, not as a failed replication. |
+| Koeppe J et al. 2026, *Sci Rep* | 42711453 | German insurance claims | Replication across insurers (not vs RCT) of a surgical comparison (RTSA vs locked plating for humeral fracture): 7/11 endpoints replicated, Pearson r 0.97. |
+
+**What this says for surgery.**
+- **Benchmarks for procedures are scarce.** Every large benchmark of explicit emulations is mostly drug trials (RCT-DUPLICATE, Wang 2026). The procedural benchmarks found are single examples: thrombus aspiration, prostatectomy vs radiotherapy, and carotid endarterectomy against older trials.
+- **Conventional registry analyses of procedures agree poorly with RCTs.** Agreement was near zero in Soni 2019 and only 45–64% in Kumar 2020, and in Forbes 2020 more than half of the 127 comparisons (49 in surgery) disagreed by a large margin.
+- **Agreement depends on emulating closely, not on the statistical method.** Wang 2023 shows this for claims and Wang 2026 for mixed sources. In Wang 2026, claims-based emulations also tended to underestimate the trial effect (ratio of ratios 0.90).
+- **Data-source caveat:** Kumar 2020, Soni 2019 and Westerberg 2024 used cancer registries, which have clinical stage but poor comorbidity data. Westerberg shows that better comorbidity measurement moved the estimate most of the way to the RCT.
+
+## 6. Known limitations of TTE, guidance, and reviews of how it is used
+
+### 6a. Guidance and frameworks (core references, PMIDs verified)
+
+| Paper | PMID | Takeaway |
+|---|---|---|
+| Hernán MA, Robins JM 2016, *AJE* | 26994063 (in 01) | Foundational TTE paper. |
+| Hernán MA et al. 2016, *J Clin Epidemiol* | 27237061 (in 01) | Aligning eligibility, assignment and time zero prevents immortal time bias. |
+| Hernán MA et al. 2022, *JAMA* (JAMA Guide) | 36508210 (in 01) | Framework for causal inference from observational data. |
+| Matthews AA et al. 2022, *BMJ* | 36041749 | "Target trial emulation: applying principles of randomised trials to observational studies." **No PubMed abstract.** |
+| Cashin AG et al. 2025, *JAMA* / *BMJ* (TARGET); Hansford 2025, *PLoS Med* (E&E) | 40899949; 40903028; 41091811 (in 01 and 02) | Reporting guideline. |
+| Messinger CJ et al. 2025, *Anesthesiology* | 40067038 | Perioperative and critical care primer: which biases the framework does and does not avoid. |
+| Wong KHF et al. 2025, *Br J Surg* | 40893042 | "Harnessing real-world data to evaluate surgery and perioperative care interventions." **No PubMed abstract.** |
+| Wanis KN et al. 2024, *JRSS-A* | 39281783 (in 02) | Natural vs stochastic grace-period strategies are different estimands. |
+| VanderWeele TJ, Ding P 2017, *Ann Intern Med* | 28693043 | The E-value: how strong an unmeasured confounder would need to be to explain away an estimate. |
+| Lipsitch M et al. 2010, *Epidemiology* | 20335814 | Negative control exposures and outcomes for detecting confounding. |
+
+### 6b. Reviews of how TTE is actually done (quality)
+
+| Paper | PMID | Takeaway |
+|---|---|---|
+| Hansford HJ et al. 2023, *JAMA Netw Open* | 37755828 | 200 TTEs (2012–2022): 43% did not describe all key emulation items, and 57% did not describe the target-trial protocol. |
+| **Ren Y et al. 2026, *JAMA Netw Open*** | 41712213 | 237 TTEs in high-impact journals: 56.5% had a prespecified protocol. 15.2% used post-baseline information for eligibility. 16.9% gave a diagram defining time zero. **Only 30.8% addressed unmeasured confounding.** |
+| McIntyre M et al. 2025, *J Clin Epidemiol* | 40639692 | 49% of 181 TTEs had misaligned eligibility, time zero and assignment. Of those, 67% did nothing about it, although 78% of these could have fixed it cheaply. |
+| Manzanilla A et al. 2026, *J Clin Epidemiol* | 42600711 | 113 CCW studies: **19.5% still had potential immortal time bias**, mostly from excluding patients on post-time-zero criteria. |
+| Scola G et al. 2023, *BMC Med Res Methodol* | 37587484 (in 01) | Only 16% of TTEs used cloning to handle immortal time. |
+
+**Surgery- and vascular-specific reviews:** none found that systematically appraises TTE in surgery or vascular surgery (this repeats the gap noted in 01). Wong 2025 *BJS* and Messinger 2025 are narrative primers, and Hoskins 2026 (01) and Hatano 2026 *Int Orthop* (42287372; abstract not read) are orthopaedic primers.
+
+### 6c. Limitations by data source (claims vs EHR vs registry), with vascular evidence
+
+| Paper | PMID | Data source | Takeaway |
+|---|---|---|---|
+| **Bose S et al. 2025, *Circ Cardiovasc Qual Outcomes*** | 40439664 | VQI-VISION (VQI linked to Medicare) | Using VQI as the gold standard, predefined ICD-10 codes separated claudication from CLTI with 80.9% sensitivity, 81.9% specificity and **81.3% agreement**. Machine learning raised agreement to 84.2%. **About 1 in 5 claims is misclassified on the key indication or outcome.** |
+| Appah-Sampong A et al. 2025, *Vasc Med* | 40340584 | Review of EHR and claims PAD studies | 84 studies: **76.2% used a single-code query** to identify PAD, despite its low yield. Only 5 used regression-based phenotypes. |
+| **Columbo JA et al. 2019, *J Vasc Surg*** | 29914838 | VQI vs VQI-Medicare vs chart review (EVAR) | **VQI alone found 12 of 30 one-year reinterventions.** VQI-Medicare found all 30 (κ 0.93 vs chart review). At 3 years, sensitivity 92% and specificity 96%. Registry follow-up alone undercounts reinterventions, and linking to claims fixes most of it. |
+| Goodney P et al. 2022, *BMJ* | 36283705 | VQI-Medicare | 20,489 EVAR patients with complete follow-up through registry and/or claims. Detected a device-specific reintervention signal (HR 1.61; surgeon-level IV 1.75) in mid-2013, years before the 2017 regulatory warnings. |
+| Mao J et al. 2022, *J Vasc Surg* | 35181518 | VQI-Medicare linkage | Indirect linkage (without SSNs) matched 93.0% of patients with 99.4% accuracy. |
+| **Mao J et al. 2024, *BMJ Open*** | 39327057 | VQI linked to EHR (INSIGHT CRN, New York) | After PVI, **75.2% were retained in the EHR through year 3.** Older, Medicaid, heart-failure, dialysis and less ambulatory patients were less likely to be retained, so attrition is informative. Amputation-free survival estimates were sensitive to this from 6–12 months onward. |
+| Jansen SCP et al. 2020, *EJVES* | 32888779 | Dutch national claims | SET captured in claims where it is reimbursed (63% → 87% primary SET). Shows that SET's visibility in claims depends on the healthcare system. |
+| Keyhani S et al. 2020 / 2022, *JAMA Neurol* / *Stroke* | 32478802; 35164531 | VA EHR (carotid imaging reports) | Emulations of carotid surgery vs medical therapy that used **imaging-report stenosis**, an EHR-only variable, for eligibility. 2022 (CAS vs MT): 5-year stroke 6.9% vs 7.1%. The population was 98.8% men (2020 paper), which limits generalizability. |
+| Sakowitz S et al. 2025, *Ann Surg* | 40607703 (in 01) | Nationwide Readmissions Database | TCAR vs CEA "national target trial": 90-day composite 4.1% vs 4.1%. Follow-up is limited to 90 days by the database. |
+| Wang Z et al. 2026, *J Vasc Surg* | 42448157 | 3-center clinical data | Open- vs closed-cell carotid stents. Median follow-up 26 months. Example of a short-follow-up, single-system emulation in vascular surgery. |
+| Heindel P et al. 2024, *Eur Heart J Open* | 39175849 | Medicare linked with academic health-system EHR | PAD drug TTE (colchicine) with a MALE/MACE/death outcome. RD −1.7% (−6.5 to 3.1). Example of a claims-plus-EHR PAD emulation. |
+| Westerberg 2024; Wang C 2026 (above) | 39159770; 42156120 | — | Comorbidity measurement drives residual confounding. Claims-based emulations tended to underestimate RCT effects. |
+
+**Summary of data-source trade-offs for an early-PVI TTE.**
+- **Claims (Medicare).**
+  - Strengths: population-wide, long and nearly complete follow-up for procedures and amputation, procedure dates, operator NPI, Part D fills, and SET billing from 2017.
+  - Weaknesses: no ABI, walking distance or symptom severity (the main triggers for crossover and confounding by indication). About 19% misclassification between claudication and CLTI (Bose 2025). Home exercise, smoking, and patient refusal are not recorded.
+- **EHR.**
+  - Strengths: ABI, imaging, symptom notes, SET referral, smoking, and refusal documentation.
+  - Weaknesses: follow-up is fragmented, and patients leave the system informatively (75% retained at 3 years; Mao 2024). Usually a single system, so fewer physicians to compare.
+- **Registry (VQI).**
+  - Strengths: rich anatomy and indication data, plus OMT and smoking status at the time of the procedure.
+  - Weaknesses: **enrolls only patients who have a procedure**, so there is no conservative arm. Long-term outcomes are undercounted unless linked to claims (12 of 30 reinterventions captured; Columbo 2019).
+- **Linked data (VQI-Medicare/VISION).** Best for outcomes after a procedure. It still has no comparator arm of patients never treated, unless claims-only patients are added, and those patients then lack registry detail.
+
+### 6d. Unmeasured confounding by indication: the limit that remains
+
+- Every guidance paper above, and every review, names unmeasured confounding as the limitation TTE does not remove. Only 30.8% of published TTEs address it (Ren 2026).
+- For procedures, Westerberg 2024 shows that adjustment moved the HR from 1.94 to 1.14, and that most of the bias came from how well comorbidity was measured.
+- In the claudication setting the main unmeasured confounders are:
+  - symptom severity and walking distance;
+  - frailty;
+  - patient preference and refusal;
+  - physician style (see 02, B6).
+- Tools exist for each: E-values (28693043), negative controls (20335814), physician-level designs (02), and linking to EHR or registry data to measure symptoms.
+
+---
+
+## Feed for Table 1 (this draft's rows)
+
+| Challenge | Evidence it is common/important | Vascular/claudication example | Can TTE solve it? | Known-truth simulation? |
+|---|---|---|---|---|
+| Crossover / contamination | 27–43% crossover in conservative arms within 1–2 years (Peul, Katz, SPORT, CODA, APPAC). Effect estimates nearer the null with more crossover (Gaudino 2019). STICH ITT not significant at 5 years. | SUPER 29% at 1 year. Swedish SFA 25% at 5 years. ERASE HR 2.50. | **Yes** (CCW/IPCW for explicit strategies), if crossover triggers are measured. Claims lack symptoms; EHR/registry partly have them. | Yes: latent progression drives deferred PVI. Observed vs unobserved. |
+| Non-adherence / refusal | STICH CABG→MED crossover mainly by patient/family choice, with worse prognosis. ACST only 50% operated by 1 month. SET recruitment 24.2%. | SUPER SET compliance 50%. US SET completion 9.8–49.9%. | **Partly.** No consent step. Refusal is not visible in claims (sometimes in EHR). SET billing is visible in Medicare from 2017. | Yes: latent "refuser" trait. |
+| Co-interventions / usual care | Usual care poorly reported (TIDieR median 1/12). Only 38.1% complete OMT before revascularization in VQI. | Comparator differs across trials (aspirin plus advice vs SET plus BMT vs cilostazol). | **Partly.** Part D statins and SET claims are visible. Home exercise and smoking are not. VQI has OMT only at procedure and no conservative arm. | Yes: unmeasured exercise or OMT that differs by arm. |
+| Comparator definition | Most "conservative" arms are "surgery-if-needed". ISCHEMIA and ACST define dynamic comparators. CCW and sequential trials give different numbers (Garcia-Albeniz 2026). | "No PVI within 6 months" vs "PVI only for CLTI" vs "never PVI". | **Yes**, and TTE's main gain: several named estimands from one dataset. Triggers must be coded (CLTI codes about 81% accurate). "Never" may lack positivity. | Yes, central: true risk under each strategy. |
+
+## Five-line summary
+
+1. Crossover from conservative care to surgery is common and large: 25–68% across hernia, spine, knee, appendicitis, cardiac surgery and claudication trials. It pulls ITT estimates toward the null, as in STICH and the Swedish SFA trial, and it is the challenge TTE handles best, through explicit dynamic strategies with CCW/IPCW.
+2. Non-adherence and refusal (to surgery and especially to SET) are substantial, and patients who refuse have a different prognosis. TTE removes the consent barrier, but refusal of an offered operation is not visible in claims and is only partly visible in EHR.
+3. Co-interventions and "usual care" vary widely and are poorly reported. Claims see drugs and Medicare SET but not home exercise or smoking. VQI records OMT only at the time of a procedure and cannot provide a no-PVI arm on its own.
+4. Of the 15 claudication revascularization trials checked (plus the Cochrane review), none was powered for hard limb outcomes. Several under-recruited (CLEVER 252 → 111) or stopped early (SUPER 400 → 240). Only the Swedish SFA trial reports CLTI and amputation by arm (about 2% each). EXITPAD could not be found in PubMed.
+5. TTE agrees well with RCTs when the emulation is close (r 0.93 in RCT-DUPLICATE; 0.83 in Wang 2026 *BMJ*), but agreement is poor otherwise (r 0.53; and 0.08 for conventional cancer-registry analyses). Procedural benchmarks are scarce, many published TTEs are of low quality (only 30.8% address unmeasured confounding), and no claudication or PVI TTE was found.
+
+---
+
+### Verification notes
+- **No PubMed abstract, so cited without numbers:** Divakaran 2021 (34293930), Divakaran 2024 (39129558), Dahabreh 2020 (32740470), Matthews 2022 *BMJ* (36041749), Wong 2025 *BJS* (40893042), Katsoulis 2024 (39024012, in 01).
+- **EXITPAD:** not found in PubMed under that name. It may be a non-indexed acronym or a confusion with ERASE or SUPER (both Dutch).
+- **Not in the abstracts** (need full text): planned sample sizes for MIMIC, IRONIC and ERASE; crossover rates for IRONIC, CLEVER and MIMIC; ERASE's long-term CLI result; whether CLEVER was formally stopped.
+- **ISCHEMIA and COURAGE crossover rates** are not in their abstracts, so they are not quoted.
+- **Creasy 1990 and Perkins 1996 are probably the same Oxford trial** (preliminary report with 36 patients, later report with 56). The abstracts do not confirm this.
+- **No claudication or PVI target trial emulation** was found by a PubMed search (2026-10-28).

pvisim/R/estimators.R

@@ -51,31 +51,45 @@ fit_pvi <- function(data, newdata, variant, extra_covs = NULL) {
     d <- data[win == w]
     nd <- newdata[i_new]
     tt_levels <- if (w == 1L) "factor(tt)" else "splines::ns(t, df = 4)"
+    # deferred period (about 2 million person-months, few PVIs): fit on every PVI and a
+    # fixed 10% sample of the other months, with offset log(1/0.1) so the coefficients
+    # are those of the full model (case-control sampling of person-months)
+    if (w == 0L) d <- sample_controls(d, frac = 0.1)
     # no exposure at all in a phase (the NCE after the window): probability 0
     out[i_new] <- if (sum(d$pvi) == 0) 0 else fit_pvi_phase(d, nd, variant, tt_levels, extra_covs)
   }
   out
 }
 
+#' Keep all events and a deterministic fraction of non-events; offset column `off`.
+sample_controls <- function(d, frac) {
+  keep <- d$pvi == 1L | (((d$id * 7919L + d$t * 104729L) %% 1000L) < frac * 1000)
+  d <- d[keep]
+  d[, off := log(1 / frac)]   # sampling raises every row's odds by 1/frac
+  d
+}
+
 fit_pvi_phase <- function(d, nd, variant, time_terms, extra_covs = NULL) {
+  if (!"off" %in% names(d)) d[, off := 0]
+  nd <- copy(nd)[, off := 0]                 # predictions are for the full population
   covs <- c(BASE_COVS, "sev_obs", "sev0", extra_covs)
   if (variant == "V4_within") covs <- setdiff(covs, "obl")
   rhs <- paste(c(covs, time_terms), collapse = " + ")
   if (variant %in% c("V1_ignore", "V2_cluster")) {
     f <- as.formula(paste("~", rhs))
     X <- model.matrix(f, d)
-    fit <- fastglm::fastglm(X, d$pvi, family = binomial(), method = 2)
+    fit <- fastglm::fastglm(X, d$pvi, family = binomial(), method = 2, offset = d$off)
     b <- coef(fit); b[is.na(b)] <- 0
     Xn <- model.matrix(f, nd)
     return(as.vector(plogis(Xn %*% b)))
   }
   if (variant == "V3_random") {
-    fit <- lme4::glmer(as.formula(paste("pvi ~", rhs, "+ (1 | phys_id)")), data = d,
+    fit <- lme4::glmer(as.formula(paste("pvi ~", rhs, "+ offset(off) + (1 | phys_id)")), data = d,
                        family = binomial(), nAGQ = 0, control = lme4::glmerControl(calc.derivs = FALSE))
     return(as.vector(predict(fit, newdata = nd, type = "response", allow.new.levels = TRUE)))
   }
   if (variant == "V4_within") {
-    fit <- fixest::feglm(as.formula(paste("pvi ~", rhs, "| phys_id")), data = d,
+    fit <- fixest::feglm(as.formula(paste("pvi ~", rhs, "| phys_id")), data = d, offset = ~off,
                          family = binomial(), notes = FALSE, warn = FALSE)
     pr <- as.vector(predict(fit, newdata = nd, type = "response"))
     pr[is.na(pr)] <- 0
@@ -160,24 +174,38 @@ restrict_variant <- function(rec, variant) {
 #' Run one design under one physician variant for several outcomes.
 #' Returns one row per outcome (and contrast), with the SE that the variant reports:
 #' patient-level for V1, physician-cluster-robust for V2-V4.
-run_design <- function(rec, design, variant, outcomes = c(male = "t_male", disease = "t_disease",
+run_design <- function(rec, design, variant, outcomes = c(limb = "t_limb", male = "t_male", disease = "t_disease",
+                                                           nco1 = "t_nco1", nco2 = "t_nco2")) {
+  run_designs(rec, design, variant, outcomes)
+}
+
+#' Several designs under one variant, sharing the PVI fit between CCW and sequential
+#' trials (the sequential propensity score is the window part of the same model).
+run_designs <- function(rec, designs, variant, outcomes = c(limb = "t_limb", male = "t_male", disease = "t_disease",
                                                              nco1 = "t_nco1", nco2 = "t_nco2")) {
   rec <- restrict_variant(rec, variant)
-  rs <- if (design %in% c("ccw", "seq")) pvi_risk_set(rec) else NULL
-  phat <- if (design == "ccw") fit_pvi(rs, rs, variant, rec$extra_covs) else NULL
+  need <- intersect(designs, c("ccw", "seq"))
+  rs <- if (length(need)) pvi_risk_set(rec) else NULL
+  phat <- if ("ccw" %in% need) fit_pvi(rs, rs, variant, rec$extra_covs)
+          else if ("seq" %in% need) { x <- rep(NA_real_, nrow(rs)); w <- rs$win == 1L
+                                      x[w] <- fit_pvi(rs[w], rs[w], variant, rec$extra_covs); x }
   cmp <- if (is.null(rec$comparators)) c("B1", "B2") else rec$comparators
-  out <- lapply(names(outcomes), function(o) {
-    r <- switch(design,
-      naive = cbind(contrast = "early_none", est_naive(rec, outcomes[[o]], variant)),
-      landmark = cbind(contrast = "early_none", est_landmark(rec, outcomes[[o]], variant, extra_covs = rec$extra_covs)),
-      ccw = est_ccw(rec, outcomes[[o]], variant, rs = copy(rs), phat = phat, comparators = cmp),
-      seq = cbind(contrast = "now_not", est_seq(rec, outcomes[[o]], variant, rs = copy(rs))))
-    r[, `:=`(outcome = o, design = design, variant = variant, n = nrow(rec$patients))]
   use_patient <- variant == "V1_ignore"
+  rbindlist(lapply(designs, function(design) {
+    r <- switch(design,
+      naive = rbindlist(lapply(names(outcomes), function(o)
+        cbind(contrast = "early_none", est_naive(rec, outcomes[[o]], variant), outcome = o))),
+      landmark = rbindlist(lapply(names(outcomes), function(o)
+        cbind(contrast = "early_none", est_landmark(rec, outcomes[[o]], variant, extra_covs = rec$extra_covs), outcome = o))),
+      landmark_m5 = rbindlist(lapply(names(outcomes), function(o)
+        cbind(contrast = "early_none", est_landmark(rec, outcomes[[o]], variant, extra_covs = rec$extra_covs,
+                                                    at_landmark = TRUE), outcome = o))),
+      ccw = est_ccw(rec, outcomes, variant, rs = copy(rs), phat = phat, comparators = cmp),
+      seq = cbind(contrast = "now_not", est_seq(rec, outcomes, variant, rs = copy(rs), phat = phat)))
+    r[, `:=`(design = design, variant = variant, n = nrow(rec$patients))]
     r[, se := if (use_patient) se_patient else se_cluster]
     r
-  })
-  rbindlist(out, fill = TRUE)
+  }), fill = TRUE)
 }
 
 #' Naive "ever early PVI" design: exposure = PVI in months 0..W-1, follow-up from month 0,
@@ -214,9 +242,11 @@ fit_pvi_ps <- function(p, variant, covs = c(BASE_COVS, "sev_obs", "sev0")) {
 }
 
 #' Landmark at month W (D13): patients alive and MALE-free at the start of month W;
-#' exposure = PVI in months 0..W-1; covariates at diagnosis and at month W-1;
-#' outcome counted in months W..horizon-1.
-est_landmark <- function(rec, outcome = "t_male", variant = "V1_ignore", horizon = 60L, extra_covs = NULL) {
+#' exposure = PVI in months 0..W-1; covariates at diagnosis; outcome counted in months
+#' W..horizon-1. With at_landmark = TRUE the propensity score also uses severity at month
+#' W-1, which a PVI has already changed (supplement, D53).
+est_landmark <- function(rec, outcome = "t_male", variant = "V1_ignore", horizon = 60L, extra_covs = NULL,
+                         at_landmark = FALSE) {
   W <- rec$par$size$window
   p <- copy(rec$patients)[(is.na(t_death) | t_death >= W) & (is.na(t_male) | t_male >= W)]
   p[, `:=`(t_out = get(outcome), arm = ifelse(!is.na(t_pvi) & t_pvi < W, "early", "none"))]
@@ -224,7 +254,7 @@ est_landmark <- function(rec, outcome = "t_male", variant = "V1_ignore", horizon
   p <- merge(p, rec$pm[t == W - 1L, .(id, sev_obs)], by = "id")
   p <- merge(p, rec$pm[t == 0, .(id, sev0 = sev_obs)], by = "id")
   p[, pvi := as.integer(arm == "early")]
-  ps <- fit_pvi_ps(p, variant, covs = c(BASE_COVS, "sev_obs", "sev0", extra_covs))
+  ps <- fit_pvi_ps(p, variant, covs = c(BASE_COVS, "sev0", if (at_landmark) "sev_obs", extra_covs))
   p[, w := ifelse(arm == "early", 1 / ps, 1 / (1 - ps))]
   d <- expand_followup(p[!is.na(w)], W, horizon)
   rd_from_aj(aj_cif(d, horizon), "early", "none")
@@ -237,13 +267,14 @@ est_ccw <- function(rec, outcome = "t_male", variant = "V1_ignore", horizon = 60
                     comparators = c("B1", "B2"), rs = NULL, phat = NULL) {
   W <- rec$par$size$window
   if (is.null(rs)) rs <- pvi_risk_set(rec)
-  if (is.null(phat)) phat <- fit_pvi(rs, rs, variant)
+  if (is.null(phat)) phat <- fit_pvi(rs, rs, variant, rec$extra_covs)
   rs[, ph := phat]
   rs <- rs[!is.na(ph)]
   # everyone eligible at diagnosis is cloned, including patients with a MALE event or
-  # death in month 0 (events in the grace period count in both arms)
+  # death in month 0 (events in the grace period count in both arms). Clones and weights
+  # are built once over follow-up to death or the horizon; each outcome then truncates.
   p <- copy(rec$patients)
-  p[, t_out := get(outcome)]
+  p[, t_out := NA_integer_]
   base <- expand_followup(p, 0L, horizon)
   # P(PVI) for each eligible month; months where a patient is not at risk of PVI have
   # probability 0 of deviating
@@ -251,7 +282,7 @@ est_ccw <- function(rec, outcome = "t_male", variant = "V1_ignore", horizon = 60
   base[is.na(ph), ph := 0]
   base[, `:=`(tp = fifelse(is.na(t_pvi), Inf, as.numeric(t_pvi)), el = pvi_elective %in% TRUE)]
 
-  out <- list()
+  arms <- list()
   # arm A: censored after month W-1 if no PVI by then (and still at risk of PVI then)
   a <- copy(base)[, arm := "A"]
   cens_A <- a[t == W - 1L & ph > 0 & tp > W - 1L, id]          # deviated at month W-1
@@ -259,38 +290,47 @@ est_ccw <- function(rec, outcome = "t_male", variant = "V1_ignore", horizon = 60
   pA <- base[t == W - 1L & ph > 0, .(id, pA = ph)]
   a <- pA[a, on = "id"]
   a[, w := ifelse(t > W - 1L & !is.na(pA) & tp == W - 1L, 1 / pA, 1)]
-  out$A <- a[, .(id, phys_id, arm, t, w, ev, dth)]
+  arms$A <- a[, .(id, phys_id, arm, t, w)]
   for (cmp in comparators) {
     b <- copy(base)[, arm := cmp]
     last <- if (cmp == "B1") W - 1L else horizon
     b <- b[!(el & tp <= last & t > tp)]                         # censored after month tp
     b[, q := ifelse(t <= last, 1 - ph, 1)]
     b[, w := 1 / shift(cumprod(q), fill = 1), by = id]
-    out[[cmp]] <- b[, .(id, phys_id, arm, t, w, ev, dth)]
+    arms[[cmp]] <- b[, .(id, phys_id, arm, t, w)]
   }
-  aj <- aj_cif(rbindlist(out), horizon)
-  rbindlist(lapply(comparators, function(cmp) cbind(contrast = paste0("A_", cmp), rd_from_aj(aj, "A", cmp))))
+  clones <- rbindlist(arms)
+  ev_times <- rec$patients[, c("id", "t_death", unname(outcome)), with = FALSE]
+  rbindlist(lapply(seq_along(outcome), function(k) {
+    d <- ev_times[, .(id, t_death, t_out = get(outcome[k]))][clones, on = "id"]
+    d <- d[is.na(t_out) | t <= t_out]
+    aj <- aj_cif(mark_events(d), horizon)
+    rbindlist(lapply(comparators, function(cmp)
+      cbind(contrast = paste0("A_", cmp), rd_from_aj(aj, "A", cmp))))[
+        , outcome := if (is.null(names(outcome))) outcome[k] else names(outcome)[k]]
+  }))
 }
 
 #' Sequential trials, ITT (D14): a trial at each month m = 0..W-1 among patients eligible
 #' for PVI at m; treatment = PVI at m; propensity score from covariates at m; outcome
 #' counted in months m+1..horizon-1; trial RDs pooled with weights = number eligible.
-est_seq <- function(rec, outcome = "t_male", variant = "V1_ignore", horizon = 60L, rs = NULL) {
+est_seq <- function(rec, outcome = "t_male", variant = "V1_ignore", horizon = 60L, rs = NULL,
+                    phat = NULL) {
   W <- rec$par$size$window
   if (is.null(rs)) rs <- pvi_risk_set(rec)
+  if (!is.null(phat)) rs[, ps := phat]
   trials <- rs[t < W]
-  trials[, tt := t]                       # trial-month indicators in the PS model
-  ps <- fit_pvi(trials, trials, variant)
-  trials[, ps := ps]
+  # the trial-baseline propensity score is the window model of the PVI fit
+  if (is.null(phat)) trials[, ps := fit_pvi(trials, trials, variant, rec$extra_covs)]
   trials <- trials[!is.na(ps)]
   trials[, `:=`(m = t, arm = ifelse(pvi == 1L, "pvi", "no"),
                 w = ifelse(pvi == 1L, 1 / ps, 1 / (1 - ps)))]
-  p <- rec$patients[, .(id, t_out = get(outcome))]
-  trials <- p[trials, on = "id"]
-  trials <- trials[is.na(t_out) | t_out > m]
+  trials <- trials[, .(id, phys_id, m, arm, w, t_death)]
+  rbindlist(lapply(seq_along(outcome), function(k) {
+    tr_k <- rec$patients[, .(id, t_out = get(outcome[k]))][trials, on = "id"][is.na(t_out) | t_out > m]
     res <- list(); infs <- list()
     for (mm in 0:(W - 1)) {
-    tr <- trials[m == mm]
+      tr <- tr_k[m == mm]
       d <- expand_followup(tr[, .(id, phys_id, arm, w, t_out, t_death)], mm + 1L, horizon)
       aj <- aj_cif(d, horizon)
       x <- dcast(aj$inf, id + phys_id ~ arm, value.var = "inf", fill = 0)
@@ -302,7 +342,9 @@ est_seq <- function(rec, outcome = "t_male", variant = "V1_ignore", horizon = 60
     infs <- res[, .(m, wt)][infs, on = "m"]
     pi <- infs[, .(inf = sum(wt * inf)), by = .(id, phys_id)]
     data.table(est = sum(res$wt * res$est), se_patient = sqrt(sum(pi$inf^2)),
-             se_cluster = sqrt(sum(tapply(pi$inf, pi$phys_id, sum)^2)), risk1 = NA_real_, risk0 = NA_real_)
+               se_cluster = sqrt(sum(tapply(pi$inf, pi$phys_id, sum)^2)), risk1 = NA_real_, risk0 = NA_real_,
+               outcome = if (is.null(names(outcome))) outcome[k] else names(outcome)[k])
+  }))
 }
 
 #' Negative-control exposure (D32, D36): the early low-value procedure (NCE, months

pvisim/R/limits.R

@@ -17,13 +17,14 @@ oracle_records <- function(rec) {
   o
 }
 
-#' Oracle versions of the landmark and naive designs (V1, patient-level SE).
-run_oracle <- function(rec, outcomes = c(male = "t_male", disease = "t_disease", nco1 = "t_nco1", nco2 = "t_nco2")) {
+#' Oracle versions of the landmark (both versions) and naive designs (V1, patient-level SE).
+run_oracle <- function(rec, outcomes = c(limb = "t_limb", male = "t_male", disease = "t_disease", nco1 = "t_nco1", nco2 = "t_nco2")) {
   o <- oracle_records(rec)
   rbindlist(lapply(names(outcomes), function(nm) {
     lm <- est_landmark(o, outcomes[[nm]], "V1_ignore", extra_covs = o$extra_covs)
+    l5 <- est_landmark(o, outcomes[[nm]], "V1_ignore", extra_covs = o$extra_covs, at_landmark = TRUE)
     nv <- est_naive(o, outcomes[[nm]], "V1_ignore", extra_covs = o$extra_covs)
-    rbind(cbind(design = "landmark", lm), cbind(design = "naive", nv))[
+    rbind(cbind(design = "landmark", lm), cbind(design = "landmark_m5", l5), cbind(design = "naive", nv))[
       , `:=`(outcome = nm, contrast = "early_none", variant = "oracle", se = se_patient)]
   }))
 }

pvisim/R/rct.R

@@ -1,16 +1,27 @@
 # Simulated randomized trials (plan section 8, D18).
 #
 # Realistic RCT: hospital-based (OBL = 0) physicians in the top third of skill and the
-# bottom half of style; 30% of their eligible patients consent; n = 240.
+# bottom half of style; on average 30% of their eligible patients consent, less often when
+# older or frailer (OR 0.7 per 10 years and per SD of frailty; D55); n = 240.
 # Ideal pragmatic RCT: all physicians, everyone consents, large n.
 # Assignment A: PVI at a scheduled month 0..2 (uniform); 10% refuse and follow usual care.
 # Assignment B: no protocol PVI in the window; crossover happens through the DGP's own
 # PVI process (usual care), and usual care continues after the window.
 
 RCT_DESIGNS <- list(
-  realistic = list(n = 240L, consent = 0.30, select = TRUE),
-  ideal = list(n = NA, consent = 1.00, select = FALSE)
+  realistic = list(n = 240L, consent = 0.30, select = TRUE, consent_or = c(age10 = 0.7, frailty = 0.7)),
+  ideal = list(n = NA, consent = 1.00, select = FALSE, consent_or = NULL)
 )
+RCT_HORIZONS <- c(12L, 24L, 60L)   # trials usually report at 1-2 years (D55)
+
+#' Probability that each eligible patient consents: mean `spec$consent`, lower for older
+#' and frailer patients (Wasmann 2019, King 2005: refusers differ moderately by covariates).
+consent_prob <- function(pat, spec) {
+  if (is.null(spec$consent_or)) return(rep(spec$consent, nrow(pat)))
+  lp <- log(spec$consent_or[["age10"]]) * (pat$age - 76) / 10 + log(spec$consent_or[["frailty"]]) * pat$frailty
+  a <- uniroot(function(a) mean(plogis(a + lp)) - spec$consent, c(-10, 10))$root
+  plogis(a + lp)
+}
 
 #' Physicians a realistic trial would recruit.
 trial_physicians <- function(phys) {
@@ -25,7 +36,7 @@ enrol_trial <- function(pop, design = c("realistic", "ideal"), n = NULL, seed =
   set.seed(seed)
   pat <- pop$pat
   if (spec$select) pat <- pat[phys_id %in% trial_physicians(pop$phys)]
-  pat <- pat[runif(.N) < spec$consent]
+  pat <- pat[runif(.N) < consent_prob(pat, spec)]
   n <- if (!is.null(n)) n else if (is.na(spec$n)) nrow(pat) else spec$n
   if (nrow(pat) < n) stop("not enough consenting patients: ", nrow(pat), " < ", n)
   pat <- pat[sort(sample.int(.N, n))]
@@ -49,7 +60,7 @@ trial_policy <- function(trial, assign = NULL) {
 
 #' Run one simulated trial and analyse it: ITT (by assignment, unadjusted) and naive
 #' as-treated (PVI in the window vs not, ignoring assignment).
-run_rct <- function(par, design = "realistic", rep = 1L, outcomes = c(male = "t_male", disease = "t_disease"),
+run_rct <- function(par, design = "realistic", rep = 1L, outcomes = c(limb = "t_limb", male = "t_male", disease = "t_disease"),
                     source_n = 60000L) {
   seed <- replicate_seed(par$world, paste0(par$scenario, "|rct|", design), rep)
   pop <- make_population(par, n = source_n, seed = seed)
@@ -58,33 +69,38 @@ run_rct <- function(par, design = "realistic", rep = 1L, outcomes = c(male = "t_
   W <- par$size$window
   ev[, `:=`(phys_id = trial$pop$pat$phys_id, assigned = trial$arm,
             treated_win = ifelse(!is.na(t_pvi) & t_pvi < W, "early", "none"))]
-  rbindlist(lapply(names(outcomes), function(o) {
+  rbindlist(lapply(names(outcomes), function(o) rbindlist(lapply(RCT_HORIZONS, function(h) {
     p <- ev[, .(id, phys_id, assigned, treated_win, t_out = get(outcomes[[o]]), t_death)]
-    itt <- rd_from_aj(aj_cif(expand_followup(copy(p)[, `:=`(arm = assigned, w = 1)], 0L)), "A", "B")
-    at <- rd_from_aj(aj_cif(expand_followup(copy(p)[, `:=`(arm = treated_win, w = 1)], 0L)), "early", "none")
+    itt <- rd_from_aj(aj_cif(expand_followup(copy(p)[, `:=`(arm = assigned, w = 1)], 0L, h), h), "A", "B")
+    at <- rd_from_aj(aj_cif(expand_followup(copy(p)[, `:=`(arm = treated_win, w = 1)], 0L, h), h), "early", "none")
     rbind(cbind(analysis = "itt", itt), cbind(analysis = "as_treated", at))[
-      , `:=`(outcome = o, design = paste0("rct_", design), n = nrow(ev),
+      , `:=`(outcome = o, horizon = h, design = paste0("rct_", design), n = nrow(ev),
              crossover_B = mean(ev$treated_win[ev$assigned == "B"] == "early"),
              nonadherence_A = mean(ev$treated_win[ev$assigned == "A"] == "none"))]
-  }))
+  }))))
 }
 
 #' ITT truth for a trial design: everyone in a large trial-eligible population assigned
 #' A vs everyone assigned B, same protocol (refusal, scheduling, natural crossover).
-#' Consent is random, so it does not change the trial population and is skipped here.
-rct_truth <- function(par, design = "realistic", n = 1e6, seed = 20261029L, horizon = 60L) {
+#' Patients are weighted by their probability of consenting, so the truth refers to the
+#' population that enrols. Horizons: 60 months (estimand rct_<design>_itt) and the
+#' shorter trial horizons (rct_<design>_itt_h12, _h24).
+rct_truth <- function(par, design = "realistic", n = 1e6, seed = 20261029L, horizons = RCT_HORIZONS) {
   pop <- truth_population(par, n = n, seed = seed)
   pat <- if (design == "realistic") pop$pat[phys_id %in% trial_physicians(pop$phys)] else pop$pat
+  pc <- consent_prob(pat, RCT_DESIGNS[[design]])
   set.seed(seed + 1L)
   k <- nrow(pat)
   trial <- list(pop = list(phys = pop$phys, pat = copy(pat)[, id := seq_len(.N)]),
                 arm = rep("A", k), refuse = runif(k) < 0.10, sched = sample(0:2, k, replace = TRUE))
   runs <- lapply(c("A", "B"), function(a)
     simulate_course(trial$pop, par, trial_policy(trial, assign = a), seed = seed + 2L, record = FALSE)$events)
-  rbindlist(lapply(names(OUTCOMES), function(o) {
-    d <- as.numeric(by_h(runs[[1]][[OUTCOMES[[o]]]], horizon)) - as.numeric(by_h(runs[[2]][[OUTCOMES[[o]]]], horizon))
+  w <- pc / sum(pc)
+  rbindlist(lapply(names(OUTCOMES), function(o) rbindlist(lapply(horizons, function(h) {
+    d <- as.numeric(by_h(runs[[1]][[OUTCOMES[[o]]]], h)) - as.numeric(by_h(runs[[2]][[OUTCOMES[[o]]]], h))
+    m <- sum(w * d)
     data.table(world = par$world, scenario = par$scenario, outcome = o,
-               estimand = paste0("rct_", design, "_itt"), value = mean(d), mcse = sd(d) / sqrt(length(d)),
-               n = length(d))
-  }))
+               estimand = paste0("rct_", design, "_itt", if (h != 60L) paste0("_h", h)), value = m,
+               mcse = sqrt(sum(w^2 * (d - m)^2)), n = length(d))
+  }))))
 }

pvisim/R/records.R

@@ -32,6 +32,8 @@ make_records <- function(pop, course, par) {
   idx <- cbind(id, t + 1L)
   if (identical(ms$severity, "exact")) {
     sev <- course$z[idx]
+  } else if (identical(ms$severity, "noisy")) {
+    sev <- course$z[idx] + rnorm(length(id), 0, ms$noise_sd)   # EHR/registry: measured with error
   } else {
     w <- ms$proxy_window
     cs <- t(apply(course$counts, 1, function(x) { x[is.na(x)] <- 0L; cumsum(x) }))
@@ -68,3 +70,24 @@ replicate_seed <- function(world, scenario, rep) {
   h <- sum(utf8ToInt(key) * seq_along(utf8ToInt(key))^2)
   as.integer((h * 2654435761) %% 2147483647)
 }
+
+#' Physician-cluster bootstrap resample of a replicate's records (D28): physicians are
+#' drawn with replacement and each draw brings all of its patients, with new patient and
+#' physician ids so repeated physicians count as separate clusters.
+bootstrap_records <- function(rec, seed) {
+  set.seed(seed)
+  phys <- unique(rec$patients$phys_id)
+  draw <- data.table(phys_id = sample(phys, length(phys), replace = TRUE), new_phys = seq_along(phys))
+  map <- rec$patients[, .(id, phys_id)][draw, on = "phys_id", allow.cartesian = TRUE]
+  map[, new_id := seq_len(.N)]
+  remap <- function(x) {
+    x <- x[map[, .(id, new_id)], on = "id", allow.cartesian = TRUE]
+    x[, id := new_id][, new_id := NULL][]
+  }
+  out <- rec
+  out$patients <- remap(rec$patients)[map[, .(new_id, new_phys)], on = c(id = "new_id"), phys_id := i.new_phys]
+  out$pm <- setkey(remap(rec$pm), id, t)
+  out$hidden <- remap(rec$hidden)
+  out$hidden_pm <- remap(rec$hidden_pm)
+  out
+}

pvisim/R/simulate.R

@@ -127,12 +127,14 @@ simulate_course <- function(pop, par, pol = policy("natural"), seed = 1L, record
   lp_rr0 <- rr$intercept + rr$skill * pat$skill + rr$style * pat$style
   lp_nco1 <- nc$nco1$intercept + nc$nco1$frailty * pat$frailty + nc$nco1$smoking * pat$smoking_true +
     nc$nco1$age10 * a10
+  nco1_z <- if (is.null(nc$nco1$z)) 0 else nc$nco1$z
   lp_nco2 <- nc$nco2$intercept + nc$nco2$style * pat$style + nc$nco2$venous * pat$venous
   lp_nce <- nc$nce$intercept + nc$nce$style * pat$style + nc$nce$frailty * pat$frailty
   kappa_post <- ef$kappa * exp(-ef$skill_kappa * pat$skill)
 
   # state
   Z <- pat$z0
+  Zn <- pat$z0     # severity on the natural course without any PVI (relief or kappa): drives NCO-1 (D53)
   alive <- rep(TRUE, N); treated <- rep(FALSE, N); clti_true <- rep(FALSE, N)
   clti_code <- rep(FALSE, N); male <- rep(FALSE, N)
   t_pvi <- t_clti_true <- t_clti_code <- t_amp <- t_bypass <- t_rr <- t_male <- t_death <-
@@ -147,8 +149,11 @@ simulate_course <- function(pop, par, pol = policy("natural"), seed = 1L, record
   for (t in 0:(M - 1)) {
     U <- matrix(runif(N * ns), N, ns, dimnames = list(NULL, STREAMS))
     a <- alive
-    if (t > 0) Z[a] <- Z[a] + pat$rate[a] + kappa_post[a] * treated[a] +
-      par$severity$eps_sd * qnorm(U[a, "eps"])
+    if (t > 0) {
+      eps <- par$severity$eps_sd * qnorm(U[a, "eps"])
+      Z[a] <- Z[a] + pat$rate[a] + kappa_post[a] * treated[a] + eps
+      Zn[a] <- Zn[a] + pat$rate[a] + eps
+    }
     if (record) {
       zmat[a, t + 1] <- Z[a]
       cmat[a, t + 1] <- qpois(U[a, "proxy"], exp(par$measurement$proxy_intercept + par$measurement$proxy_z * Z[a]))
@@ -183,7 +188,7 @@ simulate_course <- function(pop, par, pol = policy("natural"), seed = 1L, record
     }
 
     # negative controls (never depend on PVI)
-    x <- a & is.na(t_nco1) & U[, "nco1"] < plogis(lp_nco1); t_nco1[x] <- t
+    x <- a & is.na(t_nco1) & U[, "nco1"] < plogis(lp_nco1 + nco1_z * Zn); t_nco1[x] <- t
     x <- a & is.na(t_nco2) & U[, "nco2"] < plogis(lp_nco2); t_nco2[x] <- t
     if (t < W) { x <- a & is.na(t_nce) & U[, "nce"] < plogis(lp_nce); t_nce[x] <- t }
 
@@ -212,8 +217,11 @@ simulate_course <- function(pop, par, pol = policy("natural"), seed = 1L, record
 
   # disease composite (D34): coded CLTI or major amputation
   t_disease <- pmin(t_clti_code, t_amp, na.rm = TRUE)
+  # limb MALE (primary, D56): coded CLTI, major amputation or open bypass. Every component
+  # can occur with or without a prior PVI; repeat revascularization is reported separately
+  t_limb <- pmin(t_clti_code, t_amp, t_bypass, na.rm = TRUE)
   out <- data.table(id = pat$id, t_pvi, pvi_elective, t_clti_true, t_clti_code, t_amp, t_bypass,
-                    t_rr, t_male, t_disease, t_death, t_nco1, t_nco2, t_nce, n_proc)
+                    t_rr, t_male, t_disease, t_limb, t_death, t_nco1, t_nco2, t_nce, n_proc)
   list(events = out, z = zmat, counts = cmat, policy = pol)
 }
 

pvisim/R/truths.R

@@ -9,7 +9,7 @@
 # Landmark, naive and RCT estimands are not interventional in this sense; they are added
 # with the estimators (oracle large-sample limits).
 
-OUTCOMES <- c(male = "t_male", disease = "t_disease", clti = "t_clti_code", amp = "t_amp",
+OUTCOMES <- c(limb = "t_limb", male = "t_male", disease = "t_disease", clti = "t_clti_code", amp = "t_amp",
               bypass = "t_bypass", rr = "t_rr", death = "t_death", nco1 = "t_nco1", nco2 = "t_nco2")
 
 truth_population <- function(par, n, seed) {
@@ -83,7 +83,7 @@ compute_truths <- function(world, scenario, n = 1e6, seed = 20261027L, horizon =
                                        value = rd, mcse = sqrt(var / n_elig), n = n_elig)]
   }
 
-  curves <- rbindlist(lapply(names(ev), function(p) rbindlist(lapply(c("male", "disease"), function(o)
+  curves <- rbindlist(lapply(names(ev), function(p) rbindlist(lapply(c("limb", "male", "disease"), function(o)
     data.table(world = world, scenario = scenario, policy = p, outcome = o, month = seq_len(horizon),
                cuminc = vapply(seq_len(horizon), function(h) mean(by_h(ev[[p]][[OUTCOMES[[o]]]], h)), 0))))))
   if (verbose) message(sprintf("%s/%s: %.1f min", world, scenario,

pvisim/inst/params/dgp_default.yml

@@ -78,12 +78,13 @@ outcomes:
   clti_revasc: {intercept: -4.5}                      # PVI after CLTI (not elective); most CLTI patients get bypass, tibial work, amputation or nothing
 
 negative_controls:           # never depend on PVI
-  nco1: {intercept: -6.4, frailty: 0.5, smoking: 0.4, age10: 0.3}   # hip fracture / pneumonia
+  nco1: {intercept: -6.75, frailty: 0.5, smoking: 0.4, age10: 0.3, z: 0.3}   # hip fracture / pneumonia; z = PVI-free severity (D53)
   nco2: {intercept: -7.1, style: 0.8, venous: 0.5}                  # varicose-vein ablation
   nce:  {intercept: -5.7, style: 0.8, frailty: -0.2}                # low-value procedure, months 0-5
 
 measurement:                 # what the analyst sees. World W5 switches to claims proxies
-  severity: exact            # exact: analyst sees Z_t. proxy: only visit/testing counts
+  severity: exact            # exact: analyst sees Z_t. proxy: only visit/testing counts. noisy: Z_t + error (EHR/registry)
+  noise_sd: 0.75             # noisy: SD of the monthly measurement error; Z_t has SD about 1 at diagnosis, so cor(observed, true) is about 0.8
   proxy_intercept: -1.0      # monthly count ~ Poisson(exp(proxy_intercept + proxy_z * Z_t))
   proxy_z: 0.5
   proxy_window: 3            # observed severity = log1p(sum of counts over the last 3 months)
@@ -113,6 +114,8 @@ worlds:                      # one factor at a time (plan section 5.2, D24). Ove
     calibration_targets: {late_not: 0.15}
   W5:
     measurement: {severity: proxy, frailty_reliability: 0.4, smoking: coded}
+  WE:                        # EHR/registry-like: severity recorded (ABI, symptom class) but with error (D55)
+    measurement: {severity: noisy, frailty_reliability: 0.7, smoking: true}
   WR:
     effect: {skill_kappa: 0.5}
     treatment: {window: {style: 1.6}, deferred: {style: 1.6, z: 0.8}}   # deferral at the Sorber rate, but severity-driven

pvisim/tests/testthat/test-simulate.R

@@ -20,7 +20,7 @@ test_that("under the sharp null, disease, death and negative controls are identi
                       policy("seq_force", 3L)),
                  function(p) simulate_course(sw$pop, sw$par, p, seed = 5L, record = FALSE)$events)
   for (r in runs[-1]) {
-    for (col in c("t_disease", "t_clti_code", "t_amp", "t_death", "t_nco1", "t_nco2", "t_nce")) {
+    for (col in c("t_limb", "t_disease", "t_clti_code", "t_amp", "t_bypass", "t_death", "t_nco1", "t_nco2", "t_nce")) {
       expect_identical(r[[col]], runs[[1]][[col]], info = col)
     }
   }

reports/draft-report.Rmd

@@ -0,0 +1,311 @@
+---
+title: "Early vs delayed PVI for claudication: known-truth simulation, draft results"
+subtitle: "Draft scale: n = 50,000 patients, 100 replicates per world x scenario"
+date: "`r format(Sys.Date())`"
+output:
+  html_document:
+    toc: true
+    toc_float: true
+    self_contained: true
+---
+
+```{r setup, include = FALSE}
+knitr::opts_chunk$set(echo = FALSE, message = FALSE, warning = FALSE, fig.align = "center", dpi = 120)
+suppressPackageStartupMessages({ library(data.table); library(ggplot2); library(patchwork) })
+root <- normalizePath("..")
+S <- file.path(root, "results", "summary")
+perf <- fread(file.path(S, "perf.csv")); nc <- fread(file.path(S, "nc_reps.csv"))
+ncs <- fread(file.path(S, "nc_summary.csv")); reps <- fread(file.path(S, "reps.csv"))
+
+WORLDS <- c("W0", "W1", "W1_rho0", "W2", "W3", "W4", "W5", "WE", "WR")
+WLAB <- c(W0 = "W0 Clean", W1 = "W1 Skill", W1_rho0 = "W1 Skill (rho 0)", W2 = "W2 Style",
+          W3 = "W3 Style -> outcome", W4 = "W4 Deferral", W5 = "W5 Unmeasured severity", WE = "WE EHR/registry-like", WR = "WR Realistic")
+VAR <- c(V2_cluster = "Ignore physician (cluster SE)", V3_random = "Random intercept",
+         V4_within = "Within-physician (FE)", none = "Trial (no adjustment)")
+VCOL <- c("Ignore physician (cluster SE)" = "#2a78d6", "Random intercept" = "#eb6834",
+          "Within-physician (FE)" = "#1baf7a", "Trial (no adjustment)" = "#52514e")
+METH <- data.table(
+  k = c("ccw|A_B1", "ccw|A_B2", "seq|now_not", "landmark|early_none", "landmark_m5|early_none", "naive|early_none",
+          "rct_realistic|itt", "rct_ideal|itt", "rct_realistic|as_treated", "rct_ideal|as_treated"),
+  method = c("CCW, A vs B1 (reference)", "CCW, A vs B2", "Sequential trials, ITT", "Landmark at 6 months", "Landmark, + month-5 severity (suppl.)",
+             "Naive 'ever early PVI'", "Realistic RCT (n = 240), ITT", "Ideal pragmatic RCT, ITT",
+             "Realistic RCT, as-treated", "Ideal RCT, as-treated"))
+setnames(METH, "k", "key")
+pp <- function(x, k = 1) formatC(100 * x, format = "f", digits = k)
+theme_set(theme_minimal(base_size = 11) + theme(panel.grid.minor = element_blank(),
+          panel.grid.major.y = element_blank(), strip.text = element_text(face = "bold", hjust = 0),
+          legend.position = "bottom"))
+perf_h <- perf[analysis == "rct"]                      # trials at 12, 24 and 60 months (D55)
+perf <- perf[analysis %in% c("main", "rct") & horizon == 60L]   # NCE analyses are used only in Table 4
+reps <- reps[horizon == 60L]
+perf[, key := paste(design, contrast, sep = "|")]
+perf <- merge(perf, METH, by = "key")
+perf[, method := factor(method, levels = rev(METH$method))]
+perf[, wlab := factor(WLAB[world], levels = WLAB[WORLDS])]
+perf[, vlab := factor(VAR[variant], levels = VAR)]
+```
+
+# Read this first
+
+* **Run.** 8 worlds × 3 truth scenarios × 100 replicates of 50,000 patients (about 1,500 physicians per
+  replicate). Each replicate runs naive, landmark, CCW and sequential trials under 4 physician
+  variants, 3 negative controls, an oracle run, and 2 simulated RCTs. 2,400 replicate files, 0
+  errors. 16 of 288,000 estimates are missing (sequential trials with physician fixed effects, 4
+  replicates), and these are dropped.
+* **Outcome shown.** The headline outcome is **limb MALE (CLTI, major amputation or open bypass; D56)**,
+  whose true effect under the null is exactly 0 (D34). MALE is shown where it adds something.
+* **Two errors, kept separate (D27).** *Estimation bias* = mean estimate − the method's own truth.
+  *Question gap* = the method's own truth − the reference truth (CCW A vs B1). *Total error* is the sum.
+* **Variants shown.** "Ignore physician" is V1's point estimate with the cluster-robust SE (V2).
+  The V1 patient-level SE is in the supplement tables.
+* **Two open questions** came out of this run (Section 6). They affect the landmark results and the
+  negative-control results, so treat those parts as provisional.
+
+# 1. Aim 1: what each method answers (clean world W0)
+
+```{r tab2}
+t2 <- perf[world == "W0" & outcome == "limb" & variant %in% c("V2_cluster", "none") &
+             !grepl("as_treated", key)]
+t2 <- t2[, .(Scenario = scenario, Method = as.character(method), `Reference truth` = pp(ref),
+             `Own truth` = pp(own), `Question gap` = pp(gap), `Mean estimate` = pp(mean_est),
+             `Estimation bias (MCSE)` = sprintf("%s (%s)", pp(bias, 2), pp(mcse_bias, 2)),
+             `Empirical SE` = pp(emp_se), `Coverage of own truth` = sprintf("%.0f%%", 100 * cover_own))]
+t2[, Scenario := factor(Scenario, levels = c("null", "harm", "benefit_harm"))]
+t2[, Method := factor(Method, levels = METH$method)]
+knitr::kable(t2[order(Scenario, Method)], align = "llrrrrrrr",
+  caption = "Table 2. Disease composite, 5-year risk difference in percentage points, world W0.")
+```
+
+**What it shows.**
+
+* Every TTE method is close to unbiased for **its own** truth in W0 (all |bias| < 0.35 pp). H1a holds.
+* The methods answer **different questions**. Under harm (reference +5.1 pp): sequential trials target
+  +5.7, landmark +7.9 and naive +7.2. So the question gap alone is 0.6–2.8 pp. Under the null, naive
+  has a gap of +1.3 pp from immortal time even with every confounder measured. H1b holds, though the
+  gap for the sequential trials is small.
+* **CCW is right but imprecise.** Its empirical SE is about 2.8 pp at n = 50,000, against about 1.5 pp for
+  sequential trials. Under the null its 95% CI excludes 0 in 15% of replicates (Section 3).
+* The **realistic RCT (n = 240)** is unbiased for its own ITT, but its SE is about 4.2 pp. It detects
+  the 5-point harm in only 25% of replicates.
+
+```{r fig1, fig.width = 11, fig.height = 9}
+ladder <- function(sc, oc = "limb") {
+  x <- perf[scenario == sc & outcome == oc & variant != "V1_ignore" & !grepl("as_treated", key)]
+  r <- reps[scenario == sc & outcome == oc & analysis %in% c("main", "rct") & variant != "V1_ignore" &
+              variant != "oracle" & !grepl("as_treated", contrast)]
+  r[, key := paste(design, contrast, sep = "|")]
+  q <- r[, .(lo = quantile(est, 0.025), hi = quantile(est, 0.975)), by = .(world, key, variant)]
+  x <- merge(x, q, by = c("world", "key", "variant"))
+  dodge <- position_dodge(width = 0.6)
+  ggplot(x, aes(y = method, group = vlab)) +
+    geom_vline(aes(xintercept = 100 * ref), colour = "#0b0b0b", linewidth = 0.4) +
+    geom_vline(xintercept = 0, colour = "#c3c2b7", linewidth = 0.3, linetype = 2) +
+    geom_linerange(aes(xmin = 100 * lo, xmax = 100 * hi, colour = vlab), position = dodge, linewidth = 0.6) +
+    geom_point(aes(x = 100 * own), position = dodge, shape = 21, size = 2.6, fill = "white", colour = "#0b0b0b") +
+    geom_point(aes(x = 100 * mean_est, colour = vlab), position = dodge, size = 2) +
+    facet_wrap(~wlab, ncol = 4) + scale_colour_manual(values = VCOL, name = NULL) +
+    labs(x = "5-year risk difference, early − delayed (percentage points)", y = NULL,
+         caption = paste("Filled dot: mean estimate (bar: 2.5th–97.5th percentile across replicates).",
+                         "Hollow dot: the method's own truth. Solid line: the reference truth (CCW A vs B1).")) +
+    coord_cartesian(xlim = c(-15, 35))
+}
+ladder("harm") + ggtitle("Figure 1a. Estimand ladder, harm scenario (limb MALE)")
+```
+
+```{r fig1b, fig.width = 11, fig.height = 9}
+ladder("null") + ggtitle("Figure 1b. Estimand ladder, null scenario (limb MALE)")
+```
+
+# 2. Aim 2: surgery-specific threats
+
+```{r tab3}
+x <- perf[scenario == "harm" & outcome == "limb" & analysis == "main" &
+            variant %in% c("V2_cluster", "V3_random", "V4_within") & key != "ccw|A_B2"]
+x[, cell := sprintf("%s (%s)", pp(bias), pp(mcse_bias))]
+t3 <- dcast(x, method + vlab ~ wlab, value.var = "cell")
+setnames(t3, c("method", "vlab"), c("Method", "Physician variant"))
+knitr::kable(t3[order(-as.integer(Method))], caption = paste(
+  "Table 3. Estimation bias (MCSE) against each method's own truth, percentage points.",
+  "Harm scenario, limb MALE."))
+```
+
+```{r tab3b}
+x[, cell2 := pp(total)]
+t3b <- dcast(x, method + vlab ~ wlab, value.var = "cell2")
+setnames(t3b, c("method", "vlab"), c("Method", "Physician variant"))
+knitr::kable(t3b[order(-as.integer(Method))], caption = paste(
+  "Table 3b. Total error against the reference truth (CCW A vs B1), percentage points.",
+  "Harm scenario, limb MALE."))
+```
+
+**What it shows.**
+
+* **Surgeon skill (W1).** CCW and sequential trials stay within about 1 pp, and the FE variant brings
+  them to 0.2–0.3 pp. The **landmark is badly off (total error about 14–16 pp)**. Almost all of that is
+  question gap, not estimation bias, and Section 6 (Q1) explains why.
+* **Practice style (W2).** It only harms the **naive** design (bias −9 to −12 pp). The TTE designs
+  absorb style through their time-varying PVI models.
+* **Style that also changes the outcome coding (W3).** It biases landmark and sequential trials by
+  about 1.5 pp when the physician is ignored. A random intercept or FE removes most of it, which
+  supports H2b.
+* **Deferral / crossover (W4).** No design bias. The TTE designs handle crossover by design.
+* **Unmeasured severity (W5).** It biases **every** observational design by about 2.5–5.5 pp, and no
+  physician variant helps (H2c holds).
+* **Realistic world (WR).** CCW is biased by +6 to +7 pp and sequential trials by +7 to +9 pp. FE helps
+  only a little. Unmeasured severity dominates.
+
+# 3. False alarms under the null (headline for clinicians)
+
+```{r fig2, fig.width = 10, fig.height = 5.2}
+fa <- perf[scenario == "null" & outcome == "limb" & analysis == "main" & key != "ccw|A_B2" & variant != "V1_ignore"]
+fa[, row := factor(paste0(method, " · ", sub(" \\(.*", "", vlab)))]
+fa[, row := factor(row, levels = rev(sort(levels(row))))]
+p1 <- ggplot(fa, aes(wlab, row, fill = 100 * reject0)) + geom_tile(colour = "white", linewidth = 1) +
+  geom_text(aes(label = sprintf("%.0f", 100 * reject0), colour = reject0 > 0.5), size = 3) +
+  scale_colour_manual(values = c(`TRUE` = "white", `FALSE` = "#0b0b0b"), guide = "none") +
+  scale_fill_gradient(low = "#eef4fc", high = "#0d3d78", limits = c(0, 100), name = "% of replicates\nCI excludes 0") +
+  labs(x = NULL, y = NULL, title = "Figure 2. False-alarm rate, null scenario (limb MALE; nominal 5%)") +
+  theme(axis.text.x = element_text(angle = 30, hjust = 1), panel.grid = element_blank(), legend.position = "right")
+p1
+```
+
+```{r fig2b, fig.width = 8, fig.height = 3}
+m0 <- perf[scenario == "null" & world == "W0" & variant %in% c("V2_cluster", "none") &
+             outcome %in% c("male", "limb") & !grepl("as_treated", key) & key != "ccw|A_B2"]
+m0[, oc := factor(ifelse(outcome == "male", "MALE incl. repeat revascularization", "Limb MALE"), levels = c("Limb MALE", "MALE incl. repeat revascularization"))]
+ggplot(m0, aes(100 * mean_est, method, colour = oc, group = oc)) +
+  geom_vline(xintercept = 0, colour = "#c3c2b7") +
+  geom_errorbarh(aes(xmin = 100 * (mean_est - 1.96 * emp_se), xmax = 100 * (mean_est + 1.96 * emp_se)), height = 0, linewidth = 0.6, position = position_dodge(width = 0.5)) +
+  geom_point(size = 2, position = position_dodge(width = 0.5)) + scale_colour_manual(values = c("Limb MALE" = "#2a78d6", "MALE incl. repeat revascularization" = "#eb6834"), name = NULL) +
+  labs(x = "Risk difference (pp), W0 null", y = NULL,
+       title = "Figure 2 inset. Sharp null: limb MALE 0, yet MALE incl. repeat revascularization about +14 pp")
+```
+
+**What it shows.**
+
+* In the clean, skill and deferral worlds, landmark, sequential trials and naive are near 5%. **CCW
+  is at 10–15% in W0/W1**, mainly because its SE is slightly underestimated in the null scenario
+  (model/empirical SE 0.86 in W0) and its estimate sits slightly low (−0.33 pp, 1.3 MCSE). The
+  bootstrap check (D28) is meant to settle this.
+* Style that changes outcome coding (W3) pushes naive to 51–55% and sequential trials to 25% when the
+  physician is ignored. A random intercept or FE brings that back to 8–16%.
+* Unmeasured severity (W5, WR) gives **80–100% false alarms** for landmark, sequential trials and
+  naive. CCW stays at 15–24%, but only because its SE is wide. FE lowers the landmark and sequential
+  rates in WR (31% and 60%).
+* **MALE under the sharp null:** +14 pp for CCW, purely because only treated patients can have a
+  repeat revascularization. Every design "detects harm" on MALE in 93–100% of replicates.
+
+# 4. Aim 3: do negative controls detect bias?
+
+```{r tab4}
+s <- ncs[level == "cell"]
+s[, cell := ifelse(is.na(sens), "–", sprintf("%.0f%% / %s", 100 * sens, ifelse(is.na(spec), "–", sprintf("%.0f%%", 100 * spec))))]
+s[, control := factor(c(nco1 = "NCO-1 (hip fracture / pneumonia)", nco2 = "NCO-2 (vein ablation)",
+                        nce = "NCE (early low-value procedure)", any_nc = "Any of the three")[control],
+                      levels = c("NCO-1 (hip fracture / pneumonia)", "NCO-2 (vein ablation)",
+                                 "NCE (early low-value procedure)", "Any of the three"))]
+s[, wl := factor(c(WLAB, all = "All worlds")[world], levels = c(WLAB[WORLDS], "All worlds"))]
+knitr::kable(dcast(s, control ~ wl, value.var = "cell"), caption = paste(
+  "Table 4. Sensitivity / specificity of each negative control for MALE bias > 1 pp.",
+  "A cell (world × scenario × method × variant) counts as biased when its mean MALE bias exceeds 1 pp;",
+  "a replicate is flagged when the control's 95% CI excludes 0. '–': no biased (or no clean) cells in that world."))
+```
+
+```{r fig3, fig.width = 10, fig.height = 4.5}
+nc[, design_lab := factor(c(ccw = "CCW", seq = "Sequential", landmark = "Landmark", naive = "Naive")[design],
+                          levels = c("CCW", "Sequential", "Landmark", "Naive"))]
+nc[, flag := factor(ifelse(any_nc, "A negative control flagged", "All negative controls clean"),
+                    levels = c("All negative controls clean", "A negative control flagged"))]
+nc[, wlab := factor(WLAB[world], levels = WLAB[WORLDS])]
+cb <- nc[, .(absbias = abs(mean(err))), by = .(world, wlab, scenario, design_lab, variant, flag)]
+ggplot(cb, aes(wlab, 100 * absbias, colour = flag)) +
+  geom_point(position = position_jitterdodge(jitter.width = 0.15, dodge.width = 0.6), size = 1.4, alpha = 0.8) +
+  geom_hline(yintercept = 1, linetype = 2, colour = "#52514e") +
+  scale_colour_manual(values = c("All negative controls clean" = "#1baf7a", "A negative control flagged" = "#eb6834"), name = NULL) +
+  facet_wrap(~design_lab, nrow = 1) +
+  labs(x = NULL, y = "|mean MALE bias| (pp)", title = "Figure 3. How much bias sits behind a clean negative control") +
+  theme(axis.text.x = element_text(angle = 45, hjust = 1))
+```
+
+**What it shows (provisional, see Q2).**
+
+* **NCO-1 almost never flags (sensitivity about 12%), even with unmeasured severity (W5).** In the
+  current DGP, NCO-1 depends on frailty, smoking and age but **not on claudication severity**, so it
+  cannot see the main source of bias. H3a fails, by construction.
+* **NCO-2 is the working detector (48% sensitivity overall, 65–67% in WR).** It also fires when style
+  does no harm (specificity 26–45% in W2/W3), which supports H3b.
+* **The NCE almost never flags (about 5–9%).** It is adjusted for early PVI (D36), and it shares style
+  and frailty, not severity.
+* **Bias hides behind clean controls.** In WR, the cells with clean negative controls still have a
+  median |MALE bias| of 8.5 pp (H3c).
+
+# 5. Simulated RCTs
+
+```{r tabrct}
+r <- perf[analysis == "rct" & outcome == "limb" & scenario != "null"]
+r <- r[, .(World = wlab, Scenario = scenario, Trial = method, `Reference` = pp(ref), `Own ITT truth` = pp(own),
+           `Mean estimate` = pp(mean_est), `Empirical SE` = pp(emp_se), `Power (CI excludes 0)` = sprintf("%.0f%%", 100 * reject0))]
+knitr::kable(r[order(World, Scenario, Trial)], caption = "Table 5. Simulated trials, limb MALE (pp).")
+```
+
+* **Surgeon selection changes the answer.** In WR, a realistic trial recruits hospital-based,
+  top-third-skill, low-style surgeons. Its own ITT truth is **+2.1 pp under harm and −1.4 pp under
+  benefit-then-harm**, while the population effects are +6.9 and +7.7. A well-run trial answers a
+  question about selected surgeons (H1c).
+* **Small trials are uninformative.** With n = 240 the SE is about 4 pp, giving 6–29% power against
+  effects of 5–8 pp.
+* The ideal pragmatic RCT (n = 50,000) recovers its truth with an SE of about 0.3–0.6 pp.
+
+# 6. Open questions raised by the draft run
+
+**Q1. Landmark adjusts for severity at month 5, after the PVI.** As specified in D13, the landmark
+propensity model uses covariates at diagnosis **and at month W−1**. For treated patients, month-5
+severity is already changed by the PVI (relief, then faster progression), so it is a mediator.
+Conditioning on it gives extreme weights (effective treated sample about 110–140) and a large own-truth
+shift when the harm varies between patients (W1: own truth about 20 pp vs a reference of 7 pp).
+A check on 20 oracle replicates gave these landmark estimates:
+
+| World (harm) | Adjust for month-5 severity | Baseline covariates only | Reference |
+|---|---|---|---|
+| W0 | 8.6 (SD 3.4) | 5.6 (SD 1.4) | 5.1 |
+| W1 | 14.6 (SD 4.3) | 8.1 (SD 1.6) | 7.2 |
+
+**Q2. NCO-1 does not depend on severity.** D32 says NCO-1 "shares the latent patient frailty,
+severity and smoking", but the DGP's NCO-1 model has no severity term. That is why it misses the
+W5/WR bias.
+
+**Also pending:** the physician-cluster bootstrap check of SEs (D28). Model-to-empirical SE ratios
+are near 1 for CCW (0.96–1.07, except WR 0.87). They are 0.83–0.92 for landmark and sequential trials
+in W0, and 0.58 for landmark in W1.
+
+# 7. Standard errors: physician-cluster bootstrap check (D28, D53)
+
+```{r boot}
+bf <- list.files(file.path(root, "results", "bootstrap"), pattern = "\\.csv$", full.names = TRUE)
+if (length(bf)) {
+  bt <- rbindlist(lapply(bf, fread))
+  mv <- reps[analysis == "main" & variant == "V1_ignore" & rep %in% unique(bt$rep),
+             .(world, scenario, rep, design, contrast, outcome, se_patient, se_cluster)]
+  bt <- merge(bt, mv, by = c("world", "scenario", "rep", "design", "contrast", "outcome"))
+  emp <- perf[analysis == "main" & variant == "V2_cluster", .(world, scenario, design, contrast, outcome, emp_se)]
+  x <- bt[, .(reps = .N, boot = sqrt(mean(boot_se^2)), patient = sqrt(mean(se_patient^2)),
+              cluster = sqrt(mean(se_cluster^2))), by = .(world, scenario, design, contrast, outcome)]
+  x <- merge(x, emp, by = c("world", "scenario", "design", "contrast", "outcome"))
+  x <- x[outcome == "limb" & contrast != "A_B2"]
+  knitr::kable(x[order(world, scenario, design), .(World = world, Scenario = scenario, Design = design,
+    `Empirical SE (100 reps)` = pp(emp_se, 2), `Bootstrap SE` = pp(boot, 2), `Model SE, cluster` = pp(cluster, 2),
+    `Model SE, patient` = pp(patient, 2), `Bootstrap / empirical` = round(boot / emp_se, 2),
+    `Cluster / empirical` = round(cluster / emp_se, 2))],
+    caption = "Table 6. Disease composite. Bootstrap: 50 physician-cluster resamples on each of replicates 1-20.")
+} else cat("Bootstrap results not yet available.")
+```
+
+# Supplement: full performance table
+
+```{r supp}
+x <- perf[analysis == "main" & outcome %in% c("limb", "male", "disease")]
+knitr::kable(x[order(outcome, scenario, world, method, variant),
+  .(outcome, scenario, world, method = as.character(method), variant, est = pp(mean_est, 2), own = pp(own, 2),
+    ref = pp(ref, 2), bias = pp(bias, 2), mcse = pp(mcse_bias, 2), gap = pp(gap, 2), emp_se = pp(emp_se, 2),
+    se_ratio = round(se_ratio, 2), cover = round(cover_own, 2), reject0 = round(reject0, 2))])
+```

simulations/03_limits.R

@@ -11,8 +11,8 @@ jobs <- CJ(world = names(raw$worlds), scenario = names(raw$scenarios), sorted =
 res <- parallel::mclapply(seq_len(nrow(jobs)), function(i) {
   w <- jobs$world[i]; s <- jobs$scenario[i]
   par <- load_params(w, s)
-  rbind(rct_truth(par, "realistic", n = 1e6)[outcome %in% c("male", "disease", "nco1", "nco2")],
-        rct_truth(par, "ideal", n = 1e6)[outcome %in% c("male", "disease", "nco1", "nco2")])
+  rbind(rct_truth(par, "realistic", n = 1e6)[outcome %in% c("limb", "male", "disease", "rr", "nco1", "nco2")],
+        rct_truth(par, "ideal", n = 1e6)[outcome %in% c("limb", "male", "disease", "rr", "nco1", "nco2")])
 }, mc.cores = cores)
 bad <- vapply(res, inherits, TRUE, "try-error")
 if (any(bad)) stop(res[bad][[1]])

simulations/04_summarise.R

@@ -0,0 +1,100 @@
+# Performance against the known truths (plan sections 9 and 10).
+# Usage: Rscript simulations/04_summarise.R [results_dir]
+# Writes results/summary/{reps,perf,nc_reps,nc_summary}.csv.
+
+suppressPackageStartupMessages(library(data.table))
+a <- commandArgs(TRUE)
+indir <- if (length(a) >= 1) a[1] else "results/main/n50k_v2"
+outdir <- "results/summary"
+dir.create(outdir, recursive = TRUE, showWarnings = FALSE)
+
+files <- list.files(indir, pattern = "^rep[0-9]+\\.csv$", recursive = TRUE, full.names = TRUE)
+d <- rbindlist(lapply(files, fread), fill = TRUE)
+cat(sprintf("%d files, %d rows\n", length(files), nrow(d)))
+# rare failures (sequential V4: fixed effects not estimable in a trial month) are dropped and counted
+cat(sprintf("dropped %d estimates with NA\n", d[is.na(est), .N]))
+d <- d[!is.na(est)]
+# trials are analysed at 12, 24 and 60 months (D55); everything else at 60
+if (!"horizon" %in% names(d)) d[, horizon := NA_integer_]
+d[is.na(horizon), horizon := 60L]
+
+tr <- rbind(fread("simulations/truths/truths.csv"), fread("simulations/truths/truths_rct.csv"))
+tr <- tr[, .(world, scenario, outcome, estimand, truth = value, truth_mcse = mcse)]
+
+# ---- own estimand and reference for every estimate -----------------------------------
+# reference (clinically relevant) = RD(A - B1); NCE has a true effect of exactly 0 (D32)
+d[, estimand := fcase(design == "ccw" & contrast == "A_B1", "rd_A_B1",
+                      design == "ccw" & contrast == "A_B2", "rd_A_B2",
+                      design == "seq", "seq_itt",
+                      design %like% "^rct_" & contrast == "itt",
+                        paste0(design, "_itt", fifelse(horizon == 60L, "", paste0("_h", horizon))),
+                      default = NA_character_)]
+d <- merge(d, tr[, .(world, scenario, outcome, estimand, own = truth)],
+           by = c("world", "scenario", "outcome", "estimand"), all.x = TRUE)
+d <- merge(d, tr[estimand == "rd_A_B1", .(world, scenario, outcome, ref = truth)],
+           by = c("world", "scenario", "outcome"), all.x = TRUE)
+# at the shorter trial horizons the reference is RD(A - B1) at that month, from the risk curves
+cv <- dcast(fread("simulations/truths/risk_curves.csv")[policy %in% c("A", "B1")],
+            world + scenario + outcome + month ~ policy, value.var = "cuminc")
+d <- merge(d, cv[, .(world, scenario, outcome, horizon = month, ref_h = A - B1)],
+           by = c("world", "scenario", "outcome", "horizon"), all.x = TRUE)
+d[horizon != 60L, ref := ref_h][, ref_h := NULL]
+# landmark and naive: own estimand = the paired oracle estimate in the same replicate (D47)
+orc <- d[variant == "oracle", .(world, scenario, rep, design, outcome, oracle = est)]
+d <- merge(d, orc, by = c("world", "scenario", "rep", "design", "outcome"), all.x = TRUE)
+d[design %in% c("landmark", "landmark_m5", "naive") & variant != "oracle", own := oracle]
+d[analysis %like% "^nce", `:=`(own = 0, ref = 0)]
+d[, oracle := NULL]
+d[, `:=`(lo = est - 1.96 * se, hi = est + 1.96 * se)]
+fwrite(d, file.path(outdir, "reps.csv"))
+
+# ---- performance by cell (plan 9) -----------------------------------------------------
+keys <- c("world", "scenario", "analysis", "design", "contrast", "variant", "outcome", "horizon")
+perf <- d[variant != "oracle", {
+  err <- est - own; R <- .N; own_m <- mean(own)
+  .(reps = R, mean_est = mean(est), own = own_m, ref = ref[1],
+    bias = mean(err), mcse_bias = sd(err) / sqrt(R),
+    gap = own_m - ref[1], total = mean(est) - ref[1],
+    emp_se = sd(est), mod_se = sqrt(mean(se^2)),
+    se_ratio = sqrt(mean(se^2)) / sd(est),
+    cover_own = mean(lo <= own_m & hi >= own_m),
+    cover_ref = mean(lo <= ref[1] & hi >= ref[1]),
+    reject0 = mean(lo > 0 | hi < 0),
+    rmse_own = sqrt(mean((est - own_m)^2)), rmse_ref = sqrt(mean((est - ref[1])^2)))
+}, by = keys]
+perf[, `:=`(mcse_cover = sqrt(cover_own * (1 - cover_own) / reps),
+            mcse_reject = sqrt(reject0 * (1 - reject0) / reps))]
+fwrite(perf, file.path(outdir, "perf.csv"))
+
+# ---- negative controls as bias detectors (plan 10) ------------------------------------
+# One unit = world x scenario x design x variant x replicate. Bias is in the limb-MALE RD (D56)
+# against the method's own estimand; a control "flags" when its 95% CI excludes 0.
+m <- d[analysis == "main" & design != "landmark_m5" & variant %in% c("V1_ignore", "V2_cluster", "V3_random", "V4_within") &
+         contrast %in% c("early_none", "A_B1", "now_not")]
+u <- c("world", "scenario", "design", "variant", "rep")
+flag <- function(x) x$lo > 0 | x$hi < 0
+nc <- m[outcome == "limb", c(..u, "est", "own")][, err := est - own]
+nc <- merge(nc, m[outcome == "nco1", c(..u)][, nco1 := flag(m[outcome == "nco1"])], by = u, all.x = TRUE)
+nc <- merge(nc, m[outcome == "nco2", c(..u)][, nco2 := flag(m[outcome == "nco2"])], by = u, all.x = TRUE)
+ne <- d[analysis == "nce" & outcome == "limb" & contrast %in% c("early_none", "A_B1")]
+nc <- merge(nc, ne[, c(..u)][, nce := flag(ne)], by = u, all.x = TRUE)
+nc[, any_nc := nco1 | nco2 | fifelse(is.na(nce), FALSE, nce)]
+nc[, cell_bias := mean(err), by = .(world, scenario, design, variant)]
+nc[, `:=`(biased_rep = abs(err) > 0.01, biased_cell = abs(cell_bias) > 0.01)]
+fwrite(nc, file.path(outdir, "nc_reps.csv"))
+
+acc <- function(x, truth) {
+  ok <- !is.na(x)
+  list(sens = mean(x[ok & truth]), spec = mean(!x[ok & !truth]),
+    n_biased = sum(ok & truth), n_clean = sum(ok & !truth))
+}
+ncs <- rbindlist(lapply(c("nco1", "nco2", "nce", "any_nc"), function(k) rbind(
+  nc[, c(control = k, level = "replicate", acc(get(k), biased_rep)), by = world],
+  nc[, c(control = k, level = "cell", acc(get(k), biased_cell)), by = world],
+  nc[, c(world = "all", control = k, level = "replicate", acc(get(k), biased_rep))],
+  nc[, c(world = "all", control = k, level = "cell", acc(get(k), biased_cell))])))
+hide <- nc[any_nc == FALSE, .(control = "any_nc", med_absbias_clean = median(abs(cell_bias), na.rm = TRUE),
+                              p90_absbias_clean = quantile(abs(cell_bias), 0.9, na.rm = TRUE)), by = world]
+ncs <- merge(ncs, hide, by = c("world", "control"), all.x = TRUE)
+fwrite(ncs, file.path(outdir, "nc_summary.csv"))
+cat("wrote", outdir, "\n")

simulations/05_bootstrap.R

@@ -0,0 +1,35 @@
+# Physician-cluster bootstrap check of the model-based SEs (D28, reduced in D53).
+# Usage: Rscript simulations/05_bootstrap.R [reps] [B] [cores]
+# Writes results/bootstrap/<world>_<scenario>_rep<k>.csv: the estimate and the
+# bootstrap SD for each design, contrast and outcome (V1 point estimates).
+
+pkgload::load_all("pvisim", quiet = TRUE)
+suppressPackageStartupMessages(library(data.table))
+a <- commandArgs(TRUE)
+reps <- if (length(a) >= 1) as.integer(a[1]) else 20L
+B <- if (length(a) >= 2) as.integer(a[2]) else 50L
+cores <- if (length(a) >= 3) as.integer(a[3]) else 11L
+cfg <- yaml::read_yaml("simulations/config/experiments.yml")
+n <- cfg$scales$draft$n_patients
+outdir <- "results/bootstrap"; dir.create(outdir, showWarnings = FALSE, recursive = TRUE)
+designs <- c("naive", "landmark", "landmark_m5", "ccw", "seq")
+outs <- c(limb = "t_limb", male = "t_male", disease = "t_disease")
+
+jobs <- CJ(world = c("W0", "WR"), scenario = c("null", "harm"), rep = seq_len(reps), sorted = FALSE)
+jobs[, file := file.path(outdir, sprintf("%s_%s_rep%03d.csv", world, scenario, rep))]
+todo <- jobs[!file.exists(file)]
+cat(sprintf("bootstrap: %d jobs, %d to do, B = %d, %d cores\n", nrow(jobs), nrow(todo), B, cores))
+res <- parallel::mclapply(seq_len(nrow(todo)), function(i) tryCatch({
+  j <- todo[i]
+  par <- load_params(j$world, j$scenario)
+  rec <- simulate_records(par, rep = j$rep, n = n)          # the same data as the main run's replicate
+  bs <- rbindlist(lapply(seq_len(B), function(b)
+    cbind(b = b, run_designs(bootstrap_records(rec, seed = 1e6 * j$rep + b), designs, "V1_ignore", outs))), fill = TRUE)
+  out <- bs[, .(boot_se = sd(est, na.rm = TRUE), boot_mean = mean(est, na.rm = TRUE), B = sum(!is.na(est))),
+            by = .(design, contrast, outcome)]
+  fwrite(out[, `:=`(world = j$world, scenario = j$scenario, rep = j$rep)], j$file)
+  NULL
+}, error = function(e) sprintf("%s/%s/%d: %s", todo$world[i], todo$scenario[i], todo$rep[i], conditionMessage(e))),
+mc.cores = cores, mc.preschedule = FALSE)
+err <- unlist(res[!vapply(res, is.null, TRUE)])
+cat(sprintf("done; %d errors\n", length(err))); if (length(err)) cat(head(err, 3), sep = "\n")

simulations/config/calibrated.yml

@@ -102,3 +102,16 @@ WR:
       intercept: -7.5233
     death:
       intercept: -5.6489
+WE:
+  treatment:
+    window:
+      intercept: -6.3552
+    deferred:
+      intercept: -7.7172
+  outcomes:
+    repeat_revasc:
+      intercept: -5.6375
+    clti:
+      intercept: -7.4903
+    death:
+      intercept: -5.6504

simulations/config/experiments.yml

@@ -1,12 +1,16 @@
 # Experiment grids (plan sections 5.2, 7, 9). Run with simulations/run_experiment.R <grid> [--scale draft|final]
 main:
-  worlds: [W0, W1, W1_rho0, W2, W3, W4, W5, WR]
+  worlds: [W0, W1, W1_rho0, W2, W3, W4, W5, WE, WR]   # WE: EHR/registry-like measurement (D55)
   scenarios: ["null", harm, benefit_harm]
-  designs: [naive, landmark, ccw, seq]
+  designs: [naive, landmark, landmark_m5, ccw, seq]   # landmark_m5: also adjusts for month-5 severity (supplement, D53)
   variants: [V1_ignore, V3_random, V4_within]     # V1 also yields V2 (cluster-robust SE)
   nce: true                                        # negative-control exposure (landmark, CCW), D32/D36
+  # trims (D57): supplement-only or uninformative cells are skipped to save time
+  v1_only_designs: [landmark_m5]                   # month-5 landmark (supplement) only without physician terms
+  nce_variants: [V1_ignore, V4_within]             # NCE without the random intercept
+  skip_variant: {V3_random: [W0, W4]}              # worlds where the physician plays no role by design
   rct: [realistic, ideal]
 scales:
   smoke: {n_patients: 20000, reps: 1}
-  draft: {n_patients: 50000, reps: 100}
-  final: {n_patients: 50000, reps: 500}                     # to be revisited after the draft (sample size, D25/D29)
+  draft: {n_patients: 50000, reps: 100, dir: n50k_v2}   # v2: D53 (landmark baseline covariates, NCO-1 severity)
+  final: {n_patients: 50000, reps: 200, dir: n50k_v2}   # extends the draft's replicates 1-100 to 200 (D51)

simulations/run_experiment.R

@@ -2,6 +2,8 @@
 # Usage: Rscript simulations/run_experiment.R main [--scale draft] [--cores 10] [--worlds W0,W2]
 pkgload::load_all("pvisim", quiet = TRUE)
 suppressPackageStartupMessages(library(data.table))
+# one thread per worker: parallelism comes from mclapply across replicates
+setDTthreads(1); fixest::setFixest_nthreads(1)
 a <- commandArgs(TRUE)
 opt <- function(flag, default) { i <- match(flag, a); if (is.na(i)) default else a[i + 1] }
 grid_name <- a[1]
@@ -10,7 +12,7 @@ cores <- as.integer(opt("--cores", "10"))
 cfg <- yaml::read_yaml("simulations/config/experiments.yml")
 g <- cfg[[grid_name]]; sc <- cfg$scales[[scale]]
 worlds <- if (is.null(opt("--worlds", NULL))) g$worlds else strsplit(opt("--worlds", NULL), ",")[[1]]
-outdir <- file.path("results", grid_name, scale)
+outdir <- file.path("results", grid_name, if (is.null(sc$dir)) scale else sc$dir)
 jobs <- CJ(world = worlds, scenario = unlist(g$scenarios), rep = seq_len(sc$reps), sorted = FALSE)
 jobs[, file := file.path(outdir, world, scenario, sprintf("rep%03d.csv", rep))]
 todo <- jobs[!file.exists(file)]
@@ -21,25 +23,20 @@ one_job <- function(i) {
   par <- load_params(j$world, j$scenario)
   rec <- simulate_records(par, rep = j$rep, n = sc$n_patients)
   out <- list()
+  with_v2 <- function(r) if (r$variant[1] == "V1_ignore")   # same fit, two SEs
+    rbind(r, copy(r)[, `:=`(variant = "V2_cluster", se = se_cluster)]) else r
+  nce_out <- c(limb = "t_limb", male = "t_male", disease = "t_disease")
   for (v in g$variants) {
-    for (d in g$designs) {
-      r <- run_design(rec, d, v)
-      if (v == "V1_ignore") {   # same fit, two SEs
-        r2 <- copy(r)[, `:=`(variant = "V2_cluster", se = se_cluster)]
-        r <- rbind(r, r2)
-      }
-      out[[length(out) + 1]] <- r[, analysis := "main"]
-    }
-    if (isTRUE(g$nce)) for (d in c("landmark", "ccw")) {
-      r <- run_design(nce_records(rec), d, v, outcomes = c(male = "t_male", disease = "t_disease"))
-      if (v == "V1_ignore") r <- rbind(r, copy(r)[, `:=`(variant = "V2_cluster", se = se_cluster)])
-      out[[length(out) + 1]] <- r[, analysis := "nce"]
-    }
-  }
-  for (d in c("landmark", "ccw")) {     # unadjusted NCE (supplement, D36), V1/V2 only
-    r <- run_design(nce_records(rec, adjust_pvi = FALSE), d, "V1_ignore", outcomes = c(male = "t_male", disease = "t_disease"))
-    out[[length(out) + 1]] <- r[, analysis := "nce_unadjusted"]
+    if (j$world %in% unlist(g$skip_variant[[v]])) next
+    des <- unlist(g$designs)
+    if (v != "V1_ignore") des <- setdiff(des, unlist(g$v1_only_designs))
+    out[[length(out) + 1]] <- with_v2(run_designs(rec, des, v))[, analysis := "main"]
+    if (isTRUE(g$nce) && (is.null(g$nce_variants) || v %in% unlist(g$nce_variants)))
+      out[[length(out) + 1]] <- with_v2(run_designs(nce_records(rec), c("landmark", "ccw"), v, nce_out))[, analysis := "nce"]
   }
+  # unadjusted NCE (supplement, D36), V1/V2 only
+  out[[length(out) + 1]] <- with_v2(run_designs(nce_records(rec, adjust_pvi = FALSE), c("landmark", "ccw"),
+                                                "V1_ignore", nce_out))[, analysis := "nce_unadjusted"]
   out[[length(out) + 1]] <- run_oracle(rec)[, analysis := "main"]   # own estimand of landmark / naive
   for (dz in g$rct) {
     r <- run_rct(par, dz, rep = j$rep)

simulations/truths/truths_rct.csv

@@ -0,0 +1,973 @@
+world,scenario,outcome,estimand,value,mcse,n
+W0,null,limb,rct_realistic_itt_h12,0,0,138782
+W0,null,limb,rct_realistic_itt_h24,0,0,138782
+W0,null,limb,rct_realistic_itt,0,0,138782
+W0,null,male,rct_realistic_itt_h12,0.0316896791667746,0.000489980259881347,138782
+W0,null,male,rct_realistic_itt_h24,0.066135680620639,0.000696255343077188,138782
+W0,null,male,rct_realistic_itt,0.140661174578609,0.00098072361759117,138782
+W0,null,disease,rct_realistic_itt_h12,0,0,138782
+W0,null,disease,rct_realistic_itt_h24,0,0,138782
+W0,null,disease,rct_realistic_itt,0,0,138782
+W0,null,rr,rct_realistic_itt_h12,0.0329190178576472,0.000498508344475691,138782
+W0,null,rr,rct_realistic_itt_h24,0.0717591137320835,0.000722074502261802,138782
+W0,null,rr,rct_realistic_itt,0.176524278977813,0.00107249504657992,138782
+W0,null,nco1,rct_realistic_itt_h12,0,0,138782
+W0,null,nco1,rct_realistic_itt_h24,0,0,138782
+W0,null,nco1,rct_realistic_itt,0,0,138782
+W0,null,nco2,rct_realistic_itt_h12,0,0,138782
+W0,null,nco2,rct_realistic_itt_h24,0,0,138782
+W0,null,nco2,rct_realistic_itt,0,0,138782
+W0,null,limb,rct_ideal_itt_h12,0,0,1000000
+W0,null,limb,rct_ideal_itt_h24,0,0,1000000
+W0,null,limb,rct_ideal_itt,0,0,1000000
+W0,null,male,rct_ideal_itt_h12,0.0313360000000107,0.000174379055808226,1000000
+W0,null,male,rct_ideal_itt_h24,0.0641560000000435,0.000245132632800575,1000000
+W0,null,male,rct_ideal_itt,0.132398000000112,0.000338976060504823,1000000
+W0,null,disease,rct_ideal_itt_h12,0,0,1000000
+W0,null,disease,rct_ideal_itt_h24,0,0,1000000
+W0,null,disease,rct_ideal_itt,0,0,1000000
+W0,null,rr,rct_ideal_itt_h12,0.0327740000000121,0.000178207364955272,1000000
+W0,null,rr,rct_ideal_itt_h24,0.0702760000000496,0.000255728926453558,1000000
+W0,null,rr,rct_ideal_itt,0.166642000000146,0.000372733743888863,1000000
+W0,null,nco1,rct_ideal_itt_h12,0,0,1000000
+W0,null,nco1,rct_ideal_itt_h24,0,0,1000000
+W0,null,nco1,rct_ideal_itt,0,0,1000000
+W0,null,nco2,rct_ideal_itt_h12,0,0,1000000
+W0,null,nco2,rct_ideal_itt_h24,0,0,1000000
+W0,null,nco2,rct_ideal_itt,0,0,1000000
+W0,harm,limb,rct_realistic_itt_h12,-0.000634779123119918,8.59030281036993e-05,138782
+W0,harm,limb,rct_realistic_itt_h24,0.00357723457388988,0.000197648179331725,138782
+W0,harm,limb,rct_realistic_itt,0.0657009909762577,0.000715048507757777,138782
+W0,harm,male,rct_realistic_itt_h12,0.0305563168266289,0.000494042770555967,138782
+W0,harm,male,rct_realistic_itt_h24,0.0712904014827441,0.000728275405042644,138782
+W0,harm,male,rct_realistic_itt,0.209013847241733,0.00115467602996962,138782
+W0,harm,disease,rct_realistic_itt_h12,-0.000492563430128351,7.42094536253605e-05,138782
+W0,harm,disease,rct_realistic_itt_h24,0.00248801890157732,0.000166343673945743,138782
+W0,harm,disease,rct_realistic_itt,0.0539520228689354,0.000648343714778322,138782
+W0,harm,rr,rct_realistic_itt_h12,0.032397552203542,0.000494895030835064,138782
+W0,harm,rr,rct_realistic_itt_h24,0.0738721748970789,0.000732229204166303,138782
+W0,harm,rr,rct_realistic_itt,0.206069107976931,0.00113935408855565,138782
+W0,harm,nco1,rct_realistic_itt_h12,6.53400738957156e-06,6.53399013392965e-06,138782
+W0,harm,nco1,rct_realistic_itt_h24,-8.45156676626482e-06,1.58783112120704e-05,138782
+W0,harm,nco1,rct_realistic_itt,-0.00117248163179654,8.93642930544493e-05,138782
+W0,harm,nco2,rct_realistic_itt_h12,1.43513048739779e-05,1.01883274755683e-05,138782
+W0,harm,nco2,rct_realistic_itt_h24,1.97093822894091e-05,1.15113069317871e-05,138782
+W0,harm,nco2,rct_realistic_itt,-0.000146622386963238,3.5567497711738e-05,138782
+W0,harm,limb,rct_ideal_itt_h12,-0.00069000000000001,3.32193302160264e-05,1000000
+W0,harm,limb,rct_ideal_itt_h24,0.00369700000000017,7.40090007424914e-05,1000000
+W0,harm,limb,rct_ideal_itt,0.0625470000000419,0.000250125713973842,1000000
+W0,harm,male,rct_ideal_itt_h12,0.0302030000000096,0.000176195285949148,1000000
+W0,harm,male,rct_ideal_itt_h24,0.0694110000000488,0.000257408455725467,1000000
+W0,harm,male,rct_ideal_itt,0.196825000000176,0.000401807067352472,1000000
+W0,harm,disease,rct_ideal_itt_h12,-0.000527000000000006,2.88569276083573e-05,1000000
+W0,harm,disease,rct_ideal_itt_h24,0.00283300000000005,6.47531783851892e-05,1000000
+W0,harm,disease,rct_ideal_itt,0.0515180000000309,0.00022818390757615,1000000
+W0,harm,rr,rct_ideal_itt_h12,0.0322370000000116,0.000176793031060403,1000000
+W0,harm,rr,rct_ideal_itt_h24,0.0723530000000517,0.000259225854016017,1000000
+W0,harm,rr,rct_ideal_itt,0.193524000000173,0.000395447166412248,1000000
+W0,harm,nco1,rct_ideal_itt_h12,1e-05,3.1622618487401e-06,1000000
+W0,harm,nco1,rct_ideal_itt_h24,-5e-06,7.93725235838367e-06,1000000
+W0,harm,nco1,rct_ideal_itt,-0.001143,3.57448395016914e-05,1000000
+W0,harm,nco2,rct_ideal_itt_h12,1e-06,9.9999950002108e-07,1000000
+W0,harm,nco2,rct_ideal_itt_h24,-2e-06,5.47722520992005e-06,1000000
+W0,harm,nco2,rct_ideal_itt,-0.000447000000000004,2.24232065279446e-05,1000000
+W0,benefit_harm,limb,rct_realistic_itt_h12,-0.00936046865284354,0.000264129599237624,138782
+W0,benefit_harm,limb,rct_realistic_itt_h24,-0.0156253817310153,0.000346005778226494,138782
+W0,benefit_harm,limb,rct_realistic_itt,0.0476375959324454,0.000747755930004121,138782
+W0,benefit_harm,male,rct_realistic_itt_h12,0.0138456909190784,0.000496906834236644,138782
+W0,benefit_harm,male,rct_realistic_itt_h24,0.0389507656400573,0.000721058908850233,138782
+W0,benefit_harm,male,rct_realistic_itt,0.183585931781821,0.00116128147313179,138782
+W0,benefit_harm,disease,rct_realistic_itt_h12,-0.00701853844622746,0.000227256392216162,138782
+W0,benefit_harm,disease,rct_realistic_itt_h24,-0.0118343056375335,0.000299104065798181,138782
+W0,benefit_harm,disease,rct_realistic_itt,0.0400446368325273,0.000676156316495598,138782
+W0,benefit_harm,rr,rct_realistic_itt_h12,0.0237399250833049,0.000426256751172818,138782
+W0,benefit_harm,rr,rct_realistic_itt_h24,0.0573123691206079,0.000651693144083697,138782
+W0,benefit_harm,rr,rct_realistic_itt,0.188126682049733,0.00110432772565977,138782
+W0,benefit_harm,nco1,rct_realistic_itt_h12,7.59973402766115e-05,2.05244409832568e-05,138782
+W0,benefit_harm,nco1,rct_realistic_itt_h24,0.000304283539237408,3.99652071948251e-05,138782
+W0,benefit_harm,nco1,rct_realistic_itt,0.000464506673192971,0.000102228623751637,138782
+W0,benefit_harm,nco2,rct_realistic_itt_h12,2.13829664316019e-05,1.23792028465136e-05,138782
+W0,benefit_harm,nco2,rct_realistic_itt_h24,0.000101840077489285,2.47064296907095e-05,138782
+W0,benefit_harm,nco2,rct_realistic_itt,0.000204351440100052,4.71903593479565e-05,138782
+W0,benefit_harm,limb,rct_ideal_itt_h12,-0.00960499999999939,9.84110968081316e-05,1000000
+W0,benefit_harm,limb,rct_ideal_itt_h24,-0.0157609999999951,0.000127720753516602,1000000
+W0,benefit_harm,limb,rct_ideal_itt,0.0460120000000254,0.000266065585628749,1000000
+W0,benefit_harm,male,rct_ideal_itt_h12,0.0137869999999963,0.000179958102432602,1000000
+W0,benefit_harm,male,rct_ideal_itt_h24,0.037648000000017,0.000257220971335908,1000000
+W0,benefit_harm,male,rct_ideal_itt,0.173512000000153,0.000406166943334289,1000000
+W0,benefit_harm,disease,rct_ideal_itt_h12,-0.00713100000000062,8.4936145655476e-05,1000000
+W0,benefit_harm,disease,rct_ideal_itt_h24,-0.0117879999999978,0.000110837913441629,1000000
+W0,benefit_harm,disease,rct_ideal_itt,0.038659000000018,0.00024133064811169,1000000
+W0,benefit_harm,rr,rct_ideal_itt_h12,0.0239360000000033,0.00015314394504567,1000000
+W0,benefit_harm,rr,rct_ideal_itt_h24,0.0565390000000359,0.000231508836720838,1000000
+W0,benefit_harm,rr,rct_ideal_itt,0.177170000000157,0.000383445942867072,1000000
+W0,benefit_harm,nco1,rct_ideal_itt_h12,7.79999999999999e-05,8.8314164209386e-06,1000000
+W0,benefit_harm,nco1,rct_ideal_itt_h24,0.000407000000000003,2.02196525936012e-05,1000000
+W0,benefit_harm,nco1,rct_ideal_itt,0.000797000000000013,4.44788128322003e-05,1000000
+W0,benefit_harm,nco2,rct_ideal_itt_h12,3.5e-05,5.91597625080754e-06,1000000
+W0,benefit_harm,nco2,rct_ideal_itt_h24,0.000195,1.40343141977502e-05,1000000
+W0,benefit_harm,nco2,rct_ideal_itt,0.000396000000000003,2.91177468907165e-05,1000000
+W1,null,limb,rct_realistic_itt_h12,0,0,138782
+W1,null,limb,rct_realistic_itt_h24,0,0,138782
+W1,null,limb,rct_realistic_itt,0,0,138782
+W1,null,male,rct_realistic_itt_h12,0.0144235047445299,0.000334145736094463,138782
+W1,null,male,rct_realistic_itt_h24,0.0306225886076548,0.000483525366728361,138782
+W1,null,male,rct_realistic_itt,0.0682952261879501,0.000712862931193441,138782
+W1,null,disease,rct_realistic_itt_h12,0,0,138782
+W1,null,disease,rct_realistic_itt_h24,0,0,138782
+W1,null,disease,rct_realistic_itt,0,0,138782
+W1,null,rr,rct_realistic_itt_h12,0.0153147512364479,0.000343373916484683,138782
+W1,null,rr,rct_realistic_itt_h24,0.034504584650443,0.000510787639451681,138782
+W1,null,rr,rct_realistic_itt,0.0920330534174895,0.000813787097961872,138782
+W1,null,nco1,rct_realistic_itt_h12,0,0,138782
+W1,null,nco1,rct_realistic_itt_h24,0,0,138782
+W1,null,nco1,rct_realistic_itt,0,0,138782
+W1,null,nco2,rct_realistic_itt_h12,0,0,138782
+W1,null,nco2,rct_realistic_itt_h24,0,0,138782
+W1,null,nco2,rct_realistic_itt,0,0,138782
+W1,null,limb,rct_ideal_itt_h12,0,0,1000000
+W1,null,limb,rct_ideal_itt_h24,0,0,1000000
+W1,null,limb,rct_ideal_itt,0,0,1000000
+W1,null,male,rct_ideal_itt_h12,0.0315560000000109,0.000174986339076103,1000000
+W1,null,male,rct_ideal_itt_h24,0.0641110000000435,0.000245056686666286,1000000
+W1,null,male,rct_ideal_itt,0.128842000000108,0.000335081690094554,1000000
+W1,null,disease,rct_ideal_itt_h12,0,0,1000000
+W1,null,disease,rct_ideal_itt_h24,0,0,1000000
+W1,null,disease,rct_ideal_itt,0,0,1000000
+W1,null,rr,rct_ideal_itt_h12,0.0329870000000123,0.000178775999035469,1000000
+W1,null,rr,rct_ideal_itt_h24,0.0701960000000495,0.000255594447481916,1000000
+W1,null,rr,rct_ideal_itt,0.162295000000142,0.000368797143395813,1000000
+W1,null,nco1,rct_ideal_itt_h12,0,0,1000000
+W1,null,nco1,rct_ideal_itt_h24,0,0,1000000
+W1,null,nco1,rct_ideal_itt,0,0,1000000
+W1,null,nco2,rct_ideal_itt_h12,0,0,1000000
+W1,null,nco2,rct_ideal_itt_h24,0,0,1000000
+W1,null,nco2,rct_ideal_itt,0,0,1000000
+W1,harm,limb,rct_realistic_itt_h12,-0.001373599630385,0.000102517735138512,138782
+W1,harm,limb,rct_realistic_itt_h24,-0.000471771207073629,0.000138484846910791,138782
+W1,harm,limb,rct_realistic_itt,0.0265679752263632,0.000478240124365267,138782
+W1,harm,male,rct_realistic_itt_h12,0.012608142874936,0.000344952617431263,138782
+W1,harm,male,rct_realistic_itt_h24,0.0300850462927523,0.000501966846867299,138782
+W1,harm,male,rct_realistic_itt,0.0960941863029341,0.000845442821572819,138782
+W1,harm,disease,rct_realistic_itt_h12,-0.00102941131727323,8.84473969363332e-05,138782
+W1,harm,disease,rct_realistic_itt_h24,-0.00045693020065716,0.000118343791967768,138782
+W1,harm,disease,rct_realistic_itt,0.0213147720063486,0.000428165705068078,138782
+W1,harm,rr,rct_realistic_itt_h12,0.0148059979448982,0.000337744336761176,138782
+W1,harm,rr,rct_realistic_itt_h24,0.0343427964413466,0.000509697497507762,138782
+W1,harm,rr,rct_realistic_itt,0.100249385026741,0.000846286929385081,138782
+W1,harm,nco1,rct_realistic_itt_h12,6.53400738957156e-06,6.53399013392965e-06,138782
+W1,harm,nco1,rct_realistic_itt_h24,3.24449150747099e-05,1.67253770199362e-05,138782
+W1,harm,nco1,rct_realistic_itt,-0.000404092312836292,5.94467383510869e-05,138782
+W1,harm,nco2,rct_realistic_itt_h12,1.43513048739779e-05,1.01883274755683e-05,138782
+W1,harm,nco2,rct_realistic_itt_h24,2.39525242738712e-05,1.2268410518513e-05,138782
+W1,harm,nco2,rct_realistic_itt,-3.21786274097218e-05,2.67252013050681e-05,138782
+W1,harm,limb,rct_ideal_itt_h12,-0.000350000000000002,3.90368735943352e-05,1000000
+W1,harm,limb,rct_ideal_itt_h24,0.00576900000000044,8.80438449813041e-05,1000000
+W1,harm,limb,rct_ideal_itt,0.084697000000064,0.000286488076877455,1000000
+W1,harm,male,rct_ideal_itt_h12,0.0314140000000108,0.000179747491229717,1000000
+W1,harm,male,rct_ideal_itt_h24,0.0733340000000527,0.000264363621635138,1000000
+W1,harm,male,rct_ideal_itt,0.204205000000184,0.000407803038213413,1000000
+W1,harm,disease,rct_ideal_itt_h12,-0.000278,3.39988634513169e-05,1000000
+W1,harm,disease,rct_ideal_itt_h24,0.00446900000000027,7.75566118324939e-05,1000000
+W1,harm,disease,rct_ideal_itt,0.072661000000052,0.00026665591888863,1000000
+W1,harm,rr,rct_ideal_itt_h12,0.0331900000000125,0.000179338852175699,1000000
+W1,harm,rr,rct_ideal_itt_h24,0.0751360000000545,0.000263800268203047,1000000
+W1,harm,rr,rct_ideal_itt,0.198015000000177,0.000399055208928203,1000000
+W1,harm,nco1,rct_ideal_itt_h12,1e-05,3.46408718135054e-06,1000000
+W1,harm,nco1,rct_ideal_itt_h24,-2.4e-05,9.27358743966557e-06,1000000
+W1,harm,nco1,rct_ideal_itt,-0.00149599999999997,4.06664724801911e-05,1000000
+W1,harm,nco2,rct_ideal_itt_h12,1e-06,9.9999950002108e-07,1000000
+W1,harm,nco2,rct_ideal_itt_h24,-1.1e-05,5.56775349674996e-06,1000000
+W1,harm,nco2,rct_ideal_itt,-0.00067000000000001,2.67123772807722e-05,1000000
+W1,benefit_harm,limb,rct_realistic_itt_h12,-0.0100687707503497,0.000273979275962129,138782
+W1,benefit_harm,limb,rct_realistic_itt_h24,-0.0201671873778362,0.000389225369172784,138782
+W1,benefit_harm,limb,rct_realistic_itt,-0.0179862610172264,0.000543775907644638,138782
+W1,benefit_harm,male,rct_realistic_itt_h12,0.000421249459074619,0.000395540209152943,138782
+W1,benefit_harm,male,rct_realistic_itt_h24,0.00384220084773466,0.000577500168129415,138782
+W1,benefit_harm,male,rct_realistic_itt,0.0456265915942134,0.000868756295164852,138782
+W1,benefit_harm,disease,rct_realistic_itt_h12,-0.00755786823382364,0.000235610498666142,138782
+W1,benefit_harm,disease,rct_realistic_itt_h24,-0.015261928209796,0.00033704582876464,138782
+W1,benefit_harm,disease,rct_realistic_itt,-0.0134227825916772,0.000480003954380111,138782
+W1,benefit_harm,rr,rct_realistic_itt_h12,0.010850111951166,0.000289795992225478,138782
+W1,benefit_harm,rr,rct_realistic_itt_h24,0.0255359971692079,0.000443045073074274,138782
+W1,benefit_harm,rr,rct_realistic_itt,0.0812230409215406,0.000778945174193417,138782
+W1,benefit_harm,nco1,rct_realistic_itt_h12,7.59973402766115e-05,2.05244409832568e-05,138782
+W1,benefit_harm,nco1,rct_realistic_itt_h24,0.000345328509197363,4.23578338108543e-05,138782
+W1,benefit_harm,nco1,rct_realistic_itt,0.00167760516470874,0.000102598875198916,138782
+W1,benefit_harm,nco2,rct_realistic_itt_h12,2.13829664316019e-05,1.23792028465136e-05,138782
+W1,benefit_harm,nco2,rct_realistic_itt_h24,0.000107801730973639,2.54154238646691e-05,138782
+W1,benefit_harm,nco2,rct_realistic_itt,0.000341525175221389,4.76676185289784e-05,138782
+W1,benefit_harm,limb,rct_ideal_itt_h12,-0.00922399999999968,9.70304994525534e-05,1000000
+W1,benefit_harm,limb,rct_ideal_itt_h24,-0.0120759999999976,0.000135735663050895,1000000
+W1,benefit_harm,limb,rct_ideal_itt,0.0938970000000732,0.000343127313676257,1000000
+W1,benefit_harm,male,rct_ideal_itt_h12,0.0154609999999951,0.000182794850799761,1000000
+W1,benefit_harm,male,rct_ideal_itt_h24,0.0452080000000246,0.000268142194991403,1000000
+W1,benefit_harm,male,rct_ideal_itt,0.194898000000174,0.00043004507855924,1000000
+W1,benefit_harm,disease,rct_ideal_itt_h12,-0.00684900000000059,8.37979188230937e-05,1000000
+W1,benefit_harm,disease,rct_ideal_itt_h24,-0.00885899999999994,0.000118247698154484,1000000
+W1,benefit_harm,disease,rct_ideal_itt,0.0869100000000663,0.000323926923700412,1000000
+W1,benefit_harm,rr,rct_ideal_itt_h12,0.0252970000000046,0.000157280201523023,1000000
+W1,benefit_harm,rr,rct_ideal_itt_h24,0.0618910000000412,0.000241492244430772,1000000
+W1,benefit_harm,rr,rct_ideal_itt,0.192663000000172,0.000396386135515462,1000000
+W1,benefit_harm,nco1,rct_ideal_itt_h12,7.79999999999999e-05,8.83141642093955e-06,1000000
+W1,benefit_harm,nco1,rct_ideal_itt_h24,0.000370000000000003,1.98459844803684e-05,1000000
+W1,benefit_harm,nco1,rct_ideal_itt,4.3e-05,5.36562964714672e-05,1000000
+W1,benefit_harm,nco2,rct_ideal_itt_h12,3.5e-05,5.91597625080754e-06,1000000
+W1,benefit_harm,nco2,rct_ideal_itt_h24,0.000184,1.39987908050369e-05,1000000
+W1,benefit_harm,nco2,rct_ideal_itt,-7.19999999999999e-05,3.59443293997484e-05,1000000
+W1_rho0,null,limb,rct_realistic_itt_h12,0,0,106170
+W1_rho0,null,limb,rct_realistic_itt_h24,0,0,106170
+W1_rho0,null,limb,rct_realistic_itt,0,0,106170
+W1_rho0,null,male,rct_realistic_itt_h12,0.0162623427800871,0.000403467759713899,106170
+W1_rho0,null,male,rct_realistic_itt_h24,0.034257765877929,0.000584347981441931,106170
+W1_rho0,null,male,rct_realistic_itt,0.0745038473435866,0.000847319247700144,106170
+W1_rho0,null,disease,rct_realistic_itt_h12,0,0,106170
+W1_rho0,null,disease,rct_realistic_itt_h24,0,0,106170
+W1_rho0,null,disease,rct_realistic_itt,0,0,106170
+W1_rho0,null,rr,rct_realistic_itt_h12,0.0174208354954014,0.000416968225183115,106170
+W1_rho0,null,rr,rct_realistic_itt_h24,0.0388602323683751,0.000619689726151177,106170
+W1_rho0,null,rr,rct_realistic_itt,0.0999419634753957,0.000964602112897118,106170
+W1_rho0,null,nco1,rct_realistic_itt_h12,0,0,106170
+W1_rho0,null,nco1,rct_realistic_itt_h24,0,0,106170
+W1_rho0,null,nco1,rct_realistic_itt,0,0,106170
+W1_rho0,null,nco2,rct_realistic_itt_h12,0,0,106170
+W1_rho0,null,nco2,rct_realistic_itt_h24,0,0,106170
+W1_rho0,null,nco2,rct_realistic_itt,0,0,106170
+W1_rho0,null,limb,rct_ideal_itt_h12,0,0,1000000
+W1_rho0,null,limb,rct_ideal_itt_h24,0,0,1000000
+W1_rho0,null,limb,rct_ideal_itt,0,0,1000000
+W1_rho0,null,male,rct_ideal_itt_h12,0.0340940000000134,0.000181630391631942,1000000
+W1_rho0,null,male,rct_ideal_itt_h24,0.0690780000000484,0.00025368135507928,1000000
+W1_rho0,null,male,rct_ideal_itt,0.137794000000117,0.000344730058978436,1000000
+W1_rho0,null,disease,rct_ideal_itt_h12,0,0,1000000
+W1_rho0,null,disease,rct_ideal_itt_h24,0,0,1000000
+W1_rho0,null,disease,rct_ideal_itt,0,0,1000000
+W1_rho0,null,rr,rct_ideal_itt_h12,0.0355950000000149,0.000185445398905871,1000000
+W1_rho0,null,rr,rct_ideal_itt_h24,0.0754360000000548,0.000264203349532408,1000000
+W1_rho0,null,rr,rct_ideal_itt,0.172761000000152,0.000378109292240791,1000000
+W1_rho0,null,nco1,rct_ideal_itt_h12,0,0,1000000
+W1_rho0,null,nco1,rct_ideal_itt_h24,0,0,1000000
+W1_rho0,null,nco1,rct_ideal_itt,0,0,1000000
+W1_rho0,null,nco2,rct_ideal_itt_h12,0,0,1000000
+W1_rho0,null,nco2,rct_ideal_itt_h24,0,0,1000000
+W1_rho0,null,nco2,rct_ideal_itt,0,0,1000000
+W1_rho0,harm,limb,rct_realistic_itt_h12,-0.0013361050646777,0.000117190542881897,106170
+W1_rho0,harm,limb,rct_realistic_itt_h24,-0.000199473659602554,0.00016501219063867,106170
+W1_rho0,harm,limb,rct_realistic_itt,0.0279237552752964,0.00055718382017812,106170
+W1_rho0,harm,male,rct_realistic_itt_h12,0.0144587886735274,0.000414584056428262,106170
+W1_rho0,harm,male,rct_realistic_itt_h24,0.0340510024625777,0.000606763317824294,106170
+W1_rho0,harm,male,rct_realistic_itt,0.103495525655754,0.000995986223038789,106170
+W1_rho0,harm,disease,rct_realistic_itt_h12,-0.000966024324043659,9.89895229423779e-05,106170
+W1_rho0,harm,disease,rct_realistic_itt_h24,-8.31239638047122e-05,0.000141161203711099,106170
+W1_rho0,harm,disease,rct_realistic_itt,0.0228166798980784,0.000502703874328023,106170
+W1_rho0,harm,rr,rct_realistic_itt_h12,0.0168849054515258,0.00041055143195152,106170
+W1_rho0,harm,rr,rct_realistic_itt_h24,0.0387736230445496,0.000619157819043317,106170
+W1_rho0,harm,rr,rct_realistic_itt,0.108816624683441,0.00100267246838486,106170
+W1_rho0,harm,nco1,rct_realistic_itt_h12,0,0,106170
+W1_rho0,harm,nco1,rct_realistic_itt_h24,8.30510581723947e-06,8.30507910279427e-06,106170
+W1_rho0,harm,nco1,rct_realistic_itt,-0.000319062365139344,6.37576618224476e-05,106170
+W1_rho0,harm,nco2,rct_realistic_itt_h12,0,0,106170
+W1_rho0,harm,nco2,rct_realistic_itt_h24,3.50615336313888e-06,3.50615891102087e-06,106170
+W1_rho0,harm,nco2,rct_realistic_itt,-5.26083778885331e-05,3.06349260265071e-05,106170
+W1_rho0,harm,limb,rct_ideal_itt_h12,-0.000334000000000002,3.93686225820893e-05,1000000
+W1_rho0,harm,limb,rct_ideal_itt_h24,0.00581800000000045,8.84768380765336e-05,1000000
+W1_rho0,harm,limb,rct_ideal_itt,0.0849540000000643,0.000286860275893109,1000000
+W1_rho0,harm,male,rct_ideal_itt_h12,0.0340030000000134,0.000186324437451083,1000000
+W1_rho0,harm,male,rct_ideal_itt_h24,0.0784980000000578,0.000272536353533258,1000000
+W1_rho0,harm,male,rct_ideal_itt,0.212454000000192,0.000413610079528282,1000000
+W1_rho0,harm,disease,rct_ideal_itt_h12,-0.000266,3.43209738206234e-05,1000000
+W1_rho0,harm,disease,rct_ideal_itt_h24,0.00447300000000027,7.77624091120047e-05,1000000
+W1_rho0,harm,disease,rct_ideal_itt,0.0728310000000522,0.000266920672557173,1000000
+W1_rho0,harm,rr,rct_ideal_itt_h12,0.0358380000000152,0.000186042032231041,1000000
+W1_rho0,harm,rr,rct_ideal_itt_h24,0.08067200000006,0.000272477574151182,1000000
+W1_rho0,harm,rr,rct_ideal_itt,0.209534000000189,0.000407491721200427,1000000
+W1_rho0,harm,nco1,rct_ideal_itt_h12,8e-06,3.46409237750559e-06,1000000
+W1_rho0,harm,nco1,rct_ideal_itt_h24,-2.3e-05,8.99997061103611e-06,1000000
+W1_rho0,harm,nco1,rct_ideal_itt,-0.00147299999999998,4.02843675763495e-05,1000000
+W1_rho0,harm,nco2,rct_ideal_itt_h12,1e-06,9.9999950002108e-07,1000000
+W1_rho0,harm,nco2,rct_ideal_itt_h24,-7e-06,5.38516025765599e-06,1000000
+W1_rho0,harm,nco2,rct_ideal_itt,-0.000546000000000007,2.43659985224613e-05,1000000
+W1_rho0,benefit_harm,limb,rct_realistic_itt_h12,-0.00978245408702786,0.000311474250813258,106170
+W1_rho0,benefit_harm,limb,rct_realistic_itt_h24,-0.0192033040015254,0.000438243377013365,106170
+W1_rho0,benefit_harm,limb,rct_realistic_itt,-0.0153924928954827,0.000617493172659435,106170
+W1_rho0,benefit_harm,male,rct_realistic_itt_h12,0.00177395900211174,0.000460458555015892,106170
+W1_rho0,benefit_harm,male,rct_realistic_itt_h24,0.00730717553832012,0.000671928525462793,106170
+W1_rho0,benefit_harm,male,rct_realistic_itt,0.0538636617312581,0.00100963757693059,106170
+W1_rho0,benefit_harm,disease,rct_realistic_itt_h12,-0.00714369696164247,0.000263232069347728,106170
+W1_rho0,benefit_harm,disease,rct_realistic_itt_h24,-0.0139297664193082,0.000369962763837187,106170
+W1_rho0,benefit_harm,disease,rct_realistic_itt,-0.0108484078699601,0.000534054620194249,106170
+W1_rho0,benefit_harm,rr,rct_realistic_itt_h12,0.0119936567519674,0.000347437677675633,106170
+W1_rho0,benefit_harm,rr,rct_realistic_itt_h24,0.0285024847698048,0.000534906512758664,106170
+W1_rho0,benefit_harm,rr,rct_realistic_itt,0.088924625651879,0.000926453186350461,106170
+W1_rho0,benefit_harm,nco1,rct_realistic_itt_h12,0.000100116370211912,2.57359968375756e-05,106170
+W1_rho0,benefit_harm,nco1,rct_realistic_itt_h24,0.000372922895699211,5.46968380429417e-05,106170
+W1_rho0,benefit_harm,nco1,rct_realistic_itt,0.00176038422237092,0.000127814346255105,106170
+W1_rho0,benefit_harm,nco2,rct_realistic_itt_h12,1.14055757904836e-05,8.09474394691263e-06,106170
+W1_rho0,benefit_harm,nco2,rct_realistic_itt_h24,6.31004145073422e-05,2.26778385327308e-05,106170
+W1_rho0,benefit_harm,nco2,rct_realistic_itt,0.000279311699475823,4.96074893035325e-05,106170
+W1_rho0,benefit_harm,limb,rct_ideal_itt_h12,-0.0091969999999997,9.67802417397742e-05,1000000
+W1_rho0,benefit_harm,limb,rct_ideal_itt_h24,-0.0119569999999977,0.000135654082691168,1000000
+W1_rho0,benefit_harm,limb,rct_ideal_itt,0.0951390000000745,0.000343915063173529,1000000
+W1_rho0,benefit_harm,male,rct_ideal_itt_h12,0.0174999999999968,0.000187775797163343,1000000
+W1_rho0,benefit_harm,male,rct_ideal_itt_h24,0.0498540000000292,0.000275438883753021,1000000
+W1_rho0,benefit_harm,male,rct_ideal_itt,0.202967000000182,0.000434913091212629,1000000
+W1_rho0,benefit_harm,disease,rct_ideal_itt_h12,-0.00682700000000058,8.35726753857821e-05,1000000
+W1_rho0,benefit_harm,disease,rct_ideal_itt_h24,-0.008782,0.000118156152933778,1000000
+W1_rho0,benefit_harm,disease,rct_ideal_itt,0.0881540000000675,0.000324972110009958,1000000
+W1_rho0,benefit_harm,rr,rct_ideal_itt_h12,0.0273460000000067,0.000163316246233535,1000000
+W1_rho0,benefit_harm,rr,rct_ideal_itt_h24,0.0667490000000461,0.000250107119050684,1000000
+W1_rho0,benefit_harm,rr,rct_ideal_itt,0.203860000000183,0.000404745723139152,1000000
+W1_rho0,benefit_harm,nco1,rct_ideal_itt_h12,7.49999999999999e-05,8.65992927222551e-06,1000000
+W1_rho0,benefit_harm,nco1,rct_ideal_itt_h24,0.000373000000000003,1.97195555475079e-05,1000000
+W1_rho0,benefit_harm,nco1,rct_ideal_itt,2.4e-05,5.39999946668951e-05,1000000
+W1_rho0,benefit_harm,nco2,rct_ideal_itt_h12,3.6e-05,5.99989199903001e-06,1000000
+W1_rho0,benefit_harm,nco2,rct_ideal_itt_h24,0.00019,1.41408592383115e-05,1000000
+W1_rho0,benefit_harm,nco2,rct_ideal_itt,0.000117,3.49998044424634e-05,1000000
+W2,null,limb,rct_realistic_itt_h12,0,0,138782
+W2,null,limb,rct_realistic_itt_h24,0,0,138782
+W2,null,limb,rct_realistic_itt,0,0,138782
+W2,null,male,rct_realistic_itt_h12,0.0337810807940289,0.00050535197517823,138782
+W2,null,male,rct_realistic_itt_h24,0.0706124607788977,0.000718022668609694,138782
+W2,null,male,rct_realistic_itt,0.15039557347479,0.00100866830147449,138782
+W2,null,disease,rct_realistic_itt_h12,0,0,138782
+W2,null,disease,rct_realistic_itt_h24,0,0,138782
+W2,null,disease,rct_realistic_itt,0,0,138782
+W2,null,rr,rct_realistic_itt_h12,0.0350866714353561,0.000514159961326907,138782
+W2,null,rr,rct_realistic_itt_h24,0.0765060586504421,0.000743918029824463,138782
+W2,null,rr,rct_realistic_itt,0.188204683561983,0.00109988933258071,138782
+W2,null,nco1,rct_realistic_itt_h12,0,0,138782
+W2,null,nco1,rct_realistic_itt_h24,0,0,138782
+W2,null,nco1,rct_realistic_itt,0,0,138782
+W2,null,nco2,rct_realistic_itt_h12,0,0,138782
+W2,null,nco2,rct_realistic_itt_h24,0,0,138782
+W2,null,nco2,rct_realistic_itt,0,0,138782
+W2,null,limb,rct_ideal_itt_h12,0,0,1000000
+W2,null,limb,rct_ideal_itt_h24,0,0,1000000
+W2,null,limb,rct_ideal_itt,0,0,1000000
+W2,null,male,rct_ideal_itt_h12,0.0329770000000123,0.00017876665648551,1000000
+W2,null,male,rct_ideal_itt_h24,0.0673800000000467,0.000250802582918274,1000000
+W2,null,male,rct_ideal_itt,0.138373000000118,0.000345366345885349,1000000
+W2,null,disease,rct_ideal_itt_h12,0,0,1000000
+W2,null,disease,rct_ideal_itt_h24,0,0,1000000
+W2,null,disease,rct_ideal_itt,0,0,1000000
+W2,null,rr,rct_ideal_itt_h12,0.0344580000000138,0.000182605164868548,1000000
+W2,null,rr,rct_ideal_itt_h24,0.073691000000053,0.000261416595724307,1000000
+W2,null,rr,rct_ideal_itt,0.173625000000153,0.000378897557888212,1000000
+W2,null,nco1,rct_ideal_itt_h12,0,0,1000000
+W2,null,nco1,rct_ideal_itt_h24,0,0,1000000
+W2,null,nco1,rct_ideal_itt,0,0,1000000
+W2,null,nco2,rct_ideal_itt_h12,0,0,1000000
+W2,null,nco2,rct_ideal_itt_h24,0,0,1000000
+W2,null,nco2,rct_ideal_itt,0,0,1000000
+W2,harm,limb,rct_realistic_itt_h12,-0.000647095963942255,8.58503040561573e-05,138782
+W2,harm,limb,rct_realistic_itt_h24,0.0036405242539962,0.000199173446527444,138782
+W2,harm,limb,rct_realistic_itt,0.0669917748050983,0.000721672135709116,138782
+W2,harm,male,rct_realistic_itt_h12,0.032542405057567,0.000508352078311317,138782
+W2,harm,male,rct_realistic_itt_h24,0.0757891246498343,0.000748628637494755,138782
+W2,harm,male,rct_realistic_itt,0.220119775713179,0.00117638221802888,138782
+W2,harm,disease,rct_realistic_itt_h12,-0.000504880270950688,7.41485576314562e-05,138782
+W2,harm,disease,rct_realistic_itt_h24,0.00254566205020426,0.000168064607656196,138782
+W2,harm,disease,rct_realistic_itt,0.0549723983558079,0.000654344898207341,138782
+W2,harm,rr,rct_realistic_itt_h12,0.034452698741701,0.00050960970223207,138782
+W2,harm,rr,rct_realistic_itt_h24,0.0786394813495808,0.000753394477334654,138782
+W2,harm,rr,rct_realistic_itt,0.219344001226149,0.00116567947137748,138782
+W2,harm,nco1,rct_realistic_itt_h12,6.53400738957156e-06,6.53399013392965e-06,138782
+W2,harm,nco1,rct_realistic_itt_h24,-8.45156676626482e-06,1.58783112120704e-05,138782
+W2,harm,nco1,rct_realistic_itt,-0.00116717312537871,8.95220971394273e-05,138782
+W2,harm,nco2,rct_realistic_itt_h12,1.43513048739779e-05,1.01883274755683e-05,138782
+W2,harm,nco2,rct_realistic_itt_h24,1.97093822894091e-05,1.15113069317871e-05,138782
+W2,harm,nco2,rct_realistic_itt,-0.000159798329295093,3.67672360388556e-05,138782
+W2,harm,limb,rct_ideal_itt_h12,-0.00069000000000001,3.30986993702438e-05,1000000
+W2,harm,limb,rct_ideal_itt_h24,0.00370400000000017,7.39883800605641e-05,1000000
+W2,harm,limb,rct_ideal_itt,0.0625790000000419,0.00025018167150836,1000000
+W2,harm,male,rct_ideal_itt_h12,0.0317890000000111,0.000180384199638679,1000000
+W2,harm,male,rct_ideal_itt_h24,0.072679000000052,0.000262794145596412,1000000
+W2,harm,male,rct_ideal_itt,0.202625000000182,0.000406091257445419,1000000
+W2,harm,disease,rct_ideal_itt_h12,-0.000530000000000006,2.87353284301963e-05,1000000
+W2,harm,disease,rct_ideal_itt_h24,0.00283200000000005,6.47300531131894e-05,1000000
+W2,harm,disease,rct_ideal_itt,0.0515850000000309,0.0002283111643668,1000000
+W2,harm,rr,rct_ideal_itt_h12,0.0338660000000132,0.000181088635878571,1000000
+W2,harm,rr,rct_ideal_itt_h24,0.0758370000000552,0.000264918382582814,1000000
+W2,harm,rr,rct_ideal_itt,0.201188000000181,0.00040129713262865,1000000
+W2,harm,nco1,rct_ideal_itt_h12,1e-05,3.1622618487401e-06,1000000
+W2,harm,nco1,rct_ideal_itt_h24,-7e-06,7.81024653897016e-06,1000000
+W2,harm,nco1,rct_ideal_itt,-0.001151,3.58284132919553e-05,1000000
+W2,harm,nco2,rct_ideal_itt_h12,1e-06,9.9999950002108e-07,1000000
+W2,harm,nco2,rct_ideal_itt_h24,-2e-06,5.47722520992009e-06,1000000
+W2,harm,nco2,rct_ideal_itt,-0.000430000000000004,2.21317667617172e-05,1000000
+W2,benefit_harm,limb,rct_realistic_itt_h12,-0.00956931188534672,0.000266722694787558,138782
+W2,benefit_harm,limb,rct_realistic_itt_h24,-0.015997189555154,0.000349278093848526,138782
+W2,benefit_harm,limb,rct_realistic_itt,0.0482680667089776,0.000753022714736988,138782
+W2,benefit_harm,male,rct_realistic_itt_h12,0.0152903234806801,0.000510562769863247,138782
+W2,benefit_harm,male,rct_realistic_itt_h24,0.0426586754708397,0.000742059131594603,138782
+W2,benefit_harm,male,rct_realistic_itt,0.19384749564067,0.00118274574590801,138782
+W2,benefit_harm,disease,rct_realistic_itt_h12,-0.00717623900129959,0.000229450691329312,138782
+W2,benefit_harm,disease,rct_realistic_itt_h24,-0.0121039723747459,0.000302167336843336,138782
+W2,benefit_harm,disease,rct_realistic_itt,0.0405570112816336,0.000680988232258797,138782
+W2,benefit_harm,rr,rct_realistic_itt_h12,0.0254104353872551,0.000440262059614362,138782
+W2,benefit_harm,rr,rct_realistic_itt_h24,0.0615396754787447,0.00067385273770769,138782
+W2,benefit_harm,rr,rct_realistic_itt,0.200788058510693,0.00113184883009162,138782
+W2,benefit_harm,nco1,rct_realistic_itt_h12,7.59973402766115e-05,2.05244409832568e-05,138782
+W2,benefit_harm,nco1,rct_realistic_itt_h24,0.000304283539237408,3.99652071948251e-05,138782
+W2,benefit_harm,nco1,rct_realistic_itt,0.000482621722916582,0.000102446486450744,138782
+W2,benefit_harm,nco2,rct_realistic_itt_h12,2.13829664316019e-05,1.23792028465136e-05,138782
+W2,benefit_harm,nco2,rct_realistic_itt_h24,0.000101840077489285,2.47064296907095e-05,138782
+W2,benefit_harm,nco2,rct_realistic_itt,0.000204351440100052,4.71903593479565e-05,138782
+W2,benefit_harm,limb,rct_ideal_itt_h12,-0.00962099999999938,9.85212482625226e-05,1000000
+W2,benefit_harm,limb,rct_ideal_itt_h24,-0.0157609999999951,0.00012769726261298,1000000
+W2,benefit_harm,limb,rct_ideal_itt,0.0460580000000254,0.000266170360176093,1000000
+W2,benefit_harm,male,rct_ideal_itt_h12,0.0149959999999954,0.000183191484472827,1000000
+W2,benefit_harm,male,rct_ideal_itt_h24,0.0403690000000197,0.000261765818697534,1000000
+W2,benefit_harm,male,rct_ideal_itt,0.179103000000158,0.000410337806437503,1000000
+W2,benefit_harm,disease,rct_ideal_itt_h12,-0.00714100000000062,8.50294426588907e-05,1000000
+W2,benefit_harm,disease,rct_ideal_itt_h24,-0.0117909999999978,0.000110824060199874,1000000
+W2,benefit_harm,disease,rct_ideal_itt,0.0387480000000181,0.000241562812732483,1000000
+W2,benefit_harm,rr,rct_ideal_itt_h12,0.0251730000000045,0.00015694368439448,1000000
+W2,benefit_harm,rr,rct_ideal_itt_h24,0.0593490000000387,0.00023683896680697,1000000
+W2,benefit_harm,rr,rct_ideal_itt,0.184437000000164,0.000389463724920822,1000000
+W2,benefit_harm,nco1,rct_ideal_itt_h12,7.79999999999999e-05,8.8314164209376e-06,1000000
+W2,benefit_harm,nco1,rct_ideal_itt_h24,0.000404000000000003,2.0145391135619e-05,1000000
+W2,benefit_harm,nco1,rct_ideal_itt,0.000787000000000013,4.45014677399414e-05,1000000
+W2,benefit_harm,nco2,rct_ideal_itt_h12,3.3e-05,5.74446786046983e-06,1000000
+W2,benefit_harm,nco2,rct_ideal_itt_h24,0.000187,1.37464552158657e-05,1000000
+W2,benefit_harm,nco2,rct_ideal_itt,0.000370000000000003,2.87378339478796e-05,1000000
+W3,null,limb,rct_realistic_itt_h12,0,0,138782
+W3,null,limb,rct_realistic_itt_h24,0,0,138782
+W3,null,limb,rct_realistic_itt,0,0,138782
+W3,null,male,rct_realistic_itt_h12,0.00903030984809614,0.000265285956694921,138782
+W3,null,male,rct_realistic_itt_h24,0.0193240465029616,0.000386258154997006,138782
+W3,null,male,rct_realistic_itt,0.0440196832365507,0.000580334073905548,138782
+W3,null,disease,rct_realistic_itt_h12,0,0,138782
+W3,null,disease,rct_realistic_itt_h24,0,0,138782
+W3,null,disease,rct_realistic_itt,0,0,138782
+W3,null,rr,rct_realistic_itt_h12,0.00980558532326791,0.000275302372913034,138782
+W3,null,rr,rct_realistic_itt_h24,0.0224939568930911,0.000414150028887632,138782
+W3,null,rr,rct_realistic_itt,0.0632610041136034,0.000684900741084694,138782
+W3,null,nco1,rct_realistic_itt_h12,0,0,138782
+W3,null,nco1,rct_realistic_itt_h24,0,0,138782
+W3,null,nco1,rct_realistic_itt,0,0,138782
+W3,null,nco2,rct_realistic_itt_h12,0,0,138782
+W3,null,nco2,rct_realistic_itt_h24,0,0,138782
+W3,null,nco2,rct_realistic_itt,0,0,138782
+W3,null,limb,rct_ideal_itt_h12,0,0,1000000
+W3,null,limb,rct_ideal_itt_h24,0,0,1000000
+W3,null,limb,rct_ideal_itt,0,0,1000000
+W3,null,male,rct_ideal_itt_h12,0.0168199999999962,0.000128899525213432,1000000
+W3,null,male,rct_ideal_itt_h24,0.0345620000000139,0.000182837272338099,1000000
+W3,null,male,rct_ideal_itt,0.0722660000000516,0.000259020511240493,1000000
+W3,null,disease,rct_ideal_itt_h12,0,0,1000000
+W3,null,disease,rct_ideal_itt_h24,0,0,1000000
+W3,null,disease,rct_ideal_itt,0,0,1000000
+W3,null,rr,rct_ideal_itt_h12,0.0178909999999972,0.000132879314112807,1000000
+W3,null,rr,rct_ideal_itt_h24,0.0389820000000183,0.000193758622198589,1000000
+W3,null,rr,rct_ideal_itt,0.0965230000000759,0.000295442567130136,1000000
+W3,null,nco1,rct_ideal_itt_h12,0,0,1000000
+W3,null,nco1,rct_ideal_itt_h24,0,0,1000000
+W3,null,nco1,rct_ideal_itt,0,0,1000000
+W3,null,nco2,rct_ideal_itt_h12,0,0,1000000
+W3,null,nco2,rct_ideal_itt_h24,0,0,1000000
+W3,null,nco2,rct_ideal_itt,0,0,1000000
+W3,harm,limb,rct_realistic_itt_h12,-0.000588249907037527,8.24840115132283e-05,138782
+W3,harm,limb,rct_realistic_itt_h24,0.00351844644831374,0.000194074633264242,138782
+W3,harm,limb,rct_realistic_itt,0.0660614353198847,0.000716019815307085,138782
+W3,harm,male,rct_realistic_itt_h12,0.00834947836101997,0.000276711896017007,138782
+W3,harm,male,rct_realistic_itt_h24,0.0235198547064754,0.000436716645646618,138782
+W3,harm,male,rct_realistic_itt,0.111320688064598,0.000900182566646114,138782
+W3,harm,disease,rct_realistic_itt_h12,-0.00044603421404596,7.02226079620628e-05,138782
+W3,harm,disease,rct_realistic_itt_h24,0.00243844263822334,0.000162308204945514,138782
+W3,harm,disease,rct_realistic_itt,0.0537864738547063,0.000646605565247129,138782
+W3,harm,rr,rct_realistic_itt_h12,0.009669523269744,0.000273479054555858,138782
+W3,harm,rr,rct_realistic_itt_h24,0.0233429831683389,0.000422018922865152,138782
+W3,harm,rr,rct_realistic_itt,0.0778699410977019,0.000755464856924967,138782
+W3,harm,nco1,rct_realistic_itt_h12,6.53400738957156e-06,6.53399013392965e-06,138782
+W3,harm,nco1,rct_realistic_itt_h24,-8.45156676626482e-06,1.58783112120704e-05,138782
+W3,harm,nco1,rct_realistic_itt,-0.00114175384665679,8.83389863805806e-05,138782
+W3,harm,nco2,rct_realistic_itt_h12,1.43513048739779e-05,1.01883274755683e-05,138782
+W3,harm,nco2,rct_realistic_itt_h24,1.97093822894091e-05,1.15113069317871e-05,138782
+W3,harm,nco2,rct_realistic_itt,-0.000159798329295093,3.67672360388556e-05,138782
+W3,harm,limb,rct_ideal_itt_h12,-0.000649000000000009,3.24742790372589e-05,1000000
+W3,harm,limb,rct_ideal_itt_h24,0.00364500000000016,7.30596603814059e-05,1000000
+W3,harm,limb,rct_ideal_itt,0.0614270000000408,0.00024811232067428,1000000
+W3,harm,male,rct_ideal_itt_h12,0.0159559999999953,0.000132081058687124,1000000
+W3,harm,male,rct_ideal_itt_h24,0.0390860000000184,0.000198131987835618,1000000
+W3,harm,male,rct_ideal_itt,0.135530000000115,0.000347591166603257,1000000
+W3,harm,disease,rct_ideal_itt_h12,-0.000490000000000005,2.80670607653485e-05,1000000
+W3,harm,disease,rct_ideal_itt_h24,0.00277000000000004,6.3673598139629e-05,1000000
+W3,harm,disease,rct_ideal_itt,0.0503310000000297,0.000225725032816082,1000000
+W3,harm,rr,rct_ideal_itt_h12,0.017620999999997,0.000131910956173239,1000000
+W3,harm,rr,rct_ideal_itt_h24,0.0401920000000195,0.000196663680264494,1000000
+W3,harm,rr,rct_ideal_itt,0.114807000000094,0.000319321707298835,1000000
+W3,harm,nco1,rct_ideal_itt_h12,1.1e-05,3.31660654886121e-06,1000000
+W3,harm,nco1,rct_ideal_itt_h24,-7e-06,7.93725084644526e-06,1000000
+W3,harm,nco1,rct_ideal_itt,-0.001143,3.57448395016908e-05,1000000
+W3,harm,nco2,rct_ideal_itt_h12,1e-06,9.9999950002108e-07,1000000
+W3,harm,nco2,rct_ideal_itt_h24,4.2351647362715e-22,5.29150262212918e-06,1000000
+W3,harm,nco2,rct_ideal_itt,-0.000428000000000004,2.21318055297241e-05,1000000
+W3,benefit_harm,limb,rct_realistic_itt_h12,-0.00940372301744105,0.00026426342050986,138782
+W3,benefit_harm,limb,rct_realistic_itt_h24,-0.0157683072512824,0.00034646646266276,138782
+W3,benefit_harm,limb,rct_realistic_itt,0.0476437098675707,0.000749080677845296,138782
+W3,benefit_harm,male,rct_realistic_itt_h12,-0.00244380704260416,0.000352039283668794,138782
+W3,benefit_harm,male,rct_realistic_itt_h24,0.000480156976458699,0.00049420426918914,138782
+W3,benefit_harm,male,rct_realistic_itt,0.0904566992351862,0.000918464245987336,138782
+W3,benefit_harm,disease,rct_realistic_itt_h12,-0.00701065013339392,0.000226573633546181,138782
+W3,benefit_harm,disease,rct_realistic_itt_h24,-0.0118546838717681,0.000298550065502498,138782
+W3,benefit_harm,disease,rct_realistic_itt,0.039773855046475,0.000674822314350528,138782
+W3,benefit_harm,rr,rct_realistic_itt_h12,0.0073096552975273,0.000238753234295454,138782
+W3,benefit_harm,rr,rct_realistic_itt_h24,0.0177366927987561,0.000370980734206821,138782
+W3,benefit_harm,rr,rct_realistic_itt,0.0682386327080767,0.000716941848502967,138782
+W3,benefit_harm,nco1,rct_realistic_itt_h12,7.59973402766115e-05,2.05244409832568e-05,138782
+W3,benefit_harm,nco1,rct_realistic_itt_h24,0.000304283539237408,3.99652071948251e-05,138782
+W3,benefit_harm,nco1,rct_realistic_itt,0.000500793138392635,0.000102032165589858,138782
+W3,benefit_harm,nco2,rct_realistic_itt_h12,2.13829664316019e-05,1.23792028465136e-05,138782
+W3,benefit_harm,nco2,rct_realistic_itt_h24,0.000101840077489285,2.47064296907095e-05,138782
+W3,benefit_harm,nco2,rct_realistic_itt,0.000217072611910802,4.63066004091942e-05,138782
+W3,benefit_harm,limb,rct_ideal_itt_h12,-0.00939399999999955,9.73537506412589e-05,1000000
+W3,benefit_harm,limb,rct_ideal_itt_h24,-0.0154449999999951,0.000126421722717641,1000000
+W3,benefit_harm,limb,rct_ideal_itt,0.0451160000000245,0.000263800201942329,1000000
+W3,benefit_harm,male,rct_ideal_itt_h12,0.0031670000000001,0.000147895132141516,1000000
+W3,benefit_harm,male,rct_ideal_itt_h24,0.0132119999999967,0.00020868503313997,1000000
+W3,benefit_harm,male,rct_ideal_itt,0.115277000000095,0.000354632504534453,1000000
+W3,benefit_harm,disease,rct_ideal_itt_h12,-0.00691300000000059,8.36612839434453e-05,1000000
+W3,benefit_harm,disease,rct_ideal_itt_h24,-0.011476999999998,0.00010934934142963,1000000
+W3,benefit_harm,disease,rct_ideal_itt,0.0377310000000171,0.000238623912544438,1000000
+W3,benefit_harm,rr,rct_ideal_itt_h12,0.0130149999999969,0.000113717236050071,1000000
+W3,benefit_harm,rr,rct_ideal_itt_h24,0.0309500000000103,0.000173902551733555,1000000
+W3,benefit_harm,rr,rct_ideal_itt,0.103537000000083,0.000306961055563444,1000000
+W3,benefit_harm,nco1,rct_ideal_itt_h12,7.79999999999999e-05,8.83141642093734e-06,1000000
+W3,benefit_harm,nco1,rct_ideal_itt_h24,0.000402000000000003,2.00957307904308e-05,1000000
+W3,benefit_harm,nco1,rct_ideal_itt,0.000793000000000013,4.43888629161159e-05,1000000
+W3,benefit_harm,nco2,rct_ideal_itt_h12,3.3e-05,5.74446786046983e-06,1000000
+W3,benefit_harm,nco2,rct_ideal_itt_h24,0.000188,1.37827666309731e-05,1000000
+W3,benefit_harm,nco2,rct_ideal_itt,0.000366000000000002,2.86682061525463e-05,1000000
+W4,null,limb,rct_realistic_itt_h12,0,0,138782
+W4,null,limb,rct_realistic_itt_h24,0,0,138782
+W4,null,limb,rct_realistic_itt,0,0,138782
+W4,null,male,rct_realistic_itt_h12,0.0314301517471955,0.000488135642992868,138782
+W4,null,male,rct_realistic_itt_h24,0.0651870494834334,0.000691691002028623,138782
+W4,null,male,rct_realistic_itt,0.133995787257383,0.000960963787695265,138782
+W4,null,disease,rct_realistic_itt_h12,0,0,138782
+W4,null,disease,rct_realistic_itt_h24,0,0,138782
+W4,null,disease,rct_realistic_itt,0,0,138782
+W4,null,rr,rct_realistic_itt_h12,0.0326495807952769,0.000496607482977814,138782
+W4,null,rr,rct_realistic_itt_h24,0.0706865829537568,0.00071714466087967,138782
+W4,null,rr,rct_realistic_itt,0.167725642698684,0.00105114817642789,138782
+W4,null,nco1,rct_realistic_itt_h12,0,0,138782
+W4,null,nco1,rct_realistic_itt_h24,0,0,138782
+W4,null,nco1,rct_realistic_itt,0,0,138782
+W4,null,nco2,rct_realistic_itt_h12,0,0,138782
+W4,null,nco2,rct_realistic_itt_h24,0,0,138782
+W4,null,nco2,rct_realistic_itt,0,0,138782
+W4,null,limb,rct_ideal_itt_h12,0,0,1000000
+W4,null,limb,rct_ideal_itt_h24,0,0,1000000
+W4,null,limb,rct_ideal_itt,0,0,1000000
+W4,null,male,rct_ideal_itt_h12,0.0310470000000104,0.000173600356539896,1000000
+W4,null,male,rct_ideal_itt_h24,0.0625880000000419,0.000242323631239363,1000000
+W4,null,male,rct_ideal_itt,0.121135000000101,0.000326342322989634,1000000
+W4,null,disease,rct_ideal_itt_h12,0,0,1000000
+W4,null,disease,rct_ideal_itt_h24,0,0,1000000
+W4,null,disease,rct_ideal_itt,0,0,1000000
+W4,null,rr,rct_ideal_itt_h12,0.0324740000000118,0.000177418824603335,1000000
+W4,null,rr,rct_ideal_itt_h24,0.0685640000000479,0.000252829938703752,1000000
+W4,null,rr,rct_ideal_itt,0.152055000000131,0.000359157732727129,1000000
+W4,null,nco1,rct_ideal_itt_h12,0,0,1000000
+W4,null,nco1,rct_ideal_itt_h24,0,0,1000000
+W4,null,nco1,rct_ideal_itt,0,0,1000000
+W4,null,nco2,rct_ideal_itt_h12,0,0,1000000
+W4,null,nco2,rct_ideal_itt_h24,0,0,1000000
+W4,null,nco2,rct_ideal_itt,0,0,1000000
+W4,harm,limb,rct_realistic_itt_h12,-0.000627062213012661,8.62493618988047e-05,138782
+W4,harm,limb,rct_realistic_itt_h24,0.00355735445447908,0.000197263010075687,138782
+W4,harm,limb,rct_realistic_itt,0.064209964660219,0.000707931355134836,138782
+W4,harm,male,rct_realistic_itt_h12,0.0302959762106384,0.000492102326285693,138782
+W4,harm,male,rct_realistic_itt_h24,0.0702653620366293,0.000723483267829581,138782
+W4,harm,male,rct_realistic_itt,0.200766870513184,0.00113862740540386,138782
+W4,harm,disease,rct_realistic_itt_h12,-0.000484846520021094,7.46099874018648e-05,138782
+W4,harm,disease,rct_realistic_itt_h24,0.00248636075716954,0.000166080110011518,138782
+W4,harm,disease,rct_realistic_itt,0.0525976621937731,0.000641176211394885,138782
+W4,harm,rr,rct_realistic_itt_h12,0.0321195850346531,0.00049281544942983,138782
+W4,harm,rr,rct_realistic_itt_h24,0.0727389540473738,0.000726968108529196,138782
+W4,harm,rr,rct_realistic_itt,0.196531826803881,0.00112111646334625,138782
+W4,harm,nco1,rct_realistic_itt_h12,6.53400738957156e-06,6.53399013392965e-06,138782
+W4,harm,nco1,rct_realistic_itt_h24,-7.15379975309977e-06,1.90454145071415e-05,138782
+W4,harm,nco1,rct_realistic_itt,-0.00113926334365766,8.83869459965305e-05,138782
+W4,harm,nco2,rct_realistic_itt_h12,6.53400738957156e-06,6.53399013392965e-06,138782
+W4,harm,nco2,rct_realistic_itt_h24,1.18920848050027e-05,8.44994964807204e-06,138782
+W4,harm,nco2,rct_realistic_itt,-0.000142201617028631,3.35877497802296e-05,138782
+W4,harm,limb,rct_ideal_itt_h12,-0.00068600000000001,3.327956436009e-05,1000000
+W4,harm,limb,rct_ideal_itt_h24,0.00373100000000017,7.41692634389976e-05,1000000
+W4,harm,limb,rct_ideal_itt,0.059637000000039,0.000245006996288818,1000000
+W4,harm,male,rct_ideal_itt_h12,0.0299240000000093,0.000175449577441567,1000000
+W4,harm,male,rct_ideal_itt_h24,0.0678620000000472,0.000254795504193282,1000000
+W4,harm,male,rct_ideal_itt,0.182678000000162,0.000391084068095295,1000000
+W4,harm,disease,rct_ideal_itt_h12,-0.000525000000000006,2.88915969614512e-05,1000000
+W4,harm,disease,rct_ideal_itt_h24,0.00286600000000006,6.49290847306103e-05,1000000
+W4,harm,disease,rct_ideal_itt,0.0490340000000284,0.00022328382575656,1000000
+W4,harm,rr,rct_ideal_itt_h12,0.0319460000000113,0.000176021172259444,1000000
+W4,harm,rr,rct_ideal_itt_h24,0.07063500000005,0.000256381935350656,1000000
+W4,harm,rr,rct_ideal_itt,0.177898000000157,0.000383622603083671,1000000
+W4,harm,nco1,rct_ideal_itt_h12,9e-06,2.99998649996762e-06,1000000
+W4,harm,nco1,rct_ideal_itt_h24,-5e-06,7.81024807547441e-06,1000000
+W4,harm,nco1,rct_ideal_itt,-0.00109800000000001,3.52816438955922e-05,1000000
+W4,harm,nco2,rct_ideal_itt_h12,3e-06,1.73204820949881e-06,1000000
+W4,harm,nco2,rct_ideal_itt_h24,-1e-06,5.19615232638288e-06,1000000
+W4,harm,nco2,rct_ideal_itt,-0.000439000000000004,2.21992630281702e-05,1000000
+W4,benefit_harm,limb,rct_realistic_itt_h12,-0.00929276007926325,0.000263562909258414,138782
+W4,benefit_harm,limb,rct_realistic_itt_h24,-0.0152956796657341,0.000345323877379886,138782
+W4,benefit_harm,limb,rct_realistic_itt,0.0491758811390879,0.000753483858758792,138782
+W4,benefit_harm,male,rct_realistic_itt_h12,0.0137017389425249,0.00049518558763673,138782
+W4,benefit_harm,male,rct_realistic_itt_h24,0.0384161956441744,0.000716619811699705,138782
+W4,benefit_harm,male,rct_realistic_itt,0.178788067867868,0.0011506403550177,138782
+W4,benefit_harm,disease,rct_realistic_itt_h12,-0.00695082987264717,0.000226591955285682,138782
+W4,benefit_harm,disease,rct_realistic_itt_h24,-0.0115624044636492,0.000298942839282845,138782
+W4,benefit_harm,disease,rct_realistic_itt,0.0411974385194535,0.000680812020253607,138782
+W4,benefit_harm,rr,rct_realistic_itt_h12,0.0235282645331711,0.000424435802539338,138782
+W4,benefit_harm,rr,rct_realistic_itt_h24,0.0564045829754239,0.000647006977339569,138782
+W4,benefit_harm,rr,rct_realistic_itt,0.179864727156537,0.00108685035753337,138782
+W4,benefit_harm,nco1,rct_realistic_itt_h12,7.59973402766115e-05,2.05244409832568e-05,138782
+W4,benefit_harm,nco1,rct_realistic_itt_h24,0.000310392406001315,4.03922202687285e-05,138782
+W4,benefit_harm,nco1,rct_realistic_itt,0.000397348624478565,0.000105014903563497,138782
+W4,benefit_harm,nco2,rct_realistic_itt_h12,1.35656689471956e-05,9.59877335354946e-06,138782
+W4,benefit_harm,nco2,rct_realistic_itt_h24,8.38179845360632e-05,2.38086068010171e-05,138782
+W4,benefit_harm,nco2,rct_realistic_itt,0.000187238134616088,4.6073264402126e-05,138782
+W4,benefit_harm,limb,rct_ideal_itt_h12,-0.00954899999999944,9.81825676947545e-05,1000000
+W4,benefit_harm,limb,rct_ideal_itt_h24,-0.0151279999999953,0.000127574071096906,1000000
+W4,benefit_harm,limb,rct_ideal_itt,0.0485780000000279,0.000269766895514397,1000000
+W4,benefit_harm,male,rct_ideal_itt_h12,0.0136269999999964,0.000179263227883511,1000000
+W4,benefit_harm,male,rct_ideal_itt_h24,0.0370390000000164,0.000254921777177018,1000000
+W4,benefit_harm,male,rct_ideal_itt,0.165850000000145,0.000400004721849139,1000000
+W4,benefit_harm,disease,rct_ideal_itt_h12,-0.00708100000000062,8.47163469414974e-05,1000000
+W4,benefit_harm,disease,rct_ideal_itt_h24,-0.0113059999999981,0.00011062628242945,1000000
+W4,benefit_harm,disease,rct_ideal_itt,0.0405780000000199,0.000244469683023112,1000000
+W4,benefit_harm,rr,rct_ideal_itt_h12,0.0237180000000031,0.000152458048249918,1000000
+W4,benefit_harm,rr,rct_ideal_itt_h24,0.0552410000000346,0.00022899657621463,1000000
+W4,benefit_harm,rr,rct_ideal_itt,0.164206000000144,0.000372889245706426,1000000
+W4,benefit_harm,nco1,rct_ideal_itt_h12,7.79999999999999e-05,8.83141642094399e-06,1000000
+W4,benefit_harm,nco1,rct_ideal_itt_h24,0.000387000000000003,2.00212444919833e-05,1000000
+W4,benefit_harm,nco1,rct_ideal_itt,0.00067800000000001,4.5338066963389e-05,1000000
+W4,benefit_harm,nco2,rct_ideal_itt_h12,3.6e-05,5.99989199902816e-06,1000000
+W4,benefit_harm,nco2,rct_ideal_itt_h24,0.000191,1.39629337532708e-05,1000000
+W4,benefit_harm,nco2,rct_ideal_itt,0.000335000000000002,2.97470633006767e-05,1000000
+W5,null,limb,rct_realistic_itt_h12,0,0,138782
+W5,null,limb,rct_realistic_itt_h24,0,0,138782
+W5,null,limb,rct_realistic_itt,0,0,138782
+W5,null,male,rct_realistic_itt_h12,0.0316896791667746,0.000489980259881347,138782
+W5,null,male,rct_realistic_itt_h24,0.066135680620639,0.000696255343077188,138782
+W5,null,male,rct_realistic_itt,0.140661174578609,0.00098072361759117,138782
+W5,null,disease,rct_realistic_itt_h12,0,0,138782
+W5,null,disease,rct_realistic_itt_h24,0,0,138782
+W5,null,disease,rct_realistic_itt,0,0,138782
+W5,null,rr,rct_realistic_itt_h12,0.0329190178576472,0.000498508344475691,138782
+W5,null,rr,rct_realistic_itt_h24,0.0717591137320835,0.000722074502261802,138782
+W5,null,rr,rct_realistic_itt,0.176524278977813,0.00107249504657992,138782
+W5,null,nco1,rct_realistic_itt_h12,0,0,138782
+W5,null,nco1,rct_realistic_itt_h24,0,0,138782
+W5,null,nco1,rct_realistic_itt,0,0,138782
+W5,null,nco2,rct_realistic_itt_h12,0,0,138782
+W5,null,nco2,rct_realistic_itt_h24,0,0,138782
+W5,null,nco2,rct_realistic_itt,0,0,138782
+W5,null,limb,rct_ideal_itt_h12,0,0,1000000
+W5,null,limb,rct_ideal_itt_h24,0,0,1000000
+W5,null,limb,rct_ideal_itt,0,0,1000000
+W5,null,male,rct_ideal_itt_h12,0.0313360000000107,0.000174379055808226,1000000
+W5,null,male,rct_ideal_itt_h24,0.0641560000000435,0.000245132632800575,1000000
+W5,null,male,rct_ideal_itt,0.132398000000112,0.000338976060504823,1000000
+W5,null,disease,rct_ideal_itt_h12,0,0,1000000
+W5,null,disease,rct_ideal_itt_h24,0,0,1000000
+W5,null,disease,rct_ideal_itt,0,0,1000000
+W5,null,rr,rct_ideal_itt_h12,0.0327740000000121,0.000178207364955272,1000000
+W5,null,rr,rct_ideal_itt_h24,0.0702760000000496,0.000255728926453558,1000000
+W5,null,rr,rct_ideal_itt,0.166642000000146,0.000372733743888863,1000000
+W5,null,nco1,rct_ideal_itt_h12,0,0,1000000
+W5,null,nco1,rct_ideal_itt_h24,0,0,1000000
+W5,null,nco1,rct_ideal_itt,0,0,1000000
+W5,null,nco2,rct_ideal_itt_h12,0,0,1000000
+W5,null,nco2,rct_ideal_itt_h24,0,0,1000000
+W5,null,nco2,rct_ideal_itt,0,0,1000000
+W5,harm,limb,rct_realistic_itt_h12,-0.000634779123119918,8.59030281036993e-05,138782
+W5,harm,limb,rct_realistic_itt_h24,0.00357723457388988,0.000197648179331725,138782
+W5,harm,limb,rct_realistic_itt,0.0657009909762577,0.000715048507757777,138782
+W5,harm,male,rct_realistic_itt_h12,0.0305563168266289,0.000494042770555967,138782
+W5,harm,male,rct_realistic_itt_h24,0.0712904014827441,0.000728275405042644,138782
+W5,harm,male,rct_realistic_itt,0.209013847241733,0.00115467602996962,138782
+W5,harm,disease,rct_realistic_itt_h12,-0.000492563430128351,7.42094536253605e-05,138782
+W5,harm,disease,rct_realistic_itt_h24,0.00248801890157732,0.000166343673945743,138782
+W5,harm,disease,rct_realistic_itt,0.0539520228689354,0.000648343714778322,138782
+W5,harm,rr,rct_realistic_itt_h12,0.032397552203542,0.000494895030835064,138782
+W5,harm,rr,rct_realistic_itt_h24,0.0738721748970789,0.000732229204166303,138782
+W5,harm,rr,rct_realistic_itt,0.206069107976931,0.00113935408855565,138782
+W5,harm,nco1,rct_realistic_itt_h12,6.53400738957156e-06,6.53399013392965e-06,138782
+W5,harm,nco1,rct_realistic_itt_h24,-8.45156676626482e-06,1.58783112120704e-05,138782
+W5,harm,nco1,rct_realistic_itt,-0.00117248163179654,8.93642930544493e-05,138782
+W5,harm,nco2,rct_realistic_itt_h12,1.43513048739779e-05,1.01883274755683e-05,138782
+W5,harm,nco2,rct_realistic_itt_h24,1.97093822894091e-05,1.15113069317871e-05,138782
+W5,harm,nco2,rct_realistic_itt,-0.000146622386963238,3.5567497711738e-05,138782
+W5,harm,limb,rct_ideal_itt_h12,-0.00069000000000001,3.32193302160264e-05,1000000
+W5,harm,limb,rct_ideal_itt_h24,0.00369700000000017,7.40090007424914e-05,1000000
+W5,harm,limb,rct_ideal_itt,0.0625470000000419,0.000250125713973842,1000000
+W5,harm,male,rct_ideal_itt_h12,0.0302030000000096,0.000176195285949148,1000000
+W5,harm,male,rct_ideal_itt_h24,0.0694110000000488,0.000257408455725467,1000000
+W5,harm,male,rct_ideal_itt,0.196825000000176,0.000401807067352472,1000000
+W5,harm,disease,rct_ideal_itt_h12,-0.000527000000000006,2.88569276083573e-05,1000000
+W5,harm,disease,rct_ideal_itt_h24,0.00283300000000005,6.47531783851892e-05,1000000
+W5,harm,disease,rct_ideal_itt,0.0515180000000309,0.00022818390757615,1000000
+W5,harm,rr,rct_ideal_itt_h12,0.0322370000000116,0.000176793031060403,1000000
+W5,harm,rr,rct_ideal_itt_h24,0.0723530000000517,0.000259225854016017,1000000
+W5,harm,rr,rct_ideal_itt,0.193524000000173,0.000395447166412248,1000000
+W5,harm,nco1,rct_ideal_itt_h12,1e-05,3.1622618487401e-06,1000000
+W5,harm,nco1,rct_ideal_itt_h24,-5e-06,7.93725235838367e-06,1000000
+W5,harm,nco1,rct_ideal_itt,-0.001143,3.57448395016914e-05,1000000
+W5,harm,nco2,rct_ideal_itt_h12,1e-06,9.9999950002108e-07,1000000
+W5,harm,nco2,rct_ideal_itt_h24,-2e-06,5.47722520992005e-06,1000000
+W5,harm,nco2,rct_ideal_itt,-0.000447000000000004,2.24232065279446e-05,1000000
+W5,benefit_harm,limb,rct_realistic_itt_h12,-0.00936046865284354,0.000264129599237624,138782
+W5,benefit_harm,limb,rct_realistic_itt_h24,-0.0156253817310153,0.000346005778226494,138782
+W5,benefit_harm,limb,rct_realistic_itt,0.0476375959324454,0.000747755930004121,138782
+W5,benefit_harm,male,rct_realistic_itt_h12,0.0138456909190784,0.000496906834236644,138782
+W5,benefit_harm,male,rct_realistic_itt_h24,0.0389507656400573,0.000721058908850233,138782
+W5,benefit_harm,male,rct_realistic_itt,0.183585931781821,0.00116128147313179,138782
+W5,benefit_harm,disease,rct_realistic_itt_h12,-0.00701853844622746,0.000227256392216162,138782
+W5,benefit_harm,disease,rct_realistic_itt_h24,-0.0118343056375335,0.000299104065798181,138782
+W5,benefit_harm,disease,rct_realistic_itt,0.0400446368325273,0.000676156316495598,138782
+W5,benefit_harm,rr,rct_realistic_itt_h12,0.0237399250833049,0.000426256751172818,138782
+W5,benefit_harm,rr,rct_realistic_itt_h24,0.0573123691206079,0.000651693144083697,138782
+W5,benefit_harm,rr,rct_realistic_itt,0.188126682049733,0.00110432772565977,138782
+W5,benefit_harm,nco1,rct_realistic_itt_h12,7.59973402766115e-05,2.05244409832568e-05,138782
+W5,benefit_harm,nco1,rct_realistic_itt_h24,0.000304283539237408,3.99652071948251e-05,138782
+W5,benefit_harm,nco1,rct_realistic_itt,0.000464506673192971,0.000102228623751637,138782
+W5,benefit_harm,nco2,rct_realistic_itt_h12,2.13829664316019e-05,1.23792028465136e-05,138782
+W5,benefit_harm,nco2,rct_realistic_itt_h24,0.000101840077489285,2.47064296907095e-05,138782
+W5,benefit_harm,nco2,rct_realistic_itt,0.000204351440100052,4.71903593479565e-05,138782
+W5,benefit_harm,limb,rct_ideal_itt_h12,-0.00960499999999939,9.84110968081316e-05,1000000
+W5,benefit_harm,limb,rct_ideal_itt_h24,-0.0157609999999951,0.000127720753516602,1000000
+W5,benefit_harm,limb,rct_ideal_itt,0.0460120000000254,0.000266065585628749,1000000
+W5,benefit_harm,male,rct_ideal_itt_h12,0.0137869999999963,0.000179958102432602,1000000
+W5,benefit_harm,male,rct_ideal_itt_h24,0.037648000000017,0.000257220971335908,1000000
+W5,benefit_harm,male,rct_ideal_itt,0.173512000000153,0.000406166943334289,1000000
+W5,benefit_harm,disease,rct_ideal_itt_h12,-0.00713100000000062,8.4936145655476e-05,1000000
+W5,benefit_harm,disease,rct_ideal_itt_h24,-0.0117879999999978,0.000110837913441629,1000000
+W5,benefit_harm,disease,rct_ideal_itt,0.038659000000018,0.00024133064811169,1000000
+W5,benefit_harm,rr,rct_ideal_itt_h12,0.0239360000000033,0.00015314394504567,1000000
+W5,benefit_harm,rr,rct_ideal_itt_h24,0.0565390000000359,0.000231508836720838,1000000
+W5,benefit_harm,rr,rct_ideal_itt,0.177170000000157,0.000383445942867072,1000000
+W5,benefit_harm,nco1,rct_ideal_itt_h12,7.79999999999999e-05,8.8314164209386e-06,1000000
+W5,benefit_harm,nco1,rct_ideal_itt_h24,0.000407000000000003,2.02196525936012e-05,1000000
+W5,benefit_harm,nco1,rct_ideal_itt,0.000797000000000013,4.44788128322003e-05,1000000
+W5,benefit_harm,nco2,rct_ideal_itt_h12,3.5e-05,5.91597625080754e-06,1000000
+W5,benefit_harm,nco2,rct_ideal_itt_h24,0.000195,1.40343141977502e-05,1000000
+W5,benefit_harm,nco2,rct_ideal_itt,0.000396000000000003,2.91177468907165e-05,1000000
+WE,null,limb,rct_realistic_itt_h12,0,0,138782
+WE,null,limb,rct_realistic_itt_h24,0,0,138782
+WE,null,limb,rct_realistic_itt,0,0,138782
+WE,null,male,rct_realistic_itt_h12,0.0316896791667746,0.000489980259881347,138782
+WE,null,male,rct_realistic_itt_h24,0.066135680620639,0.000696255343077188,138782
+WE,null,male,rct_realistic_itt,0.140661174578609,0.00098072361759117,138782
+WE,null,disease,rct_realistic_itt_h12,0,0,138782
+WE,null,disease,rct_realistic_itt_h24,0,0,138782
+WE,null,disease,rct_realistic_itt,0,0,138782
+WE,null,rr,rct_realistic_itt_h12,0.0329190178576472,0.000498508344475691,138782
+WE,null,rr,rct_realistic_itt_h24,0.0717591137320835,0.000722074502261802,138782
+WE,null,rr,rct_realistic_itt,0.176524278977813,0.00107249504657992,138782
+WE,null,nco1,rct_realistic_itt_h12,0,0,138782
+WE,null,nco1,rct_realistic_itt_h24,0,0,138782
+WE,null,nco1,rct_realistic_itt,0,0,138782
+WE,null,nco2,rct_realistic_itt_h12,0,0,138782
+WE,null,nco2,rct_realistic_itt_h24,0,0,138782
+WE,null,nco2,rct_realistic_itt,0,0,138782
+WE,null,limb,rct_ideal_itt_h12,0,0,1000000
+WE,null,limb,rct_ideal_itt_h24,0,0,1000000
+WE,null,limb,rct_ideal_itt,0,0,1000000
+WE,null,male,rct_ideal_itt_h12,0.0313360000000107,0.000174379055808226,1000000
+WE,null,male,rct_ideal_itt_h24,0.0641560000000435,0.000245132632800575,1000000
+WE,null,male,rct_ideal_itt,0.132398000000112,0.000338976060504823,1000000
+WE,null,disease,rct_ideal_itt_h12,0,0,1000000
+WE,null,disease,rct_ideal_itt_h24,0,0,1000000
+WE,null,disease,rct_ideal_itt,0,0,1000000
+WE,null,rr,rct_ideal_itt_h12,0.0327740000000121,0.000178207364955272,1000000
+WE,null,rr,rct_ideal_itt_h24,0.0702760000000496,0.000255728926453558,1000000
+WE,null,rr,rct_ideal_itt,0.166642000000146,0.000372733743888863,1000000
+WE,null,nco1,rct_ideal_itt_h12,0,0,1000000
+WE,null,nco1,rct_ideal_itt_h24,0,0,1000000
+WE,null,nco1,rct_ideal_itt,0,0,1000000
+WE,null,nco2,rct_ideal_itt_h12,0,0,1000000
+WE,null,nco2,rct_ideal_itt_h24,0,0,1000000
+WE,null,nco2,rct_ideal_itt,0,0,1000000
+WE,harm,limb,rct_realistic_itt_h12,-0.000634779123119918,8.59030281036993e-05,138782
+WE,harm,limb,rct_realistic_itt_h24,0.00357723457388988,0.000197648179331725,138782
+WE,harm,limb,rct_realistic_itt,0.0657009909762577,0.000715048507757777,138782
+WE,harm,male,rct_realistic_itt_h12,0.0305563168266289,0.000494042770555967,138782
+WE,harm,male,rct_realistic_itt_h24,0.0712904014827441,0.000728275405042644,138782
+WE,harm,male,rct_realistic_itt,0.209013847241733,0.00115467602996962,138782
+WE,harm,disease,rct_realistic_itt_h12,-0.000492563430128351,7.42094536253605e-05,138782
+WE,harm,disease,rct_realistic_itt_h24,0.00248801890157732,0.000166343673945743,138782
+WE,harm,disease,rct_realistic_itt,0.0539520228689354,0.000648343714778322,138782
+WE,harm,rr,rct_realistic_itt_h12,0.032397552203542,0.000494895030835064,138782
+WE,harm,rr,rct_realistic_itt_h24,0.0738721748970789,0.000732229204166303,138782
+WE,harm,rr,rct_realistic_itt,0.206069107976931,0.00113935408855565,138782
+WE,harm,nco1,rct_realistic_itt_h12,6.53400738957156e-06,6.53399013392965e-06,138782
+WE,harm,nco1,rct_realistic_itt_h24,-8.45156676626482e-06,1.58783112120704e-05,138782
+WE,harm,nco1,rct_realistic_itt,-0.00117248163179654,8.93642930544493e-05,138782
+WE,harm,nco2,rct_realistic_itt_h12,1.43513048739779e-05,1.01883274755683e-05,138782
+WE,harm,nco2,rct_realistic_itt_h24,1.97093822894091e-05,1.15113069317871e-05,138782
+WE,harm,nco2,rct_realistic_itt,-0.000146622386963238,3.5567497711738e-05,138782
+WE,harm,limb,rct_ideal_itt_h12,-0.00069000000000001,3.32193302160264e-05,1000000
+WE,harm,limb,rct_ideal_itt_h24,0.00369700000000017,7.40090007424914e-05,1000000
+WE,harm,limb,rct_ideal_itt,0.0625470000000419,0.000250125713973842,1000000
+WE,harm,male,rct_ideal_itt_h12,0.0302030000000096,0.000176195285949148,1000000
+WE,harm,male,rct_ideal_itt_h24,0.0694110000000488,0.000257408455725467,1000000
+WE,harm,male,rct_ideal_itt,0.196825000000176,0.000401807067352472,1000000
+WE,harm,disease,rct_ideal_itt_h12,-0.000527000000000006,2.88569276083573e-05,1000000
+WE,harm,disease,rct_ideal_itt_h24,0.00283300000000005,6.47531783851892e-05,1000000
+WE,harm,disease,rct_ideal_itt,0.0515180000000309,0.00022818390757615,1000000
+WE,harm,rr,rct_ideal_itt_h12,0.0322370000000116,0.000176793031060403,1000000
+WE,harm,rr,rct_ideal_itt_h24,0.0723530000000517,0.000259225854016017,1000000
+WE,harm,rr,rct_ideal_itt,0.193524000000173,0.000395447166412248,1000000
+WE,harm,nco1,rct_ideal_itt_h12,1e-05,3.1622618487401e-06,1000000
+WE,harm,nco1,rct_ideal_itt_h24,-5e-06,7.93725235838367e-06,1000000
+WE,harm,nco1,rct_ideal_itt,-0.001143,3.57448395016914e-05,1000000
+WE,harm,nco2,rct_ideal_itt_h12,1e-06,9.9999950002108e-07,1000000
+WE,harm,nco2,rct_ideal_itt_h24,-2e-06,5.47722520992005e-06,1000000
+WE,harm,nco2,rct_ideal_itt,-0.000447000000000004,2.24232065279446e-05,1000000
+WE,benefit_harm,limb,rct_realistic_itt_h12,-0.00936046865284354,0.000264129599237624,138782
+WE,benefit_harm,limb,rct_realistic_itt_h24,-0.0156253817310153,0.000346005778226494,138782
+WE,benefit_harm,limb,rct_realistic_itt,0.0476375959324454,0.000747755930004121,138782
+WE,benefit_harm,male,rct_realistic_itt_h12,0.0138456909190784,0.000496906834236644,138782
+WE,benefit_harm,male,rct_realistic_itt_h24,0.0389507656400573,0.000721058908850233,138782
+WE,benefit_harm,male,rct_realistic_itt,0.183585931781821,0.00116128147313179,138782
+WE,benefit_harm,disease,rct_realistic_itt_h12,-0.00701853844622746,0.000227256392216162,138782
+WE,benefit_harm,disease,rct_realistic_itt_h24,-0.0118343056375335,0.000299104065798181,138782
+WE,benefit_harm,disease,rct_realistic_itt,0.0400446368325273,0.000676156316495598,138782
+WE,benefit_harm,rr,rct_realistic_itt_h12,0.0237399250833049,0.000426256751172818,138782
+WE,benefit_harm,rr,rct_realistic_itt_h24,0.0573123691206079,0.000651693144083697,138782
+WE,benefit_harm,rr,rct_realistic_itt,0.188126682049733,0.00110432772565977,138782
+WE,benefit_harm,nco1,rct_realistic_itt_h12,7.59973402766115e-05,2.05244409832568e-05,138782
+WE,benefit_harm,nco1,rct_realistic_itt_h24,0.000304283539237408,3.99652071948251e-05,138782
+WE,benefit_harm,nco1,rct_realistic_itt,0.000464506673192971,0.000102228623751637,138782
+WE,benefit_harm,nco2,rct_realistic_itt_h12,2.13829664316019e-05,1.23792028465136e-05,138782
+WE,benefit_harm,nco2,rct_realistic_itt_h24,0.000101840077489285,2.47064296907095e-05,138782
+WE,benefit_harm,nco2,rct_realistic_itt,0.000204351440100052,4.71903593479565e-05,138782
+WE,benefit_harm,limb,rct_ideal_itt_h12,-0.00960499999999939,9.84110968081316e-05,1000000
+WE,benefit_harm,limb,rct_ideal_itt_h24,-0.0157609999999951,0.000127720753516602,1000000
+WE,benefit_harm,limb,rct_ideal_itt,0.0460120000000254,0.000266065585628749,1000000
+WE,benefit_harm,male,rct_ideal_itt_h12,0.0137869999999963,0.000179958102432602,1000000
+WE,benefit_harm,male,rct_ideal_itt_h24,0.037648000000017,0.000257220971335908,1000000
+WE,benefit_harm,male,rct_ideal_itt,0.173512000000153,0.000406166943334289,1000000
+WE,benefit_harm,disease,rct_ideal_itt_h12,-0.00713100000000062,8.4936145655476e-05,1000000
+WE,benefit_harm,disease,rct_ideal_itt_h24,-0.0117879999999978,0.000110837913441629,1000000
+WE,benefit_harm,disease,rct_ideal_itt,0.038659000000018,0.00024133064811169,1000000
+WE,benefit_harm,rr,rct_ideal_itt_h12,0.0239360000000033,0.00015314394504567,1000000
+WE,benefit_harm,rr,rct_ideal_itt_h24,0.0565390000000359,0.000231508836720838,1000000
+WE,benefit_harm,rr,rct_ideal_itt,0.177170000000157,0.000383445942867072,1000000
+WE,benefit_harm,nco1,rct_ideal_itt_h12,7.79999999999999e-05,8.8314164209386e-06,1000000
+WE,benefit_harm,nco1,rct_ideal_itt_h24,0.000407000000000003,2.02196525936012e-05,1000000
+WE,benefit_harm,nco1,rct_ideal_itt,0.000797000000000013,4.44788128322003e-05,1000000
+WE,benefit_harm,nco2,rct_ideal_itt_h12,3.5e-05,5.91597625080754e-06,1000000
+WE,benefit_harm,nco2,rct_ideal_itt_h24,0.000195,1.40343141977502e-05,1000000
+WE,benefit_harm,nco2,rct_ideal_itt,0.000396000000000003,2.91177468907165e-05,1000000
+WR,null,limb,rct_realistic_itt_h12,0,0,138782
+WR,null,limb,rct_realistic_itt_h24,0,0,138782
+WR,null,limb,rct_realistic_itt,0,0,138782
+WR,null,male,rct_realistic_itt_h12,0.00326265613441116,0.000161011731454756,138782
+WR,null,male,rct_realistic_itt_h24,0.00708260461688506,0.000234991461002943,138782
+WR,null,male,rct_realistic_itt,0.0159597525547126,0.000354917769662178,138782
+WR,null,disease,rct_realistic_itt_h12,0,0,138782
+WR,null,disease,rct_realistic_itt_h24,0,0,138782
+WR,null,disease,rct_realistic_itt,0,0,138782
+WR,null,rr,rct_realistic_itt_h12,0.00390068897873103,0.000174372283650843,138782
+WR,null,rr,rct_realistic_itt_h24,0.00963484905166457,0.000271376362156474,138782
+WR,null,rr,rct_realistic_itt,0.030981137329801,0.00048593194676983,138782
+WR,null,nco1,rct_realistic_itt_h12,0,0,138782
+WR,null,nco1,rct_realistic_itt_h24,0,0,138782
+WR,null,nco1,rct_realistic_itt,0,0,138782
+WR,null,nco2,rct_realistic_itt_h12,0,0,138782
+WR,null,nco2,rct_realistic_itt_h24,0,0,138782
+WR,null,nco2,rct_realistic_itt,0,0,138782
+WR,null,limb,rct_ideal_itt_h12,0,0,1000000
+WR,null,limb,rct_ideal_itt_h24,0,0,1000000
+WR,null,limb,rct_ideal_itt,0,0,1000000
+WR,null,male,rct_ideal_itt_h12,0.0144499999999958,0.000119646134497257,1000000
+WR,null,male,rct_ideal_itt_h24,0.0295130000000089,0.000169416595498972,1000000
+WR,null,male,rct_ideal_itt,0.0604990000000398,0.000238509687434567,1000000
+WR,null,disease,rct_ideal_itt_h12,0,0,1000000
+WR,null,disease,rct_ideal_itt_h24,0,0,1000000
+WR,null,disease,rct_ideal_itt,0,0,1000000
+WR,null,rr,rct_ideal_itt_h12,0.0154799999999951,0.000123775480609506,1000000
+WR,null,rr,rct_ideal_itt_h24,0.033686000000013,0.000180630156408789,1000000
+WR,null,rr,rct_ideal_itt,0.0827870000000621,0.000275694237569834,1000000
+WR,null,nco1,rct_ideal_itt_h12,0,0,1000000
+WR,null,nco1,rct_ideal_itt_h24,0,0,1000000
+WR,null,nco1,rct_ideal_itt,0,0,1000000
+WR,null,nco2,rct_ideal_itt_h12,0,0,1000000
+WR,null,nco2,rct_ideal_itt_h24,0,0,1000000
+WR,null,nco2,rct_ideal_itt,0,0,1000000
+WR,harm,limb,rct_realistic_itt_h12,-0.00132523127022743,9.87182615298501e-05,138782
+WR,harm,limb,rct_realistic_itt_h24,-0.000391612509628908,0.000137328320143579,138782
+WR,harm,limb,rct_realistic_itt,0.0268614851094127,0.000479680170586925,138782
+WR,harm,male,rct_realistic_itt_h12,0.00183086191430631,0.000186524882607493,138782
+WR,harm,male,rct_realistic_itt_h24,0.00660876178106249,0.000270497788471577,138782
+WR,harm,male,rct_realistic_itt,0.0430522807826039,0.000592228316505743,138782
+WR,harm,disease,rct_realistic_itt_h12,-0.000978475276970814,8.43708423896255e-05,138782
+WR,harm,disease,rct_realistic_itt_h24,-0.000368508831361038,0.000117786559586914,138782
+WR,harm,disease,rct_realistic_itt,0.0214007067446433,0.00042827954516037,138782
+WR,harm,rr,rct_realistic_itt_h12,0.0037420432221117,0.000170623602771209,138782
+WR,harm,rr,rct_realistic_itt_h24,0.00945306147737351,0.000269083492149388,138782
+WR,harm,rr,rct_realistic_itt,0.0346859729178713,0.000513920148878995,138782
+WR,harm,nco1,rct_realistic_itt_h12,1.05565276719135e-05,7.67290992256411e-06,138782
+WR,harm,nco1,rct_realistic_itt_h24,3.64674353570518e-05,1.720227729913e-05,138782
+WR,harm,nco1,rct_realistic_itt,-0.000362712154403391,5.78770255563114e-05,138782
+WR,harm,nco2,rct_realistic_itt_h12,1.43513048739779e-05,1.01883274755683e-05,138782
+WR,harm,nco2,rct_realistic_itt_h24,2.39525242738712e-05,1.2268410518513e-05,138782
+WR,harm,nco2,rct_realistic_itt,-3.21786274097218e-05,2.67252013050681e-05,138782
+WR,harm,limb,rct_ideal_itt_h12,-0.000337000000000002,3.86637612112804e-05,1000000
+WR,harm,limb,rct_ideal_itt_h24,0.00560400000000042,8.67559518652397e-05,1000000
+WR,harm,limb,rct_ideal_itt,0.0822680000000616,0.000282962852999299,1000000
+WR,harm,male,rct_ideal_itt_h12,0.014272999999996,0.000126353794842551,1000000
+WR,harm,male,rct_ideal_itt_h24,0.0369670000000163,0.000193861911966953,1000000
+WR,harm,male,rct_ideal_itt,0.138377000000118,0.000351378436835139,1000000
+WR,harm,disease,rct_ideal_itt_h12,-0.000265,3.35101443597238e-05,1000000
+WR,harm,disease,rct_ideal_itt_h24,0.00431000000000025,7.6114544602554e-05,1000000
+WR,harm,disease,rct_ideal_itt,0.0701190000000495,0.000262564898334477,1000000
+WR,harm,rr,rct_ideal_itt_h12,0.015637999999995,0.000124424486963914,1000000
+WR,harm,rr,rct_ideal_itt_h24,0.0364470000000158,0.000187703532708246,1000000
+WR,harm,rr,rct_ideal_itt,0.107241000000087,0.000310245657376612,1000000
+WR,harm,nco1,rct_ideal_itt_h12,9e-06,3.31661257912739e-06,1000000
+WR,harm,nco1,rct_ideal_itt_h24,-2.2e-05,8.9442448535487e-06,1000000
+WR,harm,nco1,rct_ideal_itt,-0.00146099999999998,4.0110665401783e-05,1000000
+WR,harm,nco2,rct_ideal_itt_h12,1e-06,9.9999950002108e-07,1000000
+WR,harm,nco2,rct_ideal_itt_h24,-1e-05,5.47721644638051e-06,1000000
+WR,harm,nco2,rct_ideal_itt,-0.000626000000000009,2.60309070911949e-05,1000000
+WR,benefit_harm,limb,rct_realistic_itt_h12,-0.0100692054330358,0.000273838728306849,138782
+WR,benefit_harm,limb,rct_realistic_itt_h24,-0.0202388549769331,0.000389934711939864,138782
+WR,benefit_harm,limb,rct_realistic_itt,-0.0182894827099266,0.0005456625670121,138782
+WR,benefit_harm,male,rct_realistic_itt_h12,-0.00774914797382966,0.000305314608664021,138782
+WR,benefit_harm,male,rct_realistic_itt_h24,-0.0149271553871344,0.000439529129829332,138782
+WR,benefit_harm,male,rct_realistic_itt,-0.00351118384900574,0.000640293782628295,138782
+WR,benefit_harm,disease,rct_realistic_itt_h12,-0.00750716023907874,0.000234635092877723,138782
+WR,benefit_harm,disease,rct_realistic_itt_h24,-0.015204708121949,0.000336574207932989,138782
+WR,benefit_harm,disease,rct_realistic_itt,-0.0135974833244941,0.000479920260541957,138782
+WR,benefit_harm,rr,rct_realistic_itt_h12,0.00257621420842002,0.000142259721726994,138782
+WR,benefit_harm,rr,rct_realistic_itt_h24,0.00629731243916255,0.000223283735341535,138782
+WR,benefit_harm,rr,rct_realistic_itt,0.0245015631438965,0.000452100073874658,138782
+WR,benefit_harm,nco1,rct_realistic_itt_h12,8.00198605589534e-05,2.09148811829732e-05,138782
+WR,benefit_harm,nco1,rct_realistic_itt_h24,0.000349351029479705,4.25483580158384e-05,138782
+WR,benefit_harm,nco1,rct_realistic_itt,0.00169299260303213,0.000103132274584187,138782
+WR,benefit_harm,nco2,rct_realistic_itt_h12,2.13829664316019e-05,1.23792028465136e-05,138782
+WR,benefit_harm,nco2,rct_realistic_itt_h24,0.000107801730973639,2.54154238646691e-05,138782
+WR,benefit_harm,nco2,rct_realistic_itt,0.000341525175221389,4.76676185289784e-05,138782
+WR,benefit_harm,limb,rct_ideal_itt_h12,-0.0090509999999998,9.60993204924743e-05,1000000
+WR,benefit_harm,limb,rct_ideal_itt_h24,-0.0119589999999977,0.000134439511749023,1000000
+WR,benefit_harm,limb,rct_ideal_itt,0.0913770000000707,0.00033987533578012,1000000
+WR,benefit_harm,male,rct_ideal_itt_h12,0.00228599999999998,0.000143118042902288,1000000
+WR,benefit_harm,male,rct_ideal_itt_h24,0.0147399999999956,0.000209691994123175,1000000
+WR,benefit_harm,male,rct_ideal_itt,0.137433000000117,0.000386244961794193,1000000
+WR,benefit_harm,disease,rct_ideal_itt_h12,-0.00666600000000056,8.26653763315159e-05,1000000
+WR,benefit_harm,disease,rct_ideal_itt_h24,-0.00873600000000003,0.000116771924297951,1000000
+WR,benefit_harm,disease,rct_ideal_itt,0.0842650000000636,0.00032011312028207,1000000
+WR,benefit_harm,rr,rct_ideal_itt_h12,0.0118289999999978,0.000108632751778034,1000000
+WR,benefit_harm,rr,rct_ideal_itt_h24,0.0298510000000092,0.000170990987478453,1000000
+WR,benefit_harm,rr,rct_ideal_itt,0.106172000000086,0.000310949363104946,1000000
+WR,benefit_harm,nco1,rct_ideal_itt_h12,7.69999999999999e-05,8.77462654469781e-06,1000000
+WR,benefit_harm,nco1,rct_ideal_itt_h24,0.000369000000000003,1.98711811175825e-05,1000000
+WR,benefit_harm,nco1,rct_ideal_itt,7.89999999999999e-05,5.33572278051064e-05,1000000
+WR,benefit_harm,nco2,rct_ideal_itt_h12,3.3e-05,5.74446786046983e-06,1000000
+WR,benefit_harm,nco2,rct_ideal_itt_h24,0.000179,1.38191157096822e-05,1000000
+WR,benefit_harm,nco2,rct_ideal_itt,-6.99999999999999e-05,3.54400211626608e-05,1000000

simulations/validate_designs.R

@@ -15,7 +15,7 @@ par <- load_params(world, scenario)
 res <- rbindlist(parallel::mclapply(seq_len(reps), function(r) {
   rec <- simulate_records(par, rep = 1000L + r, n = n)
   rbindlist(lapply(c("naive", "landmark", "ccw", "seq"), function(d)
-    cbind(rep = r, run_design(rec, d, "V1_ignore", outcomes = c(male = "t_male", disease = "t_disease", nco1 = "t_nco1")))),
+    cbind(rep = r, run_designs(rec, d, "V1_ignore", outcomes = c(male = "t_male", disease = "t_disease", nco1 = "t_nco1")))),
     fill = TRUE)
 }, mc.cores = cores), fill = TRUE)