Compare commits
20
Commits
@@ -0,0 +1,6 @@
|
||||
.Rproj.user
|
||||
.Rhistory
|
||||
.RData
|
||||
results/
|
||||
*.parquet
|
||||
.DS_Store
|
||||
Binary file not shown.
@@ -0,0 +1,346 @@
|
||||
**Why surgical trials fail, and what each target trial emulation answers instead: a known-truth simulation of early peripheral vascular intervention for claudication**
|
||||
|
||||
*Use case: PVI within 6 months of a new claudication diagnosis, compared with initial conservative management; Medicare-like claims; 5-year major adverse limb events (MALE)*
|
||||
|
||||
Analysis plan, draft v1.0, 2026-10-27. Every decision cited as "D#" is recorded, with its reason, in `docs/decisions.md`. The readings are in `docs/literature/`.
|
||||
|
||||
# 1. The paper in one paragraph
|
||||
|
||||
Whether to perform peripheral vascular intervention (PVI) early for claudication is one of the most common decisions in vascular surgery. It has never been tested in a randomized trial on outcomes that matter to the limb. The trials that do exist measured walking distance and quality of life in 56–240 patients, many recruited poorly, and in their conservative arms patients crossed over to intervention. Observational studies in Medicare suggest early PVI leads to more interventions and more chronic limb-threatening ischemia (CLTI). They use different designs, though (landmark, clone–censor–weight), and those designs answer different questions. We build one simulated Medicare-like population of claudicants and their physicians in which the true effect of early PVI is known by construction. In it we switch on, one at a time, the features that make surgical trials hard:
|
||||
|
||||
- variation in surgeon skill;
|
||||
- surgeon practice style;
|
||||
- surgeons who also decide the outcome (reintervention, CLTI coding);
|
||||
- symptom-driven deferral and crossover;
|
||||
- unmeasured symptom severity.
|
||||
|
||||
We then run naive, landmark, sequential-trial and clone–censor–weight (CCW) analyses, with four ways of handling the physician, alongside the RCT that could realistically be run. Each method is scored against the question it actually answers and against the question a surgeon is asking. We also test whether negative controls can tell when the answer is wrong.
|
||||
|
||||
# 2. The gap
|
||||
|
||||
## 2.1 What is already known
|
||||
|
||||
- **Trials of claudication are small and short, and none measures hard outcomes.** CLEVER (n = 111; planned 252), ERASE (212), IRONIC (158), MIMIC (127), SUPER (240, stopped early, 29% crossover in year 1) and OBACT (56 randomized of 434 screened) all used walking or QoL endpoints. Where follow-up was long, the benefit faded by 2–5 years (ERASE, IRONIC). The Cochrane review pooled no hard limb outcomes (Fakhry 2018).
|
||||
|
||||
- **Surgical RCTs fail for structural reasons.** 21% of surgical RCTs stop early, mostly for poor recruitment (Chapman 2014). Surgical trials are discontinued more often than medical trials (43% vs 27%; Rosenthal 2015). Comparisons of surgery with non-surgery recruit about twice as slowly (Cook 2008). Crossover can make ITT uninterpretable (SPORT; Weinstein 2006). The intervention's effect depends on the operator (Birkmeyer 2013), and trial surgeons are selected (Rothwell 2005; Cutteridge 2025).
|
||||
|
||||
- **In timing trials, the "conservative" arm becomes delayed surgery.** Examples: hernia watchful waiting reached 68% crossover in the long term (Fitzgibbons 2013); 62% of the ADAM surveillance arm were repaired (Lederle 2002); 85% of CAESAR by 54 months (Cao 2011).
|
||||
|
||||
- **Medicare evidence on early PVI.** 3.2% of new claudicants have early PVI. The median physician rate is 0%, and 5.6% of physicians are outliers (Hicks 2020). Early PVI is associated with late PVI (22.5% vs 3.6%; aHR 6.89) and with CLTI (16.4% vs 7.8%) (Sorber 2023). A CCW emulation found more CLTI and MALE (Dun, VESS 2026). The Sorber design is in effect a 6-month landmark analysis.
|
||||
|
||||
- **Methods.** Immortal time (Suissa 2008), landmark (Anderson 1983; Dafni 2011), CCW (Hernán 2018; Maringe 2020; grace periods: Wanis 2024) and sequential trials (Hernán 2008; Keogh 2023) are each well described. Duchesneau 2022 compared landmark and CCW against a plasmode truth in cancer surgery. Unmeasured confounding at the cluster level has been studied outside TTE (Li 2020; Cafri 2020). So has bias amplification from conditioning on an instrument (Myers 2011).
|
||||
|
||||
## 2.2 The gap this paper fills
|
||||
|
||||
**No study has put landmark, sequential-trial and CCW analyses of one surgical timing question side by side, each scored against its own true estimand, in a world where the surgeon drives treatment, modifies its effect, and partly decides the outcome. The study should also run alongside the RCT that could realistically be done.** So surgeons reading a Medicare TTE cannot tell:
|
||||
|
||||
1. which question the analysis answered;
|
||||
2. how far that is from the question they care about;
|
||||
3. which surgery-specific threats could still bias it;
|
||||
4. whether a clean negative control means anything.
|
||||
|
||||
## 2.3 Why it matters
|
||||
|
||||
- Claudication PVI is high-volume, increasingly done in office-based labs, and varies widely between physicians (Hicks 2020; Dun 2024). The question will be answered by claims analyses, not trials.
|
||||
|
||||
- Journals now expect TTE (TARGET 2025). Readers in surgery need to know what each design estimates and where it breaks, in clinical language.
|
||||
|
||||
# 3. Aims and hypotheses
|
||||
|
||||
| **Aim** | **Question** | **Hypotheses** | **Primary output** |
|
||||
|---|---|---|---|
|
||||
| **Aim 1. What each method answers** | In a clean world (all confounders measured, one physician type), how far is each method's own estimand from the clinically relevant one (question gap), and is each method unbiased for its own estimand? | H1a: Each TTE method is unbiased for its own estimand. H1b: Landmark and sequential-trial ITT have large question gaps, while CCW-B1 has none by definition. H1c: The ITT of the realistic RCT is diluted towards the null by crossover, and is shifted by surgeon selection. | Fig 1 (base panel), Table 2 |
|
||||
| **Aim 2. Surgery-specific threats** | How does each threat change the estimation bias of each method × physician variant? | H2a: Cluster-robust SEs fix coverage, not bias. H2b: A physician random intercept in the weight models removes only part of the confounding by physician; within-physician analysis (FE) removes it when style affects the outcome directly, but amplifies residual patient confounding when style acts only on treatment (Z-bias). H2c: Unmeasured severity biases every observational method; no physician variant fixes it. H2d: Crossover driven by deferral hurts landmark and RCT-ITT most. | Fig 1 (threat panels), Fig 2, Table 3 |
|
||||
| **Aim 3. Do negative controls detect the problem?** | When a method's MALE estimate is biased by more than 1 percentage point, how often does each negative control flag it? | H3a: NCO-1 (patient-level) is sensitive to unmeasured severity or frailty. H3b: NCO-2 (physician-level) flags style variation even when style is not confounding, so its specificity is low. H3c: Substantial bias can sit behind a clean negative control. | Fig 3, Table 4 |
|
||||
|
||||
*Secondary (supplement): each MALE component separately; total procedures per patient; Sorber "late PVI" definition; HR scale; MALE-free survival; 3-month window; sequential trials per-protocol; preference IV; unmeasured supervised exercise; n = 200,000; full factorial if the one-at-a-time results suggest interactions.*
|
||||
|
||||
# 4. Target trial specification
|
||||
|
||||
| **Component** | **Target trial** | **Emulation in the simulated claims** |
|
||||
|---|---|---|
|
||||
| Eligibility | Medicare beneficiaries ≥66 y with a new claudication diagnosis, no PVI, bypass, CLTI or amputation in the prior 12 months | Same, at the diagnosis month (month 0) |
|
||||
| Strategies | **A:** PVI within 6 months of diagnosis. **B1 (primary comparator):** no PVI in months 0–5, then usual care. **B2:** no elective PVI at any time (PVI only after a MALE event) | Observed PVI timing compared with each strategy |
|
||||
| Assignment | Randomized at diagnosis | Exchangeable conditional on measured baseline and time-varying covariates (and, by variant, on the physician) |
|
||||
| Time zero | Diagnosis = eligibility = assignment | Same |
|
||||
| Follow-up | Months 0–59. Ends at MALE, death or month 60 | Same; no loss to follow-up (as in Medicare FFS without disenrollment) |
|
||||
| Outcome | MALE: first of CLTI, major amputation, open bypass, or any revascularization after the first (index) one (D7). Death is a competing event (D8) | Same, from simulated claims |
|
||||
| Causal contrast | Per-protocol (sustained strategies) | CCW with artificial-censoring weights |
|
||||
| Measure | 5-year cumulative incidence of MALE; risk difference A − B1 (and A − B2) | Weighted cause-specific hazards → cumulative incidence |
|
||||
|
||||
**Clinically relevant estimand (the reference, D11):** RD(A − B1) at month 60, among all new claudicants, with PVI done by each patient's own physician (D21), under full adherence.
|
||||
|
||||
# 5. Data-generating mechanism
|
||||
|
||||
## 5.1 Structure
|
||||
|
||||
Time is in months 0–59 (D16). Each replicate draws **J ≈ 1,500 physicians** and **N = 50,000 patients** (D25). Patients are assigned to physicians with lognormal panel sizes (median ≈ 20, long right tail).
|
||||
|
||||
**Physicians (j).**
|
||||
- Latent **skill** K_j ~ N(0, σ_K²) and latent **style** S_j ~ N(0, σ_S²), with corr(K, S) = ρ.
|
||||
- **Practice setting** OBL_j ∈ {0,1} is observed, with logit P(OBL_j = 1) = o₀ + o_S·S_j (high-style physicians are more often in office-based labs or ASCs).
|
||||
|
||||
**Patients (i), at diagnosis.**
|
||||
- Measured X_i: age, female, Black race, diabetes, CKD/dialysis, CHF, COPD, prior CVD, coded smoking, calendar year.
|
||||
- Latent **frailty** F_i, correlated with the measured comorbidities.
|
||||
- Latent **true smoking** (coded smoking is a misclassified version of it).
|
||||
- Latent **initial severity** Z_i0 = γ₀ + γ_X·X_i + e_i.
|
||||
- Latent **progression rate** r_i = r₀ + r_X·(diabetes, smoking) + u_i.
|
||||
|
||||
**Latent severity process.** Z_it = Z_i,t−1 + r_i + ε_it, so severity drifts upward at a patient-specific rate.
|
||||
|
||||
**What a PVI does (the treatment mechanism, D20).** At the month of PVI:
|
||||
- **Relief.** Severity drops by Δ (symptom relief, which lowers the near-term CLTI hazard).
|
||||
- **Restenosis and progression.** After PVI, the progression rate increases by κ·g(K_j), where g decreases with skill. This adds a skill-dependent hazard of repeat revascularization and of later CLTI.
|
||||
- Relief early and faster progression later produce the *benefit-then-harm* pattern (D4, factor 7). Δ and κ set the truth scenario (§5.3).
|
||||
|
||||
**PVI decision, before the first PVI.** The monthly hazard is
|
||||
|
||||
logit h_PVI(t) = α₀ + α_Z·Z_it + α_X·X_i + α_F·F_i + α_S·S_j + α_win·1[t ≥ 6]·(β_def + β_Z,def·Z_it)
|
||||
|
||||
- Inside the window (months 0–5) the decision is driven by severity, which is confounding by indication.
|
||||
- After month 6, deferred PVI is governed by β_def and β_Z,def. This is factor 4.
|
||||
|
||||
**Outcomes (monthly cause-specific hazards; pooled logistic form).**
|
||||
|
||||
| **Process** | **Depends on** |
|
||||
|---|---|
|
||||
| True CLTI | Z_it, X_i, F_i |
|
||||
| Coded CLTI | True CLTI + extra coding hazard ∝ S_j·1[factor 3] among patients with moderate Z_it |
|
||||
| Major amputation | Mainly after CLTI; small direct hazard from Z_it, diabetes, CKD |
|
||||
| Open bypass (elective) | Z_it; higher if a previous PVI failed |
|
||||
| Repeat revascularization (only after the index PVI) | Restenosis process (κ, K_j); lower threshold ∝ S_j·1[factor 3] |
|
||||
| Death | Age, X_i, F_i, Z_it (weak) |
|
||||
| NCO-1: hip-fracture or pneumonia admission | F_i, true smoking, age. **Never** PVI |
|
||||
| NCO-2: varicose-vein ablation | S_j, latent venous disease. **Never** PVI |
|
||||
| NCE: low-value procedure (vein ablation) in months 0–5 | S_j, F_i. **Never** affects any outcome |
|
||||
|
||||
**Measured vs latent (D17).**
|
||||
- What the analyst sees: X_i, a claims frailty index, the physician ID, OBL_j, all coded events, and a time-varying **severity proxy**. The proxy is a monthly count of vascular visits and noninvasive arterial tests, P_it ~ Poisson(exp(p₀ + λ·Z_it)), plus its running sum.
|
||||
- What is latent: Z_it, r_i, F_i (the frailty index is its proxy), true smoking, K_j, S_j.
|
||||
|
||||
**Table A. Variable roles**
|
||||
|
||||
| **Block** | **Causal role** | **Clinical analogue** | **Measured?** |
|
||||
|---|---|---|---|
|
||||
| X | Baseline confounders | Age, sex, race, diabetes, CKD, CHF, COPD, prior CVD, coded smoking, year | Yes |
|
||||
| F | Patient confounder (death, PVI candidacy, NCO-1) | Frailty | Through a frailty-index proxy |
|
||||
| Z_t | Time-varying confounder (PVI decision, CLTI) | Walking limitation / ABI | Through a visit/testing proxy, P_t (strength λ) |
|
||||
| r | Progression rate (prognostic) | Diabetes/smoking-driven disease progression | No |
|
||||
| K | Physician effect modifier (and confounder if ρ ≠ 0) | Technical skill, durability of PVI | No (physician ID only) |
|
||||
| S | Physician treatment driver; with factor 3 on, also an outcome driver | Practice style / "high user" | No (physician ID, OBL) |
|
||||
| NCO-1, NCO-2, NCE | Negative controls (true effect exactly 0) | Hip fracture/pneumonia; vein ablation | Yes |
|
||||
| E (supplement only) | Unmeasured co-intervention | Supervised exercise uptake | No |
|
||||
|
||||
## 5.2 Worlds (one factor at a time, D24)
|
||||
|
||||
Every world is run under each of the three truth scenarios in §5.3.
|
||||
|
||||
| **World** | **Skill (1)** | **Style (2)** | **Style → outcome (3)** | **Deferral/crossover (4)** | **Unmeasured severity (5)** |
|
||||
|---|---|---|---|---|---|
|
||||
| **W0 Base (clean)** | σ_K = 0 | Modest σ_S | Off | Low, unrelated to severity | Off: λ high, proxy ≈ Z |
|
||||
| W1 Skill | σ_K > 0, ρ = −0.3 (and ρ = 0) | as W0 | Off | as W0 | Off |
|
||||
| W2 Style | as W0 | Strong σ_S, calibrated to the Hicks 2020 distribution | Off | as W0 | Off |
|
||||
| W3 Style → outcome | as W0 | Strong | **On** | as W0 | Off |
|
||||
| W4 Deferral | as W0 | as W0 | Off | **High, severity-driven after month 6** | Off |
|
||||
| W5 Unmeasured severity | as W0 | as W0 | Off | as W0 | **On: λ low (proxy–Z correlation ≈ 0.4), frailty index noisy** |
|
||||
| **WR Realistic** | On | Strong | On | High | On |
|
||||
|
||||
W1 is run at both ρ = −0.3 and ρ = 0 (D22). WR is calibrated to the Medicare targets in §5.4.
|
||||
|
||||
## 5.3 Truth scenarios
|
||||
|
||||
| **Scenario** | **Mechanism** | **Target** |
|
||||
|---|---|---|
|
||||
| **Null (sharp disease null, D34)** | Δ = 0, κ = 0: PVI changes nothing about severity, CLTI, amputation, bypass or death | RD(A − B1) = 0 **exactly** for the disease composite (CLTI or major amputation) and every disease component. MALE is still > 0, because only patients who have had a PVI can have a repeat revascularization: this is the mechanical effect of how the outcome is defined, and it is reported as a finding |
|
||||
| **Harm** | Δ small, κ moderate | RD(A − B1) ≈ +5 percentage points |
|
||||
| **Benefit then harm** | Δ large, κ moderate | Cumulative incidence curves cross: A lower before about month 18, higher at 60 |
|
||||
|
||||
## 5.4 Calibration targets for the usual-care world (WR)
|
||||
|
||||
| **Target** | **Value** | **Source** |
|
||||
|---|---|---|
|
||||
| Early PVI (≤6 months) | 3.2% | Hicks 2020 (31208940); Sorber 2023 (37276171) |
|
||||
| Physician early-PVI rate | median 0%, range 0–58%, about 5.6% of physicians ≥2 SD outliers | Hicks 2020 |
|
||||
| OBL/ASC share of PVI | about 50% | Dun 2024 (38958132) |
|
||||
| Late PVI (>6 months, ~4.4 years) | 22.5% after early PVI vs 3.6% without | Sorber 2023 |
|
||||
| CLTI (~4.4 years) | 16.4% vs 7.8% | Sorber 2023 |
|
||||
| Mean age | about 76 | Dun 2025 (40782724) |
|
||||
| All-cause mortality | Interim target about 35% at 5 years (Swedish population cohort: 10-year mortality in IC 63% vs 27% reference). **Replace with the 5-year mortality of the authors' own Medicare claudication cohort** (Sorber 2023 / VESS 2026 data) | Sigvant 2018 *EJVES* (29478910); cardiovascular-only 5-year mortality 14–16%: Baubeta Fridh 2018 *BJS* (29116654) |
|
||||
| Amputation | about 0.4%/year after revascularization for IC | Baubeta Fridh 2017 (28797662) |
|
||||
|
||||
Calibration solves for the intercepts and the key coefficients (α₀, α_S, σ_S, β_def, the outcome intercepts) by stochastic approximation on 10⁵ patients, as in `tte-papers/simulations/01_calibrate.R`.
|
||||
|
||||
## 5.5 Computing the truths
|
||||
|
||||
Truths come from simulating 10⁶ patients, reusing the physicians, under each intervention. The latent processes stay the same; only the PVI decision is overridden.
|
||||
|
||||
| **Estimand** | **Intervention used for the truth** |
|
||||
|---|---|
|
||||
| A | Natural PVI hazard in months 0–4. Everyone still untreated, alive and MALE-free gets PVI at month 5 (the "natural grace" strategy that CCW with end-of-window censoring targets; Wanis 2024) |
|
||||
| B1 | PVI hazard = 0 in months 0–5, then natural |
|
||||
| B2 | PVI hazard = 0 at all times before MALE |
|
||||
| Sequential-trial ITT | For each m = 0–5, take the natural-course population eligible at m. Force PVI at m vs no PVI at m (natural course afterwards in both). Pool over m, weighted by the number eligible |
|
||||
| Landmark | No interventional definition, because the comparison is among survivors of the observed regime. Its own estimand is the **oracle large-sample limit**: the landmark IPW estimator on 10⁷ patients, with latent Z and F included in the weight model |
|
||||
| Naive | No own estimand. Only total error against the reference is reported |
|
||||
| Realistic RCT ITT | Simulate the trial population (hospital-based, top-third skill, low-style physicians; eligible patients), with assignment to A or B under natural adherence (refusal in A; severity-triggered crossover in B) |
|
||||
| Ideal pragmatic RCT ITT | Same, with all physicians |
|
||||
|
||||
Truths are stored in `simulations/truths/` as small committed CSVs, and each has its own Monte Carlo SE.
|
||||
|
||||
# 6. Estimands, in plain words (for the main text)
|
||||
|
||||
| **Method** | **The question it answers** |
|
||||
|---|---|
|
||||
| CCW, A vs B1 (the reference) | "If every newly diagnosed claudicant had PVI within 6 months, compared with if every one waited 6 months and was then treated as usual, how different would 5-year MALE be?" |
|
||||
| CCW, A vs B2 | "…compared with if no one ever had an elective PVI?" (abstention, not deferral) |
|
||||
| Sequential trials, ITT | "For a patient who is still eligible this month, what does doing the PVI now change, compared with not now (perhaps later)?" |
|
||||
| Landmark at 6 months | "Among patients still alive and MALE-free at 6 months, how do those who happened to have a PVI differ from those who didn't?" |
|
||||
| Naive "ever early PVI" | No well-defined question (immortal time) |
|
||||
| Realistic RCT, ITT | "Among patients that selected, skilled, low-intervention surgeons are willing to randomize, what does *assigning* early PVI change, given that some patients refuse it and some controls cross over?" |
|
||||
| Ideal pragmatic RCT, ITT | Same, for all surgeons |
|
||||
|
||||
# 7. Methods grid
|
||||
|
||||
**Adjustment (D26).** The same machinery is used everywhere. Inverse-probability weights come from pooled logistic models on the measured covariates (X, frailty index, P_t history, OBL, year). Weighted cause-specific pooled logistic hazards for MALE and death are fitted with flexible time terms and combined into standardized cumulative incidence at month 60. Weights are truncated at the 99.5th percentile in a supplementary analysis only.
|
||||
|
||||
| **Method** | **Weights** | **Notes** |
|
||||
|---|---|---|
|
||||
| Naive | Baseline PS for "PVI by month 6" | Follow-up starts at month 0 for both groups, so immortal time is present |
|
||||
| Landmark (D13) | PS for "PVI in months 0–5" among patients at risk at month 6, using covariates up to month 6 | Outcome counted through month 60 |
|
||||
| Sequential trials (D14) | A PS for treatment at each trial baseline m = 0–5 | Person-trials are stacked. ITT is primary; per-protocol (with IPCW for deviation) is in the supplement |
|
||||
| CCW-B1, CCW-B2 | Clones are artificially censored at deviation, with IPCW from the PVI-hazard model | A clones untreated at month 5 are censored at the end of the window |
|
||||
| Realistic and ideal RCT | None | ITT by assigned arm; naive per-protocol by treatment received |
|
||||
|
||||
**Physician variants (D15), crossed with every observational method.** The physician enters through the **weight models**, because that is where confounding is controlled.
|
||||
|
||||
| **Variant** | **Implementation** |
|
||||
|---|---|
|
||||
| V1 Ignore | No physician terms; model-based SE |
|
||||
| V2 Cluster-robust | As V1, with the sandwich SE clustered on the physician |
|
||||
| V3 Random intercept | Physician random intercept in the weight models (`glmmTMB`/`lme4`); cluster-robust SE |
|
||||
| V4 Within-physician | Physician fixed effects in the weight models. Physicians with no variation in early PVI are dropped, and the change to the estimand that this causes is reported. Mundlak (physician means) is the fallback if FE is too slow. Cluster-robust SE |
|
||||
| V5 Preference IV (supplement) | Instrument = the physician's early-PVI rate among the physician's *other* patients; two-stage residual inclusion |
|
||||
|
||||
**Negative controls (D32).** NCO-1 and NCO-2 replace MALE as the outcome in every method × variant. The NCE replaces early PVI as the exposure in the CCW and landmark designs. The NCE models adjust for early-PVI status (D36). Without that adjustment, the path NCE ← style → PVI → MALE would make the NCE "detect" a real effect of PVI. The unadjusted NCE analysis is in the supplement.
|
||||
|
||||
# 8. Simulated RCTs (D18)
|
||||
|
||||
| | **Realistic RCT** | **Ideal pragmatic RCT** |
|
||||
|---|---|---|
|
||||
| Physicians | Hospital-based (OBL = 0), top third of skill, bottom half of style | All physicians |
|
||||
| Patients | Eligible patients of those physicians; a constant 30% consent | All eligible patients |
|
||||
| n | 240 (120 per arm) | 50,000 |
|
||||
| Arm A | PVI at the next scheduled month in 0–2. Refusal 10% | Same |
|
||||
| Arm B | No PVI in months 0–5 unless severity crosses the "lifestyle-limiting" threshold (crossover generated by the DGP); usual care afterwards | Same |
|
||||
| Analyses | ITT (unadjusted cumulative incidence) and naive as-treated | Same |
|
||||
|
||||
Running replicates of the 240-patient trial also gives its power and CI width, which is the "small trial" result.
|
||||
|
||||
# 9. Performance measures and simulation size (D27–D29)
|
||||
|
||||
**Notation.** For method m with estimate θ̂_m, its own estimand θ_m, and the clinically relevant estimand θ*:
|
||||
|
||||
| **Measure** | **Definition** |
|
||||
|---|---|
|
||||
| Estimation bias | E[θ̂_m] − θ_m |
|
||||
| Question gap | θ_m − θ* |
|
||||
| Total error | E[θ̂_m] − θ* (the sum of the two above) |
|
||||
| Empirical SE | SD of θ̂_m across replicates |
|
||||
| Model SE | Mean of the estimated SEs |
|
||||
| Coverage | Coverage of θ_m by the 95% CI |
|
||||
| RMSE | Against θ_m and against θ* |
|
||||
| **False-alarm rate** (headline) | In the null scenario, the proportion of replicates whose 95% CI excludes 0 **for the disease composite (CLTI or major amputation)**, whose true effect is exactly 0 (D34) |
|
||||
|
||||
MCSE is reported for every measure (Morris 2019).
|
||||
|
||||
- **Variance (D28).** A sandwich SE clustered on the physician, with the delta method for the cumulative-incidence RD. Validated by a physician-cluster bootstrap (200 resamples) on 100 replicates in W0 and WR.
|
||||
- **Replicates (D29).** 100 at draft scale, 500 at final. With 500 replicates, the MCSE of coverage is about 1% and the MCSE of bias is about 0.045 × empirical SE.
|
||||
|
||||
# 10. Negative-control diagnostic analysis (Aim 3, D33)
|
||||
|
||||
For each world × scenario × method × variant × replicate, record:
|
||||
|
||||
- **Flag**: the negative control's 95% CI excludes 0.
|
||||
- **Truly biased**: |θ̂_m − θ_m| > 1 percentage point. Also |θ̄_m − θ_m| > 1 pp at the cell level.
|
||||
|
||||
**Reported:**
|
||||
- the sensitivity and specificity of each negative control, and of "any negative control", for detecting bias;
|
||||
- the distribution of |bias| among replicates with a clean negative control ("how much can hide");
|
||||
- in the supplement, each negative control's false-positive rate.
|
||||
|
||||
# 11. Mock tables
|
||||
|
||||
- **Table 1 (introduction). Randomized trials of revascularization for claudication.** Trial, year, n (planned vs actual), comparison, primary endpoint, follow-up, crossover, whether the trial stopped early, whether there was a hard limb outcome. See Appendix A.
|
||||
|
||||
- **Table 2. What each method answers (W0, all three scenarios).** Rows: methods. Columns: plain-language estimand, own truth θ_m, question gap, estimation bias, coverage.
|
||||
|
||||
- **Table 3. Bias by threat.** Rows: method × physician variant. Columns: W0–W5, WR. Cells: estimation bias (MCSE), shaded by |bias| > 1 pp.
|
||||
|
||||
- **Table 4. Negative controls as bias detectors.** Rows: NCO-1, NCO-2, NCE, "any". Columns: sensitivity, specificity, and the median |bias| when the negative control is clean, by world.
|
||||
|
||||
# 12. Mock figures
|
||||
|
||||
- **Fig 1. Estimand ladder (D30).** One row per method. Each row shows the method's own truth (hollow dot), its mean estimate and 95% interval across replicates (filled dot, coloured by physician variant), and the clinically relevant truth (vertical line). Panels: W0, W1–W5, WR. Two versions: null and harm.
|
||||
|
||||
- **Fig 2. False-alarm rate in the null scenario** (disease composite). Heat map of method × variant (rows) against world (columns). The 5% nominal level is marked. An inset shows the MALE estimate in the same null world: "harm" produced purely by how the outcome is defined.
|
||||
|
||||
- **Fig 3. What a clean negative control is worth.** For each method, |bias| plotted for flagged and clean replicates, by world.
|
||||
|
||||
- **Fig S. The DGP as a DAG**, and cumulative-incidence curves under A, B1 and B2 for each scenario.
|
||||
|
||||
# 13. Position relative to the literature
|
||||
|
||||
| **Prior work** | **What it showed** | **What this paper adds** |
|
||||
|---|---|---|
|
||||
| CLEVER, ERASE, IRONIC, SUPER; Fakhry 2018 | Small trials of walking and QoL, with crossover | The trial that could realistically be run, simulated and compared with the question surgeons ask |
|
||||
| Sorber 2023; Hicks 2020; Dun 2025/2026 | Medicare associations; physician outliers; CCW emulation | What those designs actually estimated, and which threats remain |
|
||||
| Duchesneau 2022 CEBP | Landmark vs CCW against a plasmode truth, in cancer surgery | Adds sequential trials, RCTs and the physician level; estimand-specific truths |
|
||||
| Keogh 2023; Hernán 2008/2018; Wanis 2024 | Sequential trials, CCW, grace periods in theory and simulation | The difference between deferral and abstention shown in a surgical question |
|
||||
| Li 2020 SMMR; Cafri 2020 Biom J | Unmeasured cluster confounding; within-surgeon matching | Inside TTE designs, with the surgeon acting on treatment, effect and outcome |
|
||||
| Myers 2011; Brookhart 2006 | Z-bias; preference IV | Shown for physician style in claims |
|
||||
| Paper 1 (tte-papers) | Design matters more than the adjustment method (ICU) | A surgical setting; estimand gaps; negative controls as diagnostics |
|
||||
|
||||
# 14. Out of scope
|
||||
|
||||
- PS matching and doubly robust estimators (Paper 1 territory).
|
||||
- Data-quality problems (Paper 2 territory): coding error beyond factor 3, missing data, disenrollment.
|
||||
- Patient–physician sorting, where sicker patients see particular physicians. Patients are assigned to physicians at random (D37); sorting is a possible single sensitivity world.
|
||||
- Cost-effectiveness.
|
||||
|
||||
# 15. Practicalities
|
||||
|
||||
| **Item** | **Plan** |
|
||||
|---|---|
|
||||
| Software | R (4.6.1). A new package `pvisim/` in this repo with parameters in YAML, a simulator, truths, designs, estimators and metrics (D12). Uses `fastglm`/`speedglm` for pooled logistic, `sandwich`, `glmmTMB`, `data.table`, `arrow`, `furrr`. Quarto reports; renv |
|
||||
| Validation | Before any runs: (a) every method is unbiased for its own estimand in W0 at n = 10⁶ (within MCSE); (b) a closed-form check of the CCW truth in a special case with no time-varying confounding; (c) negative-control truths are exactly 0 by construction (tests) |
|
||||
| Compute | 7 worlds (+ W1 at ρ = 0) × 3 scenarios × 500 replicates. Each replicate fits about 7 methods × 4 variants × (MALE + 3 negative controls). V3/V4 on person-month data are the bottleneck. Profile at the draft stage and switch to Mundlak if needed |
|
||||
| Sequence | 1) parameters YAML → 2) simulator + tests → 3) calibration → 4) truths → 5) estimators, validated at large n → 6) draft runs (100 replicates) → 7) report → 8) final runs → 9) manuscript |
|
||||
| Reporting | ADEMP (Morris 2019); TARGET 2025 items for the emulated trial; code and seeds public |
|
||||
| Target journal | JAMA Surgery (Original Investigation). The mathematics and the full grid go in the supplement |
|
||||
|
||||
# 16. Decisions resolved after the first draft (D34–D37)
|
||||
|
||||
1. **Null (D34).** Sharp disease null. The headline false-alarm rate is reported for the disease composite (CLTI or major amputation). MALE in the null world shows "harm" that comes purely from the outcome definition.
|
||||
2. **Strength of each threat when switched on (D35).** Use the empirical quantity where one exists (physician-style spread from the Hicks 2020 outlier distribution; deferred PVI from Sorber's late-PVI rates; early-PVI rate). Where none exists (skill, coding intensity), set the threat so that on its own it moves the naive or landmark estimate by about 2–4 percentage points in the harm scenario.
|
||||
3. **NCE (D36).** The NCE analysis is adjusted for early-PVI status. The unadjusted version goes in the supplement as a warning.
|
||||
4. **Patient–physician sorting (D37).** Not in the main grid. One sensitivity world, if time allows.
|
||||
|
||||
Still to do: replace the interim mortality target (§5.4) with the 5-year mortality of the authors' own Medicare cohort.
|
||||
|
||||
---
|
||||
|
||||
# Appendix A. Draft Table 1: randomized trials of revascularization for intermittent claudication
|
||||
|
||||
| **Trial** | **n (planned → randomized)** | **Comparison** | **Primary endpoint** | **Longest follow-up** | **Crossover / adherence** | **Stopped early / under-recruited** | **Hard limb outcome?** |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| CLEVER (Murphy 2012, 2015) | 252 → 111 | OMC vs OMC+SE vs OMC+stent (aortoiliac) | Peak walking time, 6 months | 18 months (79 completed) | — | Under-recruited (reduced to 3 arms) | No |
|
||||
| ERASE (Fakhry 2015; Klaphake 2022) | 212 | Endovascular + SET vs SET | Maximum walking distance, 12 months | ~5.4 years (post hoc) | SET arm: later revascularization HR 2.5 | No | No (procedures reported) |
|
||||
| IRONIC (Nordanstig 2014; Djerf 2020) | 158 | Revascularization + BMT + exercise vs BMT + exercise | SF-36, 1 year | 5 years (benefit lost) | — | Single centre | No |
|
||||
| MIMIC (Greenhalgh 2008) | 93 + 34 | PTA added to SE + BMT | Walking distance, 24 months | 24 months | — | Small | No |
|
||||
| SUPER (Koelemay 2022) | 240 | SET vs endovascular (iliac) | MWD + VascuQol, 1 year | ~5.5 years | **29% crossover by year 1**; 50% SET compliance | **Stopped early** | No |
|
||||
| Mazari (2012, 2017) | 178 | PTA vs SEP vs both (fem-pop) | Walking/QoL, 12 months | 5.2 years | — | — | No (reinterventions reported) |
|
||||
| Spronk (2009) | 151 | Endovascular vs SET | Rutherford, 6–12 months | 12 months | — | — | No |
|
||||
| OBACT (Nylaende 2007) | 434 screened → 56 | OMT ± PTA | QoL, ABI, walking, 2 years | 2 years | — | **56 of 434 screened** | No |
|
||||
|
||||
*Missing cells are to be filled from the full texts (crossover is not reported in several abstracts).*
|
||||
@@ -0,0 +1,121 @@
|
||||
# Decisions log
|
||||
|
||||
Each decision is listed with the date and the reason for it. Settled decisions are changed only by adding a new entry below them, never by editing them.
|
||||
|
||||
## Round 1: framing (2026-10-27)
|
||||
|
||||
| # | Decision | Why |
|
||||
|---|---|---|
|
||||
| 1 | **Paper type: a "teaching" paper combined with a "threats" paper.** The backbone is that landmark, CCW and sequential trials answer three different clinical questions. The experimental factors are the threats specific to surgery. We do not claim that "TTE works". | Earlier simulations already show that the methods work. The new contribution is the surgical setting, plus showing what each estimand means in it. |
|
||||
| 2 | **Primary outcome: MALE composite. Secondary outcomes: each component on its own.** | Matches the VESS 2026 abstract and practice in vascular surgery. The components show how outcomes that physicians generate (reintervention) behave differently from outcomes that come from disease (CLTI, amputation). The exact definition of MALE is still open (round 2). |
|
||||
| 3 | **Include a simulated RCT arm in the same DGP** (ITT and naive per-protocol, with crossover, surgeon selection and small n). | Turns "why RCTs are hard in surgery" into numbers. |
|
||||
| 4 | **Experimental factors:** (1) surgeon skill, (2) surgeon preference or style, (3) style acting on the outcome directly, (4) deferral and crossover driven by symptom progression, (5) unmeasured symptom severity. **Built in through the RCT arm:** (6) trial selection of high-skill surgeons. **Fixed feature of the DGP:** (7) treatment effect that changes over time. **Supplement / sensitivity analysis:** (8) unmeasured co-intervention (uptake of supervised exercise). | Keeps the main grid readable. |
|
||||
| 5 | **Audience: surgeons. Target journal: JAMA Surgery.** Each method in the main text has a one-sentence clinical definition. The mathematics goes in the supplement. | The user is a surgical health-services researcher. Readers are clinicians. |
|
||||
| 6 | **The RCT limitations are covered as narrative in the introduction plus one evidence table** (claudication RCTs: n, endpoint, crossover, recruitment shortfall). There is no separate systematic review. | Scope. |
|
||||
|
||||
## Round 2: the target trial (2026-10-27)
|
||||
|
||||
| # | Decision | Why |
|
||||
|---|---|---|
|
||||
| 7 | **MALE = CLTI, major amputation, open bypass, or any revascularization after the first.** The index PVI does not count as an event in either arm, whenever it happens. Total procedures per patient is a secondary outcome. **Supplement:** the Sorber "late PVI" definition (any PVI after month 6), to show how the outcome definition alone changes the answer. | Avoids an outcome that is asymmetric between arms by construction. Not counting the index procedure is defined the same way in both arms. |
|
||||
| 8 | **Death is a competing event.** The primary measure is the 5-year cumulative incidence of MALE, reported as a risk difference. HR, and MALE-free survival (MALE or death), go in the supplement. | Medicare population with a mean age of about 76. Censoring at death would answer an unrealistic hypothetical question. |
|
||||
| 9 | **Time zero is the new claudication diagnosis. Strategy A is PVI within 6 months.** A 3-month window is a supplement-only sensitivity analysis. | Matches Sorber 2023, VESS 2026, Dun 2025, and the guidelines. |
|
||||
| 10 | **Strategy B, primary: no PVI in the first 6 months, then usual care.** **Strategy B, second (main text): no elective PVI unless a MALE-qualifying event occurs.** | Deferral vs abstention: "conservative" is not a single strategy (the ADAM/CAESAR lesson). |
|
||||
| 11 | **Each method gets two errors: estimation bias (estimate vs its own estimand) and question gap (its own estimand vs the clinically relevant estimand). Total error is the sum of the two.** The clinically relevant reference is A vs B(primary) in all new claudicants under full adherence, which is the CCW estimand. | The teaching paper needs to separate "the method fails" from "the method answers another question". |
|
||||
| 12 | **R, with a new self-contained package in this repo.** It follows the `tte-papers` conventions (YAML parameters, truths from simulating the interventions in large samples, Quarto, renv) but does not reuse its code. | The DGP differs (months, not hours; physicians as clusters). |
|
||||
|
||||
## Round 3: comparators and the simulated world (2026-10-27)
|
||||
|
||||
| # | Decision | Why |
|
||||
|---|---|---|
|
||||
| 13 | **Landmark:** month 6, among patients alive and MALE-free. Exposure is PVI in months 0–6. Outcome is MALE by month 60 *since diagnosis*. Adjust for baseline covariates plus measured covariates at month 6. | Same horizon as CCW. Mirrors the Sorber 2023 design, so the paper can say what that design estimated. |
|
||||
| 14 | **Sequential trials:** monthly trials in months 0–5 among patients still eligible. **ITT is primary**; per-protocol (weighted) goes in the supplement. | Sequential ITT estimates "now vs maybe later" (deferral), in contrast to CCW's "within 6 months vs not". |
|
||||
| 15 | **Physician-level variants, crossed with every method:** (1) ignore, (2) cluster-robust SE, (3) physician random intercept, (4) within-physician (FE or Mundlak). **Supplement:** (5) physician-preference IV. | Clustering fixes SEs, not bias. A random intercept does not remove confounding. FE can, but it risks Z-bias. The IV analysis is dubious when the surgeon also decides on reintervention. |
|
||||
| 16 | **Monthly discrete time over 60 months. Pooled logistic hazard models.** | Standard for CCW and sequential trials, and simple. |
|
||||
| 17 | **Measured:** age, sex, race, diabetes, CKD/dialysis, CHF, COPD, prior CVD, coded smoking, physician ID, practice setting (OBL/ASC vs hospital), calendar year, and a time-varying **proxy for symptom progression** (repeat vascular visits / noninvasive arterial testing). **Latent:** true symptom severity, ABI, true smoking, patient preference, surgeon skill, surgeon style. **Proxy strength is the dial for factor 5.** | What is "measured" means what is in Medicare claims. |
|
||||
| 18 | **Two simulated RCTs:** a *realistic* one (n ≈ 240, hospital-based, top-third skill, low-preference surgeons) and an *ideal pragmatic* one (all physicians, large n). Crossover is **generated by the DGP** in both. Each is analysed by ITT and naive per-protocol. | Separates bias from surgeon selection and small n from bias due to crossover. |
|
||||
| 19 | **Calibrate usual care to Medicare figures:** early PVI ≈ 3.2%; the physician distribution from Hicks 2020 (median 0%, range 0–58%, about 5.6% outliers); Sorber's crude late PVI and CLTI rates; age-appropriate mortality. **Truth scenarios:** null; harm (e.g. RD +5 points at 5 years); early benefit then late harm. | The null scenario shows pure bias. Calibration keeps the simulated world credible to surgeons. |
|
||||
|
||||
## Round 4: DGP core (2026-10-27)
|
||||
|
||||
| # | Decision | Why |
|
||||
|---|---|---|
|
||||
| 20 | **Mechanistic latent-severity engine.** Each patient has a latent severity that drifts with a patient-specific rate. Severity drives the CLTI hazard and the PVI decision. PVI lowers severity right away but adds a restenosis/progression hazard that depends on skill. | Benefit followed by harm, crossover in the conservative arm, and confounding by indication all come out of one mechanism. Truths are computed by simulating the interventions, so the extra complexity costs nothing there. |
|
||||
| 21 | **Estimand: PVI done by the patient's own physician, averaged over the real physician mix.** The gap between the realistic RCT (top-third skill) and the population effect is reported as a finding (external validity), not as bias. | Skill modifies the effect of PVI. |
|
||||
| 22 | **Skill and style are separate latent traits. Base correlation −0.3; comparison level 0.** | Sorber's outliers had more late PVI and CLTI. |
|
||||
| 23 | **Factor 3 = high users have a lower threshold for repeat revascularization AND code CLTI more intensively.** The two are switched on together. | These are paths outside the treatment decision. Style acting through deferred PVI after month 6 is part of strategy B, not bias. |
|
||||
| 24 | **Main analysis: one factor at a time from a clean base world, plus one "realistic, all threats on" world, in each of the 3 truth scenarios. Supplement: full factorial**, only if the one-at-a-time results suggest interactions. | Readability: one panel per threat. |
|
||||
| 25 | **Each replicate has 50,000 patients and about 1,500 physicians, with lognormal panel sizes (median about 20). Supplement: 200,000 patients.** Number of replicates is set later, with ADEMP. | About 1,600 early-PVI patients per replicate, enough for CCW and FE positivity. |
|
||||
|
||||
## Round 5: analysis structure (2026-10-27)
|
||||
|
||||
| # | Decision | Why |
|
||||
|---|---|---|
|
||||
| 26 | **All observational methods use the same adjustment.** IP weights come from pooled logistic models on the measured covariates. Weighted cause-specific hazards for MALE and death are then combined into a standardized 5-year cumulative incidence. Landmark uses IPW at month 6. Sequential trials use IPW at each trial baseline. CCW uses artificial-censoring weights. The RCTs are analysed unadjusted (ITT) and as-treated (naive per-protocol). No matching and no doubly robust variants. | Differences between methods then reflect design and estimand only. Paper 1 already covered the choice of adjustment method. |
|
||||
| 27 | **Performance measures:** estimation bias (against the method's own estimand), question gap, total error, empirical SE, model SE, coverage of the method's own estimand, RMSE, and MCSE for each. **The headline measure for clinicians is the false-alarm rate under the null** (CI excludes 0). | Separates two problems: whether the method fails, and whether it answers a different question. |
|
||||
| 28 | **Variance:** cluster-robust sandwich SE plus the delta method in every replicate. A physician-cluster bootstrap (200 resamples) is run on about 100 replicates in the base and realistic worlds. | A full bootstrap at scale costs too much. Same approach as Paper 1. |
|
||||
| 29 | **Replicates:** 100 at draft, 500 at final. MCSE is reported. | MCSE for coverage is about 1% at 500 replicates. |
|
||||
| 30 | **Figure 1, the "estimand ladder":** one row per method. Each row shows a hollow dot for the method's own truth, a filled dot with interval for its estimate, and a line for the clinically relevant truth. Panels are worlds; colour is the physician variant. Shown for the null and harm scenarios. **Figure 2:** false-alarm rate by threat and physician variant. | One picture that shows both the question gap and the estimation bias. |
|
||||
| 31 | **Next:** the analysis plan (md plus docx), then the table of claudication RCTs, then the build. The docs are committed on the branch `analysis-plan`. | |
|
||||
|
||||
## Round 6: negative controls (2026-10-27)
|
||||
|
||||
| # | Decision | Why |
|
||||
|---|---|---|
|
||||
| 32 | **Three negative controls.** **NCO-1:** hip fracture / pneumonia admission. Shares the latent patient frailty, severity and smoking, so it targets factor 5. **NCO-2:** varicose vein ablation by the same physician. Shares physician procedural style, so it targets factors 2–3. **NCE:** early low-value procedure (e.g. vein ablation) within 6 months, which shares both. All three are generated only from latent traits and never depend on PVI, so their true effect is exactly 0. They go through the same method × physician-variant pipeline as MALE. | Negative controls are the practical bias check available in real claims. CCW and landmark condition on survival, so the controls also need testing under each design. |
|
||||
| 33 | **Main text (Figure 3 / Table 3):** how accurately the negative controls detect bias. A control is "flagged" if its CI excludes 0. Bias in the MALE risk difference counts if |bias| > 1 percentage point. Report sensitivity and specificity, plus how much bias can sit behind a clean negative control. **Supplement:** each control's own false-positive rate. | Answers a new, practical question: what does a clean negative control actually tell a surgeon? |
|
||||
|
||||
## Round 7: issues raised by drafting the plan (2026-10-27)
|
||||
|
||||
| # | Decision | Why |
|
||||
|---|---|---|
|
||||
| 34 | **Null scenario = sharp disease null** (Δ = κ = 0). The headline false-alarm rate is reported for the disease composite (CLTI or major amputation), whose true effect is exactly 0. MALE in the null world is reported as "harm" that comes from the outcome definition alone. | Repeat revascularization can only happen after an index PVI, so MALE cannot be exactly null. Showing that is a teaching point. |
|
||||
| 35 | **Strength of each threat:** use an empirical anchor where one exists (style spread from Hicks 2020; deferred PVI from Sorber late-PVI rates; the early-PVI rate). Where none exists (skill, coding), set the threat so that on its own it moves the naive or landmark estimate by about 2–4 pp in the harm scenario. | Credible to surgeons while keeping each threat's effect visible. |
|
||||
| 36 | **The NCE analysis is adjusted for early-PVI status.** The unadjusted version goes in the supplement. | Otherwise NCE ← style → PVI → MALE flags a real effect of PVI. |
|
||||
| 37 | **Patient–physician sorting is not in the main grid.** It becomes one sensitivity world if time allows. | Scope. |
|
||||
| — | **Fact (not a decision):** interim calibration target for all-cause mortality is about 35% at 5 years (Sigvant 2018 *EJVES*, PMID 29478910: 10-year mortality in IC 63%). To be replaced by the 5-year mortality in the authors' own Medicare cohort. | No published 5-year all-cause figure for Medicare claudicants was found. |
|
||||
|
||||
## Round 8: scope (2026-10-27)
|
||||
|
||||
| # | Decision | Why |
|
||||
|---|---|---|
|
||||
| 38 | **No data-quality aim.** The paper keeps the three aims of the plan. Claims data problems such as time-zero misdating, physician attribution, laterality and Medicare Advantage switching are not modelled. | Focus. Papers 1 and 2 of `tte-papers` have stopped. |
|
||||
| 39 | **Paper 1's comparison of adjustment methods and covariate vibration stay out** (see D26). | This paper is about which question each design answers. |
|
||||
|
||||
## Build notes: simulator and calibration (2026-10-27)
|
||||
|
||||
| # | Decision | Why |
|
||||
|---|---|---|
|
||||
| 40 | **Factor 5 (unmeasured confounding by indication): "off" means the analyst sees true severity Z_t, true frailty and true smoking. "On" means the analyst sees only claims proxies:** a 3-month visit/testing count for severity, a frailty index with reliability 0.4, and coded smoking (sensitivity 0.6). | In the clean worlds every confounder must be measured, so each method can be checked as unbiased for its own estimand. |
|
||||
| 41 | **Each world's intercepts are re-solved by calibration (early PVI, deferred PVI, repeat revascularization, CLTI, death) so that every world matches the same Medicare targets.** A world differs from W0 only in the threat it switches on. | A fair comparison between worlds. |
|
||||
| 42 | **Deferral:** in W0–W3, W5 and WR, deferred PVI matches Sorber (3.6% late PVI among patients without early PVI). **W4 is calibrated to 15%**, which is trial-like crossover, and in W4 deferral is driven by severity. In WR, deferral is at the Sorber rate but driven by severity. PVI after CLTI was lowered to about 1% per month, so most late PVI is elective. | At the Medicare rate, the deferral threat would be almost invisible. Trials (SUPER 29% in year 1) show much more crossover. |
|
||||
| 43 | **Physician style coefficient 1.6** (W2, W3, WR). Among physicians with ≥11 patients this gives a median early-PVI rate of 0.8%, a maximum of 75%, and 4.7% of physicians ≥2 SD outliers who perform 38% of early PVIs. Hicks 2020 reported 0%, 58% and 5.6%. | Close enough to Hicks 2020. Not tuned further. |
|
||||
| 44 | **Within a month the order is:** severity update → outcome events → death → elective PVI decision. A MALE event and death in the same month count the MALE event first. | Matches discrete-time pooled logistic analysis. |
|
||||
|
||||
## Build notes: truths, estimators, validation (2026-10-27)
|
||||
|
||||
| # | Decision | Why |
|
||||
|---|---|---|
|
||||
| 45 | **Scenario effects, tuned in W0 against the disease composite:** harm (relief 0.3, kappa 0.039) gives RD(A−B1) **+5.1 pp**. Benefit-then-harm (relief 2.0, kappa 0.075) gives **−1.2 pp at 24 months and +3.1 pp at 60 months**. The same mechanism is used in every world. Where skill varies (W1, WR) the true effects are larger (+7 pp), because exp(−skill) is convex: on average a PVI is done worse than one at average skill. | D34 sets the headline on the disease composite. MALE carries an extra +7 to +15 pp that comes purely from the outcome definition. |
|
||||
| 46 | **Truths for the negative controls are taken from the truths table, not assumed to be 0.** In the non-null scenarios PVI changes death, so a negative-control outcome's 5-year risk moves slightly through the competing risk (e.g. NCO-1 −0.08 pp). In the null scenario the effect is exactly 0. | Honest scoring. |
|
||||
| 47 | **The own estimand of landmark and naive is estimated within each replicate by an oracle run:** the same estimator with exact severity, true frailty and smoking, and the latent skill and style. Estimation bias is the paired difference (estimate − oracle). | A large-n limit was too noisy: the landmark MCSE at n = 10⁵ was 2.8 pp. |
|
||||
| 48 | **The PVI model is fitted separately inside the window (month indicators) and in the deferred period (spline in month).** | A pooled model misspecified the window probabilities that the CCW and sequential weights depend on. |
|
||||
| 49 | **V2 is the V1 fit with a physician-cluster-robust SE.** Under V4, physicians without an estimable fixed effect in a phase get probability 0, and the V4 population is restricted to physicians with variation in early PVI. | Same point estimate by construction (D15). |
|
||||
| 50 | **Validation (W0, harm, 40 replicates of 50,000):** CCW A−B1 is +5.12 vs a truth of +5.11 on the disease composite (z = 0.0); sequential ITT is +5.83 vs +5.71 (z = 0.6); MALE and NCO-1 have |z| ≤ 0.7. The model SE is 0–20% below the empirical SE for CCW, so the bootstrap check in D28 still needs doing. | Plan §15 validation (a). |
|
||||
| 51 | **Speed and scale.** The PVI model is fitted once per replicate and variant, and shared by CCW and the sequential trials. CCW clones and weights are built once for all outcomes. The deferred-period PVI model uses every PVI plus a fixed 10% sample of the other person-months, with offset log(10). On W0 validation the estimates are identical to 4 decimals. The main designs take about 39 s per replicate instead of about 130 s. **Scale:** n = 50,000 stays. Draft is 100 replicates; final is 200 replicates (replacing 500 in D29), and the final run extends the draft's replicates 1–100. Total about 12 hours on 11 cores. | User asked for a faster, smaller run. The sample size is not reduced, because CCW's strategy A already depends on about 250 patients treated in month 5 at n = 50,000. With 200 replicates, MCSE of bias ≈ empirical SE / 14 (CCW about 0.2 pp) and MCSE of coverage ≈ 1.5%. |
|
||||
|
||||
## Draft run (2026-10-28)
|
||||
|
||||
| # | Decision | Why |
|
||||
|---|---|---|
|
||||
| 52 | **Draft run done:** 8 worlds × 3 scenarios × 100 replicates of 50,000 patients; 9.4 h on 11 cores; 0 errors. 16 of 288,000 estimates are NA (sequential trials, V4, 4 replicates) and are dropped. Summary: `simulations/04_summarise.R` writes `results/summary/`. Report: `reports/draft-report.Rmd`. Two open questions: (Q1) the landmark propensity model adjusts for month-5 severity, a post-PVI mediator, which drives the landmark question gap (W1 own truth about 20 pp vs a reference of 7.2 pp); with baseline covariates only, the 20-replicate oracle mean is 8.1. (Q2) NCO-1 has no severity term in the DGP, contrary to D32, so it cannot detect W5/WR bias. The bootstrap check (D28) waits on Q1/Q2. | Surfaced by the draft results. |
|
||||
| 53 | **Answers to D52 (user, 2026-10-28).** (Q1) **The primary landmark propensity score uses baseline covariates only** (diagnosis covariates and month-0 severity). The version that also adjusts for month-5 severity becomes `landmark_m5` (supplement). It conditions on a mediator, because a PVI in months 0–5 has already changed that severity. Both versions get their own oracle. (Q2) **NCO-1 now depends on severity (coefficient 0.3)**, using the patient's **PVI-free severity path** (the same baseline, rate and monthly noise, with no relief and no kappa). That way NCO-1 shares the confounder but still cannot be affected by PVI (only through death as a competing risk, D46). Intercept −6.75 keeps 5-year NCO-1 incidence at about 9.3%. This is the best case for a patient-level negative control, and the paper says so. **Bootstrap (D28, reduced):** W0 and WR × null and harm, replicates 1–20 of the main run, 50 physician-cluster resamples each, V1 point estimates, about 8 s per resample. Compare the bootstrap SD with the model SE (patient and cluster) and the empirical SE. **The rerun goes to `results/main/n50k_v2`;** the first draft stays in `n50k`. Truths are recomputed (NCO-1 changed). | Landmark analyses in practice adjust for baseline covariates. D32 intended NCO-1 to share severity. |
|
||||
|
||||
## Paper framing (2026-10-28)
|
||||
|
||||
| # | Decision | Why |
|
||||
|---|---|---|
|
||||
| 54 | **The paper's question: "Can target trial emulation overcome the problems that make randomized trials hard in surgery?"** Early PVI for claudication is the worked example. Medicare only supplies realistic simulation inputs, and the paper makes no claims about Medicare findings or the authors' earlier papers. The structure is a scorecard: each surgical-RCT barrier (small/stopped early, short follow-up/soft endpoints, crossover, selected surgeons, practice style, unmeasured severity) against whether TTE solves it, checked against the known truth. Aims 1–3 stay. Aim 1 = "TTE is not one method", Aim 2 = the scorecard rows, Aim 3 = whether the one unfixable problem can be detected. The MALE/repeat-revascularization point is secondary, general outcome-definition guidance. The Medicare "dial" idea is dropped. **Novelty (quick PubMed check, 2026-10-28):** there are TTE guides for surgeons (e.g. JBJS Rev 2026) and immortal-time design comparisons (Duchesneau 2022 *CEBP*), but no search hit for a known-truth simulation comparing TTE with a feasible surgical RCT under surgeon-level threats. | User: the aim is to test whether TTE can solve RCT challenges in surgery, without arguing against the authors' earlier results. |
|
||||
| 55 | **Three additions from literature review 3 (user, 2026-10-28), for the overnight run.** (1) **New world WE, "EHR/registry-like":** the same DGP and calibration as W0, but the analyst sees severity with measurement error (monthly Z_t + N(0, 0.75²), correlation with true severity about 0.8), a frailty index with reliability 0.7, and true smoking. It sits between W0 (exact) and W5 (claims proxy) and answers the claims-vs-EHR row of Table 1. (2) **Non-random consent in the realistic trial:** mean 30%, OR 0.7 per 10 years of age and per SD of frailty (Wasmann 2019, King 2005). The trial's ITT truth is weighted by consent probability. (3) **Trial horizons 12, 24 and 60 months:** ITT and as-treated at each. Truths `rct_<design>_itt_h12/_h24`. The reference at the short horizons is RD(A − B1) at that month, from the risk curves. **Learning curve:** not simulated; limitation (user). | Completes Table 1 rows 3, 5 and 17. |
|
||||
| 56 | **Primary outcome changed to limb MALE: the first of coded CLTI, major amputation or open bypass** (user, 2026-10-28). D7's MALE (which also counts any revascularization after the first) moves to the supplement as "MALE including repeat revascularization". Repeat revascularization is reported on its own as a physician-driven outcome. Limb MALE also replaces the disease composite (CLTI or amputation) as the headline (D34); the disease composite stays as a secondary outcome. Negative-control bias (Aim 3) is now measured on limb MALE. Clean-world truths: 0 under the null (exactly); +6.2 under harm; −0.7 at 12 months, −1.7 at 24 and +3.6 at 60 under benefit-then-harm. Acute limb ischemia is not simulated (rare in claudication; limitation). | Standard MALE definitions (Conte 2009 *JVS* 19897335; Nordanstig 2024 *EJVES* 39121905) are built for follow-up **after** a revascularization: reintervention exists only for treated patients. D7 argued that excluding the index PVI in both arms made the outcome symmetric. It does not: under A everyone is at risk of a repeat for about 55 months, under B1 about 3.6% are. In the clean world under the null, repeat revascularization is 18.6% vs 0.7%, giving +14 pp of MALE from the definition alone. |
|
||||
| 57 | **Run-time trims** (user, 2026-10-28). One replicate is about 77 s of compute, half of it in the random-intercept variant (V3). Trims: `landmark_m5` runs under V1/V2 only (supplement); the NCE runs under V1/V2 and V4, not V3; V3 is skipped in W0 and W4, where the physician plays no role by design. V3 stays in W1, W1_rho0, W2, W3, W5, WE and WR. About 20–25% less run time. The run is started manually when the laptop is plugged in (`nohup caffeinate -i -s zsh results/run_v2.sh &`), at nice 10. | Faster, with no loss for the main text. |
|
||||
@@ -0,0 +1,168 @@
|
||||
# Literature review 1: limitations of RCTs in surgery, mapped to TTE on claims data
|
||||
|
||||
All citations were checked against PubMed through NCBI E-utilities, 2026-10-27. The numbers come from PubMed abstracts; full texts were not read. Papers with no PubMed abstract are described only in general terms.
|
||||
|
||||
**Corrections and caveats**
|
||||
- Chapman 2014 was published in *BMJ*, not *Surgery*.
|
||||
- The page numbers for McCulloch 2002 are from memory, not from PubMed.
|
||||
- BASIL-3 recruited 481 of the planned 861 patients. That the paclitaxel signal caused the shortfall is a guess, not a verified finding.
|
||||
|
||||
---
|
||||
|
||||
## 1. Classic critiques and the IDEAL framework
|
||||
|
||||
| Paper | PMID | Takeaway |
|
||||
|---|---|---|
|
||||
| McCulloch P et al. 2002, *BMJ* "Randomised trials in surgery: problems and possible solutions" | 12065273 | The founding critique. Barriers: equipoise, learning curve, quality control, blinding, surgeon preference. |
|
||||
| McLeod RS 1996, *Surgery* 119:483 | 8619200 | Problems specific to RCTs in surgery. |
|
||||
| Solomon MJ, McLeod RS 1995, *Surgery* | 7652679 | Many surgical questions could be randomized but are not. |
|
||||
| Solomon MJ, McLeod RS 1998, *Med J Aust* | 9803252 | Overview of RCTs in surgery: past, present and future. |
|
||||
| Barkun JS et al. 2009, *Lancet* 374:1089 | 19782874 | IDEAL series, paper 1: the stages of surgical innovation. |
|
||||
| Ergina PL et al. 2009, *Lancet* 374:1097 | 19782875 | Few of the challenges are unique to surgery, but in surgery they occur together: complexity, surgeon factors, and the range of outcomes. |
|
||||
| McCulloch P et al. 2009, *Lancet* 374:1105 | 19782876 | IDEAL recommendations: registries and prospective databases, with RCTs "whenever possible". |
|
||||
| McCulloch 2013 / Ergina 2013 / Cook 2013, *BMJ* (IDEAL series) | 23778427, 23778426, 23778425 | Observational studies in the Exploration and Assessment stages; RCTs and long-term registry evaluation. |
|
||||
| Hirst A et al. 2019, *Ann Surg* | 29697448 | IDEAL update: adds a Pre-IDEAL stage and covers devices. |
|
||||
| Cook JA 2009, *Trials* 10:9 | 19200379 | Checklist for the design, conduct and analysis of surgical RCTs. |
|
||||
| Pronk AJM et al. 2023, *Br J Surg* 110:1300 | 37379487 | Of 1,188 surgical RCTs, only **22.1%** published in 2019 were at low risk of bias (14.7% in 2009). |
|
||||
| Ahmed Ali U et al. 2018, *J Surg Res* 228:1 | 29907196 | Of 228 negative surgical RCTs, the median sample size was about 76–80, so most were underpowered. |
|
||||
|
||||
## 2. Recruitment failure, discontinuation, and equipoise
|
||||
|
||||
| Paper | PMID | Takeaway |
|
||||
|---|---|---|
|
||||
| **Chapman SJ et al. 2014, *BMJ* 349:g6870** | 25491195 | Of 395 surgical RCTs, **21% stopped early**. The main reason was poor recruitment (44% of stopped trials). Of completed trials, 34% were unpublished at a median of 4.9 years. |
|
||||
| **Rosenthal R et al. 2015, *Ann Surg* 262:68** | 24979608 | Discontinued: **43%** of surgical vs **27%** of medical RCTs. Stopped for slow recruitment: 18% vs 11%. Discontinuation predicted non-publication (OR 4.18). |
|
||||
| Kasenda B et al. 2014, *JAMA* 311:1045 | 24618966 | Across all fields, **24.9%** of RCTs were discontinued, most often for poor recruitment. |
|
||||
| Cook JA et al. 2008, *Contemp Clin Trials* 29:631 | 18397843 | **Trials of surgery vs medical treatment recruited about 2× slower** than trials of surgery vs surgery (median 68 vs 35 days per recruit). This applies directly to PVI vs exercise. |
|
||||
| Donovan J et al. 2002, *BMJ* 325:766 (ProtecT) | 12364308 | Qualitative work on how recruiters presented the options made an "impossible" trial feasible. |
|
||||
| Donovan JL et al. 2014, *J Clin Epidemiol* 67:912 | 24811157 | Clinicians' "hunches" undermined equipoise, and **surgeons felt this most intensely**. |
|
||||
| Rooshenas L et al. 2016, *PLoS Med* 13:e1002147 | 27755555 | In recorded consultations, the equipoise clinicians stated differed from how they actually presented the options. |
|
||||
| Donovan JL et al. 2022, *Trials* 23:258 (QRI-Two) | 35379301 | A rapid recruitment intervention for trials that are failing to recruit. |
|
||||
| Hamdy FC et al. 2023, *NEJM* 388:1547 (ProtecT at 15 years) | 36912538 | A monitor-vs-intervene trial can be completed. |
|
||||
|
||||
## 3. Surgeon skill, learning curve, expertise-based trials, and clustering
|
||||
|
||||
| Paper | PMID | Takeaway |
|
||||
|---|---|---|
|
||||
| **Devereaux PJ et al. 2005, *BMJ* 330:88** | 15637373 | Proposes expertise-based RCTs: each surgeon performs only the procedure they are expert in, to avoid differential-expertise bias. |
|
||||
| Cook JA et al. 2015, *Trials* 16:241 | 26025450 | Review of 43 expertise-based RCTs: 92% compliance, but expertise criteria were poorly reported. |
|
||||
| Cook JA et al. 2004, *Clin Trials* 1:421 | 16279280 | Statistical methods for learning curves in surgical trials. |
|
||||
| Ramsay CR et al. 2000 / 2001 | 11155830, 11319991 | Assessing learning curves in health technology. |
|
||||
| **Lee KJ, Thompson SG 2005, *BMJ* 330:142**; *Clin Trials* 2:163 | 15649931, 16279138 | Outcomes cluster by clinician even under individual randomization. Ignoring it makes the estimates look more precise than they are. |
|
||||
| Roberts C, Roberts SA 2005, *Clin Trials* 2:152 | 16279137 | Clustering induced by treatment, which can differ between arms (partially nested designs). |
|
||||
| Walters SJ 2010, *J Clin Nurs* 19:1102 | 20492055 | Therapist effects: cluster-level, random-effects, and marginal models compared. |
|
||||
| **Birkmeyer JD et al. 2013, *NEJM* 369:1434** | 24106936 | Peer ratings of surgical skill from video: the bottom quartile vs the top had **14.5% vs 5.2%** complications and **0.26% vs 0.05%** mortality. |
|
||||
| Birkmeyer JD et al. 2003, *NEJM* 349:2117 | 14645640 | Surgeon volume and mortality. |
|
||||
|
||||
## 4. Blinding and sham surgery
|
||||
|
||||
| Paper | PMID | Takeaway |
|
||||
|---|---|---|
|
||||
| **Wartolowska K et al. 2014, *BMJ* 348:g3253** | 24850821 | Across 53 placebo-controlled surgical RCTs, the placebo arm improved in 74%. Surgery was **no better than placebo in 51%**. |
|
||||
| Wartolowska K et al. 2016, *BMJ Open* | 27008687 | Placebo-controlled surgical trials are feasible but recruit slowly. |
|
||||
| Boutron I et al. 2017, *Ann Intern Med* 167:40 | 28630973 | CONSORT extension for non-pharmacologic treatments: expertise, blinding, and standardization. |
|
||||
|
||||
## 5. Standardization, crossover, and contamination
|
||||
|
||||
| Paper | PMID | Takeaway |
|
||||
|---|---|---|
|
||||
| **Garas G et al. 2018, *Ann Surg Oncol* 25:221** | 29110271 | Across 40 RCTs of minimally invasive vs open surgery, crossover averaged **12.6% (0–45%)** and worsened the ITT results. |
|
||||
| **Weinstein JN et al. 2006, *JAMA* 296:2441 (SPORT)** | 17119140 | **Only 50%** of the surgery arm had surgery, and **30%** of the nonoperative arm did, so the ITT comparison could not be interpreted. |
|
||||
| Hernán MA, Robins JM 2017, *NEJM* 377:1391 | 28976864 | Per-protocol effects in pragmatic trials. |
|
||||
|
||||
## 6. Generalizability and external validity
|
||||
|
||||
| Paper | PMID | Takeaway |
|
||||
|---|---|---|
|
||||
| **Rothwell PM 2005, *Lancet* 365:82** | 15639683 | "To whom do the results of this trial apply?" |
|
||||
| Rothwell PM 2006, *PLoS Clin Trials* | 16871331 | Factors that affect external validity. |
|
||||
| Cutteridge J et al. 2025, *BJS Open* | 40105904 | Patients in 307 vascular RCTs were **younger** than those in the UK National Vascular Registry. |
|
||||
| Jones HG et al. 2026, *EJVES* | 42526742 | Endovascular lower-limb revascularization is under-researched relative to how often it is done. Its trials are 72% efficacy trials and 39% industry-funded. |
|
||||
| Loudon K et al. 2015, *BMJ* (PRECIS-2) | 25956159 | Tool for rating how pragmatic or explanatory a trial is. |
|
||||
|
||||
## 7. Trials of timing, "when to operate" (the key analogue)
|
||||
|
||||
| Paper | PMID | Takeaway |
|
||||
|---|---|---|
|
||||
| **Fitzgibbons RJ et al. 2006, *JAMA* 295:285** | 16418463 | Hernia watchful waiting: 23% of the waiting arm crossed over to surgery, and 17% of the surgery arm crossed the other way. |
|
||||
| **Fitzgibbons RJ et al. 2013, *Ann Surg* 258:508** | 24022443 | Long-term crossover was **68%** (79% in men over 65). "Watchful waiting" works out as "delayed surgery". |
|
||||
| O'Dwyer PJ et al. 2006, *Ann Surg* | 16858177 | UK hernia trial: crossover was higher than predicted. |
|
||||
| **UK Small Aneurysm Trial 1998, *Lancet*** | 9853436 | HR 0.94. Operative mortality of 5.8% caused an early deficit in survival. |
|
||||
| **Lederle FA et al. 2002, *NEJM* (ADAM)** | 12000813 | **61.6% of the surveillance arm were eventually repaired.** Surveillance is a dynamic strategy, not "no treatment". |
|
||||
| Cao P et al. 2011, *EJVES* (CAESAR) | 20869890 | Delayed repair under surveillance: 59.7% by 36 months and 84.5% by 54 months. One in six aneurysms became unsuitable for EVAR, a harm of delay that the ITT effect on mortality does not capture. |
|
||||
| Ouriel K et al. 2010, *JVS* (PIVOTAL) | 20304589 | 31% of the surveillance arm were repaired by 20 months. HR 1.01. |
|
||||
| Ulug P et al. 2020, *Cochrane* | 32609382 | Early repair of small AAAs has no advantage. |
|
||||
|
||||
## 8. Pragmatic trials, registry trials, and TTE
|
||||
|
||||
| Paper | PMID | Takeaway |
|
||||
|---|---|---|
|
||||
| **Hernán MA, Robins JM 2016, *AJE* 183:758** | 26994063 | The foundational TTE paper. |
|
||||
| **Hernán MA et al. 2016, *J Clin Epidemiol* 79:70** | 27237061 | Specifying the target trial avoids self-inflicted biases such as immortal time. |
|
||||
| Hernán MA et al. 2022, *JAMA* 328:2446 | 36508210 | JAMA Guide to TTE. |
|
||||
| Cain LE et al. 2010, *Int J Biostat* | 21972433 | "When to start treatment?": dynamic regimes. |
|
||||
| **Maringe C et al. 2020, *Int J Epidemiol* 49:1719** | 32386426 | CCW tutorial on surgery in older patients with lung cancer. The naive 22% benefit fell to 11%. |
|
||||
| Wang SV et al. 2023, *JAMA* 329:1376 (RCT-DUPLICATE) | 37097356 | Of 32 RCTs emulated in claims data, agreement was r = 0.93 when the emulation was close and r = 0.53 when it was not. All were drug trials. |
|
||||
| Cashin AG et al. 2025, *JAMA* / *BMJ* (TARGET) | 40899949 / 40903028 | Reporting guideline for TTE. |
|
||||
| Scola G et al. 2023, *BMC Med Res Methodol* | 37587484 | Only 16% of TTEs used cloning to handle immortal time. |
|
||||
| Katsoulis M et al. 2024, *Epidemiology* 35:730 | 39024012 | TTE of bariatric surgery. |
|
||||
| Hoskins W et al. 2026, *JBJS Rev* | 42896011 | Orthopaedic primer. TTE **does not** address unmeasured confounding. |
|
||||
| Lauer MS, D'Agostino RB 2013, *NEJM* | 23991657 | The randomized registry trial. |
|
||||
| Zwarenstein M et al. 2008, *BMJ* | 19001484 | CONSORT extension for pragmatic trials. |
|
||||
| Brookhart MA et al. 2006, *Epidemiology* 17:268 | 16617275 | Physician preference as an instrumental variable. |
|
||||
|
||||
**Gap:** we found no surgery-specific systematic review of TTE methods (JAMA Surg / Ann Surg, 2022–2026). Applied examples: Sakowitz 2025, *Ann Surg* (TCAR vs CEA), PMID 40607703; Talathi 2026, *Surgery* (STRASS 2 emulated), PMID 41936769.
|
||||
|
||||
## 9. Vascular surgery
|
||||
|
||||
| Paper | PMID | Takeaway |
|
||||
|---|---|---|
|
||||
| Farber A et al. 2022, *NEJM* (BEST-CLI) | 36342173 | Cohort 1: MALE or death 42.6% with surgery vs 57.4% with endovascular (HR 0.68). Whether a vein conduit was available decided which cohort a patient entered. |
|
||||
| Bradbury AW et al. 2023, *Lancet* (BASIL-2) | 37116524 | n = 345. Vein bypass first was worse (aHR 1.35), which conflicts with BEST-CLI. |
|
||||
| Hunt 2017 *Trials*; Bradbury 2025 *BMJ* (BASIL-3) | 28526046, 39993822 | **481 of 861 planned** randomized. No benefit from drug-coated balloons or drug-eluting stents. |
|
||||
| **Katsanos K et al. 2018, *JAHA*** | 30561254 | Paclitaxel: 5-year mortality 14.7% vs 8.1% (RR 1.93). Only 3 RCTs had 5-year data. |
|
||||
| Rocha-Singh KJ et al. 2020, *Circulation* | 32370548 | Individual patient data: **23–24% lost to follow-up or withdrew**. After recovering vital status, HR 1.38 → 1.27. |
|
||||
| Secemsky EA et al. 2019, *JAMA Cardiol* | 30747949 | Medicare: no excess mortality. |
|
||||
| Freisinger E et al. 2020, *Eur Heart J* | 31593987 | German claims data: no excess mortality. |
|
||||
| Nordanstig J et al. 2020, *NEJM* (SWEDEPAD, interim) | 33296560 | Registry RCT: HR 1.06. |
|
||||
| **Nordanstig J et al. 2025, *Lancet* (SWEDEPAD 2, claudication)** | 40902614 | No benefit in quality of life. 5-year mortality HR 1.47 (1.09–1.98). |
|
||||
| Falkenberg M et al. 2025, *Lancet* (SWEDEPAD 1, CLTI) | 40902617 | The CLTI companion trial. |
|
||||
| Fakhry F et al. 2015, *JAMA* (ERASE) | 26547465 | Revascularization plus exercise beat exercise alone at 12 months. |
|
||||
| Murphy TP et al. 2012, *Circulation* (CLEVER) | 22090168 | Exercise was better for walking time and stenting for quality of life. n = 111. |
|
||||
| Nordanstig 2014, *Circulation*; Djerf 2020, *Circ CV Interv* (IRONIC) | 25095886, 31937137 | The quality-of-life benefit seen at 1 year was **lost by 5 years**, at about twice the cost. |
|
||||
| **Sorber R et al. 2023, *JVS*** | 37276171 | Medicare. Early PVI 3.2%. Late PVI 22.5% vs 3.6% (aHR 6.89). High-use physicians. |
|
||||
| Dun C et al. 2025, *Ann Vasc Surg* 110:424 | 39419322 | Market competition predicts early PVI. |
|
||||
| Weaver ML et al. 2023, *JVS* | 37141948 | Market competition and claudication management in the VQI. |
|
||||
| Madabhushi V et al. 2021, *JVS* | 33775749 | Revascularizing claudicants: CLTI HR 2.9, amputation HR 4.5. |
|
||||
| Conte MS et al. 2025, *JVS* (SVS focused update) | 40316185 | Major evidence gaps for claudication. |
|
||||
|
||||
---
|
||||
|
||||
## Taxonomy: RCT limitations in surgery → what TTE on claims can do
|
||||
|
||||
IV = internal validity, EV = external validity, F = feasibility.
|
||||
|
||||
| # | Limitation | Threatens | Can TTE on claims fix it? | Can it go in the simulation DGP? |
|
||||
|---|---|---|---|---|
|
||||
| 1 | Recruitment failure and discontinuation | F (and EV) | **Yes.** No consent is needed. The price is confounding instead of randomization. | Indirectly: compare a "would-consent" subset with the whole population. |
|
||||
| 2 | Patient and surgeon equipoise and preference | F, EV, IV | **Partly.** Preference is what makes treatment vary in practice, and can serve as an IV (high-use physicians). Patient preference is unmeasured. | **Yes.** A latent physician "intervention propensity". |
|
||||
| 3 | Surgeon skill and learning curve | IV, EV | **Partly.** Skill is not in claims, but the operator NPI is, so random or fixed effects are possible. Skill and preference cannot be separated if they are correlated. | **Yes.** Latent skill, correlated with preference. |
|
||||
| 4 | Clustering and provider heterogeneity | IV (precision), EV | **Mostly.** Mixed models, cluster-robust SEs. | **Yes.** Random intercepts and random slopes by physician. |
|
||||
| 5 | Blinding and sham | IV | **No.** No patient-reported outcomes in claims, and outcome ascertainment is not blinded. | Optional: differential ascertainment of CLTI. |
|
||||
| 6 | Standardization and co-interventions | IV, EV | **Partly.** Procedure type and Part D fills are visible. Exercise, smoking, and adherence are not. | **Yes.** Unmeasured exercise uptake. |
|
||||
| 7 | Crossover and deferral | IV | **Yes, TTE's strongest case**, using dynamic strategies with CCW or sequential trials. It needs the triggers for crossover to be measured. | **Yes, central.** |
|
||||
| 8 | Immortal time and misaligned time zero | IV | **Yes, by design.** | **Yes.** The naive design as a comparator. |
|
||||
| 9 | Selected surgeons, centers, and patients | EV | **Yes, largely.** Population-based data, plus transportability. | **Yes.** A "trial of high-skill operators" vs the whole population. |
|
||||
| 10 | Long horizon, rare outcomes, changing technology | F, IV | **Partly.** Follow-up is long and complete, but device codes are coarse and calendar time confounds. | **Yes.** Time-varying effect (early benefit, late harm). |
|
||||
|
||||
**In one line:** TTE fixes feasibility, generalizability, and (with dynamic strategies) crossover. It cannot fix unmeasured confounding by indication or the lack of patient-reported outcomes. Surgeon skill sits in between: it cannot be measured, but its structure can be modelled through the NPI.
|
||||
|
||||
## Read first
|
||||
1. Sorber 2023 *JVS* (37276171)
|
||||
2. ADAM 2002 *NEJM* (12000813) and CAESAR 2011 (20869890)
|
||||
3. Fitzgibbons 2006 *JAMA* and 2013 *Ann Surg* (16418463, 24022443)
|
||||
4. Hernán & Robins 2016 *AJE*; Hernán 2016 *J Clin Epidemiol* (26994063, 27237061)
|
||||
5. Maringe 2020 *Int J Epidemiol* (32386426)
|
||||
6. McCulloch 2002 *BMJ*; Ergina 2009 *Lancet* (12065273, 19782875)
|
||||
7. Chapman 2014 *BMJ*; Rosenthal 2015 *Ann Surg* (25491195, 24979608)
|
||||
8. Katsanos 2018; Nordanstig 2020 and 2025 (30561254, 33296560, 40902614)
|
||||
@@ -0,0 +1,208 @@
|
||||
# Literature review 2: claudication evidence, and what landmark, CCW and sequential trials each estimate
|
||||
|
||||
PMIDs were checked through NCBI E-utilities on 2026-10-27. The numbers come from PubMed abstracts. The guideline recommendation wording is from memory and needs checking against the full texts.
|
||||
|
||||
---
|
||||
|
||||
## Part A. Clinical evidence
|
||||
|
||||
### A1. Randomized trials. Every one used walking distance or QoL as the endpoint; none used hard limb outcomes.
|
||||
|
||||
| Trial | N and comparison | Result | Limitations |
|
||||
|---|---|---|---|
|
||||
| **CLEVER**: Murphy 2012 *Circulation* (22090168); 18-month results 2015 *JACC* (25766947); design (18440181); update (19660897) | 111 aortoiliac; OMC vs OMC+SE vs OMC+stent | Peak walking time at 6 months: SE better than stent (p=0.04). At 18 months: no difference (p=0.16). QoL better with stent. | **Under-recruited**: planned 252 in 4 arms, cut to 217 in 3 arms, finished with 111. Only 79 had 18-month data. |
|
||||
| **ERASE**: Fakhry 2015 *JAMA* (26547465); long-term Klaphake 2022 *Ann Surg* (33378308) | 212; endovascular + SET vs SET | MWD at 12 months +282 m with the combination | At 5.4 years the walking gap had closed (p=0.62). The SET arm had more later revascularizations (HR 2.50) but **fewer procedures in total (65 vs 149)**. |
|
||||
| **IRONIC**: Nordanstig 2014 *Circulation* (25095886); 5-year Djerf 2020 *Circ CV Interv* (31937137) | 158; revascularization + BMT + exercise vs BMT + exercise | SF-36 better with revascularization at 1 year | **No difference at 5 years.** Cost about 2× higher. |
|
||||
| **MIMIC**: Greenhalgh 2008 *EJVES* (19022184) | 93 femoropopliteal + 34 aortoiliac; PTA on top of SE + BMT | Walking distance +38% / +78% | Very small. No QoL benefit. |
|
||||
| **SUPER**: Koelemay 2022 *EJVES* (35151572) | 240 iliac; SET vs endovascular | MWD the same at 1 year | **Stopped early. 29% of the SET arm crossed over to revascularization within 1 year.** Only 50% complied with SET. |
|
||||
| **Mazari**: 2012 *BJS* (22021102); 2017 *BJS* (27763685) | 178; PTA vs SEP vs both | The same at 12 months and 5.2 years | |
|
||||
| **Spronk**: 2009 *Radiology* (19188327) | 151; endovascular vs SET | An early advantage for endovascular was gone by 6–12 months | |
|
||||
| **OBACT**: Nylaende 2007 *EJVES* (17055756) | 56; OMT vs OMT + PTA | PTA better at 2 years | **434 screened to randomize 56.** |
|
||||
| **Cochrane**: Fakhry 2018 (29518253) | 10 RCTs, 1,087 patients | Endovascular vs SET: no clear difference | Small, short trials, and **no hard limb outcomes**. |
|
||||
|
||||
**Across these trials:**
|
||||
- Revascularization improves walking and QoL early, but the advantage fades by 2–5 years.
|
||||
- Crossover and non-adherence to SET are large.
|
||||
- No trial was powered for CLTI, amputation or cumulative reintervention.
|
||||
|
||||
### A2. Observational evidence that intervention is associated with progression
|
||||
|
||||
- **Madabhushi 2021 *JVS* (33775749).** n = 1,051. Revascularization was associated with CLTI (HR 2.9) and amputation (HR 4.5). The exposure is "ever revascularized", so the result is exposed to immortal time bias.
|
||||
- **Golledge 2018 *BJS* (29566427).** n = 456. Treatment depended on *which specialist the patient saw*, which is a natural physician-preference contrast. 5-year amputation was 6.2% with early revascularization vs 0.7% with conservative care.
|
||||
- **Sorber 2023 *JVS* (37276171).** aHR 6.89 for late PVI. Patients of outlier physicians had more late PVI and more CLTI.
|
||||
- **Dun 2025 *Surgery* (40782724).** Markov decision model.
|
||||
- **Alabi 2026 *JAMA Surg* (41405976).** VQI–Medicare, 10,012 patients. CLTI within 180 days of revascularization ranged up to 11.8% across subgroups.
|
||||
- **Bose 2025 *Ann Surg* (38841837) and *EJVES* (38906366).** Infrapopliteal PVI for claudication had poor outcomes.
|
||||
- **Natural history:**
|
||||
- Sigvant 2016 *EJVES* (26777541): about 21% progress to CLI.
|
||||
- Baubeta Fridh 2017 *EJVES* (28797662): amputation after revascularization for IC is 0.4% per year.
|
||||
- **Review:** Golledge 2022 *Nat Rev Cardiol* (34997200).
|
||||
|
||||
### A3. Variation between physicians
|
||||
|
||||
- **Hicks 2020 *JVS* (31208940), "Overuse of early PVI for claudication."** n = 194,974 patients, 3.2% with early PVI. The median physician's rate was 0%, with a range of 0–58%. **320 physicians (5.6%) were ≥2 SD outliers.** Office-based lab (OBL) or ambulatory surgery center (ASC) practice predicted outlier status. **This paper is the main empirical source for the physician component of the DGP.**
|
||||
- Dun 2024 *JAHA* (38958132): the OBL/ASC share of PVI went from 12.4% to 55.7% between 2011 and 2022.
|
||||
- Dun 2025 *Ann Vasc Surg* (39419322): market competition.
|
||||
- Weaver 2023 *JVS* (37141948).
|
||||
- Tsou 2024 *JVS* (38908807).
|
||||
- Smith 2019 *Ann Vasc Surg* (30684609).
|
||||
- Brown 2021 *JVS* (33617980).
|
||||
- Iannuzzi 2025 *JVS* (39824263).
|
||||
|
||||
### A4. Guidelines
|
||||
|
||||
- SVS 2015 (25638515; introduction 25721065)
|
||||
- ESVS 2024 (37949800)
|
||||
- ESC 2024 (39210722)
|
||||
- ACC/AHA 2024 (38743805; *JACC* 38752899). Advises against revascularizing only to prevent progression to CLTI.
|
||||
|
||||
### A5. Why there is no RCT of early PVI vs initial conservative therapy on hard outcomes
|
||||
|
||||
1. **Event rates are low, so the trial would have to be large and long.** These are rough two-proportion calculations with 80% power, ≥5-year follow-up and no crossover:
|
||||
- Amputation 2% vs 1% at 5 years: about 2,300 per arm.
|
||||
- CLTI 8% vs 5%: about 1,050 per arm.
|
||||
2. **Crossover is built into the comparator.** The conservative arm includes deferred PVI, so an ITT analysis dilutes the contrast. Examples: SUPER had 29% crossover in year 1; ERASE's SET arm had a revascularization HR of 2.5; CLEVER finished with 111 of 252 planned; OBACT randomized 56 of 434 screened.
|
||||
3. **Equipoise is asymmetric.** The guidelines already recommend against early PVI, and the high-use physicians who do most early PVI would not enroll their patients.
|
||||
4. **The hard endpoints are generated by physicians.** Reintervention and the coding of CLTI are decisions made by unblinded clinicians, and patients who get early PVI are watched more closely.
|
||||
5. **No sponsor and a lot of heterogeneity.** No device company will fund a "fewer devices" hypothesis. Anatomy and devices vary, and Medicare has covered SET only since 2017.
|
||||
|
||||
---
|
||||
|
||||
## Part B. Methods: which question each one answers
|
||||
|
||||
### B1. Immortal time bias
|
||||
|
||||
**Plain version:** if "early PVI" means "had PVI at month 4", that patient had to survive event-free to month 4 to be labelled early PVI. The fix is to line up eligibility, assignment and time zero.
|
||||
|
||||
- Suissa 2008 *AJE* (18056625)
|
||||
- Lévesque 2010 *BMJ* (20228141)
|
||||
- Hernán 2016 *J Clin Epidemiol* (27237061)
|
||||
- Hernán 2025 *Epidemiology* (39494894). Reframes the problem: immortal time is a symptom of misclassification or selection.
|
||||
|
||||
### B2. Landmark analysis
|
||||
|
||||
**Estimand:** among patients alive and CLTI-free at 6 months, those who happened to have PVI in months 0–6 vs those who did not.
|
||||
|
||||
- It is conditional and descriptive.
|
||||
- The comparator mixes patients who never have PVI with patients who have it at month 7.
|
||||
- Events in months 0–6 are dropped.
|
||||
- It conditions on a post-treatment variable.
|
||||
- **Sorber 2023 is effectively a 6-month landmark design.**
|
||||
|
||||
References:
|
||||
- Anderson, Cain, Gelber 1983 *JCO* (6668489); revisited 2008 (18711176)
|
||||
- Dafni 2011 *Circ CV Qual Outcomes* (21586725)
|
||||
- van Houwelingen 2008 (18836831)
|
||||
- Putter & van Houwelingen 2017 (29225713)
|
||||
|
||||
### B3. Clone–censor–weight (CCW)
|
||||
|
||||
**Estimand:** the risk if *everyone* followed strategy A, "PVI within 6 months of diagnosis", vs strategy B, "no PVI in the first 6 months, then usual care (or PVI only for CLTI)". This is a per-protocol effect of sustained strategies starting at diagnosis.
|
||||
|
||||
- The 6-month grace period means an event inside the window, before any PVI, counts in both arms.
|
||||
- **The answer depends on how strategy B is written.**
|
||||
- Natural and stochastic grace-period strategies are different estimands (Wanis 2024).
|
||||
|
||||
References:
|
||||
- Hernán 2018 *BMJ* (29419381)
|
||||
- Cain 2010 *Int J Biostat* (21972433)
|
||||
- **Maringe 2020 *IJE* (32386426)**; correspondence 33349843, 33524137
|
||||
- Gaber 2024 *Cancer Med* (39642890)
|
||||
- **Wanis 2024 *JRSS-A* (39281783)**, on grace periods
|
||||
- Hernán & Robins 2017 *NEJM* (28976864)
|
||||
- Cole & Hernán 2008 *AJE* (18682488)
|
||||
|
||||
### B4. Sequential trial emulation
|
||||
|
||||
**Estimand:** among patients still eligible *this month* (no PVI yet, no CLTI, alive), PVI now vs not now.
|
||||
|
||||
- "Not now" includes PVI next month, so it is a *deferral* effect, not an *abstention* effect.
|
||||
- The pooled population leans toward slowly progressing patients, because they stay eligible for more trials.
|
||||
|
||||
References:
|
||||
- Hernán 2008 *Epidemiology* (18854702)
|
||||
- Danaei 2013 *SMMR* (22016461)
|
||||
- Gran 2010 *Stat Med* (20803557)
|
||||
- **Keogh 2023 *Stat Med* (37086186)**
|
||||
- Limozin/Seaman 2025 *SMMR* (40631395): CIs
|
||||
- Robins 2000 *Epidemiology* (10955408): MSMs
|
||||
- The TrialEmulation R package is on CRAN and has no PubMed citation.
|
||||
|
||||
### B5. Studies comparing methods
|
||||
|
||||
- **Duchesneau 2022 *CEBP* (35984990).** Surgery in metastatic breast cancer. Compares landmark with CCW against a plasmode known truth. This is **the closest template.**
|
||||
- Mi 2016 *Stat Med* (27350312). Landmark is unbiased under the null but biased when there is an effect.
|
||||
- Karim 2016 *AJE* (27455963)
|
||||
- Zhou 2005 *AJE* (16192344)
|
||||
- Pilleron 2023 *BJC* (36759725). Immortal time bias is larger in older patients.
|
||||
- Hoffman 2022 *JAMA Netw Open* (36190729)
|
||||
- Moreno-Betancur 2026 *Epidemiology* (41490058). "The Ideal Trial": the estimand you want vs the estimand your data allow.
|
||||
|
||||
**Gap:** we found no simulation that compares landmark, CCW and sequential trials side by side as answers to *different* estimands.
|
||||
|
||||
### B6. Physician clustering and unmeasured confounding at the surgeon level
|
||||
|
||||
There are two separate problems.
|
||||
|
||||
**1. Inference.** Clustered SEs or random effects fix only the standard errors.
|
||||
- Panageas 2003 *Ann Intern Med* (14568854); 2007 *Stat Med* (17016864)
|
||||
- Kahan & Morris 2013 *Stat Med* (23112128)
|
||||
|
||||
**2. Confounding by physician.** Physician style affects both early PVI and the outcome. Here the outcome (late PVI, coding of CLTI) is *itself* a physician decision.
|
||||
- **Within-physician comparisons** (fixed effects, Mundlak, matching within surgeon) remove time-invariant confounding between physicians:
|
||||
- Neuhaus & Kalbfleisch 1998 (9629647)
|
||||
- Begg & Parides 2003 (12898546)
|
||||
- **Li Y 2020 *SMMR* (31694489)**: closed-form bias results
|
||||
- **Cafri 2020 *Biom J* (32419247)**: matching within surgeon on device data
|
||||
- Li/Zaslavsky/Landrum 2013 (23526267)
|
||||
- Thoemmes & West 2011 (26735885)
|
||||
- Arpino & Cannas 2016 (26833893)
|
||||
- Fuentes 2022 (34128730)
|
||||
- Austin 2017 (27885709): median HR
|
||||
- **Caveats:**
|
||||
- A random effect does **not** remove confounding when it is correlated with PVI use.
|
||||
- Frailty HRs are conditional on physician and non-collapsible.
|
||||
- Fixed effects need physicians who sometimes do and sometimes do not intervene early (positivity).
|
||||
- **Preference as an instrumental variable:**
|
||||
- Brookhart 2006 (16617275); Brookhart & Schneeweiss 2007 (19655038)
|
||||
- Rassen 2009 (19345561)
|
||||
- Brookhart 2010 (20354968)
|
||||
- Ertefaie 2017 (28239929)
|
||||
- Hutchings 2022 (35588540)
|
||||
- The exclusion restriction is doubtful when the same physician also decides on reintervention.
|
||||
- **Bias amplification (Z-bias):** Myers 2011 *AJE* (22025356); Pearl 2011 *AJE* (22034488).
|
||||
- If physician preference is purely instrument-like, conditioning on the physician *amplifies* remaining unmeasured patient confounding.
|
||||
- The simulation can show where the crossover lies.
|
||||
|
||||
**Gap:** we found no paper on unmeasured surgeon-level confounding *inside* CCW or sequential trial emulation. The nearest are Cafri 2020 and Li 2020.
|
||||
|
||||
### B7. Simulation methods
|
||||
|
||||
Morris, White, Crowther 2019 *Stat Med* (30652356), the ADEMP framework. **Score each method against its own true estimand.**
|
||||
|
||||
### B8. Reporting
|
||||
|
||||
TARGET 2025: *BMJ* 40903028, *JAMA* 40899949; explanation and elaboration in *PLoS Med* 41091811.
|
||||
|
||||
---
|
||||
|
||||
## Summary table (early PVI ≤6 months)
|
||||
|
||||
| Method | Estimand | Requires | Failure mode |
|
||||
|---|---|---|---|
|
||||
| Naive "ever early PVI" | None that is well defined | — | Immortal time and confounding by indication |
|
||||
| Landmark at 6 months | Among patients event-free at 6 months: had PVI in months 0–6 vs not | Confounders measured up to the landmark | Drops early harms. Mixed comparator. Selects on survival after treatment. |
|
||||
| CCW with 6-month grace | Everyone follows A vs everyone follows B | Time-varying predictors of PVI timing; positivity; a precise definition of strategy B | Unmeasured symptom progression or physician style. Extreme weights. Sensitive to how B is defined. |
|
||||
| Sequential trials (ITT) | PVI now vs not now, among patients eligible this month | Eligibility each month; covariates at each baseline | A deferral effect, not abstention. Leans toward slow progressors. |
|
||||
| Physician-clustered models | Clustered SEs: honest SEs. FE: within-physician effect. Frailty: conditional HR | FE needs within-physician variation | RE does not remove confounding. FE can show Z-bias. Conditional vs marginal mismatch. |
|
||||
|
||||
## Read first
|
||||
|
||||
1. Hernán 2018 *BMJ*, CCW (29419381)
|
||||
2. Hernán 2008 *Epidemiology*, sequential trials (18854702)
|
||||
3. Keogh 2023 *Stat Med* (37086186)
|
||||
4. Maringe 2020 *IJE* (32386426)
|
||||
5. Duchesneau 2022 *CEBP* (35984990)
|
||||
6. Morris 2019 *Stat Med*, ADEMP (30652356)
|
||||
7. Cafri 2020 *Biom J* (32419247)
|
||||
8. Myers 2011 *AJE* (22025356)
|
||||
@@ -0,0 +1,61 @@
|
||||
# Literature review 3: challenges of surgical RCTs, and whether TTE solves them
|
||||
|
||||
*Merged from three parallel searches (2026-10-28). The evidence, with verified PMIDs and numbers taken from abstracts, is in `drafts/A-feasibility.md`, `drafts/B-surgeon-intervention.md` and `drafts/C-pathway-and-tte.md`, together with files 01 and 02. In total, 220 PMIDs were checked against PubMed; all exist, and a random sample of 15 titles matched the claims made about them. Framing: D54. TTE here means emulation on any observational data: claims, EHR or clinical registry (VQI; VQI linked to Medicare, VISION).*
|
||||
|
||||
## How to read Table 1
|
||||
|
||||
- **Can TTE solve it?** asks whether target trial emulation removes the problem *in principle*, separately for claims and for EHR/registries, because the answer often depends on the data source.
|
||||
- **Tested in our simulation** names the simulated world or analysis that measures it against a known truth. *Candidate* means not yet simulated.
|
||||
|
||||
## Draft Table 1. Why surgical RCTs struggle, and whether target trial emulation solves it
|
||||
|
||||
| # | Challenge | How common / how big | Vascular / claudication example | TTE on claims | TTE on EHR / registry | Tested in our simulation |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **A. Running the trial** | | | | | | |
|
||||
| 1 | Recruitment failure, early stopping, non-publication | Only 14.6% of surgical RCTs finish on time and at full size (Shadbolt 2023). 21% stopped early, mostly for recruitment (Chapman 2014). In vascular surgery, 22% are discontinued and 80% of those are never published (Binko 2023). | CLEVER planned 252 and randomized 111. SUPER planned 400, randomized 240, and stopped early. OBACT randomized 56 of 434. | **Yes**: no recruitment step | **Yes**, but registries hold only treated patients, so a no-PVI arm needs linkage | Realistic trial (n = 240) vs TTE on the full population |
|
||||
| 2 | Equipoise is absent, and surgeons have strong preferences | Vascular surgeons' equipoise is often absent, and preference strength predicts refusal to randomize (Young 2004). Clinicians' "hunches" undermine trials (Donovan 2014). | Physician early-PVI rates range from 0% to 58% (Hicks 2020) | **Partly**: preference becomes the source of treatment variation, and an unmeasured confounder if it tracks prognosis | **Partly**: drivers such as Rutherford class and ABI are recorded | Practice-style worlds (W2, W3) |
|
||||
| 3 | Who consents is selective | Refusal >50% in most preference trials, related to age, sex and education. Outcomes rarely differ (Wasmann 2019; King 2005; Vist 2005). Preference may modify effects, ES about 0.16 (PCRG 2008). | Whyman: 62 of 600 screened claudicants randomized | **Yes for selection** (no consent step); patient preference is still unrecorded | Same | *Candidate*: make consent in the simulated trial depend on patient characteristics (currently random) |
|
||||
| 4 | Too small to detect hard outcomes | Vascular RCT conclusions hinge on 3–7 events (Javidan 2024; Li 2023). Median negative surgical RCT has about 80 patients (Ahmed Ali 2018). | Procedure-related amputation within 1 year after revascularization for claudication is 0.2% (Djerf 2020) | **Yes**: tens of thousands of patients. A precise estimate can still be biased. | **Partly**: smaller samples | Realistic trial: power 25% for a true 5-point harm. TTE spread is 1.5–2.8 points. |
|
||||
| 5 | Short follow-up and soft endpoints | Claudication trials use 541 different outcomes and few PROs (Shwan 2024). 88% of device endpoints are surrogates (Dhruva 2009). Benefit fades by 2–5 years (IRONIC, ERASE). | No claudication RCT was powered for CLTI, amputation or death. Only the Swedish SFA trial reports them by arm (about 2% amputation per arm at 5 years, n = 100). | **Yes for hard long-term outcomes; no** for walking distance and QoL | **Partly**: walking distance, ABI and Rutherford class recorded at visits, but follow-up is short and fragmented. VQI–Medicare linkage reaches 74–82% (Hoel 2017). | Benefit-then-harm scenario (benefit at 2 years, harm at 5). *Candidate*: add a 12-month trial analysis. |
|
||||
| 6 | Cost, funding, industry, ethics of withholding or sham | IRONIC revascularization cost about 2× exercise. Trials save money when they use routine data (Griessbach 2024). | No sham-controlled PVI trial exists; ORBITA is the nearest analogue | **Yes** for cost and withholding; **no** for placebo questions | Same; symptom outcomes are prone to placebo effects | Not testable (Discussion) |
|
||||
| **B. The surgeon and the procedure** | | | | | | |
|
||||
| 7 | Learning curve | Plateau at about 250 cases (Vickers 2007). 1 of 247 surgical RCTs analysed learning effects (Conroy 2019). | In Medicare claims, carotid stenting by low-experience operators had aOR 1.7 (Nallamothu 2011) | **Partly**: cumulative operator volume is computable | **Partly**: plus technical success | *Candidate*: new learning-curve world |
|
||||
| 8 | Surgeon skill varies | Video-rated skill: bottom vs top quartile complications 14.5% vs 5.2% (Birkmeyer 2013). Skill explains 25.8% of complication variation (Stulberg). | PVI operator volume and MALE, OR 0.73 (Chunawala) | **Partly**: proxies only (volume) | **Partly** | Surgeon-skill worlds (W1, W1 ρ = 0) |
|
||||
| 9 | Trials recruit selected surgeons and centres | CEA mortality 1.4% at trial hospitals vs up to 2.5% elsewhere (Wennberg 1998). Trial patients are younger than registry patients (Cutteridge 2025). | Carotid and EVAR trials vs practice | **Yes**: all operators are included | **Partly**: centres choose to join registries | Realistic trial (selected surgeons) vs ideal trial vs TTE |
|
||||
| 10 | Procedure not standardized; device heterogeneity | 29.4% of trial interventions standardized (Blencowe) | Tibial PVI rate varies by physician, IQR 9–38% (Bose 2023) | **Partly**: modality and territory only | **Mostly**: lesion and device data | Not simulated (Discussion: "versions of treatment") |
|
||||
| 11 | Outcomes cluster by surgeon | 4% of surgical RCTs adjusted for provider (Biau 2008). Ignoring clustering overstates precision (Lee & Thompson 2005). | Hundreds of PVI operators per registry | **Yes** for SEs; confounding by physician remains | **Yes** | Physician variants V1–V4 in every world |
|
||||
| 12 | No blinding; outcome detection bias | Non-blinded assessors inflate subjective effects by 36–68%; objective outcomes are not affected (Hróbjartsson 2012; Wood 2008) | Claims codes separate claudication from CLTI with 81% agreement against VQI (Bose 2025) | **No blinding**, but amputation and death are objective. CLTI coding is at risk. | **No**: clinician-reported | Style affects CLTI coding (W3) |
|
||||
| **C. The treatment pathway** | | | | | | |
|
||||
| 13 | Crossover: "conservative" becomes delayed surgery | 27–43% crossover within 1–2 years across spine, knee and appendicitis trials, up to two-thirds by 10 years in hernia trials. Effects shrink toward the null (Gaudino 2019); STICH ITT was not significant at 5 years. | SUPER 29% at 1 year; Swedish SFA 25% by 5 years; ERASE later revascularization HR 2.50 | **Yes**: explicit strategies with CCW and sequential trials, **if** the triggers (worsening) are measured. Claims lack symptoms. | **Yes, better**: symptoms partly recorded | Deferral world (W4) |
|
||||
| 14 | Patients refuse or don't adhere | ACST: only 50% were operated on by 1 month. SET: 24.2% recruited, US completion 9.8–49.9%. | SUPER SET compliance 50% | **Partly**: refusal of an offered operation is invisible | **Partly**: sometimes noted | Trial arm A has 10% refusal |
|
||||
| 15 | Co-interventions and usual care differ | Usual care poorly reported (TIDieR median 1/12). Only 38.1% of VQI claudicants on complete OMT before revascularization. | "Conservative" = advice vs SET + BMT vs cilostazol across trials | **Partly**: drugs and Medicare SET claims are visible; home exercise and smoking are not | **Partly**: VQI has OMT only at the procedure | *Candidate*: unmeasured supervised exercise (supplement) |
|
||||
| 16 | The comparator is ill-defined | Most "conservative" arms are really "surgery if needed". CCW and sequential trials give different numbers (Garcia-Albeniz 2026). | "No PVI for 6 months" vs "PVI only for CLTI" vs "never PVI" | **Yes, TTE's main gain**: several named strategies from one dataset | Same | Aim 1: A vs B1, A vs B2, sequential trials, landmark |
|
||||
| **D. The price TTE pays** | | | | | | |
|
||||
| 17 | Confounding by indication (what randomization fixes) | TTE agrees with RCTs when the emulation is close (r = 0.93, RCT-DUPLICATE; 0.83, Wang 2026), poorly otherwise (0.53). Only 30.8% of published TTEs address unmeasured confounding (Ren 2026). 19.5% of CCW studies still contain immortal time (Manzanilla 2026). | No claudication or PVI TTE found in PubMed | **No**: severity poorly captured | **Partly**: ABI and Rutherford class recorded | Unmeasured-severity world (W5) and realistic world (WR). *Candidate*: EHR-like world with noisy severity. |
|
||||
| 18 | Misclassified eligibility | About half of registry "claudication" amputations were in misclassified CLTI patients (Djerf 2020) | Claudication vs CLTI codes 81% agreement (Bose 2025) | **Risk** | **Risk** | Not simulated (out of scope, D38) |
|
||||
|
||||
## What the review means for the simulation
|
||||
|
||||
**Already covered (no change needed).** Rows 1, 2, 4, 8, 9, 11, 13, 16 and 17 map onto existing worlds or analyses, and rows 12 and 14 are partly covered.
|
||||
|
||||
**Candidate additions, in order of value for cost:**
|
||||
|
||||
| Candidate | Table 1 row | What it adds | Cost | Recommendation |
|
||||
|---|---|---|---|---|
|
||||
| **EHR/registry-like world**: severity measured with noise (e.g., ABI and symptom class) instead of exact or only visit counts | 17 | Answers "does better data close the gap?", which is the claims-vs-EHR question | Parameters only; same DGP as W0, about 1–1.5 h of running | **Add** |
|
||||
| **Non-random consent in the simulated trial**: consent depends on age and frailty, moderately (Wasmann 2019, King 2005) | 3 | More realistic trial; the trial's own truth must then be weighted by consent probability | Small code change in `rct.R` plus RCT truths | **Add** |
|
||||
| **12-month trial analysis** in the benefit-then-harm scenario | 5 | Shows a short trial reaching the opposite conclusion from 5-year TTE | Small: ITT at 12 and 24 months plus truths at those horizons | **Add** |
|
||||
| **Learning-curve world**: skill rises with each operator's cumulative volume | 7 | TTE-specific issues: interference, delayed PVIs done by more experienced operators, physician confounding that changes over time | Needs calendar-time simulation across patients; a real build (about a day) | **Later / supplement** |
|
||||
| **Unmeasured supervised exercise** | 15 | Co-intervention that differs by arm | One more confounder | Supplement (already planned) |
|
||||
|
||||
## Points for the Introduction and Discussion
|
||||
|
||||
1. **No PubMed hit for a claudication or PVI target trial emulation**, no systematic review of TTE in surgery or vascular surgery, and only scattered benchmarks of TTE against procedural RCTs. TTE guides for surgeons exist (e.g., *JBJS Rev* 2026), but no known-truth tests.
|
||||
2. **No data source wins everywhere.** Claims give long, complete follow-up of hard outcomes but miss severity and symptoms. Registries and EHR record severity but have short, fragmented follow-up and no untreated arm unless linked. Linkage (VISION) is incomplete.
|
||||
3. **TTE complements trials rather than replacing them.** It covers long-term hard outcomes trials cannot afford, while trials remain needed for walking distance and quality of life.
|
||||
|
||||
## Open items
|
||||
|
||||
- EXITPAD could not be found in PubMed; it is left out of the trial table.
|
||||
- Some trial details are not in the abstracts and need full texts: planned n for MIMIC, IRONIC and ERASE; crossover in IRONIC, CLEVER and MIMIC; ERASE's long-term CLI result.
|
||||
- **Authors' own papers.** Bose 2023 (36058433) and Bose 2025 (40439664, plus a second 2025 paper) are cited neutrally, for practice variation and code validation. The authors should decide whether to keep them.
|
||||
@@ -0,0 +1,254 @@
|
||||
# Draft A: trial feasibility and design challenges in surgical RCTs
|
||||
|
||||
Draft input for Table 1. It covers six challenges: recruitment, equipoise, consent selection, sample size, endpoints and follow-up, and cost, funding and ethics.
|
||||
|
||||
**Method.** Every PMID below was checked with NCBI esummary, and the abstract was read with efetch, on 2026-10-28. Numbers come from the PubMed abstracts only; no full texts were read. Papers already listed in `01-surgical-rct-limitations.md` or `02-claudication-evidence-and-methods.md` are not listed again. They are cited by PMID where they help the argument and are marked "(file 01)" or "(file 02)".
|
||||
|
||||
**Data sources for TTE.** We assume three kinds of data, and they differ in what they can do:
|
||||
- **Claims** (Medicare fee-for-service): population-based, with long and nearly complete follow-up for death, procedures, amputation and diagnoses. They contain no walking distance, ABI, Rutherford class, or patient-reported outcomes (PROs).
|
||||
- **EHR and clinical registries** (VQI): severity at the time of the procedure (Rutherford class, ABI, anatomy) and some PROs. Follow-up is short or fragmented, and VQI captures patients only when they have a procedure, so patients managed without one are mostly missing.
|
||||
- **Linked registry–claims data** (VQI–Medicare, VISION): registry severity plus claims follow-up. Linkage is incomplete, and the cohort is still defined by having a procedure.
|
||||
|
||||
---
|
||||
|
||||
## 1. Recruitment, slow accrual, early discontinuation, non-publication
|
||||
|
||||
**What it is.** Surgical trials often fail to recruit the planned number of patients, run years late, or stop early. Trials that stop early are rarely published, and so are many completed trials. The evidence base is therefore smaller and more biased than the registered trial activity suggests.
|
||||
|
||||
**How common / how big.**
|
||||
- Completion on time and at full size is rare. Of 2,542 surgical RCTs registered in 2010–2014, only **14.6%** finished both on time and at full enrollment. **21.5%** were discontinued, and when they stopped they were a median **92.7%** short of their enrollment target (36648945).
|
||||
- Surgical trials are discontinued more often than nonsurgical ones. Among 88,498 US trials, discontinuation was **15.97% vs 11.07%**, and poor recruitment was cited more often (**44.65% vs 34.74%**) (29804847).
|
||||
- Industry funding predicted discontinuation (OR 1.25 in surgical trials) (29804847).
|
||||
- These figures build on Chapman 2014 (25491195), Rosenthal 2015 (24979608) and Kasenda 2014 (24618966) (file 01).
|
||||
|
||||
**Vascular / claudication example.**
|
||||
- Vascular surgery RCTs: of 108 registered in 2010–2019, **22.2%** were discontinued, most often for poor enrollment (**45.8%**). Discontinued trials reached only **28.4%** of planned enrollment, and **80%** of them were never published. Of completed trials, **25%** were unpublished; industry funding cut the odds of publication (OR 0.18). In total, **4,788 enrollees** have no public results (37311508).
|
||||
- PAD exercise trials: one recruitment effort spent $1,330 on community outreach and got two referrals and no enrollments. Health-system referral produced 44 referrals and one enrollment (30139448).
|
||||
- Cardiac surgery is a neighbouring field: **22.6%** of trials were terminated early, and aortic surgery trials had higher odds of termination (OR 2.86) (39133169).
|
||||
- Under-representation of women and minorities is a related problem. Across 110 North American PAD trials (57,451 patients), **28%** of patients were women and **5.0%** were Black (42051894).
|
||||
- BASIL-3 reaching 481 of 861 planned patients is noted in file 01.
|
||||
|
||||
| Paper | PMID | Takeaway |
|
||||
|---|---|---|
|
||||
| Shadbolt C et al. 2023, *JAMA Netw Open* | 36648945 | 2,542 surgical RCTs. 14.6% finished on time and at full size, 20.4% on time, and 45.9% met their enrollment target. Median delay was 12.2 months (66.7% longer than planned). 21.5% were discontinued, with a median shortfall of 92.7%. |
|
||||
| Mouw TJ et al. 2018, *J Surg Res* | 29804847 | 5,779 surgical vs 82,719 nonsurgical US trials. Discontinued: 15.97% vs 11.07%. Poor recruitment was cited in 44.65% vs 34.74%. Industry funding predicted discontinuation. |
|
||||
| **Binko MA et al. 2023, *Ann Vasc Surg*** | 37311508 | Vascular surgery RCTs: 22.2% discontinued, 45.8% of them for poor enrollment. Of trials stopped after enrollment began, 80% were unpublished. Of completed trials, 25% were unpublished (industry funding OR 0.18 for publication). |
|
||||
| Cancelli G et al. 2024, *Eur J Cardiothorac Surg* | 39133169 | 746 cardiac surgery trials: 22.6% terminated early, 29.6% of them for slow recruitment. Odds of termination were higher for valve (OR 4.30), LVAD (OR 3.65) and aortic (OR 2.86) trials. |
|
||||
| Caldieraro-Bentley AJ et al. 2018, *J Vasc Nurs* | 30139448 | PAD exercise trials. Indirect recruitment ($1,330): 2 referrals, 0 enrollments. Direct recruitment: 44 referrals, 1 enrollment. Barriers were comorbidity, transport, and strict eligibility criteria. |
|
||||
| Shah T et al. 2026, *JSCAI* | 42051894 | 110 North American PAD trials (2000–2023): 28% women, 5.0% Black, 1.3% Hispanic. 80% were industry-funded. |
|
||||
|
||||
**Can TTE solve it? Yes for feasibility; partly for representativeness, depending on the data source.**
|
||||
- TTE has no recruitment, consent or discontinuation step, so all eligible patients in the data enter the emulation.
|
||||
- **Claims:** Medicare covers nearly every older US claudicant, including women, Black patients and rural patients whom trials under-enroll. Patients under 65 and those in Medicare Advantage are missing.
|
||||
- **Registries (VQI):** these include only patients who had a procedure, at participating centres. A registry cannot supply patients managed without a procedure (the "delayed / never" arm) unless it is linked to claims or an EHR.
|
||||
- Non-publication is a separate problem. TTE analyses can also go unpublished selectively, so a protocol should be registered in advance (TARGET, 40899949, file 01).
|
||||
|
||||
**Testable in a known-truth simulation?** Partly. Simulate a trial that recruits a small, under-powered and selected subset (see §3) and compare it with a TTE run on the full simulated population. The selective-publication part is not testable.
|
||||
|
||||
---
|
||||
|
||||
## 2. Equipoise: surgeon and patient treatment preferences
|
||||
|
||||
**What it is.**
|
||||
- Randomization is ethical only under equipoise, which is genuine uncertainty about which option is better. Freedman's "clinical equipoise" places that uncertainty in the expert community rather than in each individual clinician.
|
||||
- Surgeons often hold strong individual beliefs and tolerate uncertainty poorly, so they may not offer the trial or may selectively enrol patients.
|
||||
- Patients also bring strong preferences, especially when one arm is "an operation" and the other is "no operation".
|
||||
|
||||
**How common / how big.**
|
||||
- Surgeons were significantly more intolerant of uncertainty than physicians (P = 0.007), but their attitudes to RCTs did not differ (16078288).
|
||||
- 62.6% of Australian colorectal surgeons named difficulty convincing patients to accept random allocation as a barrier (11598468).
|
||||
- In a US national survey, there was consensus (≥60% agreement) on whether to operate in 19 of 30 cases. Variation was largest when the modal answer was "do not operate" or "equipoise" (38328985).
|
||||
- In spine surgery, surgeons judged 12 of 20 cases eligible for randomization (17304133).
|
||||
- File 01 covers Donovan 2014 (24811157) on clinicians' "hunches" and Rooshenas 2016 (27755555) on how equipoise is presented in consultations.
|
||||
|
||||
**Vascular / claudication example.**
|
||||
- In a survey of vascular surgeons in Australia and New Zealand (77% response), **24%** were in individual equipoise on endarterectomy vs stenting for high-risk carotid patients. **89%** favoured endarterectomy for average-risk patients, so there was no community equipoise. Strength of preference predicted unwillingness to join 4 of 6 hypothetical trials (15523403).
|
||||
- Patients with severe claudication were **risk-seeking** about invasive treatment, and patients who could walk only short distances accepted more risk (18455642).
|
||||
- In Medicare, the physician-level rate of tibial PVI for claudication varied widely: median **20.0%**, IQR **9.1–37.5%** (36058433). This is practice variation that no trial would capture.
|
||||
|
||||
| Paper | PMID | Takeaway |
|
||||
|---|---|---|
|
||||
| **Freedman B 1987, *NEJM*** | 3600702 | Defines "clinical equipoise": uncertainty in the expert community. Requiring individual equipoise "presents nearly insuperable obstacles" to starting a trial and can cause recruitment failure. |
|
||||
| McCulloch P et al. 2005, *Br J Surg* | 16078288 | 2,000 consultants surveyed (36.5% responded). Surgeons were more intolerant of uncertainty (P = 0.007) but not more hostile to RCTs. Intolerance may block decisions to enrol individual patients. |
|
||||
| **Young J et al. 2004, *Surgery*** | 15523403 | Vascular surgeons: 24% in equipoise on carotid stenting in high-risk patients, 89% preferred endarterectomy in average-risk patients, and 88% favoured a minimally invasive approach for high-risk AAA. Preference strength predicted unwillingness to randomize. |
|
||||
| Gattellari M et al. 2001, *Dis Colon Rectum* | 11598468 | 89% response. 66.7% said RCTs should be the design of choice. Barriers: funding (74.4%), convincing patients to accept random allocation (62.6%), lack of wider surgical support (55.9%). |
|
||||
| Zychowski KL et al. 2024, *Ann Surg* | 38328985 | 580 US surgeons. Consensus on 19 of 30 cases, with most variation where the mode was against surgery or at equipoise. Surgeons often "let patients decide". |
|
||||
| Ghogawala Z et al. 2007, *Spine* | 17304133 | Spine surgeons split 51% / 38% / 11% between approaches. Only 12 of 20 real cases were judged eligible for randomization. |
|
||||
| Letterstål A et al. 2008, *J Vasc Surg* | 18455642 | 50 patients with severe claudication: EQ-5D 0.51, standard gamble 0.88. Patients were risk-seeking about surgery, more so if their walking distance was short. |
|
||||
| Bose S et al. 2023, *J Vasc Surg* | 36058433 | Medicare, 59,930 PVIs for claudication: 27.7% tibial. Physician-level rate median 20.0% (IQR 9.1–37.5%). Practice setting (office-based lab aOR 2.37) and specialty predicted use. |
|
||||
|
||||
**Can TTE solve it? Partly.**
|
||||
- TTE needs no one to be in equipoise. It instead relies on preference-driven variation between physicians and regions, which is exactly what makes trials hard to run. Physician preference (for example, a physician's past PVI rate) can serve as an instrument or as a source of positivity.
|
||||
- Patient preference, and the clinical judgement behind a surgeon's choice for a particular patient, are not recorded. If they predict outcomes, they are unmeasured confounders.
|
||||
- **EHR and registries** partly measure what drives the choice (Rutherford class, ABI, anatomy), which narrows the problem. **Claims** do not.
|
||||
- Wherever community consensus is near-total, TTE suffers positivity violations, just as a trial would struggle to recruit.
|
||||
|
||||
**Testable in a known-truth simulation? Yes.** Generate a latent physician "intervention propensity" (already in the data-generating process, or DGP) and a latent patient preference that affects both treatment and outcome. Vary how strongly each confounds the outcome, and check how much TTE bias each produces.
|
||||
|
||||
---
|
||||
|
||||
## 3. Patient preference and consent selection: who agrees to be randomized
|
||||
|
||||
**What it is.** Eligible patients who decline randomization usually do so because they prefer one treatment. Trial participants can therefore differ from the eligible population, which limits external validity. If preference also changes how patients respond, it can affect the outcome. Comprehensive-cohort and partially randomized preference designs follow patients who decline, so the two groups can be compared directly.
|
||||
|
||||
**How common / how big.**
|
||||
- Partially randomized preference trials (44 trials, 24,873 patients): refusal of randomization was **over 50% in 26 of 44 trials** (range 19–99%). Decliners differed by education, sex, age, race and prior experience of one treatment. Primary outcomes were similar in the randomized and preference cohorts (mean difference 0.093, 95% CI −0.178 to 0.364) (31619428).
|
||||
- King 2005 (32 comprehensive cohorts and two-stage trials): preferences led "a substantial proportion" to refuse randomization, but there was little evidence of bias in who agreed, or in outcomes (15741531).
|
||||
- Musculoskeletal trials, patient-level data (n = 1,594): patients randomized to their preferred treatment did better than indifferent patients (effect size **0.162**, 95% CI 0.011–0.314) (18977792).
|
||||
- Trial participants vs comparable non-participants (31,140 vs 20,380 patients): no overall difference in outcomes (SMD 0.01), but heterogeneity was high (I² = 89%) for dichotomous outcomes (15905256).
|
||||
|
||||
**Surgical examples (no vascular example with a non-participant cohort was found).**
|
||||
- SPORT disc herniation: the 743 patients who declined randomization formed an observational cohort. Their self-selected surgery-vs-nonoperative difference was large (bodily pain 14.8 points at 3 months), whereas the randomized ITT difference was not significant (17119141; ITT trial 17119140, file 01).
|
||||
- PREOPANC (pancreatic cancer): only **33%** of 455 eligible patients were randomized, and 55% of the rest refused. Survival (15.2 vs 15.5 months) and resection rates were the same in the two groups (33273349).
|
||||
- ProtecT (prostate cancer): men who chose their treatment were similar to randomized men, apart from occupation (29288137).
|
||||
- Meniscal tear: willingness to join a planned surgery-vs-nonoperative RCT fell from **62% to 46%** after a course of physiotherapy (41367623).
|
||||
|
||||
| Paper | PMID | Takeaway |
|
||||
|---|---|---|
|
||||
| **King M et al. 2005, *JAMA*** | 15741531 | 32 preference-design studies. Preference drives refusal, but there was little evidence that it biases who enrols or the trial's internal validity. |
|
||||
| **Preference Collaborative Review Group 2008, *BMJ*** | 18977792 | Patient-level data, n = 1,594. Randomized to preferred treatment vs indifferent: ES 0.162 (0.011–0.314). Allocated to an unwanted treatment: OR 1.70 for loss to first follow-up compared with indifferent patients. |
|
||||
| **Wasmann KA et al. 2019, *BMJ Open*** | 31619428 | 44 partially randomized preference trials (n = 24,873). Refusal over 50% in 26 trials. Decliners differed by education, sex, age, race and prior treatment. Outcomes similar between cohorts. Loss to follow-up and crossover were higher in the randomized cohort. |
|
||||
| Vist GE et al. 2005, *BMJ* | 15905256 | Trial participants vs non-participants: no consistent effect of participation (SMD 0.01), but large heterogeneity (I² = 89%). |
|
||||
| **Weinstein JN et al. 2006, *JAMA* (SPORT observational cohort)** | 17119141 | Patients who declined randomization and chose surgery improved more (treatment effect 14.8 to 15.4 points at 3 months). The authors warn that this comparison is confounded. |
|
||||
| Versteijne E et al. 2022, *Ann Surg* (PREOPANC) | 33273349 | 33% of eligible patients randomized. Non-randomized patients had the same survival and resection rates. |
|
||||
| Donovan JL et al. 2018, *J Clin Epidemiol* (ProtecT) | 29288137 | Extended comprehensive cohort: 1,643 randomized, 997 chose treatment, and 557 were excluded. Those who chose were similar to the randomized men; those excluded were sicker. |
|
||||
| Fox KB et al. 2025, *Osteoarthr Cartil Open* | 41367623 | Advance preference assessment: willingness to enrol in a surgery-vs-nonoperative RCT fell from 62% to 46%. The strongest preference was to avoid surgery (47%). |
|
||||
|
||||
**Can TTE solve it? Mostly for selection into the study; no for preference as a confounder.**
|
||||
- TTE does not need consent, so it estimates the effect in everyone eligible rather than in those willing to be randomized.
|
||||
- The same preference that drives refusal in a trial drives treatment choice in observational data, as SPORT's observational cohort shows. Neither claims nor EHR record patient preference.
|
||||
- If preference also modifies the treatment effect, as PCRG 2008 suggests, the trial and the TTE estimate different estimands even when both are unbiased.
|
||||
|
||||
**Testable in a known-truth simulation? Yes, and this is directly relevant to our design.** The evidence supports making consent to the simulated trial **non-random but only moderately selective**:
|
||||
- consent depends on covariates (age, sex, prior treatment) and on a latent preference;
|
||||
- refusal rates are well over 50% (Wasmann 2019);
|
||||
- consent has only a small effect on outcomes (King 2005, Vist 2005);
|
||||
- add an optional scenario with preference × treatment effect modification of about 0.15 SD (PCRG 2008).
|
||||
|
||||
The trial's ITT estimate for consenters can then be compared with the TTE estimate for the whole population, against the known truth for each.
|
||||
|
||||
---
|
||||
|
||||
## 4. Sample size, underpowering, imprecision
|
||||
|
||||
**What it is.** Surgical trials are small because recruitment is hard, funding is limited, and the evidence is spread across many devices and techniques. Small trials miss real effects, over-estimate the effects they do find, and depend on a handful of events. Hard outcomes in claudication, such as amputation and progression to CLTI, are rare, so trials powered for them would need thousands of patients.
|
||||
|
||||
**How common / how big.**
|
||||
- Only **53%** of surgical RCTs in leading journals reported their sample-size calculation adequately, falling to **12%** against the full SPIRIT items (31787318).
|
||||
- File 01 already covers Ahmed Ali 2018 (29907196): the median size of negative surgical RCTs is about 76–80.
|
||||
|
||||
**Vascular / claudication example.**
|
||||
- Fragility: in positive open-vs-endovascular RCTs, a median of **3** event changes would reverse significance, and **70%** of outcomes had more patients lost to follow-up than that (37315910). In negative trials, a median of **7** changes would make them significant (7 for PAD), and **47%** had more patients lost than that (36572321).
|
||||
- Cochrane (file 02): **10 RCTs, 1,087 participants** in total for endovascular revascularisation in claudication. Evidence was downgraded for "small sample size", and the endovascular vs no-specific-treatment comparison rested on 3 studies with **125 participants** (29518253).
|
||||
- Device dispersal: 112 PAD endovascular RCTs tested **228 different devices or techniques** (37678660).
|
||||
- Rare hard outcomes: major amputation within 1 year of revascularisation for claudication was **0.2%** (9 of 5,860; Swedvasc) (31866238).
|
||||
- Progression to CLTI under conservative management ranged from **1.1% to 36.7%** across studies, with I² = 99%. Rates that uncertain make power calculations unreliable (39694186).
|
||||
|
||||
| Paper | PMID | Takeaway |
|
||||
|---|---|---|
|
||||
| **Javidan A et al. 2024, *J Vasc Surg*** | 37315910 | 21 positive open-vs-endovascular RCTs. Median fragility index 3 (3, 20). In 70% of outcomes, loss to follow-up exceeded the fragility index. |
|
||||
| **Li A et al. 2023, *J Vasc Surg*** | 36572321 | 49 negative vascular RCTs. Median reverse fragility index 7 (PAD 7). In 47%, loss to follow-up exceeded it. |
|
||||
| Speich B et al. 2020, *Surgery* | 31787318 | 85 surgical RCTs (2017). Sample-size reporting adequate in 53% by CONSORT and 12% with SPIRIT items. |
|
||||
| Zywicka EM et al. 2024, *EJVES* | 37678660 | 112 PAD endovascular RCTs covering 228 devices or techniques. Interventions were described well enough to replicate in only 21%. |
|
||||
| Fakhry F et al. 2018, *Cochrane* (file 02) | 29518253 | 10 RCTs, n = 1,087, downgraded for small size and heterogeneity. Long-term data on walking distance came from only 2–3 studies (103–184 patients). |
|
||||
| Djerf H et al. 2020, *EJVES* | 31866238 | Procedure-related major amputation 0.2% at 1 year after revascularisation for claudication. Of 109 registered amputations, 51 were in patients with CLTI misclassified as claudication. |
|
||||
| Froud JLJ et al. 2025, *Ann Vasc Surg* | 39694186 | 9 cohorts (n = 4,115): progression to CLTI 1.1–36.7%, I² = 99%. Women made up 22.7% of patients. |
|
||||
|
||||
**Can TTE solve it? Yes for size; partly for precision on what matters.**
|
||||
- **Claims** offer very large samples (tens of thousands of claudicants), which makes rare hard outcomes (amputation, CLTI, death) estimable with narrow confidence intervals.
|
||||
- Large samples do not remove bias. A precise but confounded estimate can mislead more than a wide trial estimate does.
|
||||
- **Registries** are smaller and centre-limited but record severity. Djerf 2020 shows that registry indication codes can be wrong: about half of the "claudication" amputations were in misclassified CLTI patients. Misclassification of eligibility is therefore a real risk in claims and in registries.
|
||||
- Effective sample size can be much smaller than the raw count after cloning, censoring and weighting, especially with extreme weights.
|
||||
|
||||
**Testable in a known-truth simulation? Yes.** Compare the sampling variability and coverage of a small trial (n ≈ 100–200) with a population-scale TTE across replicates. Show the bias–variance trade-off: unbiased but noisy trial vs precise TTE with possible bias.
|
||||
|
||||
---
|
||||
|
||||
## 5. Short follow-up and soft or surrogate endpoints
|
||||
|
||||
**What it is.**
|
||||
- Claudication trials mostly measure treadmill walking distance, ABI, patency or quality of life at 6–12 months. Patients and payers care about durability, reintervention, CLTI, amputation and death over years.
|
||||
- Device trials rely heavily on surrogate endpoints such as patency.
|
||||
- Outcome definitions vary so much that trials cannot be pooled.
|
||||
- Long follow-up raises costs and loss to follow-up, especially for patient-reported outcomes.
|
||||
|
||||
**How common / how big.**
|
||||
- Of 213 primary endpoints in FDA premarket approvals of cardiovascular devices, **88%** were surrogates. Only **27%** of the supporting studies were randomized (20040556).
|
||||
- In surgical RCTs, loss to follow-up was far higher when outcomes were patient-reported (RR **14.21**) and when follow-up lasted 3 months or more (OR **10.09**) (38795671).
|
||||
|
||||
**Vascular / claudication example.**
|
||||
- **408 claudication studies** used **541 unique outcomes**. 386 of these were used only once, ABI was the most common, PROs appeared in only **36.2%** of studies, and definitions of MALE and primary patency varied (39485886).
|
||||
- A new core outcome set for chronic symptomatic PAD has 11 outcomes, a mix of hard and patient-centred ones: death, MI, major amputation, acute limb ischaemia, pain-free walking distance, quality of life, and others (40992592).
|
||||
- Trials in file 02: CLEVER (6-month treadmill, n = 111; 22090168), ERASE (12 months, n = 212; 26547465) and IRONIC (1 year; benefit lost by 5 years; 25095886, 31937137).
|
||||
- The paclitaxel mortality signal appeared only once long-term data were pooled (30561254, file 02).
|
||||
- SWEDEPAD 2 is the counter-example: a registry-based RCT with **98.3%** follow-up and a median of **7.1 years** for mortality (40902614, file 02).
|
||||
|
||||
| Paper | PMID | Takeaway |
|
||||
|---|---|---|
|
||||
| **Shwan A et al. 2024, *BJS Open*** | 39485886 | 408 claudication studies with 541 unique outcomes; 386 were used only once. PROs in 36.2%. MALE and patency defined inconsistently. Only 38.9% of studies included claudicants alone. |
|
||||
| Zwetsloot SLM et al. 2026, *EJVES* | 40992592 | Delphi core outcome set for chronic symptomatic PAD: 11 outcomes, including pain-free walking distance, physical activity, quality of life, amputation, and death. |
|
||||
| **Dhruva SS et al. 2009, *JAMA*** | 20040556 | 78 cardiovascular device premarket approvals: 27% of studies randomized, 14% blinded, 65% based on a single study, 88% of primary endpoints surrogate. |
|
||||
| Walker RJB et al. 2024, *J Surg Res* | 38795671 | 200 surgical RCTs. Loss to follow-up 3.10 per 100 patient-years. PROs RR 14.21; follow-up ≥3 months OR 10.09; standard-of-care assessments RR 0.17. |
|
||||
| Hoel AW et al. 2017, *J Vasc Surg* | 28222989 | VQI matched to Medicare: 82.4% of EVAR and 74.4% of open repair patients matched. Agreement was good for mortality (κ 0.64–0.82) but poor for reintervention (κ 0.16–0.26). |
|
||||
|
||||
**Can TTE solve it? Partly, and the answer depends on the data source.**
|
||||
- **Claims:** strong on long, complete follow-up of hard outcomes (death, amputation, reintervention, CLTI diagnosis codes) over 5–10 years. No walking distance, ABI, Rutherford class or quality of life, so the trials' main efficacy endpoints cannot be emulated.
|
||||
- **EHR and registries (VQI):** record walking distance, ABI, Rutherford class, and sometimes PROs, so soft and functional endpoints and baseline severity are partly available. Follow-up is short and fragmented, and functional outcomes are measured only at clinical visits, so they are missing not at random.
|
||||
- **Linked VQI–Medicare (VISION):** combines registry severity with claims follow-up. Linkage is incomplete (74–82% in Hoel 2017), and reintervention agrees poorly between sources.
|
||||
- No source captures the main claudication benefit (walking and quality of life) with long follow-up. TTE complements trials by covering the long-term hard outcomes they cannot afford to measure, rather than replacing them for symptom endpoints.
|
||||
|
||||
**Testable in a known-truth simulation? Yes, for hard outcomes and time-varying effects.** The DGP already has an early benefit and late harm on CLTI and amputation. Show that a 12-month trial and a long-follow-up TTE reach different conclusions. PRO endpoints could be added as a latent outcome that is observed only at visits, to show missingness in EHR data, but this is optional.
|
||||
|
||||
---
|
||||
|
||||
## 6. Cost, funding, industry sponsorship, and ethics (sham surgery, randomizing to no treatment)
|
||||
|
||||
**What it is.**
|
||||
- Surgical RCTs are expensive and are rarely funded by public bodies. Device makers fund many vascular trials, which affects which questions get asked, which endpoints are used, and whether and how results are published.
|
||||
- Placebo (sham) procedures are needed to separate the specific effect of a symptom-relieving procedure from the placebo effect, but they raise ethical and recruitment problems.
|
||||
- Randomizing patients to "no procedure" can feel unethical to surgeons and patients when practice favours intervention.
|
||||
|
||||
**How common / how big.**
|
||||
- Cost: the median investigator-initiated RCT cost **$645,824**, and **69.7%** exceeded their budget by more than 50%. Using routinely collected data for the primary outcome was associated with lower costs (39307405).
|
||||
- Public funding in vascular surgery: only **1%** of US vascular surgeons (41 of 4,037) hold NIH funding, and only **25%** of their projects are clinical or health-services research (37327950).
|
||||
- Industry outcomes across fields: industry-sponsored studies more often have favourable efficacy results (RR **1.27**) and favourable conclusions (RR **1.34**), with no difference between drug and device studies (28207928).
|
||||
- Industry outcomes in surgery: industry funding was associated with pro-industry significant results in surgical trials (OR **8.0**, 95% CI 1.1–53.2) (14970094).
|
||||
- Device publication: only **49%** of 177 premarket approval studies for cardiovascular devices were published, and when published, primary endpoints were often relabelled or missing (26063311).
|
||||
- Sham surgery: placebo-controlled surgical trials recruit as well as non-operative-control trials, but have about twice the attrition (IRR **1.8**) (38637129).
|
||||
- Willingness for sham trials: **31%** of surgeons and **40%** of patients would not take part in any placebo-controlled knee trial design offered (39615522).
|
||||
|
||||
**Vascular / claudication example.**
|
||||
- Funding: **80%** of PAD trials were industry-funded (42051894, §1). Industry-sponsored vascular RCTs rose from **6.7% to 28%** over a decade (26298221). Industry funding predicted non-publication of completed vascular RCTs (OR 0.18; 37311508, §1). Jones 2026 found 39% industry-funded (42526742, file 01).
|
||||
- Sham procedures: ORBITA showed that a placebo-controlled trial of a symptom-relieving revascularization is possible. PCI did not improve exercise time beyond placebo (**16.6 s**, 95% CI −8.9 to 42.0) (29103656). No sham-controlled trial of PVI for claudication was found; this is an inference, since the search was not exhaustive.
|
||||
- Ethics of "no procedure": in claudication, randomizing to exercise alone is ethically easy, because exercise is guideline first-line therapy.
|
||||
|
||||
| Paper | PMID | Takeaway |
|
||||
|---|---|---|
|
||||
| Griessbach A et al. 2024, *J Clin Epidemiol* | 39307405 | 93 investigator-initiated RCTs. Median cost $645,824. 69.7% overran their budget by more than 50%. Routinely collected outcome data were linked to lower cost. |
|
||||
| **Mirzaie AA et al. 2023, *J Vasc Surg*** | 37327950 | 1% of US vascular surgeons hold NIH funding (41 of 4,037). 75% of projects are basic or translational. |
|
||||
| **Lundh A et al. 2017, *Cochrane*** | 28207928 | 75 studies. Industry sponsorship: favourable results RR 1.27, favourable conclusions RR 1.34, same for devices as for drugs. Bias not explained by standard risk-of-bias tools. |
|
||||
| Bhandari M et al. 2004, *CMAJ* | 14970094 | 332 RCTs. Industry funding linked to pro-industry results (adjusted OR 1.8). In surgical trials, OR 8.0 (1.1–53.2). |
|
||||
| Dhruva 2009 (see §5); Chang L et al. 2015, *BMJ* | 26063311 | 177 premarket approval studies for cardiovascular devices: 49% published. Of 152 FDA primary endpoints, 28% were unlabelled and 10% missing in publications. 11% of results were substantially different. |
|
||||
| Hajibandeh S et al. 2015, *EJVES* | 26298221 | Vascular RCTs: share industry-sponsored rose from 6.67% to 28%. Methodological quality did not improve. |
|
||||
| **Beard DJ et al. 2020, *Lancet* (ASPIRE)** | 32145797 | Guidance: surgical placebo controls are justified when evidence of efficacy is poor and there is a real risk of bias from placebo effects. Feasibility work is recommended. |
|
||||
| **Al-Lamee R et al. 2018, *Lancet* (ORBITA)** | 29103656 | 200 patients randomized to PCI or placebo procedure. Exercise time difference 16.6 s (−8.9 to 42.0), not significant. Shows that a sham-controlled revascularization trial is feasible. |
|
||||
| Natarajan P et al. 2024, *BMJ Open* | 38637129 | 61 placebo-control vs 38 non-operative-control surgical RCTs. Recruitment similar; attrition IRR 1.8 (1.1–3.0) for placebo trials. |
|
||||
| Wall L et al. 2024, *Bone Joint J* | 39615522 | Discrete choice experiment. 31% of surgeons and 40% of patients would join no sham-controlled knee trial design. |
|
||||
|
||||
**Can TTE solve it? Yes for cost and for withholding treatment; partly for industry influence; no for sham and placebo questions.**
|
||||
- **Cost:** TTE on existing claims or registry data costs a small fraction of an RCT. Griessbach 2024 shows that even trials save money by using routinely collected data.
|
||||
- **Ethics of withholding treatment:** no one is assigned anything, so randomizing to "no procedure" is not an issue.
|
||||
- **Industry influence:** independent investigators can run TTE on population data. However, claims do not identify devices well (file 01, limitation #10), so device-specific questions such as paclitaxel need registries or linked device identifiers.
|
||||
- **Sham and placebo:** TTE cannot separate the specific effect of a procedure from the placebo effect. Hard outcomes in claims (amputation, death) are not prone to placebo effects. Symptom outcomes in EHR or registries are prone to them, and observational data cannot blind.
|
||||
|
||||
**Testable in a known-truth simulation?** Only partly. Cost and ethics are not testable. A placebo component could be added to a simulated PRO outcome that only a blinded trial removes, but it is outside the current DGP; this is optional.
|
||||
|
||||
---
|
||||
|
||||
## Summary: the most important new papers
|
||||
|
||||
1. **Shadbolt 2023 *JAMA Netw Open* (36648945) and Binko 2023 *Ann Vasc Surg* (37311508).** Only 14.6% of surgical RCTs finish on time and at full size. In vascular surgery, 22% are discontinued, mostly for poor enrollment, and 80% of those are never published.
|
||||
2. **Wasmann 2019 *BMJ Open* (31619428), King 2005 *JAMA* (15741531) and PCRG 2008 *BMJ* (18977792).** Most eligible patients often refuse randomization, and those who refuse differ by covariates. Outcomes rarely differ, but preference may modify effects (ES about 0.16). These set the parameters for non-random consent in our simulated trial.
|
||||
3. **Young 2004 *Surgery* (15523403) and Freedman 1987 *NEJM* (3600702).** Vascular surgeons' equipoise can be measured and is often absent, and preference strength predicts refusal to randomize. Clinical equipoise is the ethical standard.
|
||||
4. **Javidan 2024 / Li 2023 *JVS* (37315910, 36572321) and Djerf 2020 *EJVES* (31866238).** Vascular RCT results hinge on 3–7 events, and procedure-related amputation within 1 year of revascularisation for claudication is 0.2%. Hard outcomes need population-scale data.
|
||||
5. **Shwan 2024 *BJS Open* (39485886), Dhruva 2009 *JAMA* (20040556) and Hoel 2017 *JVS* (28222989).** Claudication trials use 541 different outcomes and few PROs, and 88% of device endpoints are surrogates. Linking VQI to Medicare restores long-term hard outcomes but not PROs.
|
||||
@@ -0,0 +1,271 @@
|
||||
# Draft B: surgeon and intervention challenges of surgical RCTs, and what TTE can do about them
|
||||
|
||||
Scope: learning curve; skill and expertise variation; selection of surgeons and centres into trials; standardization and device heterogeneity; clustering by surgeon; blinding, sham and outcome assessment.
|
||||
|
||||
All PMIDs were checked with NCBI E-utilities esummary on 2026-10-28. All numbers come from the PubMed abstracts fetched with efetch; full texts were not read. Papers already listed in `01-surgical-rct-limitations.md` are cited by PMID only and are not repeated in the tables.
|
||||
|
||||
**Data sources for "TTE" in this draft.** TTE can run on three kinds of data, and they differ on exactly the problems in this chapter:
|
||||
- **Claims** (Medicare FFS): the operator NPI, CPT/HCPCS codes (vessel territory; angioplasty vs stent vs atherectomy), place of service, and objective outcomes (amputation, death, repeat procedures). There is no lesion anatomy, no device brand, no technical success, and no patient-reported outcomes.
|
||||
- **Clinical registries** (e.g., the Vascular Quality Initiative, VQI): de-identified physician and centre identifiers, lesion characteristics, devices, and technical details. Some outcomes are clinician-reported. Centres choose to join.
|
||||
- **Registry–claims linkage** (VQI–Medicare, "VISION"): registry procedure detail plus long-term claims follow-up. Mao 2022 (PMID 35181518) matched 93.0% of VQI patients to Medicare with 99.4% accuracy.
|
||||
|
||||
---
|
||||
|
||||
## 1. Learning curve
|
||||
|
||||
**What it is.** Surgeons get better at a procedure the more often they do it, so outcomes depend on where on their learning curve the operator is. If a trial starts while a new procedure is still early on its curve, the new procedure looks worse than it will be in practice. If the trial waits until surgeons are proficient, equipoise may already be gone (Papachristofi 2016). Learning curves are measured and reported inconsistently.
|
||||
|
||||
**How common / how big.**
|
||||
- A review of 592 learning-curve studies in minimally invasive abdominal surgery found that 86% used operative time as the proxy and only 6% used patient-oriented outcomes (Harrysson 2014).
|
||||
- An umbrella review of 62 reviews (1,990 studies) gave a definition of the learning curve in only 63% (so 37% gave none, by subtraction), 712 different data items were used, and 60 of the 62 reviews were of low or critically low quality (Ramirez 2026).
|
||||
- For radical prostatectomy, the learning curve did not plateau until about 250 prior operations. Predicted 5-year recurrence was 17.9% after 10 prior cases and 10.7% after 250 (Vickers 2007).
|
||||
- TAVR in the TVT registry: from the 1st to the 400th case, risk-adjusted mortality fell from 3.57% to 2.15% and bleeding from 9.56% to 5.08% (Carroll 2017).
|
||||
- Only 1 of 247 surgical RCTs analysed learning, and 2% used an expertise-based design (Conroy 2019; see Section 5).
|
||||
|
||||
**Vascular example.**
|
||||
- **Carotid stenting in Medicare** (Nallamothu 2011): 1,792 new operators started after the CMS coverage decision. 30-day mortality was 2.3% for a new operator's 1st–11th procedures vs 1.4% from the 12th on (aOR 1.7, 1.2–2.4). This is a learning curve measured in claims through the operator identifier.
|
||||
- **Endovascular device trials** handle the learning curve with "roll-in" patients who are excluded from the randomized analysis. The IntraCoil femoropopliteal trial used 93 roll-in patients across 22 sites (Ansel 2002).
|
||||
|
||||
| Paper | PMID | Takeaway (numbers from abstract) |
|
||||
|---|---|---|
|
||||
| Harrysson IJ et al. 2014, *Ann Surg* | 24670849 | 592 learning-curve studies. Operative time was the proxy in 86%, postoperative outcomes in 52%, patient-oriented outcomes in 6%. No increase over time in the complexity of analysis. |
|
||||
| Ramirez J et al. 2026, *Surg Innov* (umbrella review) | 42527319 | 62 reviews / 1,990 studies. A definition was given in 63%, surgeon expertise reported in 63%, 712 unique data items, and 60/62 reviews were low or critically low quality. |
|
||||
| Papachristofi O et al. 2016, *Trials* | 27245050 | Two-phase model that estimates initial skill, learning rate, final skill, and **duration of learning**. Frames the "when to run the definitive RCT" dilemma. |
|
||||
| Biau DJ et al. 2008, *Br J Surg* (LC-CUSUM) | 18498126 | Individual learning-curve monitoring. ERCP success rose from 82.0% (first 100) to 96.1% (last 129). Competence signalled at case 79. |
|
||||
| Vickers AJ et al. 2007, *JNCI* | 17652279 | 7,765 prostatectomies by 72 surgeons. Plateau at about 250 cases. 5-year recurrence 17.9% vs 10.7% (10 vs 250 prior cases). |
|
||||
| **Nallamothu BK et al. 2011, *JAMA*** | 21954477 | Medicare CAS, 24,701 procedures, 2,339 operators. Median operator volume was 3.0/year. Early vs late experience: 30-day mortality 2.3% vs 1.4% (aOR 1.7). Very-low vs high volume: aOR 1.9. |
|
||||
| Carroll JD et al. 2017, *JACC* (TVT Registry) | 28662805 | 42,988 TAVRs at 395 hospitals. Risk-adjusted mortality 3.57% → 2.15% from case 1 to 400. Vascular and bleeding risk higher in the first 100 cases. |
|
||||
| Ansel GM et al. 2002, *Catheter Cardiovasc Interv* | 12124949 | Femoropopliteal stent trial. 93 roll-in ("learning phase") patients across 22 sites, reported separately from the randomized comparison. |
|
||||
| Luft HS et al. 1987, *Health Serv Res* | 3112042 | Volume–outcome links reflect both "practice makes perfect" and "selective referral" (better surgeons attract more patients). The relative importance varies by procedure. |
|
||||
|
||||
Already in 01: Cook 2004 (16279280), Ramsay 2000/2001 (11155830, 11319991).
|
||||
|
||||
**Can TTE solve it? Partly.**
|
||||
- **Claims.** TTE does not remove the learning curve, but it changes which question is answered: it estimates the effect *as delivered* across the real mix of operator experience, not at a chosen point on the curve.
|
||||
- **Claims and registries.** Cumulative operator volume can be built from the operator identifier (NPI in claims, physician ID in VQI) and used as a time-varying covariate or effect modifier (Nallamothu 2011 shows this is feasible). Caveats:
|
||||
- Medicare FFS volume is only part of an operator's total volume.
|
||||
- Volume is left-truncated at the start of the data.
|
||||
- Volume partly reflects selective referral (Luft 1987).
|
||||
- **Registries only.** These add technical success and procedural detail, which are closer to direct measures of proficiency.
|
||||
- **What TTE cannot do** is target "the effect once everyone is proficient" unless proficient operators can be identified and positivity holds among them.
|
||||
|
||||
**Testable in a known-truth simulation? Yes.** Let operator skill rise with cumulative PVI volume (e.g., skill_j(t) = s_j + λ·log(1 + V_j(t))). This world shows three things that are specific to TTE:
|
||||
1. **Interference.** One patient's early PVI raises the operator's skill for later patients, which violates no-interference/SUTVA. The individual-level TTE contrast then differs from the effect of a population policy of "early PVI for all", which would also change volumes.
|
||||
2. **Strategy effects and skill are entangled.** In the delayed arm, PVIs happen later, when operators are more experienced. Part of any "benefit of waiting" is then skill gain, not timing. The simulation can split the two by comparing truths with λ = 0 and λ > 0.
|
||||
3. **Time-varying confounding by physician.** High-preference physicians build volume faster, so skill becomes correlated with preference *and* changes over time. A physician fixed effect cannot remove this; adjusting for cumulative volume might. The simulation can check whether a time-varying volume covariate recovers the truth.
|
||||
|
||||
---
|
||||
|
||||
## 2. Surgeon skill, expertise, and volume–outcome
|
||||
|
||||
**What it is.** Surgeons differ in technical skill. In a conventional RCT, each surgeon does both procedures, often with unequal expertise ("differential expertise bias"; Devereaux 2005, in 01). Volume is a crude, widely used proxy for skill, and it does not separate skill from referral patterns.
|
||||
|
||||
**How common / how big.**
|
||||
- **Video-rated skill.** In a meta-analysis of 7 studies (151 surgeons, 17,932 procedures), top- vs bottom-quartile skill was associated with less reoperation (OR 0.44) and fewer medical complications (OR 0.23) (Woods 2023). Peer-rated skill explained 25.8% of the variation in post-colectomy complications (Stulberg 2020).
|
||||
- **Volume.** A systematic review found that 71% of hospital-volume and 69% of physician-volume studies reported significant volume–outcome associations. AAA surgery was among the strongest, at a median of 3.3–13 excess deaths per 100 cases across the strongest conditions (Halm 2002).
|
||||
- **Reporting of expertise in trials.** Only 34.4% of surgical interventions in RCTs came with any data on operator expertise (Blencowe 2015, BJS; Section 4).
|
||||
|
||||
**Vascular example.**
|
||||
- **PVI.** In the NCDR PVI registry (555 operators), high- vs low-volume operators had lower in-hospital MALE (OR 0.73) and MACE (OR 0.64), while hospital volume was not associated (Chunawala 2025).
|
||||
- **AAA.** Elective AAA repair: OR 0.66 for mortality at high-volume hospitals (Holt 2007).
|
||||
- **Carotid stenting.** In pooled trial data, operator *in-trial* annual volume mattered (≤3.2/year: aRR 2.30 vs >5.6/year) but lifetime experience did not (Calvet 2014).
|
||||
|
||||
| Paper | PMID | Takeaway (numbers from abstract) |
|
||||
|---|---|---|
|
||||
| Stulberg JJ et al. 2020, *JAMA Surg* | 32838425 | 17 surgeons, video-rated skill 2.8–4.6 out of 5. Higher skill: complications 15.5% vs 20.6%. Skill explained 25.8% of the variation in complications. |
|
||||
| Woods MS et al. 2023, *JSLS* (meta-analysis) | 36818767 | 7 studies, 151 surgeons. Top vs bottom skill quartile: reoperation OR 0.44, any medical complication OR 0.23. Mortality not significant. Few studies exist. |
|
||||
| Halm EA et al. 2002, *Ann Intern Med* (systematic review) | 12230353 | 71% / 69% of hospital / physician volume studies were significant. Effect is larger for AAA and pancreatic surgery, and smaller for CEA and angioplasty. Clinical risk adjustment weakens associations. |
|
||||
| Holt PJ et al. 2007, *Br J Surg* (meta-analysis) | 17380547 | 421,299 elective AAA repairs. High-volume OR 0.66 (threshold 43/year). Ruptured: OR 0.78. |
|
||||
| Chunawala Z et al. 2025, *JAHA* (NCDR PVI) | 41147410 | 55,785 PVIs, 97 hospitals, 555 operators. Operator volume: MALE OR 0.73, MACE OR 0.64. Hospital volume: no significant association. |
|
||||
| Setogawa N et al. 2025, *J Endovasc Ther* (Japan) | 37675778 | Hospital EVT volume was not associated with 12-month amputation (Q4 vs Q1 OR 1.00), but was associated with death (OR 0.71). |
|
||||
| Calvet D et al. 2014, *Stroke* (CSTC pooled RCTs) | 24347422 | 1,546 CAS patients in EVA-3S/SPACE/ICSS. 30-day stroke or death 10.1% (low in-trial volume) vs 5.1% (high). aRR 2.30. Lifetime experience had no effect. |
|
||||
| Bednarska E et al. 2008, *Clin Orthop Relat Res* | 18446421 | Surgeon survey (n = 276): 53.4% willing to join an expertise-based RCT vs 18.3% a conventional RCT. |
|
||||
| Cook JA et al. 2018, *Trials* (qualitative) | 30189868 | Surgeons and methodologists: expertise-based designs help participation, but surgeons in each arm may differ systematically. Patient transfer between surgeons is a practical problem. |
|
||||
|
||||
Already in 01: Devereaux 2005 (15637373), Cook 2015 (26025450), Birkmeyer 2013 (24106936), Birkmeyer 2003 (14645640).
|
||||
|
||||
**Can TTE solve it? Partly.**
|
||||
- **What TTE gains.** It needs no "both-procedures" surgeon. Each operator does what they normally do, which resembles an expertise-based design: early-PVI operators and deferring operators are compared as they practise. This also means **skill is bundled with strategy and physician**.
|
||||
- **Claims.** These contain no skill measure. Volume and specialty are proxies, and the NPI enables random or fixed effects.
|
||||
- **Registries** add technical success and lesion complexity, which partly measure skill. Video-based skill is not available in any routine source.
|
||||
- **Remaining risk.** If skill is correlated with intervention preference, confounding by physician remains, unless within-physician comparisons are possible (see 02, B6).
|
||||
|
||||
**Testable in a known-truth simulation? Yes.** Give physicians a latent skill correlated with preference. Compare three estimands: (i) the population as-practised effect (what TTE targets), (ii) the effect if every PVI were done by top-skill operators (an "expert trial"), and (iii) the effect under differential expertise.
|
||||
|
||||
---
|
||||
|
||||
## 3. Selection of surgeons and centres into trials (generalizability)
|
||||
|
||||
**What it is.** Trials enrol centres and surgeons who are interested, have high volume, and are often credentialed with explicit audit thresholds. Their results can overstate the benefit, or understate the harm, of a procedure in routine practice. The classic case is carotid endarterectomy after NASCET/ACAS.
|
||||
|
||||
**How common / how big.**
|
||||
- **Medicare CEA, 1992–93** (Wennberg 1998). Mortality was 1.4% at the 86 NASCET/ACAS trial hospitals vs 1.7%, 1.9%, and 2.5% at high-, average-, and low-volume non-trial hospitals. These were all above the trials' own 0.6% (NASCET) and 0.1% (ACAS).
|
||||
- **Hospitals that take part in trials** (acute coronary syndromes) had lower in-hospital mortality: 5.9% at non-participants vs 3.5% at high enrollers (Majumdar 2008).
|
||||
- **Counterpoint.** A Cochrane review found no consistent difference in outcomes between patients inside vs outside RCTs (63 of 73 comparisons not significant; Vist 2007). So the gap reflects *who delivers* care more than trial participation itself.
|
||||
- **Not all settings show a gap.** Population-based CEA in Olmsted County (Hallett 1998) and a statewide Kentucky registry (Mayo 1998; stroke plus death 2.8%) roughly matched trial results. However, Mayo's registry covered only 58% of the state's procedures and 10 of 17 hospitals.
|
||||
|
||||
**Vascular example.**
|
||||
- **CEA:** Wennberg 1998 (above).
|
||||
- **Carotid stenting:** trial thresholds vs practice. Calvet 2014 recommends ≥6 CAS/year, while the median Medicare operator did 3.0/year (Nallamothu 2011).
|
||||
- **EVAR:** in Medicare (Schermerhorn 2008), the early advantage of EVAR (perioperative mortality 1.2% vs 4.8%) carried into practice, showing that population data can confirm the direction of trial results.
|
||||
|
||||
| Paper | PMID | Takeaway (numbers from abstract) |
|
||||
|---|---|---|
|
||||
| **Wennberg DE et al. 1998, *JAMA*** | 9565008 | 113,300 Medicare CEAs. Trial hospitals 1.4% vs non-trial 1.7–2.5% mortality (trend p < .001). All were above the trials' 0.6%/0.1%. "Caution … translating efficacy … to effectiveness". |
|
||||
| Hallett JW et al. 1998, *J Vasc Surg* | 9620136 | Olmsted County, all CEAs 1970–95 (n = 297 patients). Freedom from ipsilateral stroke at 2 years was 97% vs NASCET 91%. Higher 30-day stroke rate in patients over 80. |
|
||||
| Mayo SW et al. 1998, *J Vasc Surg* | 9652463 | Kentucky statewide registry: stroke plus death 2.8%, comparable to NASCET/ACAS. Covered 58% of CEAs. Surgeon stroke rates ranged 0–10%. |
|
||||
| Vist GE et al. 2007, *Cochrane* | 17443630 | 55 studies, 31,140 trial vs 20,380 non-trial patients. 63 of 73 comparisons not significant. Participation itself is not associated with better or worse outcomes. |
|
||||
| Majumdar SR et al. 2008, *Arch Intern Med* | 18362259 | Non-ST-elevation acute coronary syndromes (non-surgical). In-hospital mortality 5.9% / 4.4% / 3.5% by trial-enrolment tertile. High-enrolling hospitals aOR 0.8. |
|
||||
| Schermerhorn ML et al. 2008, *NEJM* | 18234751 | 22,830 matched pairs in Medicare. EVAR vs open perioperative mortality 1.2% vs 4.8%. By 4 years: rupture 1.8% vs 0.5% and AAA reintervention 9.0% vs 1.7%. Practice data add the late harms that short trials miss. |
|
||||
| Mao J et al. 2022, *J Vasc Surg* | 35181518 | VQI–Medicare indirect linkage: 93.0% matched, 99.4% accuracy. Registry procedure detail plus claims follow-up. |
|
||||
|
||||
Already in 01: Rothwell 2005/2006 (15639683, 16871331), Cutteridge 2025 (40105904), Jones 2026 (42526742).
|
||||
|
||||
**Can TTE solve it? Yes for claims, partly for registries.**
|
||||
- **Claims.** Medicare FFS includes essentially every operator who bills Medicare, so no surgeons or centres are selected in. The TTE estimand is the effect as practised, which is what policy needs.
|
||||
- **VQI.** Centres choose to join, and they tend to be engaged in quality improvement. Registry TTEs therefore carry a milder version of the trial's centre selection. VISION linkage helps with follow-up but not with which centres take part.
|
||||
- **What TTE cannot give** is the "ideal-operator" effect, unless operators are restricted to high-volume ones (a transportability or subgroup analysis).
|
||||
|
||||
**Testable in a known-truth simulation? Yes.** Run a simulated RCT restricted to the top skill/volume quintile of physicians and compare its estimate with the population truth and with TTE on everyone. The gap is the generalizability bias, and TTE should close it.
|
||||
|
||||
---
|
||||
|
||||
## 4. Standardization, quality control, technique variation, device heterogeneity
|
||||
|
||||
**What it is.** A surgical "treatment" is a bundle of steps, devices, and peri-operative care. Trials must decide how much to standardize. Too little, and nobody knows what was tested; too much, and the result may not transfer to practice. In endovascular trials, "PVI" can mean plain balloon, stent, drug-coated device, or atherectomy, in different vessel territories, and these may not share one effect.
|
||||
|
||||
**How common / how big.**
|
||||
- Of 160 interventions in surgical RCTs, standardization was mentioned for only 29.4%, and only 28% of articles measured adherence (Blencowe 2015).
|
||||
- In placebo-controlled surgical trials, only 7% attempted to standardize and 4% monitored delivery (Cousins 2020; Section 6).
|
||||
- In arteriovenous-graft RCTs, quality assurance had "not improved over the past three decades", and no trial assessed technical performance (Kingsmore 2025).
|
||||
|
||||
**Vascular / claudication example.**
|
||||
- **Tibial PVI for claudication.** In Medicare 2017–19, 27.7% of index claudication PVIs included tibial treatment. The physician-level rate had a median of 20.0% (IQR 9.1–37.5%) and was associated with early-career status (aOR 2.97) and office-based practice (aOR 2.37) (Bose 2023). Infrapopliteal vs femoropopliteal PVI had a higher risk of CLTI (aHR 1.31) and amputation (aHR 1.72) (Bose 2025). The "intervention" is therefore technically heterogeneous, and the heterogeneity varies by physician.
|
||||
- **Devices.** A network meta-analysis of femoropopliteal devices found that technical success and restenosis differed substantially by device class (Katsanos 2014).
|
||||
|
||||
| Paper | PMID | Takeaway (numbers from abstract) |
|
||||
|---|---|---|
|
||||
| **Blencowe NS et al. 2015, *Br J Surg*** | 26041565 | 80 surgical RCTs / 160 interventions. Described 80.6%, standardization mentioned 29.4%, adherence measured in 28% of articles, operator expertise reported for 34.4%. Pragmatic and explanatory trials were similar. |
|
||||
| Blencowe NS et al. 2016, *Br J Surg* | 27462835 | Typology for standardizing surgical interventions: purpose → components and steps → monitoring of deviations. |
|
||||
| Butterworth JW et al. 2021, *Eur J Surg Oncol* | 33059943 | 34 oncology studies. Quality-assurance challenges fall into credentialing (volume is a weak criterion), standardization (variation between centres), and monitoring. |
|
||||
| Kingsmore DB et al. 2025, *J Vasc Access* | 38501338 | Vascular-access (AV graft) RCTs: quality assurance highly variable, no pre-trial monitoring, none assessed technical performance. |
|
||||
| Katsanos K et al. 2014, *J Vasc Surg* (network meta-analysis) | 24661896 | 16 femoropopliteal RCTs. Technical success OR 13.6 for covered stents vs balloon. Target lesion revascularization RR 0.36 for paclitaxel-coated balloons. Device classes are not interchangeable. |
|
||||
| **Bose S et al. 2023, *J Vasc Surg*** | 36058433 | Medicare claudication PVI (n = 59,930; 1,542 physicians). Tibial PVI 27.7%. Physician-level rate IQR 9.1–37.5%. Early career aOR 2.97, office-based aOR 2.37. |
|
||||
| Bose S et al. 2025, *Eur J Vasc Endovasc Surg* | 38906366 | Medicare claudication PVI (n = 36,147). Infrapopliteal vs femoropopliteal: CLTI aHR 1.31, repeat PVI aHR 1.12, amputation aHR 1.72. |
|
||||
| Sansosti AA et al. 2024, *J Vasc Surg* (VQI) | 38944400 | Femoropopliteal atherectomy. Embolic protection used in 40% (hospital) vs 4.4% (office). Office setting had more reintervention at 2 years. Technique varies by setting. |
|
||||
|
||||
Already in 01: Boutron 2017 (28630973), Garas 2018 (29110271), Jones 2026 (42526742).
|
||||
|
||||
**Can TTE solve it? Partly.**
|
||||
- **TTE cannot standardize anything.** The "early PVI" strategy is whatever PVI was actually done.
|
||||
- **Claims.** CPT codes record vessel territory and modality (angioplasty, stent, atherectomy) and the setting, so the strategy can be defined more finely, or modality treated as a version of treatment. Device brand, drug coating (only partly, via HCPCS pass-through codes), lesion length, and technical success are not recorded.
|
||||
- **Registries** (VQI) record lesion characteristics, devices, and technical details, so technique variation is largely measurable there, and "versions of treatment" can be modelled explicitly.
|
||||
- **The open question** is whether a single "early PVI" effect is meaningful when the technique mix differs by physician. This is a consistency (well-defined intervention) problem.
|
||||
|
||||
**Testable in a known-truth simulation? Yes.** Give each physician a probability of a high-risk technique (e.g., tibial PVI) with its own effect. Then compare a TTE that ignores versions (estimating the effect of the as-practised mixture) with one stratified on modality.
|
||||
|
||||
---
|
||||
|
||||
## 5. Clustering of outcomes by surgeon
|
||||
|
||||
**What it is.** Patients treated by the same surgeon or centre have correlated outcomes, even under individual randomization. Ignoring this understates the standard errors and, when provider effects are tied to volume or treatment, can bias the effect estimate. Few trials handle it.
|
||||
|
||||
**How common / how big.**
|
||||
- **Trials rarely adjust.** Of 68 large interventional RCTs, 16% reported the number of providers, 9% stratified randomization by provider, and 4% adjusted the analysis for provider (Biau 2008, *JCE*).
|
||||
- **The picture is little better ten years later.** Of 247 surgical RCTs, 40% of trials stratified by surgeon did not adjust the analysis for surgeon (Conroy 2019).
|
||||
- **Re-analysis changes results.** Re-analysing three surgical RCTs found significant provider effects. In the hip-replacement trial, the treatment effect *became* significant after adjusting for provider and volume (Biau 2008, *Ann Surg*).
|
||||
- **Size of clustering.** Cook 2012 reported ICCs for 198 outcomes from 10 trials and concluded that clustering is "more of an issue than has been previously acknowledged" (no single pooled ICC in the abstract).
|
||||
|
||||
**Vascular example.** Physician-level variation in claudication PVI technique is wide (tibial PVI rate IQR 9.1–37.5%; Bose 2023), and the analysis used hierarchical models. In NCDR, 55,785 PVIs were performed by 555 operators, an average of about 100 per operator (Chunawala 2025). Clustering in claims PVI data is therefore substantial in both technique and volume.
|
||||
|
||||
| Paper | PMID | Takeaway (numbers from abstract) |
|
||||
|---|---|---|
|
||||
| **Biau DJ et al. 2008, *J Clin Epidemiol*** | 18394535 | 68 multicentre interventional RCTs (n ≥ 200). Number of providers reported 16%. Stratified by provider 9%. Analysis adjusted for provider 4% and for centre 6%. |
|
||||
| **Biau DJ et al. 2008, *Ann Surg*** | 18438129 | Patient-level re-analysis of 3 RCTs (22–88 providers each). Significant provider effects (ophthalmology p < .0001; orthopaedics p = .0037). Treatment × volume interaction. Adjustment changed the conclusion. |
|
||||
| Cook JA et al. 2012, *Trials* | 22217216 | ICCs for 198 outcomes in 10 surgical trials. Median cluster size 32 (centre) and 6 (surgeon). ICCs varied widely by outcome type. |
|
||||
| **Conroy EJ et al. 2019, *J Clin Epidemiol*** | 30439544 | 247 surgical RCTs. Expertise-based design 2%, credentials predefined 39%, only 1 trial analysed learning. One-third of centre-stratified and 40% of surgeon-stratified trials did not adjust the analysis. |
|
||||
| Conroy EJ et al. 2019, *J Clin Epidemiol* (funded applications) | 31121302 | 50 funded UK trials. 60% of surgeon-stratified trials planned to adjust. One-third of referee responses involved funder-driven changes on learning or clustering. |
|
||||
| Conroy EJ et al. 2022, *Trials* (guidance review) | 36221107 | 28 guidance documents. Clustering by centre is covered well, by surgeon poorly. Analysis methods for clustering in 5 documents and for learning in 4. No single guidance document exists. |
|
||||
|
||||
Already in 01: Lee & Thompson 2005 (15649931, 16279138), Roberts 2005 (16279137), Walters 2010 (20492055). Analysis methods in 02 B6 (Panageas, Kahan & Morris, within-physician designs).
|
||||
|
||||
**Can TTE solve it? Yes for inference, partly for confounding.**
|
||||
- **Claims and registries** both have operator identifiers (NPI; VQI physician ID), so cluster-robust SEs, random effects, physician fixed effects, or within-physician designs are standard.
|
||||
- **Clustering is larger in TTE than in a trial**, because physicians choose the strategy. Physician-level clustering therefore becomes **confounding by physician** (preference correlated with outcome style), not just a precision problem (see 02 B6).
|
||||
- **Within-physician comparisons** need physicians who sometimes intervene early and sometimes defer.
|
||||
|
||||
**Testable in a known-truth simulation? Yes.** Random intercepts and random slopes by physician, with the intercept correlated with intervention preference. Check (a) the coverage of naive vs cluster-robust vs mixed-model SEs and (b) bias with and without within-physician adjustment.
|
||||
|
||||
---
|
||||
|
||||
## 6. Blinding, sham control, and outcome-assessment (detection) bias
|
||||
|
||||
**What it is.**
|
||||
- Surgeons cannot be blinded, patients rarely are, and outcome assessors often are not.
|
||||
- Unblinded subjective outcomes (pain, symptom scores, self-limited walking distance) are prone to expectation and observer bias.
|
||||
- Sham-controlled trials of invasive cardiovascular procedures have repeatedly shrunk effects seen in open-label trials.
|
||||
- Observational data have their own version of the problem. Outcomes are recorded when patients are looked for (surveillance bias), and diagnoses are coded by the treating physician.
|
||||
|
||||
**How common / how big.**
|
||||
- **Non-blinded assessors exaggerate effects.** By 36% for subjective binary outcomes (Hróbjartsson 2012) and by 68% for measurement-scale outcomes (Hróbjartsson 2013). In an updated 2025 analysis the figure was 29% overall and larger in non-drug trials (ratio of odds ratios 0.62; Salazar 2025).
|
||||
- **The bias is in subjective outcomes, not objective ones.** In a meta-epidemiological study, lack of blinding biased subjective outcomes (ROR 0.75) but not objective ones (ROR 1.01) (Wood 2008).
|
||||
- **Blinding is under-used in surgery.** In 378 surgical RCTs only 24.0% of patients were blinded. Failure to blind surgeons where it was feasible was associated with more significant results (OR 13.7; very wide CI 1.3–142.8) (Probst 2019).
|
||||
|
||||
**Vascular / claudication example.**
|
||||
- **Sham analogue for claudication.** The closest is ORBITA, which tested revascularization for an exertional symptom against a sham procedure. PCI did not improve exercise time beyond placebo (16.6 s, −8.9 to 42.0). ORBITA-2, in patients off antianginals, did show a symptom benefit (OR 2.21).
|
||||
- **Renal denervation.** SYMPLICITY HTN-3 found that the large blood-pressure falls in unblinded studies largely disappeared against sham (difference −2.39 mmHg, not significant).
|
||||
- **Claudication trials.** Claudication revascularization trials (ERASE, CLEVER, IRONIC, SWEDEPAD 2; in 01) use unblinded walking distance or quality of life.
|
||||
- **Claims outcomes.** The usual claims endpoints (amputation, death, repeat PVI) are objective in Wood's sense. Even so:
|
||||
- ICD-10 coding distinguishes claudication from CLTI with only 81.3% agreement against VQI (Bose 2025).
|
||||
- About half of PVI patients get no surveillance imaging in year 1 (Brooke 2018), so restenosis and repeat PVI depend on who looks.
|
||||
|
||||
| Paper | PMID | Takeaway (numbers from abstract) |
|
||||
|---|---|---|
|
||||
| **Hróbjartsson A et al. 2012, *BMJ*** | 22371859 | 21 trials with both blinded and non-blinded assessors. Non-blinded assessors exaggerated the OR by 36% (ROR 0.64), driven by misclassifying a median of only 3% of patients. |
|
||||
| Hróbjartsson A et al. 2013, *CMAJ* | 23359047 | Measurement-scale outcomes, 16 trials. Non-blinded assessors exaggerated effect size by 68% (95% CI 14–230%). |
|
||||
| Salazar J et al. 2025, *J Clin Epidemiol* (update) | 40258524 | 66 trials. Pooled ROR 0.71, about a 29% exaggeration. Larger in non-drug trials (ROR 0.62) and industry-funded trials (0.57). |
|
||||
| **Wood L et al. 2008, *BMJ*** | 18316340 | Lack of blinding: ROR 0.75 for subjective outcomes vs 1.01 for objective outcomes. Little bias for all-cause mortality. |
|
||||
| Probst P et al. 2019, *Langenbecks Arch Surg* | 30824993 | 378 surgical RCTs. 24.0% of patients blinded. Assessor blinding was possible in 69%. Not blinding feasible surgeons was associated with significant results (OR 13.7). |
|
||||
| Cousins S et al. 2020, *J Clin Epidemiol* | 31786153 | 96 placebo-controlled surgical RCTs. 68% randomized fewer than 100. 34% gave no rationale for the placebo. Standardization 7%, monitoring 4%. |
|
||||
| **Al-Lamee R et al. 2018, *Lancet* (ORBITA)** | 29103656 | PCI vs sham in stable angina, n = 200. Exercise-time difference 16.6 s (−8.9 to 42.0), not significant. |
|
||||
| Rajkumar CA et al. 2023, *NEJM* (ORBITA-2) | 38015442 | n = 301, off antianginals. Angina score 2.9 vs 5.6 (OR 2.21). A sham can show *real* benefit. |
|
||||
| Bhatt DL et al. 2014, *NEJM* (SYMPLICITY HTN-3) | 24678939 | n = 535. Systolic BP −14.1 vs −11.7 mmHg with sham. Difference −2.39 (−6.89 to 2.12). |
|
||||
| **Bose S et al. 2025, *Circ Cardiovasc Qual Outcomes*** | 40439664 | VQI-VISION, n = 54,180 PVIs. ICD-10 claudication vs CLTI: sensitivity 80.9%, specificity 81.9%, agreement 81.3%. Machine learning raised the AUC to 0.892. |
|
||||
| Bilimoria KY et al. 2013, *JAMA* | 24100354 | Surveillance bias: hospital VTE rates rose from 5.0 to 13.5 per 1,000 across quartiles of imaging use. More looking, more events found. |
|
||||
| Brooke BS et al. 2018, *JAMA Surg* | 29049809 | Medicare. Only 52–58% of patients had imaging follow-up within 1 year after bypass or PVI. Joining VQI did not change this (RR about 0.99). |
|
||||
| Jacob-Brassard J et al. 2021, *Clin Invest Med* (Ontario) | 34152705 | Administrative PAD revascularization codes: PPV 94.3%, or 98.1% with physician claims. Procedures are coded accurately. |
|
||||
|
||||
Already in 01: Wartolowska 2014/2016 (24850821, 27008687), Boutron 2017 (28630973).
|
||||
|
||||
**Can TTE solve it? No for blinding, partly for outcome bias.**
|
||||
- **No blinding.** TTE cannot blind anyone.
|
||||
- **Claims.** These have no patient-reported outcomes or walking distance, so the subjective endpoints most open to bias are simply not available. The objective claims endpoints (death, major amputation) are the ones Wood 2008 found free of blinding bias. Physician-recorded endpoints remain at risk:
|
||||
- CLTI diagnosis codes (81% agreement with the registry).
|
||||
- Repeat PVI, which is itself a physician decision.
|
||||
- Restenosis, which depends on surveillance (Brooke 2018).
|
||||
- **Registries** record walking or symptom status and patency, but these are clinician-reported and unblinded. Registries add detail without solving detection bias.
|
||||
- **Practical rule for the paper:** make death and major amputation primary; treat CLTI and repeat PVI as at risk of detection or decision bias.
|
||||
|
||||
**Testable in a known-truth simulation? Yes, optionally.** Make the probability that CLTI is *recorded* (or that repeat PVI is triggered) depend on prior PVI or physician style (differential ascertainment), and keep amputation and death recorded perfectly. TTE should be unbiased for the objective outcomes and biased for the ascertainment-dependent ones. Blinding itself has no counterpart in the simulation.
|
||||
|
||||
---
|
||||
|
||||
## Table 1 inputs (one line each)
|
||||
|
||||
| Challenge | Evidence it is common | Vascular example | TTE: claims | TTE: registry (VQI/VISION) | Simulation |
|
||||
|---|---|---|---|---|---|
|
||||
| Learning curve | Plateau at about 250 cases (Vickers); 1/247 RCTs analysed learning (Conroy) | CAS Medicare early-experience aOR 1.7 (Nallamothu) | Partly: cumulative NPI volume | Partly: plus technical success | Yes. Interference, delayed arm treated by more skilled operators, time-varying physician confounding |
|
||||
| Skill and expertise | Skill explains 25.8% of complication variation (Stulberg) | PVI operator volume MALE OR 0.73 (Chunawala) | Partly: proxies only | Partly | Yes: latent skill × preference |
|
||||
| Surgeon and centre selection | CEA trial vs non-trial hospitals 1.4% vs up to 2.5% (Wennberg) | CEA, CAS volume thresholds | Yes: all operators | Partly: centres self-select | Yes: "expert-only RCT" vs population |
|
||||
| Standardization and devices | 29.4% of trial interventions standardized (Blencowe) | Tibial PVI physician IQR 9–38% (Bose 2023) | Partly: CPT modality and territory | Mostly: lesion and device data | Yes: versions of treatment |
|
||||
| Clustering | 4% of RCTs adjusted for provider (Biau) | 555 operators in NCDR PVI | Yes (inference); confounding remains | Yes | Yes: random intercepts and slopes |
|
||||
| Blinding and detection | Non-blinded assessors +36% / +68% (Hróbjartsson) | ORBITA-type sham; CLTI code agreement 81% | No blinding; objective endpoints are fine | No; clinician-reported | Optional: differential ascertainment |
|
||||
|
||||
---
|
||||
|
||||
## Summary: most important new papers
|
||||
1. **Wennberg 1998 *JAMA* (9565008):** the canonical trial vs non-trial centre gap for CEA in Medicare. It anchors the argument that claims-based TTE estimates the effect as practised.
|
||||
2. **Nallamothu 2011 *JAMA* (21954477):** a learning curve and volume effect measured *in Medicare claims* through operator identifiers, for an endovascular procedure. It shows that operator experience can be modelled in claims-based TTE.
|
||||
3. **Biau 2008 *JCE* / *Ann Surg* (18394535, 18438129) and Conroy 2019 *JCE* (30439544):** surgical RCTs almost never account for provider effects or learning, and doing so can change conclusions.
|
||||
4. **Hróbjartsson 2012 *BMJ* (22371859) and Wood 2008 *BMJ* (18316340):** unblinded subjective outcomes are inflated by about 30–70%, while objective outcomes are not. This justifies TTE using death and amputation and is cautious about walking distance and CLTI coding.
|
||||
5. **Bose 2023 *JVS* (36058433) and Bose 2025 *CCQO* (40439664):** in claudication PVI, technique varies widely by physician, and the claims coding of CLTI is about 81% accurate against VQI. These define the "versions of treatment" and outcome-misclassification limits for claims vs registry TTE.
|
||||
@@ -0,0 +1,330 @@
|
||||
# Draft C: treatment-pathway challenges, claudication trials, and evidence on TTE itself
|
||||
|
||||
Every PMID below was checked with NCBI E-utilities (esummary) on 2026-10-28. Every number comes from the PubMed abstract fetched with efetch; full texts were not read. If a paper has no PubMed abstract, the entry says so and gives no numbers. Papers already in `01-surgical-rct-limitations.md` or `02-claudication-evidence-and-methods.md` are cited by PMID only and are not listed again as new entries, apart from short reminders in the claudication table.
|
||||
|
||||
"TTE" here covers emulation in **claims** (e.g., Medicare), **EHR** (e.g., VA, health-system data), and **clinical registries** (e.g., Vascular Quality Initiative [VQI], and VQI linked to Medicare as VQI-VISION). Where the data source changes the answer, the "Can TTE solve it?" paragraphs say so.
|
||||
|
||||
---
|
||||
|
||||
# Part 1. Treatment-pathway challenges
|
||||
|
||||
## Challenge 1. Crossover and contamination
|
||||
|
||||
**What it is.** Patients receive the other arm's treatment. In timing trials and conservative-vs-surgery trials, crossover runs mostly one way: the conservative or "watchful waiting" arm later has the operation. The intention-to-treat (ITT) contrast then compares "operate now" with "operate later if needed", not with "never operate", and the difference shrinks toward the null. Naive per-protocol or as-treated analyses (excluding or reclassifying crossovers) remove the dilution but break randomization, because the patients who cross over are the ones who are getting worse.
|
||||
|
||||
**How common / how big.**
|
||||
- Crossover from conservative care to surgery is the norm, not the exception: 39% by about 19 weeks for sciatica, 30% by 6 months for meniscal tear, 43% by 2 years for spinal stenosis, 27–29% within 90 days to 1 year for appendicitis, and two-thirds by 10 years for hernia.
|
||||
- In cardiac surgery RCTs the median crossover was 7.0% from experimental to control. Trials with more of this crossover had RRs nearer the null (1.01 vs 0.80).
|
||||
- STICH is the clearest case where crossover changed the conclusion. The ITT result was not significant at 5 years (HR 0.86, P = 0.12). Per-protocol and time-dependent analyses favored CABG. Benefit was significant by ITT only after 10 years of follow-up.
|
||||
- Per-protocol estimates were on average only 2% larger than ITT across 156 RCTs in general medical journals. The gap grows with non-adherence, and surgical timing trials sit at the high end of non-adherence.
|
||||
|
||||
**Vascular / claudication example.**
|
||||
- SUPER (iliac disease): 29% of the SET arm had endovascular revascularization within 1 year, and 49% had an additional intervention by 5.5 years.
|
||||
- Swedish SFA stenting trial: 25% of the control arm had crossed over to stenting by 60 months. The authors say this "affected the power of the final analysis".
|
||||
- ERASE: the exercise arm had a higher hazard of later revascularization (HR 2.50).
|
||||
- ACST-1 (carotid): the "deferral" arm still had CEA at 4% per year.
|
||||
- Already listed elsewhere: ADAM 61.6% (12000813) and CAESAR 84.5% at 54 months (20869890).
|
||||
|
||||
| Paper | PMID | Takeaway |
|
||||
|---|---|---|
|
||||
| Peul WC et al. 2007, *NEJM* (sciatica: early surgery vs prolonged conservative care) | 17538084 | 89% of the early-surgery arm had surgery (mean 2.2 weeks). 39% of the conservative arm had surgery (mean 18.7 weeks). Disability at 1 year was no different by ITT, but recovery was faster with early surgery (HR 1.97). |
|
||||
| Katz JN et al. 2013, *NEJM* (METEOR) | 23506518 | **30%** of the physical-therapy arm had surgery within 6 months, and 6% of the surgery arm did not have surgery. No ITT difference. |
|
||||
| Weinstein JN et al. 2008, *NEJM* (SPORT stenosis) | 18287602 | At 2 years, 67% of the surgery arm and **43% of the nonsurgical arm** had surgery. ITT was significant for only 1 of 3 primary outcomes. The as-treated analysis (adjusted, combined with the observational cohort) favored surgery on all of them. |
|
||||
| CODA Collaborative 2020, *NEJM* | 33017106 | 29% of the antibiotics arm had appendectomy by 90 days (41% of those with an appendicolith). |
|
||||
| Salminen P et al. 2015, *JAMA* (APPAC) | 26080338 | 27.3% of the antibiotics arm had appendectomy within 1 year. |
|
||||
| van de Wall BJM et al. 2017, *Lancet Gastroenterol Hepatol* (DIRECT) | 28404008 | Stopped early for poor recruitment (n = 109). 23% of the conservative arm went on to elective resection. |
|
||||
| Velazquez EJ et al. 2011, *NEJM* (STICH) | 21463150 | 17% of the medical-therapy arm had CABG by a median of 56 months, and 91% of the CABG arm had CABG. ITT on all-cause death: HR 0.86 (0.72–1.04). |
|
||||
| Velazquez EJ et al. 2016, *NEJM* (STICHES) | 27040723 | At a median of 9.8 years, ITT HR 0.84 (0.73–0.97). The benefit appeared only with long follow-up. |
|
||||
| Doenst T et al. 2013, *Circ Heart Fail* (STICH crossover) | 23515275 | Crossover in year 1: 10.8% from MED to CABG (driven by worsening symptoms) and 9.0% from CABG to MED (mainly patient or family decision). Per-protocol and time-dependent analyses favored CABG. Crossover "diminished the impact of CABG" in the ITT analysis. |
|
||||
| **Gaudino M et al. 2019, *JAHA*** | 31663420 | 47 cardiac surgery RCTs. Median crossover was 7.0% from experimental to control and 1.3% from control to experimental. Experimental-to-control crossover was associated with effect estimates nearer the null. |
|
||||
| Reistrup H et al. 2021, *Hernia* (systematic review) | 32910297 | Watchful waiting for hernia: about one-third crossed over by 3 years and more than two-thirds by 10 years, mainly for pain. |
|
||||
| Halliday A et al. 2004, *Lancet* (ACST-1) | 15135594 | Immediate-CEA arm: half had CEA by 1 month and 88% by 1 year. Deferral arm: 4% per year had CEA. The authors note that full compliance would have produced "slightly bigger differences". |
|
||||
| Hernán MA, Hernández-Díaz S 2012, *Clin Trials* | 21948059 | ITT does not guarantee an estimate of clinical effectiveness. With differential adherence, ITT can even overestimate. Adherence-adjusted per-protocol analyses need IPW or g-estimation and time-varying data. |
|
||||
| Murray EJ, Hernán MA 2016, *Clin Trials* | 26951361 | Coronary Drug Project: placebo adherers vs non-adherers differed by 9.4 percentage points in 5-year mortality. With modern adjustment (IPW for post-randomization covariates) the gap shrank to 2.5 (−2.1 to 7.0). Valid per-protocol estimation is possible. |
|
||||
| Mostazir M et al. 2021, *J Clin Epidemiol* | 34161805 | Across 156 RCTs, per-protocol estimates were 2% larger than ITT (ratio of ORs 1.02), and the gap grew with non-adherence. |
|
||||
| Peduzzi P et al. 1993, *Stat Med* (VA CABG trial) | 8210821 | Compares ITT with four naive ways of handling non-adherers (exclude, transfer at randomization, censor, transfer at switch) and describes the bias in each. |
|
||||
| Morden JP et al. 2011, *BMC Med Res Methodol* | 21223539 | Simulation study (16 scenarios, 1,000 datasets each). Simple switching methods are biased when switchers' prognosis differs. Adjusted (AFT-type) methods do better. |
|
||||
| Latimer NR et al. 2014, *Med Decis Making* | 24449433 | IPCW relies on no unmeasured confounding of switching. RPSFTM relies on a common treatment effect. Both can be biased, and ITT is sometimes the least biased. |
|
||||
|
||||
**Can TTE solve it? Yes, in principle; this is TTE's strongest case.** TTE defines the strategies explicitly ("PVI within 6 months" vs "no PVI for 6 months, then usual care", or "no PVI unless CLTI"). Anyone who deviates is artificially censored, and IPCW corrects for the selection this causes (Hernán 2006, 16611197; Hernán 2018, 29419381). This estimates the per-protocol effect that the RCTs above could only approximate with naive as-treated analyses. The price is that the weights are valid only if the *triggers* for crossover are measured. In timing trials the main trigger is symptom progression (STICH: "progressive symptoms"; hernia: pain).
|
||||
- **Claims** do not record symptom severity, walking distance or ABI, so the weights miss the main reason patients cross over.
|
||||
- **EHR and VQI** may record ABI, Rutherford class and symptoms. But VQI records them only at the time of a procedure, so they are missing for patients who stay conservative.
|
||||
- The same IPCW-with-measured-triggers assumption sinks trial switching adjustments (Latimer 2014). TTE does not escape it.
|
||||
|
||||
**Testable in a known-truth simulation? Yes.** Generate deferred PVI driven by a latent progression variable, then vary whether that variable is observed (EHR-like) or not (claims-like), and compare ITT-analogue, naive as-treated, and CCW estimates against the true per-protocol effect.
|
||||
|
||||
---
|
||||
|
||||
## Challenge 2. Non-adherence and refusal of the assigned surgery (and of SET)
|
||||
|
||||
**What it is.** Some patients assigned to surgery never have it: they decline, become unfit, or die waiting. Some patients assigned to a conservative package never start it or drop out. In claudication the conservative package is supervised exercise, which needs about 3 sessions a week for 12 weeks, so adherence is a major problem. Refusal of the assigned operation differs from crossover in both cause and direction: it is often driven by patient preference or frailty, not by disease progression.
|
||||
|
||||
**How common / how big.**
|
||||
- **Refusal or delay of the assigned operation:** ACST-1 managed CEA within 1 month in only half of the immediate-CEA arm. In STICH, 9.0% of the CABG arm received medical therapy only, mainly because of patient or family decision, and that group had higher 5-year mortality (59% vs 33%). SPORT disc herniation: only 50% of the surgery arm had surgery (01 list, 17119140).
|
||||
- **SET has low uptake and adherence:**
|
||||
- Across 23 reports that gave screening numbers, only **24.2%** of screened claudicants were recruited to supervised exercise (Harwood 2016).
|
||||
- In a health system with a strong SET program, 773 of 5,320 PAD patients were referred, 415 enrolled, and **49.9%** of those completed (Whipple 2024).
|
||||
- Of 51 Chicago PAD patients who had started SET, **9.8% completed 12 weeks** (Cetlin 2023).
|
||||
|
||||
**Vascular / claudication example.**
|
||||
- SUPER: compliance with SET was **50%** at 6 months, and 8% of the endovascular arm never received the intervention.
|
||||
- CLEVER: only 79 of 111 randomized patients completed the 18-month assessment (attrition, not refusal).
|
||||
- In US practice, 85.1% of surveyed PAD patients said SET had never been prescribed (Cetlin 2023), and 49% of surveyed vascular physicians had never referred a patient (Dua 2020).
|
||||
|
||||
| Paper | PMID | Takeaway |
|
||||
|---|---|---|
|
||||
| Doenst T et al. 2013, *Circ Heart Fail* | 23515275 | CABG-to-MED crossover was mostly patient or family choice. These patients had worse prognosis (5-year mortality 59% vs 33%), so they were not a random subset. |
|
||||
| Halliday A et al. 2004, *Lancet* (ACST-1) | 15135594 | Even in the "immediate" arm, only half had CEA by 1 month. "Immediate" in practice meant a distribution of delays. |
|
||||
| Koelemay MJW et al. 2022, *EJVES* (SUPER) | 35151572 | SET compliance was 50% at 6 months. 8% of the ER arm did not get ER. |
|
||||
| **Harwood AE et al. 2016, *Ann Vasc Surg*** | 27126713 | Systematic review: 24.2% of 7,517 screened claudicants were recruited to supervised exercise, and 75.1% of 4,012 completed it. Only one article defined a minimum attendance. About 1 in 3 screened patients was suitable and willing. |
|
||||
| Whipple MO et al. 2024, *Vasc Med* | 38084864 | One US health system, 2017–2022: of 5,320 PAD patients, 773 were referred to SET, 415 enrolled, and 49.9% completed. |
|
||||
| Cetlin MD et al. 2023, *J Vasc Surg* | 36150636 | Of 489 people with PAD, 85.1% had never had SET prescribed. Of those willing to travel, 59.9% were unwilling or unable to pay the $11 copay per session. Of 51 who started, 9.8% completed 12 weeks. |
|
||||
| Dua A et al. 2020, *J Vasc Surg* | 31699514 | Survey of 135 physicians: 54% had no SET program at their facility, 49% had never referred a patient, and 26% did not know CMS covered SET. |
|
||||
| Kahan BC et al. 2024, *BMJ* (estimands primer) | 38262663 | The ICH E9(R1) estimands framework, worked through on an emergency bowel surgery trial. It names how "intercurrent events" (refusal, switching) are handled: the treatment-policy (ITT) estimand vs the hypothetical (per-protocol) estimand. |
|
||||
|
||||
**Can TTE solve it? Partly.** TTE avoids the *consent* step, so trial refusal does not exist. But refusing an *offered* operation still happens in routine care.
|
||||
- **Claims** cannot separate "offered surgery and declined" from "never offered", so the comparator arm absorbs refusers, who may be frailer or more risk-averse. This is unmeasured confounding.
|
||||
- **EHR** notes sometimes record refusal or shared-decision documentation.
|
||||
- **Adherence to SET** is partly visible. SET has been a Medicare-covered, billable service since 2017 (Dua 2020), so claims can count sessions. Home exercise and walking advice cannot be seen in any of these data sources.
|
||||
- **Adherence to the surgery itself** is easy to observe in all three sources: a procedure either happens or it does not.
|
||||
|
||||
**Testable in a known-truth simulation? Yes.** Add a latent "refuser" trait that lowers the chance of PVI and independently raises the risk of the outcome. A claims-like scenario leaves it unobserved; an EHR-like scenario observes it with misclassification.
|
||||
|
||||
---
|
||||
|
||||
## Challenge 3. Co-interventions and heterogeneous usual care in the comparator arm
|
||||
|
||||
**What it is.** The outcome depends on everything else patients receive, not just the index procedure: statins, antiplatelets, smoking cessation, supervised or home exercise, cilostazol, and surveillance. In claudication trials the "conservative" arm ranges from "aspirin plus advice" to "12 weeks of supervised treadmill training plus best medical therapy". Usual care is undefined by design and varies between sites, so the comparator is a moving target. Co-interventions can also differ *by arm* because nobody is blinded: patients who had a procedure get more follow-up visits and more statin prescriptions.
|
||||
|
||||
**How common / how big.**
|
||||
- Usual-care comparators are reported very poorly. In 67 knee-OA exercise RCTs, the median TIDieR score was 1 of 12 for the usual-care arm vs 4 for the exercise arm.
|
||||
- Exercise arms in claudication RCTs scored on average 10 of 19 on CERT, with adherence and fidelity the least reported items.
|
||||
- In practice, only **38.1%** of VQI claudicants undergoing elective revascularization had complete optimal medical therapy (antiplatelet, statin, non-smoking). Complete OMT was associated with lower 1-year reintervention (aOR 0.72).
|
||||
- Preoperative statin-plus-antiplatelet use in VQI held at about 62% from 2013 to 2023.
|
||||
|
||||
**Vascular / claudication example: the comparator differs from trial to trial.**
|
||||
- Whyman: aspirin plus advice on smoking and exercise.
|
||||
- CLEVER: "optimal medical care" (claudication pharmacotherapy), with or without SE.
|
||||
- IRONIC: BMT plus a structured *non-supervised* exercise program (Djerf 2018).
|
||||
- MIMIC: SE plus BMT in both arms.
|
||||
- Cochrane 2018: "conservative" meant supervised exercise in some trials and cilostazol in others.
|
||||
- A meta-analysis therefore pools different comparators.
|
||||
|
||||
| Paper | PMID | Takeaway |
|
||||
|---|---|---|
|
||||
| Thompson BT, Schoenfeld D 2007, *Proc Am Thorac Soc* | 17878473 | Usual care is "the full spectrum of patient care practices". Whether to use it as a control depends on the question and on how uniform usual care is. |
|
||||
| Freedland KE et al. 2011, *Psychosom Med* | 21536837 | Extrinsic, non-study care has "important but under-recognized" effects on trials with usual-care control groups. |
|
||||
| Ribeiro da Silva Neto J et al. 2026, *Musculoskeletal Care* | 42366407 | 67 RCTs: median TIDieR score 1 for the usual-care arm vs 4 for the exercise arm. No improvement over time. |
|
||||
| Ibeggazene S et al. 2023, *Vascular* | 35130092 | 73 claudication exercise RCTs: mean CERT 10/19. Adherence and fidelity were the least reported items. |
|
||||
| Fakhry F et al. 2018, *Cochrane* | 29518253 | Across 10 RCTs (1,087 patients), "conservative" meant supervised exercise or cilostazol. Effects differed by comparator: ER vs no specific treatment MWD SMD 0.70; ER vs SET −0.42 (not significant). |
|
||||
| **Jarosinski MC et al. 2025, *J Vasc Surg* (VQI)** | 39208918 | Of 39,088 claudication revascularizations: 38.1% complete OMT, 56.4% partial, 5.4% none. Complete OMT was associated with lower reintervention (aOR 0.72) and MALE (aOR 0.70). |
|
||||
| Ramkumar N et al. 2026, *J Vasc Surg* (VQI) | 42248488 | Of 257,129 patients, preop statin plus antiplatelet use held at 62%. Postop use rose from 68% to 81%. Lower in women (OR 0.88). |
|
||||
| Jansen SCP et al. 2020, *EJVES* (Dutch national claims) | 32888779 | 54,504 new IC patients. Primary SET rose from 63% to 87% (2013–2017). **Claims captured SET because it is reimbursed there.** Primary ER was associated with more secondary revascularizations (HR 1.44) and higher mortality (HR 1.38). These are observational associations. |
|
||||
| Djerf H et al. 2018, *Br J Surg* (IRONIC cost) | 30357819 | Describes the IRONIC control arm as BMT "including a structured, non-supervised exercise programme". |
|
||||
| Divakaran S et al. 2021, *Circ Cardiovasc Qual Outcomes*; 2024, *Vasc Med* | 34293930; 39129558 | Medicare SET participation 2017–2018, and "underutilization" of SET in Medicare. **No PubMed abstract (research letters), so no numbers are given here.** |
|
||||
|
||||
**Can TTE solve it? Partly.** TTE cannot randomize co-interventions. It can either (a) build them into the strategy definition ("PVI plus statin" vs "no PVI plus statin") or (b) treat them as time-varying confounders or mediators. Both need the co-intervention to be measured.
|
||||
- **Claims.** Part D captures statin and P2Y12 fills, but not OTC aspirin. Medicare SET is billable from 2017. Smoking is captured poorly, and home exercise not at all.
|
||||
- **EHR.** May record SET referral, smoking status, ABI and symptom class.
|
||||
- **VQI.** Records OMT and smoking only *at the time of a procedure* (Jarosinski 2025). Patients who are never revascularized are not in the registry, so **VQI alone cannot form a "no PVI" comparator arm**. Linking it to claims or EHR is required.
|
||||
- **A known advantage over RCTs.** The usual-care comparator in an emulation is real-world usual care, so it is representative. It is just not standardized.
|
||||
|
||||
**Testable in a known-truth simulation? Yes.** Add an unmeasured exercise or OMT variable that differs by arm (more OMT after PVI) and affects outcomes. Compare scenarios where it is observed (EHR/Part D) and unobserved (exercise in claims).
|
||||
|
||||
---
|
||||
|
||||
## Challenge 4. Defining the comparator: "conservative" vs "delayed surgery" vs "never surgery"
|
||||
|
||||
**What it is.** "Conservative management" is a *dynamic strategy*: start without the operation, and operate if and when a trigger occurs (symptoms, patient request, progression). It is not "never operate". Trials label it "watchful waiting", "deferral", "surveillance", "initial conservative", or "prolonged conservative treatment with surgery if needed". These labels name different estimands. A TTE has the same problem in sharper form, because the analyst has to write the strategy down: the trigger, the grace period, and what happens after the grace period.
|
||||
|
||||
**How common / how big.**
|
||||
- Nearly every trial in Challenge 1 used a "surgery-if-needed" comparator, and 23–68% of the comparator arm had the operation.
|
||||
- ISCHEMIA explicitly defined its comparator as "medical therapy alone and angiography if medical therapy failed".
|
||||
- The same data analysed as a CCW emulation vs a sequential emulation can answer different questions with different numbers. Example (mAb timing): CCW 4-year survival difference 4.3% vs sequential −2.8%. A naive design gave an implausible RR of 0.13.
|
||||
|
||||
**Vascular / claudication example.**
|
||||
- IRONIC and the Gothenburg trial (Nordanstig 2011) allowed patients in the noninvasive arm to "request invasive treatment if they deteriorated".
|
||||
- SUPER: the authors suggest it is "reasonable to start with SET … and accept a 30% failure rate".
|
||||
- In PVI-for-claudication terms, three candidate comparators answer different questions:
|
||||
- "no PVI within 6 months" (a deferral estimand);
|
||||
- "no PVI unless CLTI develops" (a progression-triggered dynamic strategy);
|
||||
- "never PVI" (an abstention estimand, hard to emulate because positivity fails once CLTI appears).
|
||||
|
||||
| Paper | PMID | Takeaway |
|
||||
|---|---|---|
|
||||
| Maron DJ et al. 2020, *NEJM* (ISCHEMIA) | 32227755 | Comparator defined as "medical therapy alone and angiography if medical therapy failed". Early harm from the invasive arm (5.3% vs 3.4% at 6 months) crossed over to a non-significant later benefit (16.4% vs 18.2% at 5 years). The timing of effects depends on the strategy definition. |
|
||||
| Halliday A et al. 2004, *Lancet* (ACST-1) | 15135594 | Comparator was "indefinite deferral of any CEA", which still produced CEA at 4% per year. |
|
||||
| Nordanstig J et al. 2011, *EJVES* | 21397530 | n = 201, invasive vs noninvasive. Noninvasive patients could request invasive treatment if they deteriorated. 7% of the invasive arm crossed over. |
|
||||
| Hernán MA et al. 2006, *Basic Clin Pharmacol Toxicol* | 16611197 | Comparing dynamic regimes by artificial censoring plus IPW. Valid if all joint predictors of survival and of deviating from the regime are measured. |
|
||||
| Young JG et al. 2011, *Stat Biosci* | 24039638 | Parametric g-formula for "when to start" dynamic regimes, with measured time-varying confounders affected by earlier treatment. |
|
||||
| Kahan BC et al. 2024, *BMJ* | 38262663 | The estimands language (treatment-policy vs hypothetical strategies) for writing down what "conservative" means. |
|
||||
| Garcia-Albeniz X et al. 2026, *Cancer Med* | 42321955 | "When to start" emulation: aligning eligibility with time zero, CCW vs sequential trials (4-year differences 4.3% vs −2.8%). Biased designs gave a 6-month RR of 0.13. |
|
||||
| Ahn J et al. 2026, *JAMA Netw Open* (registry) | 42832200 | Surgical timing TTE: early MV surgery (≤6 months) vs watchful waiting, CCW with a 6-month grace period, n = 802. RMST gain of 66 days at 5 years; HR 0.69. Shows the grace-period comparator in a surgical setting. |
|
||||
| Cheung LC et al. 2026, *Ann Intern Med* (Kaiser EHR) | 42330497 | "Delayed" was explicitly defined as "after 6 months or no" treatment. Immediate excision gave more unnecessary excisions (36.2% vs 7.8%) and similar 3-year cancer risk. |
|
||||
| Murray J et al. 2025, *J Clin Epidemiol* (TAA cohort linked to HES) | 39929323 | Repair within 12 months vs no repair within 12 months (CCW), with competing risks. 7-year survival 57.4% vs 49.9%. |
|
||||
| Yoshiyama T et al. 2026, *Clin Orthop Relat Res* (Japanese administrative data) | 42821763 | Early (≤2 days) vs delayed (3–10 days) surgery by CCW, explicitly to remove the immortal time in "delayed" groups. |
|
||||
|
||||
**Can TTE solve it? Yes, and this is where TTE adds most.** A trial chooses one comparator, and crossover then blurs it. A TTE can emulate *several* comparators from the same data: deferral (no PVI for 6 months), progression-triggered PVI, and, where positivity allows, abstention. It reports each as a separate, named estimand.
|
||||
- **Triggers must be measurable.** "PVI only after CLTI" is emulable in claims because CLTI has codes; ICD-10 discriminates claudication from CLTI with 81.3% agreement (Bose 2025). "PVI only if walking distance deteriorates" needs EHR or registry symptom data.
|
||||
- **"Never PVI" is hard to emulate.** It may lack positivity once CLTI develops.
|
||||
|
||||
**Testable in a known-truth simulation? Yes, and central.** The DGP can produce the true risk under each named comparator by intervening on the simulated data, so landmark, CCW and sequential estimates can each be scored against their own truth (02, B7).
|
||||
|
||||
---
|
||||
|
||||
# Part 2. RCTs of revascularization vs exercise or medical therapy for intermittent claudication
|
||||
|
||||
All rows come from abstracts. "NR" means not reported in the abstract (it may be in the full text). **EXITPAD could not be verified:** searches for "EXITPAD" and "EXIT-PAD" in PubMed returned no trial report or protocol, so it is left out.
|
||||
|
||||
| Trial (first report) | PMID(s) | n planned → randomized | Comparison | Primary endpoint | Follow-up | Crossover / non-adherence | Stopped early? | Hard limb outcome reported? |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| Lundgren 1989, *Ann Surg* | 2647051 | NR → 75 | Surgery vs surgery + training vs training | Walking performance | ~13 months | NR | NR | No |
|
||||
| Creasy 1990, *Eur J Vasc Surg* (Oxford) | 2140987 | NR → 36 (20 PTA, 16 exercise) | PTA vs supervised exercise | ABPI and treadmill distances | 12 months (preliminary) | 2 PTA patients later had surgery. 1 exercise patient later had PTA. | NR | No |
|
||||
| Perkins 1996, *EJVES* (Oxford long term) | 8846172 | NR → 56 | PTA vs exercise training | ABPI, treadmill distances | 15 months and ~6 years (37 reviewed) | NR | NR | 2 amputations (4%) in the whole cohort. Not compared by arm. |
|
||||
| Whyman 1996 / 1997, *EJVES* / *JVS* (Edinburgh) | 8760978; 9357454 | 600 screened → 62 | PTA + medical (aspirin, advice) vs medical | Walking, ABPI, QoL, duplex | 6 months; 2 years | NR | NR | No. Only 10% of claudicants had lesions suitable for PTA. |
|
||||
| Gelin 2001 / Taft 2001, *EJVES* (Gothenburg) | 11472042; 11472043 | NR → 264 (HRQL paper: 253) | Invasive (open or endovascular) vs supervised exercise vs observation | Maximum walking power; HRQL | 1 year | NR. HRQL data at 1 year available for 171. | NR | No |
|
||||
| OBACT, Nylaende 2007, *EJVES* | 17055756 | 434 considered → 56 | OMT + PTA vs OMT | QoL | 2 years | NR | NR | Mortality was a secondary measure. No limb outcome. |
|
||||
| MIMIC, Greenhalgh 2008, *EJVES* | 19022184 | NR → 93 femoropopliteal + 34 aortoiliac | PTA vs no PTA, on top of SE + BMT | ICD / AWD on treadmill | 24 months | NR | NR | No (ABPI only) |
|
||||
| Spronk 2009, *Radiology*; long term Fakhry 2013, *BJS* | 19188327; 23842830 | NR → 151 | Endovascular vs hospital-based SET | Rutherford-based clinical success | 12 months; ~7 years | Secondary interventions significantly higher after SET (P = 0.001). Total interventions still higher after ER. | NR | No (secondary interventions only) |
|
||||
| Nordanstig 2011, *EJVES* (Gothenburg INV vs NON) | 21397530 | NR → 201 | Invasive (endovascular and/or surgical) vs noninvasive. NON could request invasive treatment. | Maximal walking performance | 2 years | 7% of the INV arm crossed over | NR | No |
|
||||
| CLEVER, Murphy 2012, *Circulation*; 2015, *JACC* | 22090168; 25766947 (design 18440181; update 19660897) | **252 planned (design) → 217 planned (update) → 111** | OMC vs OMC + SE vs OMC + stent (aortoiliac) | Change in peak walking time at 6 months | 6 and 18 months (79 completed at 18 months) | NR | Fewer patients than planned. Abstract does not say whether it was formally stopped. | No |
|
||||
| Mazari 2012 / 2017, *BJS* (Hull) | 22021102; 27763685 | NR → 178 | PTA vs SEP vs PTA + SEP (femoropopliteal) | Clinical and QoL outcomes | 12 months; median 5.2 years | Withdrawals at 1 year: 8 PTA, 14 SEP, 11 combined | NR | 18 (16.2%) had a major cardiovascular event. Reinterventions: PTA 29, SEP 17, combined 9. No amputation data in abstract. |
|
||||
| IRONIC, Nordanstig 2014, *Circulation*; Djerf 2020, *Circ CV Interv* | 25095886; 31937137 | NR → 158 | Revascularization + BMT + structured exercise vs BMT + structured exercise | SF-36 at 1 year | 1, 2, 5 years | NR in abstract | No | No. Benefit lost by 5 years; cost about 2× ($13,098 vs $6,965). |
|
||||
| ERASE, Fakhry 2015, *JAMA*; Klaphake 2022, *Ann Surg* | 26547465; 33378308 | NR → 212 | ER + SET vs SET | MWD at 12 months (+282 m) | 12 months; median 5.4 years | SET arm had higher hazard of later revascularization (HR 2.50). Total procedures 65 (SET) vs 149 (combined). | No | Progression to CLI was a secondary endpoint at long-term follow-up. **The result is not given in the abstract.** |
|
||||
| Swedish SFA stenting trial, Lindgren 2017, *EJVES*; Gunnarsson 2023, *EJVES* | 28372983; 36642399 (cost 34454817) | NR → 100 | Primary nitinol stent + BMT vs BMT (SFA) | HRQoL (SF-36, EQ-5D) at 12 months | 12, 24, 36, 60 months | **25% control-to-stent crossover by 60 months** | No | **Yes:** no differences in CLTI progression, amputation (2.1% vs 1.9%) or mortality (14.6% vs 15.4%). |
|
||||
| SUPER, Koelemay 2022, *EJVES* | 35151572 (design 22326696) | **400 planned → 240** | SET vs ER (iliac) | MWD and VascuQol at 1 year | 1 year; mean 5.5 years | **29% of the SET arm had ER within 1 year.** SET compliance 50%. 8% of the ER arm did not get ER. Additional interventions by 5.5 years: 49% (SET) vs 27% (ER). | **Yes, terminated prematurely** | No (interventions only) |
|
||||
| Cochrane, Fakhry 2018 | 29518253 | 10 RCTs, 1,087 participants | ER vs no treatment, SET, or cilostazol | MWD, PFWD, QoL | Short and long term | — | — | No hard limb outcomes were synthesized. Secondary invasive interventions were reported (e.g., ER + SET vs SET: OR 0.27). |
|
||||
|
||||
**Reading across the table.**
|
||||
1. No trial was designed or powered for CLTI, amputation or death. Only the Swedish SFA trial reports CLTI or amputation by arm in its abstract, with about 2% amputation per arm at 5 years in 100 patients.
|
||||
2. Under-recruitment is common:
|
||||
- CLEVER 252 → 111;
|
||||
- SUPER 400 → 240 (stopped);
|
||||
- OBACT 56 of 434;
|
||||
- Whyman 62 of 600.
|
||||
3. Crossover from the conservative arm to revascularization is 25–49% wherever it is reported (SUPER, Swedish SFA, ERASE by hazard ratio).
|
||||
4. "Conservative" means different things across rows (see Challenge 3).
|
||||
|
||||
---
|
||||
|
||||
# Part 3. Evidence on TTE itself
|
||||
|
||||
## 5. How well does TTE or observational analysis reproduce RCT results?
|
||||
|
||||
| Paper | PMID | Data source | Agreement metrics (from abstract) |
|
||||
|---|---|---|---|
|
||||
| **Wang SV et al. 2023, *JAMA* (RCT-DUPLICATE, 32 trials)** | 37097356 (in 01) | US claims (Optum, MarketScan, Medicare) | Overall Pearson r **0.82** (0.64–0.91). 75% agreed on statistical significance, 66% on the estimate, 75% on standardized difference. **Close emulations (16): r 0.93**, with 94% / 88% / 88% agreement. **Emulations that could not closely match PICOT (16): r 0.53**, with 56% / 50% / 69%. All drug trials, deliberately selected for feasibility. |
|
||||
| Franklin JM et al. 2021, *Circulation* (first 10 of RCT-DUPLICATE) | 33327727 | US claims | Regulatory conclusion agreed in 6/10. HR inside the RCT 95% CI in 8/10. Standardized difference <2 in 9/10. Worst agreement where a sulfonylurea was used as a stand-in for placebo. |
|
||||
| **Wang C et al. 2026, *BMJ* (systematic review and meta-analysis)** | 42156120 | Mixed (claims, EHR, registries) | 107 TTE–RCT pairs: r **0.59** (0.45–0.70). Standardized difference agreement 79%. Ratio of ratios 0.96. **Closer emulations (63): r 0.83**, 87% agreement. **Claims-based emulations underestimated effects (ratio of ratios 0.90, 0.82–0.99).** Worse agreement with imbalanced baselines, treatment started in hospital, and poor outcome emulation. |
|
||||
| Dahabreh IJ et al. 2020, *Epidemiology* | 32740470 | Commentary | "Benchmarking observational methods by comparing randomized trials and their emulations." **No PubMed abstract, so no numbers.** |
|
||||
| Matthews AA et al. 2022, *Am J Epidemiol* (TASTE benchmark) | 35641151 | Swedish registries (SWEDEHEART) | Procedure (thrombus aspiration). The emulation reproduced the null 1-year trial result (RD 0.7 for death, −0.2 for MI), then extended follow-up to 3 years and to under-represented subgroups. Benchmark first, then extend. |
|
||||
| Westerberg M et al. 2024, *J Clin Epidemiol* | 39159770 | Swedish prostate cancer registry | Prostatectomy vs radiotherapy (treatment chosen by surgeon and patient). Crude HR 1.94. After Charlson adjustment 1.32. After richer comorbidity indices **1.14 (0.91–1.44), in line with the RCT.** Shows how strong confounding by indication is for procedures and how much depends on comorbidity measurement. |
|
||||
| Forbes SP et al. 2020, *J Gen Intern Med* | 32193818 | Published PS-based studies | 127 observational-vs-RCT comparisons, **49 in surgery**. Large disagreement (relative RR <0.7 or >1.43) in **54%**. These were not explicit emulations, and the authors call for target-trial-based benchmarking. |
|
||||
| Soni PD et al. 2019, *J Clin Oncol* | 30897037 | SEER, SEER-Medicare, NCDB | 350 comparisons vs 121 RCTs. **Concordance correlation 0.083**. 40% agreed. No modifiable factor (PS, IV, sensitivity analyses) improved agreement. Not TTE: conventional registry analyses. |
|
||||
| Kumar A et al. 2020, *JAMA Netw Open* | 32729921 | NCDB (cancer registry) | 141 RCTs emulated by eligibility. HRs concordant in 64% of PS models, P values concordant in 45%. No trial characteristic (including type of intervention) predicted concordance. |
|
||||
| Keyhani S et al. 2020, *JAMA Neurol* | 32478802 | VA EHR and other VA databases | **Vascular surgery vs initial medical therapy (asymptomatic carotid).** 5-year stroke risk 5.6% vs 7.8% (RD −2.3%). Accounting for competing death: RD −0.8% (not significant). The RCT-eligible subset gave similar numbers. The authors read this as a benefit smaller than in the trials of 20 years earlier, not as a failed replication. |
|
||||
| Koeppe J et al. 2026, *Sci Rep* | 42711453 | German insurance claims | Replication across insurers (not vs RCT) of a surgical comparison (RTSA vs locked plating for humeral fracture): 7/11 endpoints replicated, Pearson r 0.97. |
|
||||
|
||||
**What this says for surgery.**
|
||||
- **Benchmarks for procedures are scarce.** Every large benchmark of explicit emulations is mostly drug trials (RCT-DUPLICATE, Wang 2026). The procedural benchmarks found are single examples: thrombus aspiration, prostatectomy vs radiotherapy, and carotid endarterectomy against older trials.
|
||||
- **Conventional registry analyses of procedures agree poorly with RCTs.** Agreement was near zero in Soni 2019 and only 45–64% in Kumar 2020, and in Forbes 2020 more than half of the 127 comparisons (49 in surgery) disagreed by a large margin.
|
||||
- **Agreement depends on emulating closely, not on the statistical method.** Wang 2023 shows this for claims and Wang 2026 for mixed sources. In Wang 2026, claims-based emulations also tended to underestimate the trial effect (ratio of ratios 0.90).
|
||||
- **Data-source caveat:** Kumar 2020, Soni 2019 and Westerberg 2024 used cancer registries, which have clinical stage but poor comorbidity data. Westerberg shows that better comorbidity measurement moved the estimate most of the way to the RCT.
|
||||
|
||||
## 6. Known limitations of TTE, guidance, and reviews of how it is used
|
||||
|
||||
### 6a. Guidance and frameworks (core references, PMIDs verified)
|
||||
|
||||
| Paper | PMID | Takeaway |
|
||||
|---|---|---|
|
||||
| Hernán MA, Robins JM 2016, *AJE* | 26994063 (in 01) | Foundational TTE paper. |
|
||||
| Hernán MA et al. 2016, *J Clin Epidemiol* | 27237061 (in 01) | Aligning eligibility, assignment and time zero prevents immortal time bias. |
|
||||
| Hernán MA et al. 2022, *JAMA* (JAMA Guide) | 36508210 (in 01) | Framework for causal inference from observational data. |
|
||||
| Matthews AA et al. 2022, *BMJ* | 36041749 | "Target trial emulation: applying principles of randomised trials to observational studies." **No PubMed abstract.** |
|
||||
| Cashin AG et al. 2025, *JAMA* / *BMJ* (TARGET); Hansford 2025, *PLoS Med* (E&E) | 40899949; 40903028; 41091811 (in 01 and 02) | Reporting guideline. |
|
||||
| Messinger CJ et al. 2025, *Anesthesiology* | 40067038 | Perioperative and critical care primer: which biases the framework does and does not avoid. |
|
||||
| Wong KHF et al. 2025, *Br J Surg* | 40893042 | "Harnessing real-world data to evaluate surgery and perioperative care interventions." **No PubMed abstract.** |
|
||||
| Wanis KN et al. 2024, *JRSS-A* | 39281783 (in 02) | Natural vs stochastic grace-period strategies are different estimands. |
|
||||
| VanderWeele TJ, Ding P 2017, *Ann Intern Med* | 28693043 | The E-value: how strong an unmeasured confounder would need to be to explain away an estimate. |
|
||||
| Lipsitch M et al. 2010, *Epidemiology* | 20335814 | Negative control exposures and outcomes for detecting confounding. |
|
||||
|
||||
### 6b. Reviews of how TTE is actually done (quality)
|
||||
|
||||
| Paper | PMID | Takeaway |
|
||||
|---|---|---|
|
||||
| Hansford HJ et al. 2023, *JAMA Netw Open* | 37755828 | 200 TTEs (2012–2022): 43% did not describe all key emulation items, and 57% did not describe the target-trial protocol. |
|
||||
| **Ren Y et al. 2026, *JAMA Netw Open*** | 41712213 | 237 TTEs in high-impact journals: 56.5% had a prespecified protocol. 15.2% used post-baseline information for eligibility. 16.9% gave a diagram defining time zero. **Only 30.8% addressed unmeasured confounding.** |
|
||||
| McIntyre M et al. 2025, *J Clin Epidemiol* | 40639692 | 49% of 181 TTEs had misaligned eligibility, time zero and assignment. Of those, 67% did nothing about it, although 78% of these could have fixed it cheaply. |
|
||||
| Manzanilla A et al. 2026, *J Clin Epidemiol* | 42600711 | 113 CCW studies: **19.5% still had potential immortal time bias**, mostly from excluding patients on post-time-zero criteria. |
|
||||
| Scola G et al. 2023, *BMC Med Res Methodol* | 37587484 (in 01) | Only 16% of TTEs used cloning to handle immortal time. |
|
||||
|
||||
**Surgery- and vascular-specific reviews:** none found that systematically appraises TTE in surgery or vascular surgery (this repeats the gap noted in 01). Wong 2025 *BJS* and Messinger 2025 are narrative primers, and Hoskins 2026 (01) and Hatano 2026 *Int Orthop* (42287372; abstract not read) are orthopaedic primers.
|
||||
|
||||
### 6c. Limitations by data source (claims vs EHR vs registry), with vascular evidence
|
||||
|
||||
| Paper | PMID | Data source | Takeaway |
|
||||
|---|---|---|---|
|
||||
| **Bose S et al. 2025, *Circ Cardiovasc Qual Outcomes*** | 40439664 | VQI-VISION (VQI linked to Medicare) | Using VQI as the gold standard, predefined ICD-10 codes separated claudication from CLTI with 80.9% sensitivity, 81.9% specificity and **81.3% agreement**. Machine learning raised agreement to 84.2%. **About 1 in 5 claims is misclassified on the key indication or outcome.** |
|
||||
| Appah-Sampong A et al. 2025, *Vasc Med* | 40340584 | Review of EHR and claims PAD studies | 84 studies: **76.2% used a single-code query** to identify PAD, despite its low yield. Only 5 used regression-based phenotypes. |
|
||||
| **Columbo JA et al. 2019, *J Vasc Surg*** | 29914838 | VQI vs VQI-Medicare vs chart review (EVAR) | **VQI alone found 12 of 30 one-year reinterventions.** VQI-Medicare found all 30 (κ 0.93 vs chart review). At 3 years, sensitivity 92% and specificity 96%. Registry follow-up alone undercounts reinterventions, and linking to claims fixes most of it. |
|
||||
| Goodney P et al. 2022, *BMJ* | 36283705 | VQI-Medicare | 20,489 EVAR patients with complete follow-up through registry and/or claims. Detected a device-specific reintervention signal (HR 1.61; surgeon-level IV 1.75) in mid-2013, years before the 2017 regulatory warnings. |
|
||||
| Mao J et al. 2022, *J Vasc Surg* | 35181518 | VQI-Medicare linkage | Indirect linkage (without SSNs) matched 93.0% of patients with 99.4% accuracy. |
|
||||
| **Mao J et al. 2024, *BMJ Open*** | 39327057 | VQI linked to EHR (INSIGHT CRN, New York) | After PVI, **75.2% were retained in the EHR through year 3.** Older, Medicaid, heart-failure, dialysis and less ambulatory patients were less likely to be retained, so attrition is informative. Amputation-free survival estimates were sensitive to this from 6–12 months onward. |
|
||||
| Jansen SCP et al. 2020, *EJVES* | 32888779 | Dutch national claims | SET captured in claims where it is reimbursed (63% → 87% primary SET). Shows that SET's visibility in claims depends on the healthcare system. |
|
||||
| Keyhani S et al. 2020 / 2022, *JAMA Neurol* / *Stroke* | 32478802; 35164531 | VA EHR (carotid imaging reports) | Emulations of carotid surgery vs medical therapy that used **imaging-report stenosis**, an EHR-only variable, for eligibility. 2022 (CAS vs MT): 5-year stroke 6.9% vs 7.1%. The population was 98.8% men (2020 paper), which limits generalizability. |
|
||||
| Sakowitz S et al. 2025, *Ann Surg* | 40607703 (in 01) | Nationwide Readmissions Database | TCAR vs CEA "national target trial": 90-day composite 4.1% vs 4.1%. Follow-up is limited to 90 days by the database. |
|
||||
| Wang Z et al. 2026, *J Vasc Surg* | 42448157 | 3-center clinical data | Open- vs closed-cell carotid stents. Median follow-up 26 months. Example of a short-follow-up, single-system emulation in vascular surgery. |
|
||||
| Heindel P et al. 2024, *Eur Heart J Open* | 39175849 | Medicare linked with academic health-system EHR | PAD drug TTE (colchicine) with a MALE/MACE/death outcome. RD −1.7% (−6.5 to 3.1). Example of a claims-plus-EHR PAD emulation. |
|
||||
| Westerberg 2024; Wang C 2026 (above) | 39159770; 42156120 | — | Comorbidity measurement drives residual confounding. Claims-based emulations tended to underestimate RCT effects. |
|
||||
|
||||
**Summary of data-source trade-offs for an early-PVI TTE.**
|
||||
- **Claims (Medicare).**
|
||||
- Strengths: population-wide, long and nearly complete follow-up for procedures and amputation, procedure dates, operator NPI, Part D fills, and SET billing from 2017.
|
||||
- Weaknesses: no ABI, walking distance or symptom severity (the main triggers for crossover and confounding by indication). About 19% misclassification between claudication and CLTI (Bose 2025). Home exercise, smoking, and patient refusal are not recorded.
|
||||
- **EHR.**
|
||||
- Strengths: ABI, imaging, symptom notes, SET referral, smoking, and refusal documentation.
|
||||
- Weaknesses: follow-up is fragmented, and patients leave the system informatively (75% retained at 3 years; Mao 2024). Usually a single system, so fewer physicians to compare.
|
||||
- **Registry (VQI).**
|
||||
- Strengths: rich anatomy and indication data, plus OMT and smoking status at the time of the procedure.
|
||||
- Weaknesses: **enrolls only patients who have a procedure**, so there is no conservative arm. Long-term outcomes are undercounted unless linked to claims (12 of 30 reinterventions captured; Columbo 2019).
|
||||
- **Linked data (VQI-Medicare/VISION).** Best for outcomes after a procedure. It still has no comparator arm of patients never treated, unless claims-only patients are added, and those patients then lack registry detail.
|
||||
|
||||
### 6d. Unmeasured confounding by indication: the limit that remains
|
||||
|
||||
- Every guidance paper above, and every review, names unmeasured confounding as the limitation TTE does not remove. Only 30.8% of published TTEs address it (Ren 2026).
|
||||
- For procedures, Westerberg 2024 shows that adjustment moved the HR from 1.94 to 1.14, and that most of the bias came from how well comorbidity was measured.
|
||||
- In the claudication setting the main unmeasured confounders are:
|
||||
- symptom severity and walking distance;
|
||||
- frailty;
|
||||
- patient preference and refusal;
|
||||
- physician style (see 02, B6).
|
||||
- Tools exist for each: E-values (28693043), negative controls (20335814), physician-level designs (02), and linking to EHR or registry data to measure symptoms.
|
||||
|
||||
---
|
||||
|
||||
## Feed for Table 1 (this draft's rows)
|
||||
|
||||
| Challenge | Evidence it is common/important | Vascular/claudication example | Can TTE solve it? | Known-truth simulation? |
|
||||
|---|---|---|---|---|
|
||||
| Crossover / contamination | 27–43% crossover in conservative arms within 1–2 years (Peul, Katz, SPORT, CODA, APPAC). Effect estimates nearer the null with more crossover (Gaudino 2019). STICH ITT not significant at 5 years. | SUPER 29% at 1 year. Swedish SFA 25% at 5 years. ERASE HR 2.50. | **Yes** (CCW/IPCW for explicit strategies), if crossover triggers are measured. Claims lack symptoms; EHR/registry partly have them. | Yes: latent progression drives deferred PVI. Observed vs unobserved. |
|
||||
| Non-adherence / refusal | STICH CABG→MED crossover mainly by patient/family choice, with worse prognosis. ACST only 50% operated by 1 month. SET recruitment 24.2%. | SUPER SET compliance 50%. US SET completion 9.8–49.9%. | **Partly.** No consent step. Refusal is not visible in claims (sometimes in EHR). SET billing is visible in Medicare from 2017. | Yes: latent "refuser" trait. |
|
||||
| Co-interventions / usual care | Usual care poorly reported (TIDieR median 1/12). Only 38.1% complete OMT before revascularization in VQI. | Comparator differs across trials (aspirin plus advice vs SET plus BMT vs cilostazol). | **Partly.** Part D statins and SET claims are visible. Home exercise and smoking are not. VQI has OMT only at procedure and no conservative arm. | Yes: unmeasured exercise or OMT that differs by arm. |
|
||||
| Comparator definition | Most "conservative" arms are "surgery-if-needed". ISCHEMIA and ACST define dynamic comparators. CCW and sequential trials give different numbers (Garcia-Albeniz 2026). | "No PVI within 6 months" vs "PVI only for CLTI" vs "never PVI". | **Yes**, and TTE's main gain: several named estimands from one dataset. Triggers must be coded (CLTI codes about 81% accurate). "Never" may lack positivity. | Yes, central: true risk under each strategy. |
|
||||
|
||||
## Five-line summary
|
||||
|
||||
1. Crossover from conservative care to surgery is common and large: 25–68% across hernia, spine, knee, appendicitis, cardiac surgery and claudication trials. It pulls ITT estimates toward the null, as in STICH and the Swedish SFA trial, and it is the challenge TTE handles best, through explicit dynamic strategies with CCW/IPCW.
|
||||
2. Non-adherence and refusal (to surgery and especially to SET) are substantial, and patients who refuse have a different prognosis. TTE removes the consent barrier, but refusal of an offered operation is not visible in claims and is only partly visible in EHR.
|
||||
3. Co-interventions and "usual care" vary widely and are poorly reported. Claims see drugs and Medicare SET but not home exercise or smoking. VQI records OMT only at the time of a procedure and cannot provide a no-PVI arm on its own.
|
||||
4. Of the 15 claudication revascularization trials checked (plus the Cochrane review), none was powered for hard limb outcomes. Several under-recruited (CLEVER 252 → 111) or stopped early (SUPER 400 → 240). Only the Swedish SFA trial reports CLTI and amputation by arm (about 2% each). EXITPAD could not be found in PubMed.
|
||||
5. TTE agrees well with RCTs when the emulation is close (r 0.93 in RCT-DUPLICATE; 0.83 in Wang 2026 *BMJ*), but agreement is poor otherwise (r 0.53; and 0.08 for conventional cancer-registry analyses). Procedural benchmarks are scarce, many published TTEs are of low quality (only 30.8% address unmeasured confounding), and no claudication or PVI TTE was found.
|
||||
|
||||
---
|
||||
|
||||
### Verification notes
|
||||
- **No PubMed abstract, so cited without numbers:** Divakaran 2021 (34293930), Divakaran 2024 (39129558), Dahabreh 2020 (32740470), Matthews 2022 *BMJ* (36041749), Wong 2025 *BJS* (40893042), Katsoulis 2024 (39024012, in 01).
|
||||
- **EXITPAD:** not found in PubMed under that name. It may be a non-indexed acronym or a confusion with ERASE or SUPER (both Dutch).
|
||||
- **Not in the abstracts** (need full text): planned sample sizes for MIMIC, IRONIC and ERASE; crossover rates for IRONIC, CLEVER and MIMIC; ERASE's long-term CLI result; whether CLEVER was formally stopped.
|
||||
- **ISCHEMIA and COURAGE crossover rates** are not in their abstracts, so they are not quoted.
|
||||
- **Creasy 1990 and Perkins 1996 are probably the same Oxford trial** (preliminary report with 36 patients, later report with 56). The abstracts do not confirm this.
|
||||
- **No claudication or PVI target trial emulation** was found by a PubMed search (2026-10-28).
|
||||
@@ -0,0 +1,24 @@
|
||||
Package: pvisim
|
||||
Title: Known-Truth Simulator for Target Trial Emulation of Early Peripheral Vascular Intervention
|
||||
Version: 0.1.0
|
||||
Authors@R: person("Chen", "Dun", email = "[email protected]", role = c("aut", "cre"))
|
||||
Description: Data-generating process (claudicants nested in physicians, latent
|
||||
severity, physician skill and style), truths by simulating interventions,
|
||||
target trial emulation designs (naive, landmark, sequential trials,
|
||||
clone-censor-weight), simulated RCTs, physician-level variants, negative
|
||||
controls and performance metrics. Not for CRAN.
|
||||
License: MIT
|
||||
Encoding: UTF-8
|
||||
Depends: R (>= 4.3)
|
||||
Imports:
|
||||
stats,
|
||||
utils,
|
||||
yaml,
|
||||
data.table
|
||||
Suggests:
|
||||
testthat (>= 3.0.0),
|
||||
fastglm,
|
||||
sandwich,
|
||||
glmmTMB,
|
||||
pkgload
|
||||
Config/testthat/edition: 3
|
||||
@@ -0,0 +1,3 @@
|
||||
exportPattern("^[a-z]")
|
||||
import(data.table)
|
||||
importFrom(stats, plogis, qlogis, qnorm, qpois, rnorm, runif, rbinom, rlnorm, rgamma, setNames)
|
||||
@@ -0,0 +1,363 @@
|
||||
# Estimators (plan sections 7 and 9; D13-D15, D26, D28).
|
||||
#
|
||||
# One machinery for every observational design: inverse-probability weights from
|
||||
# pooled logistic models of elective PVI, then a weighted Aalen-Johansen estimator of
|
||||
# the cumulative incidence of the outcome (death competing), nonparametric in time.
|
||||
# Standard errors come from the influence function of the weighted Aalen-Johansen
|
||||
# estimator with the weights treated as known (usually conservative for IPW; checked
|
||||
# against a physician-cluster bootstrap, D28). Summed by patient for the naive SE, by
|
||||
# physician for the cluster-robust SE. Clones and person-trials of one patient are summed
|
||||
# together, so cloning and stacking are accounted for.
|
||||
|
||||
BASE_COVS <- c("age", "female", "black", "year", "diabetes", "ckd", "chf", "copd", "prior_cvd",
|
||||
"smoking", "frailty_index", "obl")
|
||||
VARIANTS <- c("V1_ignore", "V2_cluster", "V3_random", "V4_within")
|
||||
|
||||
# ---- inputs -------------------------------------------------------------------------
|
||||
|
||||
#' Person-month analysis table: eligibility for elective PVI and the PVI decision.
|
||||
#'
|
||||
#' A row is month t for a patient who survived month t's events and death and had no
|
||||
#' MALE and no PVI before the decision in month t: the risk set of the PVI model.
|
||||
pvi_risk_set <- function(rec) {
|
||||
p <- rec$patients
|
||||
extra <- intersect("t_pvi_orig", names(p))
|
||||
pm <- merge(rec$pm, p[, c("id", "phys_id", BASE_COVS, "t_pvi", "pvi_elective", "t_male", "t_death", extra),
|
||||
with = FALSE], by = "id")
|
||||
pm <- pm[(is.na(t_death) | t_death > t) & (is.na(t_male) | t_male > t) & (is.na(t_pvi) | t_pvi >= t)]
|
||||
pm[, pvi := as.integer(!is.na(t_pvi) & t_pvi == t)]
|
||||
sev0 <- rec$pm[t == 0, .(id, sev0 = sev_obs)]
|
||||
pm <- merge(pm, sev0, by = "id")
|
||||
pm[, `:=`(win = as.integer(t < rec$par$size$window), tt = pmin(t, 12))]
|
||||
if (length(extra)) pm[, pvi_before := as.integer(!is.na(t_pvi_orig) & t_pvi_orig < t)]
|
||||
pm[]
|
||||
}
|
||||
|
||||
# ---- treatment models ------------------------------------------------------------------
|
||||
|
||||
#' Fit P(PVI at t | eligible, history) and return predictions for `newdata`.
|
||||
#'
|
||||
#' The decision window (months 0..W-1) and the deferred period are fitted as separate
|
||||
#' models: practice inside and after the window differs, and a pooled model would
|
||||
#' misspecify the window probabilities that the CCW and sequential weights depend on.
|
||||
#' variant: V1/V2 no physician terms; V3 physician random intercept (glmer, nAGQ = 0);
|
||||
#' V4 physician fixed effects (fixest). Under V4 a physician with no PVI in a phase has
|
||||
#' no estimable intercept for it; the maximum-likelihood probability there is 0.
|
||||
fit_pvi <- function(data, newdata, variant, extra_covs = NULL) {
|
||||
out <- numeric(nrow(newdata))
|
||||
for (w in c(1L, 0L)) {
|
||||
i_new <- newdata$win == w
|
||||
if (!any(i_new)) next
|
||||
d <- data[win == w]
|
||||
nd <- newdata[i_new]
|
||||
tt_levels <- if (w == 1L) "factor(tt)" else "splines::ns(t, df = 4)"
|
||||
# deferred period (about 2 million person-months, few PVIs): fit on every PVI and a
|
||||
# fixed 10% sample of the other months, with offset log(1/0.1) so the coefficients
|
||||
# are those of the full model (case-control sampling of person-months)
|
||||
if (w == 0L) d <- sample_controls(d, frac = 0.1)
|
||||
# no exposure at all in a phase (the NCE after the window): probability 0
|
||||
out[i_new] <- if (sum(d$pvi) == 0) 0 else fit_pvi_phase(d, nd, variant, tt_levels, extra_covs)
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
#' Keep all events and a deterministic fraction of non-events; offset column `off`.
|
||||
sample_controls <- function(d, frac) {
|
||||
keep <- d$pvi == 1L | (((d$id * 7919L + d$t * 104729L) %% 1000L) < frac * 1000)
|
||||
d <- d[keep]
|
||||
d[, off := log(1 / frac)] # sampling raises every row's odds by 1/frac
|
||||
d
|
||||
}
|
||||
|
||||
fit_pvi_phase <- function(d, nd, variant, time_terms, extra_covs = NULL) {
|
||||
if (!"off" %in% names(d)) d[, off := 0]
|
||||
nd <- copy(nd)[, off := 0] # predictions are for the full population
|
||||
covs <- c(BASE_COVS, "sev_obs", "sev0", extra_covs)
|
||||
if (variant == "V4_within") covs <- setdiff(covs, "obl")
|
||||
rhs <- paste(c(covs, time_terms), collapse = " + ")
|
||||
if (variant %in% c("V1_ignore", "V2_cluster")) {
|
||||
f <- as.formula(paste("~", rhs))
|
||||
X <- model.matrix(f, d)
|
||||
fit <- fastglm::fastglm(X, d$pvi, family = binomial(), method = 2, offset = d$off)
|
||||
b <- coef(fit); b[is.na(b)] <- 0
|
||||
Xn <- model.matrix(f, nd)
|
||||
return(as.vector(plogis(Xn %*% b)))
|
||||
}
|
||||
if (variant == "V3_random") {
|
||||
fit <- lme4::glmer(as.formula(paste("pvi ~", rhs, "+ offset(off) + (1 | phys_id)")), data = d,
|
||||
family = binomial(), nAGQ = 0, control = lme4::glmerControl(calc.derivs = FALSE))
|
||||
return(as.vector(predict(fit, newdata = nd, type = "response", allow.new.levels = TRUE)))
|
||||
}
|
||||
if (variant == "V4_within") {
|
||||
fit <- fixest::feglm(as.formula(paste("pvi ~", rhs, "| phys_id")), data = d, offset = ~off,
|
||||
family = binomial(), notes = FALSE, warn = FALSE)
|
||||
pr <- as.vector(predict(fit, newdata = nd, type = "response"))
|
||||
pr[is.na(pr)] <- 0
|
||||
return(pr)
|
||||
}
|
||||
stop("unknown variant ", variant)
|
||||
}
|
||||
|
||||
#' Physicians with variation in early (window) PVI: the V4 analysis population.
|
||||
physicians_with_variation <- function(rec) {
|
||||
W <- rec$par$size$window
|
||||
x <- rec$patients[, .(e = sum(pvi_elective %in% TRUE & t_pvi < W), n = .N), by = phys_id]
|
||||
x[e > 0 & e < n, phys_id]
|
||||
}
|
||||
|
||||
# ---- weighted Aalen-Johansen with influence functions --------------------------------
|
||||
|
||||
#' Cumulative incidence at `horizon` from person-month rows.
|
||||
#'
|
||||
#' @param d data.table with id, phys_id, arm, t, w (weight for the row's risk set),
|
||||
#' ev (1 if the outcome occurs in month t), dth (1 if death without the outcome in t).
|
||||
#' @return list(F = named vector of cumulative incidence by arm at the horizon,
|
||||
#' inf = data.table(id, phys_id, arm, inf) of influence-function contributions summed
|
||||
#' by patient, curve = cumulative incidence by arm and month).
|
||||
aj_cif <- function(d, horizon = 60L) {
|
||||
d <- d[t < horizon & w > 0]
|
||||
s <- d[, .(R = sum(w), E = sum(w * ev), D = sum(w * dth)), keyby = .(arm, t)]
|
||||
s[, `:=`(he = E / R, hd = D / R)]
|
||||
s[, h := he + hd]
|
||||
s[, S_prev := shift(cumprod(1 - h), fill = 1), by = arm]
|
||||
s[, Fcum := cumsum(S_prev * he), by = arm]
|
||||
s[, FH := Fcum[.N], by = arm]
|
||||
s[, `:=`(a = S_prev - (FH - Fcum) / (1 - h), b = -(FH - Fcum) / (1 - h))]
|
||||
s[!is.finite(a), a := 0]; s[!is.finite(b), b := 0]
|
||||
d <- s[d, on = .(arm, t), .(id, phys_id, arm, x = w * ((ev - he) * a + (dth - hd) * b) / R)]
|
||||
inf <- d[, .(inf = sum(x)), by = .(id, phys_id, arm)]
|
||||
FH <- s[, .(FH = FH[1]), by = arm]
|
||||
list(F = setNames(FH$FH, FH$arm), inf = inf, curve = s[, .(arm, t, cuminc = Fcum)])
|
||||
}
|
||||
|
||||
#' Risk difference arm1 - arm0 with naive (patient) and cluster-robust (physician) SEs.
|
||||
rd_from_aj <- function(aj, arm1, arm0) {
|
||||
x <- dcast(aj$inf, id + phys_id ~ arm, value.var = "inf", fill = 0)
|
||||
diff <- x[[arm1]] - x[[arm0]]
|
||||
se_pat <- sqrt(sum(diff^2))
|
||||
se_phys <- sqrt(sum(tapply(diff, x$phys_id, sum)^2))
|
||||
data.table(est = aj$F[[arm1]] - aj$F[[arm0]], se_patient = se_pat, se_cluster = se_phys,
|
||||
risk1 = aj$F[[arm1]], risk0 = aj$F[[arm0]])
|
||||
}
|
||||
|
||||
# outcome and death indicators for a person-month table that has columns t_out, t_death
|
||||
mark_events <- function(d) {
|
||||
d[, ev := as.integer(!is.na(t_out) & t_out == t)]
|
||||
d[, dth := as.integer(ev == 0L & !is.na(t_death) & t_death == t)]
|
||||
d
|
||||
}
|
||||
|
||||
# person-months from `start` to the outcome, death or the horizon, one row per month
|
||||
expand_followup <- function(p, start, horizon = 60L) {
|
||||
end <- pmin(ifelse(is.na(p$t_out), horizon - 1L, p$t_out),
|
||||
ifelse(is.na(p$t_death), horizon - 1L, p$t_death), horizon - 1L)
|
||||
keep <- end >= start
|
||||
p <- p[keep]; st <- if (length(start) > 1) start[keep] else rep(start, nrow(p)); end <- end[keep]
|
||||
nrep <- end - st + 1L
|
||||
d <- p[rep(seq_len(nrow(p)), nrep)]
|
||||
d[, t := st[rep(seq_along(nrep), nrep)] + sequence(nrep) - 1L]
|
||||
mark_events(d)
|
||||
}
|
||||
|
||||
# ---- designs --------------------------------------------------------------------------
|
||||
|
||||
#' For the within-physician variant, restrict to physicians whose patients include both
|
||||
#' early PVI and no early PVI (positivity); the estimand changes accordingly (plan 7).
|
||||
restrict_variant <- function(rec, variant) {
|
||||
if (variant != "V4_within") return(rec)
|
||||
keep <- physicians_with_variation(rec)
|
||||
rec$patients <- rec$patients[phys_id %in% keep]
|
||||
rec$pm <- rec$pm[id %in% rec$patients$id]
|
||||
rec
|
||||
}
|
||||
|
||||
#' Run one design under one physician variant for several outcomes.
|
||||
#' Returns one row per outcome (and contrast), with the SE that the variant reports:
|
||||
#' patient-level for V1, physician-cluster-robust for V2-V4.
|
||||
run_design <- function(rec, design, variant, outcomes = c(limb = "t_limb", male = "t_male", disease = "t_disease",
|
||||
nco1 = "t_nco1", nco2 = "t_nco2")) {
|
||||
run_designs(rec, design, variant, outcomes)
|
||||
}
|
||||
|
||||
#' Several designs under one variant, sharing the PVI fit between CCW and sequential
|
||||
#' trials (the sequential propensity score is the window part of the same model).
|
||||
run_designs <- function(rec, designs, variant, outcomes = c(limb = "t_limb", male = "t_male", disease = "t_disease",
|
||||
nco1 = "t_nco1", nco2 = "t_nco2")) {
|
||||
rec <- restrict_variant(rec, variant)
|
||||
need <- intersect(designs, c("ccw", "seq"))
|
||||
rs <- if (length(need)) pvi_risk_set(rec) else NULL
|
||||
phat <- if ("ccw" %in% need) fit_pvi(rs, rs, variant, rec$extra_covs)
|
||||
else if ("seq" %in% need) { x <- rep(NA_real_, nrow(rs)); w <- rs$win == 1L
|
||||
x[w] <- fit_pvi(rs[w], rs[w], variant, rec$extra_covs); x }
|
||||
cmp <- if (is.null(rec$comparators)) c("B1", "B2") else rec$comparators
|
||||
use_patient <- variant == "V1_ignore"
|
||||
rbindlist(lapply(designs, function(design) {
|
||||
r <- switch(design,
|
||||
naive = rbindlist(lapply(names(outcomes), function(o)
|
||||
cbind(contrast = "early_none", est_naive(rec, outcomes[[o]], variant), outcome = o))),
|
||||
landmark = rbindlist(lapply(names(outcomes), function(o)
|
||||
cbind(contrast = "early_none", est_landmark(rec, outcomes[[o]], variant, extra_covs = rec$extra_covs), outcome = o))),
|
||||
landmark_m5 = rbindlist(lapply(names(outcomes), function(o)
|
||||
cbind(contrast = "early_none", est_landmark(rec, outcomes[[o]], variant, extra_covs = rec$extra_covs,
|
||||
at_landmark = TRUE), outcome = o))),
|
||||
ccw = est_ccw(rec, outcomes, variant, rs = copy(rs), phat = phat, comparators = cmp),
|
||||
seq = cbind(contrast = "now_not", est_seq(rec, outcomes, variant, rs = copy(rs), phat = phat)))
|
||||
r[, `:=`(design = design, variant = variant, n = nrow(rec$patients))]
|
||||
r[, se := if (use_patient) se_patient else se_cluster]
|
||||
r
|
||||
}), fill = TRUE)
|
||||
}
|
||||
|
||||
#' Naive "ever early PVI" design: exposure = PVI in months 0..W-1, follow-up from month 0,
|
||||
#' baseline propensity score. Immortal time is present by construction.
|
||||
est_naive <- function(rec, outcome = "t_male", variant = "V1_ignore", horizon = 60L, extra_covs = NULL) {
|
||||
W <- rec$par$size$window
|
||||
p <- copy(rec$patients)
|
||||
p[, `:=`(t_out = get(outcome), arm = ifelse(!is.na(t_pvi) & t_pvi < W, "early", "none"))]
|
||||
p <- merge(p, rec$pm[t == 0, .(id, sev_obs)], by = "id")
|
||||
p[, pvi := as.integer(arm == "early")]
|
||||
ps <- fit_pvi_ps(p, variant, covs = c(BASE_COVS, "sev_obs", extra_covs)) # sev_obs is the month-0 value here
|
||||
p[, w := ifelse(arm == "early", 1 / ps, 1 / (1 - ps))]
|
||||
d <- expand_followup(p[!is.na(w)], 0L, horizon)
|
||||
rd_from_aj(aj_cif(d, horizon), "early", "none")
|
||||
}
|
||||
|
||||
# a point-treatment propensity score with the same covariates and physician variants
|
||||
fit_pvi_ps <- function(p, variant, covs = c(BASE_COVS, "sev_obs", "sev0")) {
|
||||
if (variant == "V4_within") covs <- setdiff(covs, "obl")
|
||||
rhs <- paste(covs, collapse = " + ")
|
||||
if (variant %in% c("V1_ignore", "V2_cluster")) {
|
||||
X <- model.matrix(as.formula(paste("~", rhs)), p)
|
||||
fit <- fastglm::fastglm(X, p$pvi, family = binomial(), method = 2)
|
||||
return(as.vector(plogis(X %*% coef(fit))))
|
||||
}
|
||||
if (variant == "V3_random") {
|
||||
fit <- lme4::glmer(as.formula(paste("pvi ~", rhs, "+ (1 | phys_id)")), data = p,
|
||||
family = binomial(), nAGQ = 0, control = lme4::glmerControl(calc.derivs = FALSE))
|
||||
return(as.vector(predict(fit, type = "response")))
|
||||
}
|
||||
fit <- fixest::feglm(as.formula(paste("pvi ~", rhs, "| phys_id")), data = p, family = binomial(),
|
||||
notes = FALSE, warn = FALSE)
|
||||
as.vector(predict(fit, newdata = p, type = "response"))
|
||||
}
|
||||
|
||||
#' Landmark at month W (D13): patients alive and MALE-free at the start of month W;
|
||||
#' exposure = PVI in months 0..W-1; covariates at diagnosis; outcome counted in months
|
||||
#' W..horizon-1. With at_landmark = TRUE the propensity score also uses severity at month
|
||||
#' W-1, which a PVI has already changed (supplement, D53).
|
||||
est_landmark <- function(rec, outcome = "t_male", variant = "V1_ignore", horizon = 60L, extra_covs = NULL,
|
||||
at_landmark = FALSE) {
|
||||
W <- rec$par$size$window
|
||||
p <- copy(rec$patients)[(is.na(t_death) | t_death >= W) & (is.na(t_male) | t_male >= W)]
|
||||
p[, `:=`(t_out = get(outcome), arm = ifelse(!is.na(t_pvi) & t_pvi < W, "early", "none"))]
|
||||
p <- p[is.na(t_out) | t_out >= W]
|
||||
p <- merge(p, rec$pm[t == W - 1L, .(id, sev_obs)], by = "id")
|
||||
p <- merge(p, rec$pm[t == 0, .(id, sev0 = sev_obs)], by = "id")
|
||||
p[, pvi := as.integer(arm == "early")]
|
||||
ps <- fit_pvi_ps(p, variant, covs = c(BASE_COVS, "sev0", if (at_landmark) "sev_obs", extra_covs))
|
||||
p[, w := ifelse(arm == "early", 1 / ps, 1 / (1 - ps))]
|
||||
d <- expand_followup(p[!is.na(w)], W, horizon)
|
||||
rd_from_aj(aj_cif(d, horizon), "early", "none")
|
||||
}
|
||||
|
||||
#' Clone-censor-weight (D10, D26). Arms: A (PVI by month W-1), B1 (no PVI in months
|
||||
#' 0..W-1), B2 (no elective PVI ever). A clone is censored after the PVI decision of the
|
||||
#' month in which it deviates; weights are 1 / P(not censored through the previous month).
|
||||
est_ccw <- function(rec, outcome = "t_male", variant = "V1_ignore", horizon = 60L,
|
||||
comparators = c("B1", "B2"), rs = NULL, phat = NULL) {
|
||||
W <- rec$par$size$window
|
||||
if (is.null(rs)) rs <- pvi_risk_set(rec)
|
||||
if (is.null(phat)) phat <- fit_pvi(rs, rs, variant, rec$extra_covs)
|
||||
rs[, ph := phat]
|
||||
rs <- rs[!is.na(ph)]
|
||||
# everyone eligible at diagnosis is cloned, including patients with a MALE event or
|
||||
# death in month 0 (events in the grace period count in both arms). Clones and weights
|
||||
# are built once over follow-up to death or the horizon; each outcome then truncates.
|
||||
p <- copy(rec$patients)
|
||||
p[, t_out := NA_integer_]
|
||||
base <- expand_followup(p, 0L, horizon)
|
||||
# P(PVI) for each eligible month; months where a patient is not at risk of PVI have
|
||||
# probability 0 of deviating
|
||||
base <- rs[, .(id, t, ph)][base, on = .(id, t)]
|
||||
base[is.na(ph), ph := 0]
|
||||
base[, `:=`(tp = fifelse(is.na(t_pvi), Inf, as.numeric(t_pvi)), el = pvi_elective %in% TRUE)]
|
||||
|
||||
arms <- list()
|
||||
# arm A: censored after month W-1 if no PVI by then (and still at risk of PVI then)
|
||||
a <- copy(base)[, arm := "A"]
|
||||
cens_A <- a[t == W - 1L & ph > 0 & tp > W - 1L, id] # deviated at month W-1
|
||||
a <- a[!(id %in% cens_A & t > W - 1L)]
|
||||
pA <- base[t == W - 1L & ph > 0, .(id, pA = ph)]
|
||||
a <- pA[a, on = "id"]
|
||||
a[, w := ifelse(t > W - 1L & !is.na(pA) & tp == W - 1L, 1 / pA, 1)]
|
||||
arms$A <- a[, .(id, phys_id, arm, t, w)]
|
||||
for (cmp in comparators) {
|
||||
b <- copy(base)[, arm := cmp]
|
||||
last <- if (cmp == "B1") W - 1L else horizon
|
||||
b <- b[!(el & tp <= last & t > tp)] # censored after month tp
|
||||
b[, q := ifelse(t <= last, 1 - ph, 1)]
|
||||
b[, w := 1 / shift(cumprod(q), fill = 1), by = id]
|
||||
arms[[cmp]] <- b[, .(id, phys_id, arm, t, w)]
|
||||
}
|
||||
clones <- rbindlist(arms)
|
||||
ev_times <- rec$patients[, c("id", "t_death", unname(outcome)), with = FALSE]
|
||||
rbindlist(lapply(seq_along(outcome), function(k) {
|
||||
d <- ev_times[, .(id, t_death, t_out = get(outcome[k]))][clones, on = "id"]
|
||||
d <- d[is.na(t_out) | t <= t_out]
|
||||
aj <- aj_cif(mark_events(d), horizon)
|
||||
rbindlist(lapply(comparators, function(cmp)
|
||||
cbind(contrast = paste0("A_", cmp), rd_from_aj(aj, "A", cmp))))[
|
||||
, outcome := if (is.null(names(outcome))) outcome[k] else names(outcome)[k]]
|
||||
}))
|
||||
}
|
||||
|
||||
#' Sequential trials, ITT (D14): a trial at each month m = 0..W-1 among patients eligible
|
||||
#' for PVI at m; treatment = PVI at m; propensity score from covariates at m; outcome
|
||||
#' counted in months m+1..horizon-1; trial RDs pooled with weights = number eligible.
|
||||
est_seq <- function(rec, outcome = "t_male", variant = "V1_ignore", horizon = 60L, rs = NULL,
|
||||
phat = NULL) {
|
||||
W <- rec$par$size$window
|
||||
if (is.null(rs)) rs <- pvi_risk_set(rec)
|
||||
if (!is.null(phat)) rs[, ps := phat]
|
||||
trials <- rs[t < W]
|
||||
# the trial-baseline propensity score is the window model of the PVI fit
|
||||
if (is.null(phat)) trials[, ps := fit_pvi(trials, trials, variant, rec$extra_covs)]
|
||||
trials <- trials[!is.na(ps)]
|
||||
trials[, `:=`(m = t, arm = ifelse(pvi == 1L, "pvi", "no"),
|
||||
w = ifelse(pvi == 1L, 1 / ps, 1 / (1 - ps)))]
|
||||
trials <- trials[, .(id, phys_id, m, arm, w, t_death)]
|
||||
rbindlist(lapply(seq_along(outcome), function(k) {
|
||||
tr_k <- rec$patients[, .(id, t_out = get(outcome[k]))][trials, on = "id"][is.na(t_out) | t_out > m]
|
||||
res <- list(); infs <- list()
|
||||
for (mm in 0:(W - 1)) {
|
||||
tr <- tr_k[m == mm]
|
||||
d <- expand_followup(tr[, .(id, phys_id, arm, w, t_out, t_death)], mm + 1L, horizon)
|
||||
aj <- aj_cif(d, horizon)
|
||||
x <- dcast(aj$inf, id + phys_id ~ arm, value.var = "inf", fill = 0)
|
||||
res[[mm + 1L]] <- data.table(m = mm, n = nrow(tr), est = aj$F[["pvi"]] - aj$F[["no"]])
|
||||
infs[[mm + 1L]] <- data.table(id = x$id, phys_id = x$phys_id, m = mm, inf = x$pvi - x$no)
|
||||
}
|
||||
res <- rbindlist(res); infs <- rbindlist(infs)
|
||||
res[, wt := n / sum(n)]
|
||||
infs <- res[, .(m, wt)][infs, on = "m"]
|
||||
pi <- infs[, .(inf = sum(wt * inf)), by = .(id, phys_id)]
|
||||
data.table(est = sum(res$wt * res$est), se_patient = sqrt(sum(pi$inf^2)),
|
||||
se_cluster = sqrt(sum(tapply(pi$inf, pi$phys_id, sum)^2)), risk1 = NA_real_, risk0 = NA_real_,
|
||||
outcome = if (is.null(names(outcome))) outcome[k] else names(outcome)[k])
|
||||
}))
|
||||
}
|
||||
|
||||
#' Negative-control exposure (D32, D36): the early low-value procedure (NCE, months
|
||||
#' 0..W-1) takes the place of early PVI as the exposure; early-PVI status is adjusted
|
||||
#' for (time-varying "PVI before t" in CCW, "PVI in the window" at the landmark).
|
||||
#' Only the landmark and CCW (A vs B1) designs are run.
|
||||
nce_records <- function(rec, adjust_pvi = TRUE) {
|
||||
W <- rec$par$size$window
|
||||
p <- copy(rec$patients)
|
||||
p[, `:=`(t_pvi_orig = t_pvi, pvi_before = as.integer(!is.na(t_pvi) & t_pvi < W))]
|
||||
p[, `:=`(t_pvi = t_nce, pvi_elective = !is.na(t_nce))]
|
||||
rec$patients <- p
|
||||
rec$extra_covs <- if (adjust_pvi) "pvi_before" else NULL
|
||||
rec$comparators <- "B1"
|
||||
rec
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
# Own estimands of the non-interventional designs (plan 5.5). The landmark and naive
|
||||
# contrasts have no intervention that defines them; their own estimand is what the same
|
||||
# estimator gives with every confounder measured (oracle): exact severity, true frailty
|
||||
# and smoking, and the latent physician traits. A large-n limit is too noisy (the
|
||||
# landmark estimator has few treated patients), so the oracle estimator is run on every
|
||||
# replicate's own data: estimation bias = mean(estimate - oracle estimate), paired, and
|
||||
# the oracle mean estimates the own estimand.
|
||||
|
||||
#' The replicate's records with the hidden truth in place of the claims proxies.
|
||||
oracle_records <- function(rec) {
|
||||
o <- rec
|
||||
o$pm <- merge(rec$pm[, .(id, t)], rec$hidden_pm, by = c("id", "t"))[, .(id, t, sev_obs = z)]
|
||||
h <- rec$hidden
|
||||
o$patients <- merge(copy(rec$patients)[, c("frailty_index", "smoking") := NULL],
|
||||
h[, .(id, frailty_index = frailty, smoking = smoking_true, skill, style)], by = "id")
|
||||
o$extra_covs <- c("skill", "style")
|
||||
o
|
||||
}
|
||||
|
||||
#' Oracle versions of the landmark (both versions) and naive designs (V1, patient-level SE).
|
||||
run_oracle <- function(rec, outcomes = c(limb = "t_limb", male = "t_male", disease = "t_disease", nco1 = "t_nco1", nco2 = "t_nco2")) {
|
||||
o <- oracle_records(rec)
|
||||
rbindlist(lapply(names(outcomes), function(nm) {
|
||||
lm <- est_landmark(o, outcomes[[nm]], "V1_ignore", extra_covs = o$extra_covs)
|
||||
l5 <- est_landmark(o, outcomes[[nm]], "V1_ignore", extra_covs = o$extra_covs, at_landmark = TRUE)
|
||||
nv <- est_naive(o, outcomes[[nm]], "V1_ignore", extra_covs = o$extra_covs)
|
||||
rbind(cbind(design = "landmark", lm), cbind(design = "landmark_m5", l5), cbind(design = "naive", nv))[
|
||||
, `:=`(outcome = nm, contrast = "early_none", variant = "oracle", se = se_patient)]
|
||||
}))
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
# Parameters: read the YAML, then apply a world and a truth scenario as overrides.
|
||||
|
||||
#' Recursively merge `override` into `base` (lists only; scalars replace).
|
||||
deep_merge <- function(base, override) {
|
||||
for (nm in names(override)) {
|
||||
if (is.list(override[[nm]]) && is.list(base[[nm]])) {
|
||||
base[[nm]] <- deep_merge(base[[nm]], override[[nm]])
|
||||
} else {
|
||||
base[[nm]] <- override[[nm]]
|
||||
}
|
||||
}
|
||||
base
|
||||
}
|
||||
|
||||
#' Path of the default parameter file, inside the installed or loaded package.
|
||||
default_params_path <- function() {
|
||||
p <- system.file("params", "dgp_default.yml", package = "pvisim")
|
||||
if (p == "") stop("dgp_default.yml not found; load the package with pkgload::load_all('pvisim')")
|
||||
p
|
||||
}
|
||||
|
||||
#' Load parameters for one world and one truth scenario.
|
||||
#'
|
||||
#' @param world name in `worlds:` (W0, W1, W1_rho0, W2, W3, W4, W5, WR).
|
||||
#' @param scenario name in `scenarios:` (null, harm, benefit_harm).
|
||||
#' @param path YAML file; defaults to the package's dgp_default.yml.
|
||||
#' @param overrides optional list merged last (used by calibration and tests).
|
||||
#' @param calibrated apply the world's entry in `calibration_file` (intercepts solved by
|
||||
#' simulations/01_calibrate.R under the null scenario), after the world and before the
|
||||
#' scenario. Calibration itself runs with FALSE.
|
||||
load_params <- function(world = "W0", scenario = "null", path = default_params_path(),
|
||||
overrides = NULL, calibrated = TRUE,
|
||||
calibration_file = "simulations/config/calibrated.yml") {
|
||||
raw <- yaml::read_yaml(path)
|
||||
if (!world %in% names(raw$worlds)) stop("unknown world: ", world)
|
||||
if (!scenario %in% names(raw$scenarios)) stop("unknown scenario: ", scenario)
|
||||
par <- raw[setdiff(names(raw), c("worlds", "scenarios"))]
|
||||
par <- deep_merge(par, raw$worlds[[world]])
|
||||
if (calibrated && file.exists(calibration_file)) {
|
||||
cal <- yaml::read_yaml(calibration_file)[[world]]
|
||||
if (!is.null(cal)) par <- deep_merge(par, cal)
|
||||
par$calibrated <- !is.null(cal)
|
||||
} else {
|
||||
par$calibrated <- FALSE
|
||||
}
|
||||
par <- deep_merge(par, raw$scenarios[[scenario]])
|
||||
if (!is.null(overrides)) par <- deep_merge(par, overrides)
|
||||
par$world <- world
|
||||
par$scenario <- scenario
|
||||
par
|
||||
}
|
||||
+106
@@ -0,0 +1,106 @@
|
||||
# Simulated randomized trials (plan section 8, D18).
|
||||
#
|
||||
# Realistic RCT: hospital-based (OBL = 0) physicians in the top third of skill and the
|
||||
# bottom half of style; on average 30% of their eligible patients consent, less often when
|
||||
# older or frailer (OR 0.7 per 10 years and per SD of frailty; D55); n = 240.
|
||||
# Ideal pragmatic RCT: all physicians, everyone consents, large n.
|
||||
# Assignment A: PVI at a scheduled month 0..2 (uniform); 10% refuse and follow usual care.
|
||||
# Assignment B: no protocol PVI in the window; crossover happens through the DGP's own
|
||||
# PVI process (usual care), and usual care continues after the window.
|
||||
|
||||
RCT_DESIGNS <- list(
|
||||
realistic = list(n = 240L, consent = 0.30, select = TRUE, consent_or = c(age10 = 0.7, frailty = 0.7)),
|
||||
ideal = list(n = NA, consent = 1.00, select = FALSE, consent_or = NULL)
|
||||
)
|
||||
RCT_HORIZONS <- c(12L, 24L, 60L) # trials usually report at 1-2 years (D55)
|
||||
|
||||
#' Probability that each eligible patient consents: mean `spec$consent`, lower for older
|
||||
#' and frailer patients (Wasmann 2019, King 2005: refusers differ moderately by covariates).
|
||||
consent_prob <- function(pat, spec) {
|
||||
if (is.null(spec$consent_or)) return(rep(spec$consent, nrow(pat)))
|
||||
lp <- log(spec$consent_or[["age10"]]) * (pat$age - 76) / 10 + log(spec$consent_or[["frailty"]]) * pat$frailty
|
||||
a <- uniroot(function(a) mean(plogis(a + lp)) - spec$consent, c(-10, 10))$root
|
||||
plogis(a + lp)
|
||||
}
|
||||
|
||||
#' Physicians a realistic trial would recruit.
|
||||
trial_physicians <- function(phys) {
|
||||
phys[obl == 0 & skill >= quantile(skill, 2 / 3) & style <= median(style), phys_id]
|
||||
}
|
||||
|
||||
#' Draw a trial population from `pop` (patients at diagnosis) and its random assignment.
|
||||
#' @return list(pop = population restricted to enrolled patients, arm, refuse, sched).
|
||||
enrol_trial <- function(pop, design = c("realistic", "ideal"), n = NULL, seed = 1L) {
|
||||
design <- match.arg(design)
|
||||
spec <- RCT_DESIGNS[[design]]
|
||||
set.seed(seed)
|
||||
pat <- pop$pat
|
||||
if (spec$select) pat <- pat[phys_id %in% trial_physicians(pop$phys)]
|
||||
pat <- pat[runif(.N) < consent_prob(pat, spec)]
|
||||
n <- if (!is.null(n)) n else if (is.na(spec$n)) nrow(pat) else spec$n
|
||||
if (nrow(pat) < n) stop("not enough consenting patients: ", nrow(pat), " < ", n)
|
||||
pat <- pat[sort(sample.int(.N, n))]
|
||||
pat[, id := seq_len(.N)]
|
||||
arm <- sample(rep(c("A", "B"), length.out = n))
|
||||
list(pop = list(phys = pop$phys, pat = pat), arm = arm,
|
||||
refuse = runif(n) < 0.10, sched = sample(0:2, n, replace = TRUE))
|
||||
}
|
||||
|
||||
#' The trial protocol as a patient-specific policy.
|
||||
trial_policy <- function(trial, assign = NULL) {
|
||||
arm <- if (is.null(assign)) trial$arm else rep(assign, length(trial$arm))
|
||||
force <- arm == "A" & !trial$refuse
|
||||
policy("custom", decide = function(t, natural, idx) {
|
||||
go <- natural
|
||||
f <- force[idx]
|
||||
go[f] <- t >= trial$sched[idx][f] # PVI at the scheduled month, or the first month eligible after it
|
||||
go
|
||||
})
|
||||
}
|
||||
|
||||
#' Run one simulated trial and analyse it: ITT (by assignment, unadjusted) and naive
|
||||
#' as-treated (PVI in the window vs not, ignoring assignment).
|
||||
run_rct <- function(par, design = "realistic", rep = 1L, outcomes = c(limb = "t_limb", male = "t_male", disease = "t_disease"),
|
||||
source_n = 60000L) {
|
||||
seed <- replicate_seed(par$world, paste0(par$scenario, "|rct|", design), rep)
|
||||
pop <- make_population(par, n = source_n, seed = seed)
|
||||
trial <- enrol_trial(pop, design, n = if (design == "ideal") par$size$n_patients else NULL, seed = seed + 1L)
|
||||
ev <- simulate_course(trial$pop, par, trial_policy(trial), seed = seed + 2L, record = FALSE)$events
|
||||
W <- par$size$window
|
||||
ev[, `:=`(phys_id = trial$pop$pat$phys_id, assigned = trial$arm,
|
||||
treated_win = ifelse(!is.na(t_pvi) & t_pvi < W, "early", "none"))]
|
||||
rbindlist(lapply(names(outcomes), function(o) rbindlist(lapply(RCT_HORIZONS, function(h) {
|
||||
p <- ev[, .(id, phys_id, assigned, treated_win, t_out = get(outcomes[[o]]), t_death)]
|
||||
itt <- rd_from_aj(aj_cif(expand_followup(copy(p)[, `:=`(arm = assigned, w = 1)], 0L, h), h), "A", "B")
|
||||
at <- rd_from_aj(aj_cif(expand_followup(copy(p)[, `:=`(arm = treated_win, w = 1)], 0L, h), h), "early", "none")
|
||||
rbind(cbind(analysis = "itt", itt), cbind(analysis = "as_treated", at))[
|
||||
, `:=`(outcome = o, horizon = h, design = paste0("rct_", design), n = nrow(ev),
|
||||
crossover_B = mean(ev$treated_win[ev$assigned == "B"] == "early"),
|
||||
nonadherence_A = mean(ev$treated_win[ev$assigned == "A"] == "none"))]
|
||||
}))))
|
||||
}
|
||||
|
||||
#' ITT truth for a trial design: everyone in a large trial-eligible population assigned
|
||||
#' A vs everyone assigned B, same protocol (refusal, scheduling, natural crossover).
|
||||
#' Patients are weighted by their probability of consenting, so the truth refers to the
|
||||
#' population that enrols. Horizons: 60 months (estimand rct_<design>_itt) and the
|
||||
#' shorter trial horizons (rct_<design>_itt_h12, _h24).
|
||||
rct_truth <- function(par, design = "realistic", n = 1e6, seed = 20261029L, horizons = RCT_HORIZONS) {
|
||||
pop <- truth_population(par, n = n, seed = seed)
|
||||
pat <- if (design == "realistic") pop$pat[phys_id %in% trial_physicians(pop$phys)] else pop$pat
|
||||
pc <- consent_prob(pat, RCT_DESIGNS[[design]])
|
||||
set.seed(seed + 1L)
|
||||
k <- nrow(pat)
|
||||
trial <- list(pop = list(phys = pop$phys, pat = copy(pat)[, id := seq_len(.N)]),
|
||||
arm = rep("A", k), refuse = runif(k) < 0.10, sched = sample(0:2, k, replace = TRUE))
|
||||
runs <- lapply(c("A", "B"), function(a)
|
||||
simulate_course(trial$pop, par, trial_policy(trial, assign = a), seed = seed + 2L, record = FALSE)$events)
|
||||
w <- pc / sum(pc)
|
||||
rbindlist(lapply(names(OUTCOMES), function(o) rbindlist(lapply(horizons, function(h) {
|
||||
d <- as.numeric(by_h(runs[[1]][[OUTCOMES[[o]]]], h)) - as.numeric(by_h(runs[[2]][[OUTCOMES[[o]]]], h))
|
||||
m <- sum(w * d)
|
||||
data.table(world = par$world, scenario = par$scenario, outcome = o,
|
||||
estimand = paste0("rct_", design, "_itt", if (h != 60L) paste0("_h", h)), value = m,
|
||||
mcse = sqrt(sum(w^2 * (d - m)^2)), n = length(d))
|
||||
}))))
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
# What the analyst sees: simulated Medicare claims (plan D17).
|
||||
# Latent quantities (Z_t, true smoking, frailty, skill, style, progression rate) never
|
||||
# leave this function except in `hidden`, which analysis code must not touch.
|
||||
|
||||
HIDDEN_COLS <- c("skill", "style", "frailty", "fi_noise", "smoking_true", "z0", "rate", "venous", "z")
|
||||
|
||||
#' Turn a population and its simulated course into claims-like records.
|
||||
#'
|
||||
#' @return list(patients = one row per patient: measured baseline and coded event times;
|
||||
#' pm = person-months (id, t, sev_obs) from month 0 to death or month 59;
|
||||
#' hidden = latent traits, for oracle analyses and checks only).
|
||||
make_records <- function(pop, course, par) {
|
||||
pat <- pop$pat; ev <- course$events; ms <- par$measurement
|
||||
M <- par$size$months
|
||||
smoking <- if (isTRUE(ms$smoking) || identical(ms$smoking, "true")) pat$smoking_true else pat$smoking_coded
|
||||
rel <- ms$frailty_reliability
|
||||
fi <- sqrt(rel) * pat$frailty + sqrt(1 - rel) * pat$fi_noise
|
||||
|
||||
patients <- data.table(id = pat$id, phys_id = pat$phys_id, obl = pat$obl, age = pat$age,
|
||||
female = pat$female, black = pat$black, year = pat$year,
|
||||
diabetes = pat$diabetes, ckd = pat$ckd, chf = pat$chf, copd = pat$copd,
|
||||
prior_cvd = pat$prior_cvd, smoking = smoking, frailty_index = fi)
|
||||
patients <- cbind(patients, ev[, !"id"])
|
||||
# t_clti_true is latent: only the coded date is in claims
|
||||
patients[, t_clti_true := NULL]
|
||||
|
||||
# observed severity: exact Z_t (clean worlds) or a claims proxy (world W5, WR)
|
||||
last <- pmin(ifelse(is.na(ev$t_death), M - 1L, ev$t_death), M - 1L)
|
||||
nrep <- last + 1L
|
||||
id <- rep(pat$id, nrep)
|
||||
t <- sequence(nrep) - 1L
|
||||
idx <- cbind(id, t + 1L)
|
||||
if (identical(ms$severity, "exact")) {
|
||||
sev <- course$z[idx]
|
||||
} else if (identical(ms$severity, "noisy")) {
|
||||
sev <- course$z[idx] + rnorm(length(id), 0, ms$noise_sd) # EHR/registry: measured with error
|
||||
} else {
|
||||
w <- ms$proxy_window
|
||||
cs <- t(apply(course$counts, 1, function(x) { x[is.na(x)] <- 0L; cumsum(x) }))
|
||||
lagged <- cbind(matrix(0, nrow(cs), w), cs)[, seq_len(M)]
|
||||
sev <- log1p(cs[idx] - lagged[idx])
|
||||
}
|
||||
pm <- data.table(id = id, t = t, sev_obs = sev)
|
||||
|
||||
hidden <- data.table(id = pat$id, skill = pat$skill, style = pat$style, frailty = pat$frailty,
|
||||
smoking_true = pat$smoking_true, rate = pat$rate,
|
||||
t_clti_true = ev$t_clti_true)
|
||||
hidden_pm <- data.table(id = id, t = t, z = course$z[idx])
|
||||
list(patients = patients, pm = pm, hidden = hidden, hidden_pm = hidden_pm, par = par)
|
||||
}
|
||||
|
||||
#' Stop if an analysis table carries a latent column.
|
||||
assert_no_hidden <- function(dt) {
|
||||
bad <- intersect(names(dt), HIDDEN_COLS)
|
||||
if (length(bad)) stop("latent columns in analysis data: ", paste(bad, collapse = ", "))
|
||||
invisible(TRUE)
|
||||
}
|
||||
|
||||
#' One replicate's records: population, natural course, claims view.
|
||||
simulate_records <- function(par, rep = 1L, n = par$size$n_patients) {
|
||||
seed <- replicate_seed(par$world, par$scenario, rep)
|
||||
pop <- make_population(par, n = n, seed = seed)
|
||||
course <- simulate_course(pop, par, policy("natural"), seed = seed + 104729L, record = TRUE)
|
||||
make_records(pop, course, par)
|
||||
}
|
||||
|
||||
#' Reproducible seed for a world x scenario x replicate.
|
||||
replicate_seed <- function(world, scenario, rep) {
|
||||
key <- paste(world, scenario, rep, sep = "|")
|
||||
h <- sum(utf8ToInt(key) * seq_along(utf8ToInt(key))^2)
|
||||
as.integer((h * 2654435761) %% 2147483647)
|
||||
}
|
||||
|
||||
#' Physician-cluster bootstrap resample of a replicate's records (D28): physicians are
|
||||
#' drawn with replacement and each draw brings all of its patients, with new patient and
|
||||
#' physician ids so repeated physicians count as separate clusters.
|
||||
bootstrap_records <- function(rec, seed) {
|
||||
set.seed(seed)
|
||||
phys <- unique(rec$patients$phys_id)
|
||||
draw <- data.table(phys_id = sample(phys, length(phys), replace = TRUE), new_phys = seq_along(phys))
|
||||
map <- rec$patients[, .(id, phys_id)][draw, on = "phys_id", allow.cartesian = TRUE]
|
||||
map[, new_id := seq_len(.N)]
|
||||
remap <- function(x) {
|
||||
x <- x[map[, .(id, new_id)], on = "id", allow.cartesian = TRUE]
|
||||
x[, id := new_id][, new_id := NULL][]
|
||||
}
|
||||
out <- rec
|
||||
out$patients <- remap(rec$patients)[map[, .(new_id, new_phys)], on = c(id = "new_id"), phys_id := i.new_phys]
|
||||
out$pm <- setkey(remap(rec$pm), id, t)
|
||||
out$hidden <- remap(rec$hidden)
|
||||
out$hidden_pm <- remap(rec$hidden_pm)
|
||||
out
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
# The simulated world: physicians, patients, and the monthly course of disease and care.
|
||||
# Analysis plan v1.0, section 5.1.
|
||||
#
|
||||
# Common random numbers. simulate_course() draws the same uniforms for every patient
|
||||
# in every month, in a fixed order, whatever the patient's state or the policy. So two
|
||||
# runs on the same population with the same seed differ only through the policy. This
|
||||
# keeps truths under different strategies paired, and makes the sharp null exact at the
|
||||
# patient level (tested).
|
||||
|
||||
STREAMS <- c("eps", "proxy", "clti", "false_code", "amp", "bypass", "rr", "clti_revasc",
|
||||
"nco1", "nco2", "nce", "pvi", "death")
|
||||
|
||||
#' Physicians: latent skill k and style s (standard normal, correlated), setting, panel weight.
|
||||
make_physicians <- function(par, seed) {
|
||||
set.seed(seed)
|
||||
J <- par$size$n_physicians
|
||||
rho <- par$physician$rho_skill_style
|
||||
z1 <- rnorm(J); z2 <- rnorm(J)
|
||||
s <- z1
|
||||
k <- rho * z1 + sqrt(1 - rho^2) * z2
|
||||
obl <- rbinom(J, 1, plogis(par$physician$obl_intercept + par$physician$obl_style * s))
|
||||
w <- rlnorm(J, 0, par$size$panel_sdlog)
|
||||
data.table(phys_id = seq_len(J), skill = k, style = s, obl = obl, weight = w / sum(w))
|
||||
}
|
||||
|
||||
#' Patients at diagnosis: measured baseline, latent traits, and assigned physician.
|
||||
make_patients <- function(par, n, phys, seed) {
|
||||
set.seed(seed)
|
||||
pp <- par$patient
|
||||
phys_id <- sample(phys$phys_id, n, replace = TRUE, prob = phys$weight)
|
||||
age <- pp$age_min + rgamma(n, shape = pp$age_shape, scale = pp$age_scale)
|
||||
frailty <- rnorm(n)
|
||||
a10 <- (age - 76) / 10
|
||||
comorb <- lapply(pp$comorbidity, function(cm)
|
||||
rbinom(n, 1, plogis(qlogis(cm$prev) + cm$age_coef * a10 + cm$frailty_coef * frailty)))
|
||||
smoking <- rbinom(n, 1, plogis(qlogis(pp$smoking$prev) + pp$smoking$age_coef * a10))
|
||||
sv <- par$severity
|
||||
z0 <- sv$z0_diabetes * comorb$diabetes + sv$z0_smoking * smoking + sv$z0_ckd * comorb$ckd +
|
||||
rnorm(n, 0, sv$z0_sd)
|
||||
rate <- sv$r0 + sv$r_diabetes * comorb$diabetes + sv$r_smoking * smoking + rnorm(n, 0, sv$r_sd)
|
||||
venous <- rnorm(n)
|
||||
year <- sample(pp$year_min:pp$year_max, n, replace = TRUE)
|
||||
# what claims show for smoking and frailty (the analyst's versions; plan D17)
|
||||
ms <- par$measurement
|
||||
smoking_coded <- ifelse(smoking == 1, rbinom(n, 1, ms$smoking_sens), rbinom(n, 1, 1 - ms$smoking_spec))
|
||||
fi_noise <- rnorm(n)
|
||||
pat <- data.table(id = seq_len(n), phys_id = phys_id, age = age,
|
||||
female = rbinom(n, 1, pp$p_female), black = rbinom(n, 1, pp$p_black),
|
||||
year = year)
|
||||
for (nm in names(comorb)) set(pat, j = nm, value = comorb[[nm]])
|
||||
pat[, `:=`(smoking_true = smoking, smoking_coded = smoking_coded, frailty = frailty,
|
||||
fi_noise = fi_noise, z0 = z0, rate = rate, venous = venous)]
|
||||
pat <- merge(pat, phys[, .(phys_id, skill, style, obl)], by = "phys_id", sort = FALSE)
|
||||
setorder(pat, id)
|
||||
pat[]
|
||||
}
|
||||
|
||||
#' A full population for one replicate (or for the truths).
|
||||
make_population <- function(par, n = par$size$n_patients, seed = 1L, phys = NULL) {
|
||||
if (is.null(phys)) phys <- make_physicians(par, seed)
|
||||
pat <- make_patients(par, n, phys, seed + 7919L)
|
||||
list(phys = phys, pat = pat)
|
||||
}
|
||||
|
||||
# ---- policies -----------------------------------------------------------------------
|
||||
# A policy overrides the natural elective-PVI decision among patients eligible in month t
|
||||
# (alive, no PVI yet, no MALE yet). Non-elective revascularization after CLTI is never
|
||||
# overridden: it happens after a MALE event, which every strategy allows.
|
||||
# natural observed practice
|
||||
# A natural in months 0..(w-2); everyone still eligible gets PVI at w-1
|
||||
# B1 no PVI in months 0..(w-1); natural afterwards
|
||||
# B2 no elective PVI ever
|
||||
# seq_force / seq_forbid (m) natural except PVI forced / forbidden at month m
|
||||
|
||||
# custom pol$decide(t, natural, idx): idx are the eligible patients' row
|
||||
# numbers, so the decision can differ by patient (simulated RCTs)
|
||||
|
||||
policy <- function(type = "natural", m = NA_integer_, decide = NULL) {
|
||||
stopifnot(type %in% c("natural", "A", "B1", "B2", "seq_force", "seq_forbid", "custom"))
|
||||
list(type = type, m = m, decide = decide)
|
||||
}
|
||||
|
||||
apply_policy <- function(pol, t, natural, window, idx = NULL) {
|
||||
if (pol$type == "custom") return(pol$decide(t, natural, idx))
|
||||
switch(pol$type,
|
||||
natural = natural,
|
||||
A = if (t < window - 1) natural else if (t == window - 1) rep(TRUE, length(natural)) else natural,
|
||||
B1 = if (t < window) rep(FALSE, length(natural)) else natural,
|
||||
B2 = rep(FALSE, length(natural)),
|
||||
seq_force = if (t == pol$m) rep(TRUE, length(natural)) else natural,
|
||||
seq_forbid = if (t == pol$m) rep(FALSE, length(natural)) else natural)
|
||||
}
|
||||
|
||||
# ---- the monthly course -------------------------------------------------------------
|
||||
|
||||
#' Simulate months 0..(months-1) for every patient under a policy.
|
||||
#'
|
||||
#' Within a month the order is: severity update; outcome events (CLTI, coding, amputation,
|
||||
#' bypass, repeat revascularization, revascularization after CLTI, negative controls);
|
||||
#' death; then the elective PVI decision among survivors still eligible. A MALE event
|
||||
#' and death in the same month count the MALE event first.
|
||||
#'
|
||||
#' @return list(pat = patient table with event times, z = N x months matrix of Z_t
|
||||
#' (record = TRUE only), counts = N x months matrix of proxy counts, pvi_month = vector).
|
||||
simulate_course <- function(pop, par, pol = policy("natural"), seed = 1L, record = TRUE) {
|
||||
pat <- pop$pat
|
||||
N <- nrow(pat)
|
||||
M <- par$size$months
|
||||
W <- par$size$window
|
||||
tr <- par$treatment; oc <- par$outcomes; nc <- par$negative_controls; ef <- par$effect
|
||||
a10 <- (pat$age - 76) / 10
|
||||
|
||||
# time-fixed parts of each linear predictor
|
||||
lp_win <- tr$window$intercept + tr$window$frailty * pat$frailty + tr$window$age10 * a10 +
|
||||
tr$window$diabetes * pat$diabetes + tr$window$ckd * pat$ckd + tr$window$smoking * pat$smoking_true +
|
||||
tr$window$style * pat$style + tr$window$obl * pat$obl + tr$window$year * (pat$year - par$patient$year_min)
|
||||
lp_def <- tr$deferred$intercept + tr$deferred$style * pat$style + tr$deferred$obl * pat$obl
|
||||
d <- oc$death
|
||||
lp_death0 <- d$intercept + d$chf * pat$chf + d$ckd * pat$ckd + d$copd * pat$copd + d$frailty * pat$frailty
|
||||
cl <- oc$clti
|
||||
lp_clti0 <- cl$intercept + cl$diabetes * pat$diabetes + cl$ckd * pat$ckd + cl$smoking * pat$smoking_true +
|
||||
cl$frailty * pat$frailty
|
||||
lp_false <- oc$clti_false_code$intercept + oc$clti_false_code$style * pat$style
|
||||
am <- oc$amputation
|
||||
lp_amp0 <- am$intercept + am$diabetes * pat$diabetes + am$ckd * pat$ckd
|
||||
rr <- oc$repeat_revasc
|
||||
lp_rr0 <- rr$intercept + rr$skill * pat$skill + rr$style * pat$style
|
||||
lp_nco1 <- nc$nco1$intercept + nc$nco1$frailty * pat$frailty + nc$nco1$smoking * pat$smoking_true +
|
||||
nc$nco1$age10 * a10
|
||||
nco1_z <- if (is.null(nc$nco1$z)) 0 else nc$nco1$z
|
||||
lp_nco2 <- nc$nco2$intercept + nc$nco2$style * pat$style + nc$nco2$venous * pat$venous
|
||||
lp_nce <- nc$nce$intercept + nc$nce$style * pat$style + nc$nce$frailty * pat$frailty
|
||||
kappa_post <- ef$kappa * exp(-ef$skill_kappa * pat$skill)
|
||||
|
||||
# state
|
||||
Z <- pat$z0
|
||||
Zn <- pat$z0 # severity on the natural course without any PVI (relief or kappa): drives NCO-1 (D53)
|
||||
alive <- rep(TRUE, N); treated <- rep(FALSE, N); clti_true <- rep(FALSE, N)
|
||||
clti_code <- rep(FALSE, N); male <- rep(FALSE, N)
|
||||
t_pvi <- t_clti_true <- t_clti_code <- t_amp <- t_bypass <- t_rr <- t_male <- t_death <-
|
||||
t_nco1 <- t_nco2 <- t_nce <- rep(NA_integer_, N)
|
||||
pvi_elective <- rep(NA, N)
|
||||
n_proc <- integer(N)
|
||||
zmat <- if (record) matrix(NA_real_, N, M) else NULL
|
||||
cmat <- if (record) matrix(NA_integer_, N, M) else NULL
|
||||
ns <- length(STREAMS)
|
||||
|
||||
set.seed(seed)
|
||||
for (t in 0:(M - 1)) {
|
||||
U <- matrix(runif(N * ns), N, ns, dimnames = list(NULL, STREAMS))
|
||||
a <- alive
|
||||
if (t > 0) {
|
||||
eps <- par$severity$eps_sd * qnorm(U[a, "eps"])
|
||||
Z[a] <- Z[a] + pat$rate[a] + kappa_post[a] * treated[a] + eps
|
||||
Zn[a] <- Zn[a] + pat$rate[a] + eps
|
||||
}
|
||||
if (record) {
|
||||
zmat[a, t + 1] <- Z[a]
|
||||
cmat[a, t + 1] <- qpois(U[a, "proxy"], exp(par$measurement$proxy_intercept + par$measurement$proxy_z * Z[a]))
|
||||
}
|
||||
|
||||
# true CLTI, and its coding (true onset always coded; false codes by physician style)
|
||||
new_clti <- a & !clti_true & U[, "clti"] < plogis(lp_clti0 + cl$z * Z)
|
||||
clti_true[new_clti] <- TRUE; t_clti_true[new_clti] <- t
|
||||
new_code <- a & !clti_code & (new_clti | U[, "false_code"] < plogis(lp_false))
|
||||
clti_code[new_code] <- TRUE; t_clti_code[new_code] <- t
|
||||
|
||||
new_amp <- a & is.na(t_amp) & U[, "amp"] < plogis(lp_amp0 + am$clti * clti_true + am$z * Z)
|
||||
t_amp[new_amp] <- t
|
||||
new_byp <- a & is.na(t_bypass) & U[, "bypass"] < plogis(oc$bypass$intercept + oc$bypass$z * Z)
|
||||
t_bypass[new_byp] <- t
|
||||
n_proc[new_byp] <- n_proc[new_byp] + 1L
|
||||
|
||||
# repeat revascularization: only after the index PVI (from the month after it)
|
||||
new_rr <- a & treated & !is.na(t_pvi) & t > t_pvi & U[, "rr"] < plogis(lp_rr0 + rr$z * Z)
|
||||
t_rr[new_rr & is.na(t_rr)] <- t
|
||||
n_proc[new_rr] <- n_proc[new_rr] + 1L
|
||||
|
||||
# revascularization after a CLTI diagnosis (not elective; allowed by every strategy)
|
||||
cr <- a & clti_code & U[, "clti_revasc"] < plogis(oc$clti_revasc$intercept)
|
||||
first <- cr & !treated
|
||||
rep_ <- cr & treated & !new_rr
|
||||
t_rr[rep_ & is.na(t_rr)] <- t
|
||||
n_proc[cr & !new_rr] <- n_proc[cr & !new_rr] + 1L
|
||||
if (any(first)) {
|
||||
treated[first] <- TRUE; t_pvi[first] <- t; pvi_elective[first] <- FALSE
|
||||
Z[first] <- Z[first] - ef$relief
|
||||
}
|
||||
|
||||
# negative controls (never depend on PVI)
|
||||
x <- a & is.na(t_nco1) & U[, "nco1"] < plogis(lp_nco1 + nco1_z * Zn); t_nco1[x] <- t
|
||||
x <- a & is.na(t_nco2) & U[, "nco2"] < plogis(lp_nco2); t_nco2[x] <- t
|
||||
if (t < W) { x <- a & is.na(t_nce) & U[, "nce"] < plogis(lp_nce); t_nce[x] <- t }
|
||||
|
||||
# MALE: coded CLTI, amputation, bypass, or repeat revascularization
|
||||
new_male <- a & !male & (clti_code | !is.na(t_amp) | !is.na(t_bypass) | !is.na(t_rr))
|
||||
male[new_male] <- TRUE; t_male[new_male] <- t
|
||||
|
||||
# death
|
||||
dies <- a & U[, "death"] < plogis(lp_death0 + d$age10 * a10 + d$age10 * t / 120 + d$z * Z + d$clti * clti_true)
|
||||
alive[dies] <- FALSE; t_death[dies] <- t
|
||||
|
||||
# elective PVI among survivors still eligible
|
||||
elig <- alive & !treated & !male
|
||||
if (any(elig)) {
|
||||
lp <- if (t < W) lp_win[elig] + tr$window$z * Z[elig] else lp_def[elig] + tr$deferred$z * Z[elig]
|
||||
natural <- U[elig, "pvi"] < plogis(lp)
|
||||
go <- apply_policy(pol, t, natural, W, which(elig))
|
||||
idx <- which(elig)[go]
|
||||
if (length(idx)) {
|
||||
treated[idx] <- TRUE; t_pvi[idx] <- t; pvi_elective[idx] <- TRUE
|
||||
n_proc[idx] <- n_proc[idx] + 1L
|
||||
Z[idx] <- Z[idx] - ef$relief
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# disease composite (D34): coded CLTI or major amputation
|
||||
t_disease <- pmin(t_clti_code, t_amp, na.rm = TRUE)
|
||||
# limb MALE (primary, D56): coded CLTI, major amputation or open bypass. Every component
|
||||
# can occur with or without a prior PVI; repeat revascularization is reported separately
|
||||
t_limb <- pmin(t_clti_code, t_amp, t_bypass, na.rm = TRUE)
|
||||
out <- data.table(id = pat$id, t_pvi, pvi_elective, t_clti_true, t_clti_code, t_amp, t_bypass,
|
||||
t_rr, t_male, t_disease, t_limb, t_death, t_nco1, t_nco2, t_nce, n_proc)
|
||||
list(events = out, z = zmat, counts = cmat, policy = pol)
|
||||
}
|
||||
|
||||
#' Cumulative incidence by month `h` (event in months 0..h-1, before death or in the same month).
|
||||
cum_inc <- function(t_event, h = 60L) mean(!is.na(t_event) & t_event < h)
|
||||
@@ -0,0 +1,92 @@
|
||||
# Truths by simulating the interventions in a large population (plan section 5.5).
|
||||
#
|
||||
# The truth population keeps the replicate's panel structure (patients per physician)
|
||||
# but has proportionally more physicians, so the truth averages over the physician
|
||||
# distribution rather than over one draw of 1,500 physicians. Every policy is run on the
|
||||
# same population with the same course seed (common random numbers), so risk
|
||||
# differences are paired and their Monte Carlo SE is sd(paired difference) / sqrt(n).
|
||||
#
|
||||
# Landmark, naive and RCT estimands are not interventional in this sense; they are added
|
||||
# with the estimators (oracle large-sample limits).
|
||||
|
||||
OUTCOMES <- c(limb = "t_limb", male = "t_male", disease = "t_disease", clti = "t_clti_code", amp = "t_amp",
|
||||
bypass = "t_bypass", rr = "t_rr", death = "t_death", nco1 = "t_nco1", nco2 = "t_nco2")
|
||||
|
||||
truth_population <- function(par, n, seed) {
|
||||
scale <- n / par$size$n_patients
|
||||
par$size$n_physicians <- max(1L, round(par$size$n_physicians * scale))
|
||||
make_population(par, n = n, seed = seed)
|
||||
}
|
||||
|
||||
# event indicator by month h (event in months 0..h-1)
|
||||
by_h <- function(t, h) !is.na(t) & t < h
|
||||
|
||||
#' Truths for one world x scenario.
|
||||
#'
|
||||
#' @return list(truths = long table: outcome, estimand, value, mcse, n;
|
||||
#' curves = cumulative incidence by month under natural, A, B1, B2 for MALE and disease).
|
||||
compute_truths <- function(world, scenario, n = 1e6, seed = 20261027L, horizon = 60L,
|
||||
window = NULL, verbose = TRUE) {
|
||||
par <- load_params(world, scenario)
|
||||
W <- if (is.null(window)) par$size$window else window
|
||||
par$size$window <- W
|
||||
t0 <- Sys.time()
|
||||
pop <- truth_population(par, n, seed)
|
||||
cseed <- seed + 1L
|
||||
run <- function(p) simulate_course(pop, par, p, seed = cseed, record = FALSE)$events
|
||||
|
||||
ev <- list(natural = run(policy("natural")), A = run(policy("A")),
|
||||
B1 = run(policy("B1")), B2 = run(policy("B2")))
|
||||
rows <- list()
|
||||
add <- function(outcome, estimand, d, nn = length(d)) {
|
||||
rows[[length(rows) + 1L]] <<- data.table(world = world, scenario = scenario, outcome = outcome,
|
||||
estimand = estimand, value = mean(d),
|
||||
mcse = sd(d) / sqrt(nn), n = nn)
|
||||
}
|
||||
for (o in names(OUTCOMES)) {
|
||||
col <- OUTCOMES[[o]]
|
||||
y <- lapply(ev, function(e) as.numeric(by_h(e[[col]], horizon)))
|
||||
for (p in names(y)) add(o, paste0("risk_", p), y[[p]])
|
||||
add(o, "rd_A_B1", y$A - y$B1)
|
||||
add(o, "rd_A_B2", y$A - y$B2)
|
||||
}
|
||||
# total procedures per patient (secondary outcome, D7)
|
||||
np <- lapply(ev, function(e) e$n_proc)
|
||||
for (p in names(np)) add("n_proc", paste0("mean_", p), np[[p]])
|
||||
add("n_proc", "diff_A_B1", np$A - np$B1)
|
||||
|
||||
# sequential trials, ITT (D14): among patients eligible at month m under the natural
|
||||
# course, PVI forced vs forbidden at m, natural afterwards; risk by month `horizon`
|
||||
# since diagnosis; pooled over m = 0..W-1 weighted by the number eligible.
|
||||
nat <- ev$natural
|
||||
seq_rows <- list()
|
||||
for (m in 0:(W - 1)) {
|
||||
elig <- (is.na(nat$t_death) | nat$t_death > m) & (is.na(nat$t_pvi) | nat$t_pvi >= m) &
|
||||
(is.na(nat$t_male) | nat$t_male > m)
|
||||
f <- run(policy("seq_force", m)); b <- run(policy("seq_forbid", m))
|
||||
for (o in names(OUTCOMES)) {
|
||||
col <- OUTCOMES[[o]]
|
||||
d <- as.numeric(by_h(f[[col]][elig], horizon) & f[[col]][elig] > m) -
|
||||
as.numeric(by_h(b[[col]][elig], horizon) & b[[col]][elig] > m)
|
||||
seq_rows[[length(seq_rows) + 1L]] <- data.table(outcome = o, m = m, n_elig = sum(elig),
|
||||
rd = mean(d), var = var(d))
|
||||
}
|
||||
}
|
||||
seq_dt <- rbindlist(seq_rows)
|
||||
pooled <- seq_dt[, .(value = sum(rd * n_elig) / sum(n_elig),
|
||||
mcse = sqrt(sum(n_elig^2 * var / n_elig)) / sum(n_elig),
|
||||
n = sum(n_elig)), by = outcome]
|
||||
rows[[length(rows) + 1L]] <- pooled[, .(world, scenario, outcome, estimand = "seq_itt", value, mcse, n)]
|
||||
for (mm in 0:(W - 1)) {
|
||||
x <- seq_dt[m == mm]
|
||||
rows[[length(rows) + 1L]] <- x[, .(world, scenario, outcome, estimand = paste0("seq_itt_m", mm),
|
||||
value = rd, mcse = sqrt(var / n_elig), n = n_elig)]
|
||||
}
|
||||
|
||||
curves <- rbindlist(lapply(names(ev), function(p) rbindlist(lapply(c("limb", "male", "disease"), function(o)
|
||||
data.table(world = world, scenario = scenario, policy = p, outcome = o, month = seq_len(horizon),
|
||||
cuminc = vapply(seq_len(horizon), function(h) mean(by_h(ev[[p]][[OUTCOMES[[o]]]], h)), 0))))))
|
||||
if (verbose) message(sprintf("%s/%s: %.1f min", world, scenario,
|
||||
as.numeric(Sys.time() - t0, units = "mins")))
|
||||
list(truths = rbindlist(rows), curves = curves)
|
||||
}
|
||||
@@ -0,0 +1,129 @@
|
||||
# Every number that defines the simulated world. Analysis plan v1.0, section 5.
|
||||
# Time unit: month. Hazards are monthly, on the logit scale.
|
||||
# Starting values are placeholders, to be tuned by calibration
|
||||
# (simulations/01_calibrate.R, targets in plan section 5.4).
|
||||
# Physician traits (skill k, style s) and patient frailty F are standard normal;
|
||||
# their influence is set by the coefficients that multiply them.
|
||||
|
||||
size:
|
||||
n_patients: 50000
|
||||
n_physicians: 1500
|
||||
panel_sdlog: 1.0 # lognormal panel weights; median panel about 20 at n = 50,000
|
||||
months: 60 # follow-up months 0-59
|
||||
window: 6 # strategy A: PVI in months 0-5
|
||||
|
||||
physician:
|
||||
rho_skill_style: -0.3 # corr(k, s). Has no effect unless skill matters (world W1)
|
||||
obl_intercept: 0.0 # logit P(office-based lab / ASC); about 50% of physicians
|
||||
obl_style: 0.8 # high-style physicians more often work in OBL/ASC
|
||||
|
||||
patient:
|
||||
age_min: 66
|
||||
age_shape: 2.5 # age = 66 + Gamma(shape, scale); mean about 76
|
||||
age_scale: 4.0
|
||||
p_female: 0.50
|
||||
p_black: 0.10
|
||||
# comorbidities: logit P = qlogis(prev) + age_coef * (age - 76) / 10 + frailty_coef * F
|
||||
comorbidity:
|
||||
diabetes: {prev: 0.40, age_coef: -0.10, frailty_coef: 0.4}
|
||||
ckd: {prev: 0.20, age_coef: 0.30, frailty_coef: 0.5}
|
||||
chf: {prev: 0.20, age_coef: 0.40, frailty_coef: 0.6}
|
||||
copd: {prev: 0.25, age_coef: 0.10, frailty_coef: 0.4}
|
||||
prior_cvd: {prev: 0.45, age_coef: 0.20, frailty_coef: 0.3}
|
||||
smoking: {prev: 0.25, age_coef: -0.40}
|
||||
year_min: 2015
|
||||
year_max: 2019
|
||||
|
||||
severity: # latent Z_t: walking limitation / ABI on a standard scale
|
||||
z0_diabetes: 0.3
|
||||
z0_smoking: 0.3
|
||||
z0_ckd: 0.2
|
||||
z0_sd: 1.0
|
||||
r0: 0.02 # mean drift per month
|
||||
r_diabetes: 0.01
|
||||
r_smoking: 0.01
|
||||
r_sd: 0.01 # patient-specific progression rate
|
||||
eps_sd: 0.05 # month-to-month noise
|
||||
|
||||
treatment:
|
||||
window: # elective PVI hazard, months 0-5, among the eligible
|
||||
intercept: -5.4
|
||||
z: 0.9 # confounding by indication
|
||||
frailty: -0.3 # frail patients are less often offered PVI
|
||||
age10: -0.3
|
||||
diabetes: 0.1
|
||||
ckd: -0.1
|
||||
smoking: 0.1
|
||||
style: 0.5 # physician style; world W2 sets it to the strong level
|
||||
obl: 0.3
|
||||
year: 0.05 # per calendar year after year_min
|
||||
deferred: # elective PVI hazard from month 6 on (deferral / crossover)
|
||||
intercept: -7.2
|
||||
z: 0.0 # world W4 makes deferral severity-driven
|
||||
style: 0.5
|
||||
obl: 0.3
|
||||
|
||||
effect: # what a PVI does. Set by the truth scenario
|
||||
relief: 0.0 # immediate drop in Z at PVI
|
||||
kappa: 0.0 # extra Z drift per month after PVI
|
||||
skill_kappa: 0.0 # kappa * exp(-skill_kappa * k): skill slows post-PVI progression
|
||||
|
||||
outcomes:
|
||||
death: {intercept: -5.0, age10: 0.7, chf: 0.4, ckd: 0.4, copd: 0.3, frailty: 0.4, z: 0.1, clti: 0.8}
|
||||
clti: {intercept: -6.6, z: 0.6, diabetes: 0.4, ckd: 0.3, smoking: 0.3, frailty: 0.2}
|
||||
clti_false_code: {intercept: -9.0, style: 0.0} # factor 3 sets style
|
||||
amputation: {intercept: -8.5, clti: 3.0, diabetes: 0.4, ckd: 0.4, z: 0.2}
|
||||
bypass: {intercept: -7.8, z: 0.5}
|
||||
repeat_revasc: {intercept: -5.3, skill: 0.0, style: 0.0, z: 0.2} # only after the index PVI
|
||||
clti_revasc: {intercept: -4.5} # PVI after CLTI (not elective); most CLTI patients get bypass, tibial work, amputation or nothing
|
||||
|
||||
negative_controls: # never depend on PVI
|
||||
nco1: {intercept: -6.75, frailty: 0.5, smoking: 0.4, age10: 0.3, z: 0.3} # hip fracture / pneumonia; z = PVI-free severity (D53)
|
||||
nco2: {intercept: -7.1, style: 0.8, venous: 0.5} # varicose-vein ablation
|
||||
nce: {intercept: -5.7, style: 0.8, frailty: -0.2} # low-value procedure, months 0-5
|
||||
|
||||
measurement: # what the analyst sees. World W5 switches to claims proxies
|
||||
severity: exact # exact: analyst sees Z_t. proxy: only visit/testing counts. noisy: Z_t + error (EHR/registry)
|
||||
noise_sd: 0.75 # noisy: SD of the monthly measurement error; Z_t has SD about 1 at diagnosis, so cor(observed, true) is about 0.8
|
||||
proxy_intercept: -1.0 # monthly count ~ Poisson(exp(proxy_intercept + proxy_z * Z_t))
|
||||
proxy_z: 0.5
|
||||
proxy_window: 3 # observed severity = log1p(sum of counts over the last 3 months)
|
||||
frailty_reliability: 1.0 # frailty index = sqrt(rel) * F + sqrt(1 - rel) * noise
|
||||
smoking: true # true: analyst sees true smoking. coded: sensitivity / specificity below
|
||||
smoking_sens: 0.6
|
||||
smoking_spec: 0.98
|
||||
|
||||
worlds: # one factor at a time (plan section 5.2, D24). Overrides on the default.
|
||||
# Intercepts are then re-solved per world by calibration, so every world
|
||||
# matches the same Medicare targets unless `calibration_targets` says otherwise.
|
||||
W0: {}
|
||||
W1:
|
||||
effect: {skill_kappa: 0.5}
|
||||
outcomes: {repeat_revasc: {skill: -0.6}}
|
||||
W1_rho0:
|
||||
physician: {rho_skill_style: 0.0}
|
||||
effect: {skill_kappa: 0.5}
|
||||
outcomes: {repeat_revasc: {skill: -0.6}}
|
||||
W2:
|
||||
treatment: {window: {style: 1.6}, deferred: {style: 1.6}}
|
||||
W3:
|
||||
treatment: {window: {style: 1.6}, deferred: {style: 1.6}}
|
||||
outcomes: {clti_false_code: {style: 0.8}, repeat_revasc: {style: 0.6}}
|
||||
W4: # deferral common (as in trial conservative arms) and severity-driven
|
||||
treatment: {deferred: {z: 0.8}}
|
||||
calibration_targets: {late_not: 0.15}
|
||||
W5:
|
||||
measurement: {severity: proxy, frailty_reliability: 0.4, smoking: coded}
|
||||
WE: # EHR/registry-like: severity recorded (ABI, symptom class) but with error (D55)
|
||||
measurement: {severity: noisy, frailty_reliability: 0.7, smoking: true}
|
||||
WR:
|
||||
effect: {skill_kappa: 0.5}
|
||||
treatment: {window: {style: 1.6}, deferred: {style: 1.6, z: 0.8}} # deferral at the Sorber rate, but severity-driven
|
||||
outcomes: {repeat_revasc: {skill: -0.6, style: 0.6}, clti_false_code: {style: 0.8}}
|
||||
measurement: {severity: proxy, frailty_reliability: 0.4, smoking: coded}
|
||||
|
||||
scenarios: # truth scenarios (plan section 5.3, D34)
|
||||
"null": {effect: {relief: 0.0, kappa: 0.0}} # sharp disease null
|
||||
harm: {effect: {relief: 0.3, kappa: 0.039}} # W0: disease-composite RD(A-B1) +5.1 pp at 60 months
|
||||
benefit_harm: {effect: {relief: 2.0, kappa: 0.075}} # W0: RD -1.2 pp at 24 months, +3.1 pp at 60
|
||||
# tuned in W0 with simulations/explore_scenarios.R (2e5 patients); the same mechanism is used in every world
|
||||
@@ -0,0 +1,55 @@
|
||||
test_that("every world and scenario loads", {
|
||||
raw <- yaml::read_yaml(default_params_path())
|
||||
for (w in names(raw$worlds)) for (s in names(raw$scenarios)) {
|
||||
par <- load_params(w, s)
|
||||
expect_equal(par$world, w)
|
||||
expect_true(is.numeric(par$effect$relief))
|
||||
}
|
||||
expect_equal(load_params("W2")$treatment$window$style, 1.6)
|
||||
expect_equal(load_params("W2")$treatment$window$z, 0.9) # untouched by the override
|
||||
})
|
||||
|
||||
small_world <- function(world = "WR", scenario = "null", n = 4000) {
|
||||
par <- load_params(world, scenario, overrides = list(size = list(n_physicians = 150)))
|
||||
list(par = par, pop = make_population(par, n = n, seed = 11L))
|
||||
}
|
||||
|
||||
test_that("under the sharp null, disease, death and negative controls are identical under every policy", {
|
||||
sw <- small_world("WR", "null")
|
||||
runs <- lapply(list(policy("natural"), policy("A"), policy("B1"), policy("B2"),
|
||||
policy("seq_force", 3L)),
|
||||
function(p) simulate_course(sw$pop, sw$par, p, seed = 5L, record = FALSE)$events)
|
||||
for (r in runs[-1]) {
|
||||
for (col in c("t_limb", "t_disease", "t_clti_code", "t_amp", "t_bypass", "t_death", "t_nco1", "t_nco2", "t_nce")) {
|
||||
expect_identical(r[[col]], runs[[1]][[col]], info = col)
|
||||
}
|
||||
}
|
||||
# MALE is not null: repeat revascularization needs an index PVI (D34)
|
||||
expect_gt(cum_inc(runs[[2]]$t_male), cum_inc(runs[[3]]$t_male))
|
||||
})
|
||||
|
||||
test_that("policies are followed", {
|
||||
sw <- small_world("W0", "harm")
|
||||
W <- sw$par$size$window
|
||||
a <- simulate_course(sw$pop, sw$par, policy("A"), seed = 5L, record = FALSE)$events
|
||||
# under A, everyone who survives month W-1 without a MALE event has had a PVI by then
|
||||
# (death and MALE in a month come before that month's PVI decision)
|
||||
free <- (is.na(a$t_death) | a$t_death > W - 1) & (is.na(a$t_male) | a$t_male > W - 1 | (!is.na(a$t_pvi) & a$t_pvi < a$t_male))
|
||||
expect_true(all(!is.na(a$t_pvi[free]) & a$t_pvi[free] <= W - 1))
|
||||
b1 <- simulate_course(sw$pop, sw$par, policy("B1"), seed = 5L, record = FALSE)$events
|
||||
expect_false(any(b1$pvi_elective %in% TRUE & b1$t_pvi < W))
|
||||
b2 <- simulate_course(sw$pop, sw$par, policy("B2"), seed = 5L, record = FALSE)$events
|
||||
expect_false(any(b2$pvi_elective %in% TRUE))
|
||||
})
|
||||
|
||||
test_that("records carry no latent columns and person-months end at death", {
|
||||
par <- load_params("W5", "harm", overrides = list(size = list(n_physicians = 100)))
|
||||
rec <- simulate_records(par, rep = 1L, n = 2000)
|
||||
expect_true(assert_no_hidden(rec$patients))
|
||||
expect_true(assert_no_hidden(rec$pm))
|
||||
last <- rec$pm[, .(tmax = max(t)), by = id]
|
||||
dead <- rec$patients[!is.na(t_death), .(id, t_death)]
|
||||
chk <- merge(last, dead, by = "id")
|
||||
expect_true(all(chk$tmax == chk$t_death))
|
||||
expect_true(all(is.finite(rec$pm$sev_obs)))
|
||||
})
|
||||
@@ -0,0 +1,311 @@
|
||||
---
|
||||
title: "Early vs delayed PVI for claudication: known-truth simulation, draft results"
|
||||
subtitle: "Draft scale: n = 50,000 patients, 100 replicates per world x scenario"
|
||||
date: "`r format(Sys.Date())`"
|
||||
output:
|
||||
html_document:
|
||||
toc: true
|
||||
toc_float: true
|
||||
self_contained: true
|
||||
---
|
||||
|
||||
```{r setup, include = FALSE}
|
||||
knitr::opts_chunk$set(echo = FALSE, message = FALSE, warning = FALSE, fig.align = "center", dpi = 120)
|
||||
suppressPackageStartupMessages({ library(data.table); library(ggplot2); library(patchwork) })
|
||||
root <- normalizePath("..")
|
||||
S <- file.path(root, "results", "summary")
|
||||
perf <- fread(file.path(S, "perf.csv")); nc <- fread(file.path(S, "nc_reps.csv"))
|
||||
ncs <- fread(file.path(S, "nc_summary.csv")); reps <- fread(file.path(S, "reps.csv"))
|
||||
|
||||
WORLDS <- c("W0", "W1", "W1_rho0", "W2", "W3", "W4", "W5", "WE", "WR")
|
||||
WLAB <- c(W0 = "W0 Clean", W1 = "W1 Skill", W1_rho0 = "W1 Skill (rho 0)", W2 = "W2 Style",
|
||||
W3 = "W3 Style -> outcome", W4 = "W4 Deferral", W5 = "W5 Unmeasured severity", WE = "WE EHR/registry-like", WR = "WR Realistic")
|
||||
VAR <- c(V2_cluster = "Ignore physician (cluster SE)", V3_random = "Random intercept",
|
||||
V4_within = "Within-physician (FE)", none = "Trial (no adjustment)")
|
||||
VCOL <- c("Ignore physician (cluster SE)" = "#2a78d6", "Random intercept" = "#eb6834",
|
||||
"Within-physician (FE)" = "#1baf7a", "Trial (no adjustment)" = "#52514e")
|
||||
METH <- data.table(
|
||||
k = c("ccw|A_B1", "ccw|A_B2", "seq|now_not", "landmark|early_none", "landmark_m5|early_none", "naive|early_none",
|
||||
"rct_realistic|itt", "rct_ideal|itt", "rct_realistic|as_treated", "rct_ideal|as_treated"),
|
||||
method = c("CCW, A vs B1 (reference)", "CCW, A vs B2", "Sequential trials, ITT", "Landmark at 6 months", "Landmark, + month-5 severity (suppl.)",
|
||||
"Naive 'ever early PVI'", "Realistic RCT (n = 240), ITT", "Ideal pragmatic RCT, ITT",
|
||||
"Realistic RCT, as-treated", "Ideal RCT, as-treated"))
|
||||
setnames(METH, "k", "key")
|
||||
pp <- function(x, k = 1) formatC(100 * x, format = "f", digits = k)
|
||||
theme_set(theme_minimal(base_size = 11) + theme(panel.grid.minor = element_blank(),
|
||||
panel.grid.major.y = element_blank(), strip.text = element_text(face = "bold", hjust = 0),
|
||||
legend.position = "bottom"))
|
||||
perf_h <- perf[analysis == "rct"] # trials at 12, 24 and 60 months (D55)
|
||||
perf <- perf[analysis %in% c("main", "rct") & horizon == 60L] # NCE analyses are used only in Table 4
|
||||
reps <- reps[horizon == 60L]
|
||||
perf[, key := paste(design, contrast, sep = "|")]
|
||||
perf <- merge(perf, METH, by = "key")
|
||||
perf[, method := factor(method, levels = rev(METH$method))]
|
||||
perf[, wlab := factor(WLAB[world], levels = WLAB[WORLDS])]
|
||||
perf[, vlab := factor(VAR[variant], levels = VAR)]
|
||||
```
|
||||
|
||||
# Read this first
|
||||
|
||||
* **Run.** 8 worlds × 3 truth scenarios × 100 replicates of 50,000 patients (about 1,500 physicians per
|
||||
replicate). Each replicate runs naive, landmark, CCW and sequential trials under 4 physician
|
||||
variants, 3 negative controls, an oracle run, and 2 simulated RCTs. 2,400 replicate files, 0
|
||||
errors. 16 of 288,000 estimates are missing (sequential trials with physician fixed effects, 4
|
||||
replicates), and these are dropped.
|
||||
* **Outcome shown.** The headline outcome is **limb MALE (CLTI, major amputation or open bypass; D56)**,
|
||||
whose true effect under the null is exactly 0 (D34). MALE is shown where it adds something.
|
||||
* **Two errors, kept separate (D27).** *Estimation bias* = mean estimate − the method's own truth.
|
||||
*Question gap* = the method's own truth − the reference truth (CCW A vs B1). *Total error* is the sum.
|
||||
* **Variants shown.** "Ignore physician" is V1's point estimate with the cluster-robust SE (V2).
|
||||
The V1 patient-level SE is in the supplement tables.
|
||||
* **Two open questions** came out of this run (Section 6). They affect the landmark results and the
|
||||
negative-control results, so treat those parts as provisional.
|
||||
|
||||
# 1. Aim 1: what each method answers (clean world W0)
|
||||
|
||||
```{r tab2}
|
||||
t2 <- perf[world == "W0" & outcome == "limb" & variant %in% c("V2_cluster", "none") &
|
||||
!grepl("as_treated", key)]
|
||||
t2 <- t2[, .(Scenario = scenario, Method = as.character(method), `Reference truth` = pp(ref),
|
||||
`Own truth` = pp(own), `Question gap` = pp(gap), `Mean estimate` = pp(mean_est),
|
||||
`Estimation bias (MCSE)` = sprintf("%s (%s)", pp(bias, 2), pp(mcse_bias, 2)),
|
||||
`Empirical SE` = pp(emp_se), `Coverage of own truth` = sprintf("%.0f%%", 100 * cover_own))]
|
||||
t2[, Scenario := factor(Scenario, levels = c("null", "harm", "benefit_harm"))]
|
||||
t2[, Method := factor(Method, levels = METH$method)]
|
||||
knitr::kable(t2[order(Scenario, Method)], align = "llrrrrrrr",
|
||||
caption = "Table 2. Disease composite, 5-year risk difference in percentage points, world W0.")
|
||||
```
|
||||
|
||||
**What it shows.**
|
||||
|
||||
* Every TTE method is close to unbiased for **its own** truth in W0 (all |bias| < 0.35 pp). H1a holds.
|
||||
* The methods answer **different questions**. Under harm (reference +5.1 pp): sequential trials target
|
||||
+5.7, landmark +7.9 and naive +7.2. So the question gap alone is 0.6–2.8 pp. Under the null, naive
|
||||
has a gap of +1.3 pp from immortal time even with every confounder measured. H1b holds, though the
|
||||
gap for the sequential trials is small.
|
||||
* **CCW is right but imprecise.** Its empirical SE is about 2.8 pp at n = 50,000, against about 1.5 pp for
|
||||
sequential trials. Under the null its 95% CI excludes 0 in 15% of replicates (Section 3).
|
||||
* The **realistic RCT (n = 240)** is unbiased for its own ITT, but its SE is about 4.2 pp. It detects
|
||||
the 5-point harm in only 25% of replicates.
|
||||
|
||||
```{r fig1, fig.width = 11, fig.height = 9}
|
||||
ladder <- function(sc, oc = "limb") {
|
||||
x <- perf[scenario == sc & outcome == oc & variant != "V1_ignore" & !grepl("as_treated", key)]
|
||||
r <- reps[scenario == sc & outcome == oc & analysis %in% c("main", "rct") & variant != "V1_ignore" &
|
||||
variant != "oracle" & !grepl("as_treated", contrast)]
|
||||
r[, key := paste(design, contrast, sep = "|")]
|
||||
q <- r[, .(lo = quantile(est, 0.025), hi = quantile(est, 0.975)), by = .(world, key, variant)]
|
||||
x <- merge(x, q, by = c("world", "key", "variant"))
|
||||
dodge <- position_dodge(width = 0.6)
|
||||
ggplot(x, aes(y = method, group = vlab)) +
|
||||
geom_vline(aes(xintercept = 100 * ref), colour = "#0b0b0b", linewidth = 0.4) +
|
||||
geom_vline(xintercept = 0, colour = "#c3c2b7", linewidth = 0.3, linetype = 2) +
|
||||
geom_linerange(aes(xmin = 100 * lo, xmax = 100 * hi, colour = vlab), position = dodge, linewidth = 0.6) +
|
||||
geom_point(aes(x = 100 * own), position = dodge, shape = 21, size = 2.6, fill = "white", colour = "#0b0b0b") +
|
||||
geom_point(aes(x = 100 * mean_est, colour = vlab), position = dodge, size = 2) +
|
||||
facet_wrap(~wlab, ncol = 4) + scale_colour_manual(values = VCOL, name = NULL) +
|
||||
labs(x = "5-year risk difference, early − delayed (percentage points)", y = NULL,
|
||||
caption = paste("Filled dot: mean estimate (bar: 2.5th–97.5th percentile across replicates).",
|
||||
"Hollow dot: the method's own truth. Solid line: the reference truth (CCW A vs B1).")) +
|
||||
coord_cartesian(xlim = c(-15, 35))
|
||||
}
|
||||
ladder("harm") + ggtitle("Figure 1a. Estimand ladder, harm scenario (limb MALE)")
|
||||
```
|
||||
|
||||
```{r fig1b, fig.width = 11, fig.height = 9}
|
||||
ladder("null") + ggtitle("Figure 1b. Estimand ladder, null scenario (limb MALE)")
|
||||
```
|
||||
|
||||
# 2. Aim 2: surgery-specific threats
|
||||
|
||||
```{r tab3}
|
||||
x <- perf[scenario == "harm" & outcome == "limb" & analysis == "main" &
|
||||
variant %in% c("V2_cluster", "V3_random", "V4_within") & key != "ccw|A_B2"]
|
||||
x[, cell := sprintf("%s (%s)", pp(bias), pp(mcse_bias))]
|
||||
t3 <- dcast(x, method + vlab ~ wlab, value.var = "cell")
|
||||
setnames(t3, c("method", "vlab"), c("Method", "Physician variant"))
|
||||
knitr::kable(t3[order(-as.integer(Method))], caption = paste(
|
||||
"Table 3. Estimation bias (MCSE) against each method's own truth, percentage points.",
|
||||
"Harm scenario, limb MALE."))
|
||||
```
|
||||
|
||||
```{r tab3b}
|
||||
x[, cell2 := pp(total)]
|
||||
t3b <- dcast(x, method + vlab ~ wlab, value.var = "cell2")
|
||||
setnames(t3b, c("method", "vlab"), c("Method", "Physician variant"))
|
||||
knitr::kable(t3b[order(-as.integer(Method))], caption = paste(
|
||||
"Table 3b. Total error against the reference truth (CCW A vs B1), percentage points.",
|
||||
"Harm scenario, limb MALE."))
|
||||
```
|
||||
|
||||
**What it shows.**
|
||||
|
||||
* **Surgeon skill (W1).** CCW and sequential trials stay within about 1 pp, and the FE variant brings
|
||||
them to 0.2–0.3 pp. The **landmark is badly off (total error about 14–16 pp)**. Almost all of that is
|
||||
question gap, not estimation bias, and Section 6 (Q1) explains why.
|
||||
* **Practice style (W2).** It only harms the **naive** design (bias −9 to −12 pp). The TTE designs
|
||||
absorb style through their time-varying PVI models.
|
||||
* **Style that also changes the outcome coding (W3).** It biases landmark and sequential trials by
|
||||
about 1.5 pp when the physician is ignored. A random intercept or FE removes most of it, which
|
||||
supports H2b.
|
||||
* **Deferral / crossover (W4).** No design bias. The TTE designs handle crossover by design.
|
||||
* **Unmeasured severity (W5).** It biases **every** observational design by about 2.5–5.5 pp, and no
|
||||
physician variant helps (H2c holds).
|
||||
* **Realistic world (WR).** CCW is biased by +6 to +7 pp and sequential trials by +7 to +9 pp. FE helps
|
||||
only a little. Unmeasured severity dominates.
|
||||
|
||||
# 3. False alarms under the null (headline for clinicians)
|
||||
|
||||
```{r fig2, fig.width = 10, fig.height = 5.2}
|
||||
fa <- perf[scenario == "null" & outcome == "limb" & analysis == "main" & key != "ccw|A_B2" & variant != "V1_ignore"]
|
||||
fa[, row := factor(paste0(method, " · ", sub(" \\(.*", "", vlab)))]
|
||||
fa[, row := factor(row, levels = rev(sort(levels(row))))]
|
||||
p1 <- ggplot(fa, aes(wlab, row, fill = 100 * reject0)) + geom_tile(colour = "white", linewidth = 1) +
|
||||
geom_text(aes(label = sprintf("%.0f", 100 * reject0), colour = reject0 > 0.5), size = 3) +
|
||||
scale_colour_manual(values = c(`TRUE` = "white", `FALSE` = "#0b0b0b"), guide = "none") +
|
||||
scale_fill_gradient(low = "#eef4fc", high = "#0d3d78", limits = c(0, 100), name = "% of replicates\nCI excludes 0") +
|
||||
labs(x = NULL, y = NULL, title = "Figure 2. False-alarm rate, null scenario (limb MALE; nominal 5%)") +
|
||||
theme(axis.text.x = element_text(angle = 30, hjust = 1), panel.grid = element_blank(), legend.position = "right")
|
||||
p1
|
||||
```
|
||||
|
||||
```{r fig2b, fig.width = 8, fig.height = 3}
|
||||
m0 <- perf[scenario == "null" & world == "W0" & variant %in% c("V2_cluster", "none") &
|
||||
outcome %in% c("male", "limb") & !grepl("as_treated", key) & key != "ccw|A_B2"]
|
||||
m0[, oc := factor(ifelse(outcome == "male", "MALE incl. repeat revascularization", "Limb MALE"), levels = c("Limb MALE", "MALE incl. repeat revascularization"))]
|
||||
ggplot(m0, aes(100 * mean_est, method, colour = oc, group = oc)) +
|
||||
geom_vline(xintercept = 0, colour = "#c3c2b7") +
|
||||
geom_errorbarh(aes(xmin = 100 * (mean_est - 1.96 * emp_se), xmax = 100 * (mean_est + 1.96 * emp_se)), height = 0, linewidth = 0.6, position = position_dodge(width = 0.5)) +
|
||||
geom_point(size = 2, position = position_dodge(width = 0.5)) + scale_colour_manual(values = c("Limb MALE" = "#2a78d6", "MALE incl. repeat revascularization" = "#eb6834"), name = NULL) +
|
||||
labs(x = "Risk difference (pp), W0 null", y = NULL,
|
||||
title = "Figure 2 inset. Sharp null: limb MALE 0, yet MALE incl. repeat revascularization about +14 pp")
|
||||
```
|
||||
|
||||
**What it shows.**
|
||||
|
||||
* In the clean, skill and deferral worlds, landmark, sequential trials and naive are near 5%. **CCW
|
||||
is at 10–15% in W0/W1**, mainly because its SE is slightly underestimated in the null scenario
|
||||
(model/empirical SE 0.86 in W0) and its estimate sits slightly low (−0.33 pp, 1.3 MCSE). The
|
||||
bootstrap check (D28) is meant to settle this.
|
||||
* Style that changes outcome coding (W3) pushes naive to 51–55% and sequential trials to 25% when the
|
||||
physician is ignored. A random intercept or FE brings that back to 8–16%.
|
||||
* Unmeasured severity (W5, WR) gives **80–100% false alarms** for landmark, sequential trials and
|
||||
naive. CCW stays at 15–24%, but only because its SE is wide. FE lowers the landmark and sequential
|
||||
rates in WR (31% and 60%).
|
||||
* **MALE under the sharp null:** +14 pp for CCW, purely because only treated patients can have a
|
||||
repeat revascularization. Every design "detects harm" on MALE in 93–100% of replicates.
|
||||
|
||||
# 4. Aim 3: do negative controls detect bias?
|
||||
|
||||
```{r tab4}
|
||||
s <- ncs[level == "cell"]
|
||||
s[, cell := ifelse(is.na(sens), "–", sprintf("%.0f%% / %s", 100 * sens, ifelse(is.na(spec), "–", sprintf("%.0f%%", 100 * spec))))]
|
||||
s[, control := factor(c(nco1 = "NCO-1 (hip fracture / pneumonia)", nco2 = "NCO-2 (vein ablation)",
|
||||
nce = "NCE (early low-value procedure)", any_nc = "Any of the three")[control],
|
||||
levels = c("NCO-1 (hip fracture / pneumonia)", "NCO-2 (vein ablation)",
|
||||
"NCE (early low-value procedure)", "Any of the three"))]
|
||||
s[, wl := factor(c(WLAB, all = "All worlds")[world], levels = c(WLAB[WORLDS], "All worlds"))]
|
||||
knitr::kable(dcast(s, control ~ wl, value.var = "cell"), caption = paste(
|
||||
"Table 4. Sensitivity / specificity of each negative control for MALE bias > 1 pp.",
|
||||
"A cell (world × scenario × method × variant) counts as biased when its mean MALE bias exceeds 1 pp;",
|
||||
"a replicate is flagged when the control's 95% CI excludes 0. '–': no biased (or no clean) cells in that world."))
|
||||
```
|
||||
|
||||
```{r fig3, fig.width = 10, fig.height = 4.5}
|
||||
nc[, design_lab := factor(c(ccw = "CCW", seq = "Sequential", landmark = "Landmark", naive = "Naive")[design],
|
||||
levels = c("CCW", "Sequential", "Landmark", "Naive"))]
|
||||
nc[, flag := factor(ifelse(any_nc, "A negative control flagged", "All negative controls clean"),
|
||||
levels = c("All negative controls clean", "A negative control flagged"))]
|
||||
nc[, wlab := factor(WLAB[world], levels = WLAB[WORLDS])]
|
||||
cb <- nc[, .(absbias = abs(mean(err))), by = .(world, wlab, scenario, design_lab, variant, flag)]
|
||||
ggplot(cb, aes(wlab, 100 * absbias, colour = flag)) +
|
||||
geom_point(position = position_jitterdodge(jitter.width = 0.15, dodge.width = 0.6), size = 1.4, alpha = 0.8) +
|
||||
geom_hline(yintercept = 1, linetype = 2, colour = "#52514e") +
|
||||
scale_colour_manual(values = c("All negative controls clean" = "#1baf7a", "A negative control flagged" = "#eb6834"), name = NULL) +
|
||||
facet_wrap(~design_lab, nrow = 1) +
|
||||
labs(x = NULL, y = "|mean MALE bias| (pp)", title = "Figure 3. How much bias sits behind a clean negative control") +
|
||||
theme(axis.text.x = element_text(angle = 45, hjust = 1))
|
||||
```
|
||||
|
||||
**What it shows (provisional, see Q2).**
|
||||
|
||||
* **NCO-1 almost never flags (sensitivity about 12%), even with unmeasured severity (W5).** In the
|
||||
current DGP, NCO-1 depends on frailty, smoking and age but **not on claudication severity**, so it
|
||||
cannot see the main source of bias. H3a fails, by construction.
|
||||
* **NCO-2 is the working detector (48% sensitivity overall, 65–67% in WR).** It also fires when style
|
||||
does no harm (specificity 26–45% in W2/W3), which supports H3b.
|
||||
* **The NCE almost never flags (about 5–9%).** It is adjusted for early PVI (D36), and it shares style
|
||||
and frailty, not severity.
|
||||
* **Bias hides behind clean controls.** In WR, the cells with clean negative controls still have a
|
||||
median |MALE bias| of 8.5 pp (H3c).
|
||||
|
||||
# 5. Simulated RCTs
|
||||
|
||||
```{r tabrct}
|
||||
r <- perf[analysis == "rct" & outcome == "limb" & scenario != "null"]
|
||||
r <- r[, .(World = wlab, Scenario = scenario, Trial = method, `Reference` = pp(ref), `Own ITT truth` = pp(own),
|
||||
`Mean estimate` = pp(mean_est), `Empirical SE` = pp(emp_se), `Power (CI excludes 0)` = sprintf("%.0f%%", 100 * reject0))]
|
||||
knitr::kable(r[order(World, Scenario, Trial)], caption = "Table 5. Simulated trials, limb MALE (pp).")
|
||||
```
|
||||
|
||||
* **Surgeon selection changes the answer.** In WR, a realistic trial recruits hospital-based,
|
||||
top-third-skill, low-style surgeons. Its own ITT truth is **+2.1 pp under harm and −1.4 pp under
|
||||
benefit-then-harm**, while the population effects are +6.9 and +7.7. A well-run trial answers a
|
||||
question about selected surgeons (H1c).
|
||||
* **Small trials are uninformative.** With n = 240 the SE is about 4 pp, giving 6–29% power against
|
||||
effects of 5–8 pp.
|
||||
* The ideal pragmatic RCT (n = 50,000) recovers its truth with an SE of about 0.3–0.6 pp.
|
||||
|
||||
# 6. Open questions raised by the draft run
|
||||
|
||||
**Q1. Landmark adjusts for severity at month 5, after the PVI.** As specified in D13, the landmark
|
||||
propensity model uses covariates at diagnosis **and at month W−1**. For treated patients, month-5
|
||||
severity is already changed by the PVI (relief, then faster progression), so it is a mediator.
|
||||
Conditioning on it gives extreme weights (effective treated sample about 110–140) and a large own-truth
|
||||
shift when the harm varies between patients (W1: own truth about 20 pp vs a reference of 7 pp).
|
||||
A check on 20 oracle replicates gave these landmark estimates:
|
||||
|
||||
| World (harm) | Adjust for month-5 severity | Baseline covariates only | Reference |
|
||||
|---|---|---|---|
|
||||
| W0 | 8.6 (SD 3.4) | 5.6 (SD 1.4) | 5.1 |
|
||||
| W1 | 14.6 (SD 4.3) | 8.1 (SD 1.6) | 7.2 |
|
||||
|
||||
**Q2. NCO-1 does not depend on severity.** D32 says NCO-1 "shares the latent patient frailty,
|
||||
severity and smoking", but the DGP's NCO-1 model has no severity term. That is why it misses the
|
||||
W5/WR bias.
|
||||
|
||||
**Also pending:** the physician-cluster bootstrap check of SEs (D28). Model-to-empirical SE ratios
|
||||
are near 1 for CCW (0.96–1.07, except WR 0.87). They are 0.83–0.92 for landmark and sequential trials
|
||||
in W0, and 0.58 for landmark in W1.
|
||||
|
||||
# 7. Standard errors: physician-cluster bootstrap check (D28, D53)
|
||||
|
||||
```{r boot}
|
||||
bf <- list.files(file.path(root, "results", "bootstrap"), pattern = "\\.csv$", full.names = TRUE)
|
||||
if (length(bf)) {
|
||||
bt <- rbindlist(lapply(bf, fread))
|
||||
mv <- reps[analysis == "main" & variant == "V1_ignore" & rep %in% unique(bt$rep),
|
||||
.(world, scenario, rep, design, contrast, outcome, se_patient, se_cluster)]
|
||||
bt <- merge(bt, mv, by = c("world", "scenario", "rep", "design", "contrast", "outcome"))
|
||||
emp <- perf[analysis == "main" & variant == "V2_cluster", .(world, scenario, design, contrast, outcome, emp_se)]
|
||||
x <- bt[, .(reps = .N, boot = sqrt(mean(boot_se^2)), patient = sqrt(mean(se_patient^2)),
|
||||
cluster = sqrt(mean(se_cluster^2))), by = .(world, scenario, design, contrast, outcome)]
|
||||
x <- merge(x, emp, by = c("world", "scenario", "design", "contrast", "outcome"))
|
||||
x <- x[outcome == "limb" & contrast != "A_B2"]
|
||||
knitr::kable(x[order(world, scenario, design), .(World = world, Scenario = scenario, Design = design,
|
||||
`Empirical SE (100 reps)` = pp(emp_se, 2), `Bootstrap SE` = pp(boot, 2), `Model SE, cluster` = pp(cluster, 2),
|
||||
`Model SE, patient` = pp(patient, 2), `Bootstrap / empirical` = round(boot / emp_se, 2),
|
||||
`Cluster / empirical` = round(cluster / emp_se, 2))],
|
||||
caption = "Table 6. Disease composite. Bootstrap: 50 physician-cluster resamples on each of replicates 1-20.")
|
||||
} else cat("Bootstrap results not yet available.")
|
||||
```
|
||||
|
||||
# Supplement: full performance table
|
||||
|
||||
```{r supp}
|
||||
x <- perf[analysis == "main" & outcome %in% c("limb", "male", "disease")]
|
||||
knitr::kable(x[order(outcome, scenario, world, method, variant),
|
||||
.(outcome, scenario, world, method = as.character(method), variant, est = pp(mean_est, 2), own = pp(own, 2),
|
||||
ref = pp(ref, 2), bias = pp(bias, 2), mcse = pp(mcse_bias, 2), gap = pp(gap, 2), emp_se = pp(emp_se, 2),
|
||||
se_ratio = round(se_ratio, 2), cover = round(cover_own, 2), reject0 = round(reject0, 2))])
|
||||
```
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,80 @@
|
||||
# Calibrate each world's natural course to the Medicare targets (plan section 5.4).
|
||||
#
|
||||
# For every world, under the null scenario, solve one intercept per target, cycling
|
||||
# until all targets hold. Every world gets the same targets, so worlds differ only in
|
||||
# the threat they switch on, not in base rates.
|
||||
#
|
||||
# Usage: Rscript simulations/01_calibrate.R [n] [world ...]
|
||||
# Writes simulations/config/calibrated.yml (merged over existing entries).
|
||||
|
||||
pkgload::load_all("pvisim", quiet = TRUE)
|
||||
suppressPackageStartupMessages(library(data.table))
|
||||
|
||||
args <- commandArgs(TRUE)
|
||||
n <- if (length(args) && grepl("^[0-9e.]+$", args[1])) as.numeric(args[1]) else 1e5
|
||||
worlds <- setdiff(args, as.character(args[1]))
|
||||
if (!length(worlds)) worlds <- names(yaml::read_yaml(default_params_path())$worlds)
|
||||
H <- 53L # Sorber 2023 median follow-up, about 4.4 years
|
||||
|
||||
# Natural-course statistics that the targets are defined on.
|
||||
stats_of <- function(par) {
|
||||
pop <- make_population(par, n = n, seed = 20261027L)
|
||||
ev <- simulate_course(pop, par, policy("natural"), seed = 20261028L, record = FALSE)$events
|
||||
early <- ev$pvi_elective %in% TRUE & ev$t_pvi < par$size$window
|
||||
late_early <- !is.na(ev$t_rr) & ev$t_rr < H
|
||||
late_not <- !is.na(ev$t_pvi) & ev$t_pvi >= par$size$window & ev$t_pvi < H
|
||||
list(early = mean(early),
|
||||
late_not = mean(late_not[!early]),
|
||||
late_early = mean(late_early[early]),
|
||||
clti_not = mean((!is.na(ev$t_clti_code) & ev$t_clti_code < H)[!early]),
|
||||
death = cum_inc(ev$t_death))
|
||||
}
|
||||
|
||||
# target name -> parameter path and value (plan 5.4)
|
||||
targets <- list(
|
||||
early = list(path = c("treatment", "window", "intercept"), value = 0.032),
|
||||
late_not = list(path = c("treatment", "deferred", "intercept"), value = 0.036),
|
||||
late_early = list(path = c("outcomes", "repeat_revasc", "intercept"), value = 0.225),
|
||||
clti_not = list(path = c("outcomes", "clti", "intercept"), value = 0.078),
|
||||
death = list(path = c("outcomes", "death", "intercept"), value = 0.35)
|
||||
)
|
||||
|
||||
set_path <- function(lst, path, value) {
|
||||
if (length(path) == 1) { lst[[path]] <- value; return(lst) }
|
||||
lst[[path[1]]] <- set_path(if (is.null(lst[[path[1]]])) list() else lst[[path[1]]], path[-1], value)
|
||||
lst
|
||||
}
|
||||
get_path <- function(lst, path) Reduce(function(x, p) x[[p]], path, lst)
|
||||
|
||||
# Each target depends mainly on its own intercept, and for rare monthly events
|
||||
# logit(rate) moves about one-for-one with the intercept. So update all intercepts
|
||||
# together: b <- b - damp * (logit(stat) - logit(target)), until every target is within
|
||||
# `tol` on the logit scale. Common random numbers make each evaluation deterministic.
|
||||
calibrate_world <- function(world, max_iter = 25, tol = 0.02, damp = 0.9) {
|
||||
base <- load_params(world, "null", calibrated = FALSE)
|
||||
for (tg in names(base$calibration_targets)) targets[[tg]]$value <- base$calibration_targets[[tg]]
|
||||
ov <- list()
|
||||
for (tg in names(targets)) ov <- set_path(ov, targets[[tg]]$path, get_path(base, targets[[tg]]$path))
|
||||
for (it in seq_len(max_iter)) {
|
||||
s <- stats_of(load_params(world, "null", overrides = ov, calibrated = FALSE))
|
||||
gap <- sapply(names(targets), function(tg) qlogis(s[[tg]]) - qlogis(targets[[tg]]$value))
|
||||
cat(sprintf("%s iter %2d: %s\n", world, it,
|
||||
paste(sprintf("%s %.3f", names(s), unlist(s)), collapse = " ")))
|
||||
if (all(abs(gap) < tol)) break
|
||||
for (tg in names(targets)) {
|
||||
b <- get_path(ov, targets[[tg]]$path)
|
||||
ov <- set_path(ov, targets[[tg]]$path, round(b - damp * gap[[tg]], 4))
|
||||
}
|
||||
}
|
||||
if (any(abs(gap) >= tol)) warning(world, ": not converged; largest logit gap ", round(max(abs(gap)), 3))
|
||||
ov
|
||||
}
|
||||
|
||||
out_path <- "simulations/config/calibrated.yml"
|
||||
cal <- if (file.exists(out_path)) yaml::read_yaml(out_path) else list()
|
||||
for (w in worlds) {
|
||||
t0 <- Sys.time()
|
||||
cal[[w]] <- calibrate_world(w)
|
||||
cat(sprintf("%s done in %.1f min\n", w, as.numeric(Sys.time() - t0, units = "mins")))
|
||||
yaml::write_yaml(cal, out_path)
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
# Interventional truths for every world x scenario (plan section 5.5).
|
||||
# Usage: Rscript simulations/02_truths.R [n] [cores]
|
||||
# Writes simulations/truths/truths.csv and simulations/truths/risk_curves.csv.
|
||||
|
||||
pkgload::load_all("pvisim", quiet = TRUE)
|
||||
suppressPackageStartupMessages(library(data.table))
|
||||
args <- commandArgs(TRUE)
|
||||
n <- if (length(args) >= 1) as.numeric(args[1]) else 1e6
|
||||
cores <- if (length(args) >= 2) as.integer(args[2]) else 6L
|
||||
|
||||
raw <- yaml::read_yaml(default_params_path())
|
||||
jobs <- CJ(world = names(raw$worlds), scenario = names(raw$scenarios), sorted = FALSE)
|
||||
res <- parallel::mclapply(seq_len(nrow(jobs)), function(i)
|
||||
compute_truths(jobs$world[i], jobs$scenario[i], n = n), mc.cores = cores)
|
||||
bad <- vapply(res, inherits, TRUE, "try-error")
|
||||
if (any(bad)) stop("failed: ", paste(jobs$world[bad], jobs$scenario[bad], collapse = "; "), "\n", res[bad][[1]])
|
||||
dir.create("simulations/truths", showWarnings = FALSE)
|
||||
fwrite(rbindlist(lapply(res, `[[`, "truths")), "simulations/truths/truths.csv")
|
||||
fwrite(rbindlist(lapply(res, `[[`, "curves")), "simulations/truths/risk_curves.csv")
|
||||
cat("wrote simulations/truths/truths.csv\n")
|
||||
@@ -0,0 +1,20 @@
|
||||
# RCT ITT truths for every world x scenario (the landmark and naive own estimands are
|
||||
# estimated within replicates by run_oracle()). Writes simulations/truths/truths_rct.csv.
|
||||
# Usage: Rscript simulations/03_limits.R [n] [cores]
|
||||
pkgload::load_all("pvisim", quiet = TRUE)
|
||||
suppressPackageStartupMessages(library(data.table))
|
||||
a <- commandArgs(TRUE)
|
||||
n <- if (length(a) >= 1) as.numeric(a[1]) else 5e5
|
||||
cores <- if (length(a) >= 2) as.integer(a[2]) else 6L
|
||||
raw <- yaml::read_yaml(default_params_path())
|
||||
jobs <- CJ(world = names(raw$worlds), scenario = names(raw$scenarios), sorted = FALSE)
|
||||
res <- parallel::mclapply(seq_len(nrow(jobs)), function(i) {
|
||||
w <- jobs$world[i]; s <- jobs$scenario[i]
|
||||
par <- load_params(w, s)
|
||||
rbind(rct_truth(par, "realistic", n = 1e6)[outcome %in% c("limb", "male", "disease", "rr", "nco1", "nco2")],
|
||||
rct_truth(par, "ideal", n = 1e6)[outcome %in% c("limb", "male", "disease", "rr", "nco1", "nco2")])
|
||||
}, mc.cores = cores)
|
||||
bad <- vapply(res, inherits, TRUE, "try-error")
|
||||
if (any(bad)) stop(res[bad][[1]])
|
||||
fwrite(rbindlist(res), "simulations/truths/truths_rct.csv")
|
||||
cat("wrote simulations/truths/truths_rct.csv\n")
|
||||
@@ -0,0 +1,100 @@
|
||||
# Performance against the known truths (plan sections 9 and 10).
|
||||
# Usage: Rscript simulations/04_summarise.R [results_dir]
|
||||
# Writes results/summary/{reps,perf,nc_reps,nc_summary}.csv.
|
||||
|
||||
suppressPackageStartupMessages(library(data.table))
|
||||
a <- commandArgs(TRUE)
|
||||
indir <- if (length(a) >= 1) a[1] else "results/main/n50k_v2"
|
||||
outdir <- "results/summary"
|
||||
dir.create(outdir, recursive = TRUE, showWarnings = FALSE)
|
||||
|
||||
files <- list.files(indir, pattern = "^rep[0-9]+\\.csv$", recursive = TRUE, full.names = TRUE)
|
||||
d <- rbindlist(lapply(files, fread), fill = TRUE)
|
||||
cat(sprintf("%d files, %d rows\n", length(files), nrow(d)))
|
||||
# rare failures (sequential V4: fixed effects not estimable in a trial month) are dropped and counted
|
||||
cat(sprintf("dropped %d estimates with NA\n", d[is.na(est), .N]))
|
||||
d <- d[!is.na(est)]
|
||||
# trials are analysed at 12, 24 and 60 months (D55); everything else at 60
|
||||
if (!"horizon" %in% names(d)) d[, horizon := NA_integer_]
|
||||
d[is.na(horizon), horizon := 60L]
|
||||
|
||||
tr <- rbind(fread("simulations/truths/truths.csv"), fread("simulations/truths/truths_rct.csv"))
|
||||
tr <- tr[, .(world, scenario, outcome, estimand, truth = value, truth_mcse = mcse)]
|
||||
|
||||
# ---- own estimand and reference for every estimate -----------------------------------
|
||||
# reference (clinically relevant) = RD(A - B1); NCE has a true effect of exactly 0 (D32)
|
||||
d[, estimand := fcase(design == "ccw" & contrast == "A_B1", "rd_A_B1",
|
||||
design == "ccw" & contrast == "A_B2", "rd_A_B2",
|
||||
design == "seq", "seq_itt",
|
||||
design %like% "^rct_" & contrast == "itt",
|
||||
paste0(design, "_itt", fifelse(horizon == 60L, "", paste0("_h", horizon))),
|
||||
default = NA_character_)]
|
||||
d <- merge(d, tr[, .(world, scenario, outcome, estimand, own = truth)],
|
||||
by = c("world", "scenario", "outcome", "estimand"), all.x = TRUE)
|
||||
d <- merge(d, tr[estimand == "rd_A_B1", .(world, scenario, outcome, ref = truth)],
|
||||
by = c("world", "scenario", "outcome"), all.x = TRUE)
|
||||
# at the shorter trial horizons the reference is RD(A - B1) at that month, from the risk curves
|
||||
cv <- dcast(fread("simulations/truths/risk_curves.csv")[policy %in% c("A", "B1")],
|
||||
world + scenario + outcome + month ~ policy, value.var = "cuminc")
|
||||
d <- merge(d, cv[, .(world, scenario, outcome, horizon = month, ref_h = A - B1)],
|
||||
by = c("world", "scenario", "outcome", "horizon"), all.x = TRUE)
|
||||
d[horizon != 60L, ref := ref_h][, ref_h := NULL]
|
||||
# landmark and naive: own estimand = the paired oracle estimate in the same replicate (D47)
|
||||
orc <- d[variant == "oracle", .(world, scenario, rep, design, outcome, oracle = est)]
|
||||
d <- merge(d, orc, by = c("world", "scenario", "rep", "design", "outcome"), all.x = TRUE)
|
||||
d[design %in% c("landmark", "landmark_m5", "naive") & variant != "oracle", own := oracle]
|
||||
d[analysis %like% "^nce", `:=`(own = 0, ref = 0)]
|
||||
d[, oracle := NULL]
|
||||
d[, `:=`(lo = est - 1.96 * se, hi = est + 1.96 * se)]
|
||||
fwrite(d, file.path(outdir, "reps.csv"))
|
||||
|
||||
# ---- performance by cell (plan 9) -----------------------------------------------------
|
||||
keys <- c("world", "scenario", "analysis", "design", "contrast", "variant", "outcome", "horizon")
|
||||
perf <- d[variant != "oracle", {
|
||||
err <- est - own; R <- .N; own_m <- mean(own)
|
||||
.(reps = R, mean_est = mean(est), own = own_m, ref = ref[1],
|
||||
bias = mean(err), mcse_bias = sd(err) / sqrt(R),
|
||||
gap = own_m - ref[1], total = mean(est) - ref[1],
|
||||
emp_se = sd(est), mod_se = sqrt(mean(se^2)),
|
||||
se_ratio = sqrt(mean(se^2)) / sd(est),
|
||||
cover_own = mean(lo <= own_m & hi >= own_m),
|
||||
cover_ref = mean(lo <= ref[1] & hi >= ref[1]),
|
||||
reject0 = mean(lo > 0 | hi < 0),
|
||||
rmse_own = sqrt(mean((est - own_m)^2)), rmse_ref = sqrt(mean((est - ref[1])^2)))
|
||||
}, by = keys]
|
||||
perf[, `:=`(mcse_cover = sqrt(cover_own * (1 - cover_own) / reps),
|
||||
mcse_reject = sqrt(reject0 * (1 - reject0) / reps))]
|
||||
fwrite(perf, file.path(outdir, "perf.csv"))
|
||||
|
||||
# ---- negative controls as bias detectors (plan 10) ------------------------------------
|
||||
# One unit = world x scenario x design x variant x replicate. Bias is in the limb-MALE RD (D56)
|
||||
# against the method's own estimand; a control "flags" when its 95% CI excludes 0.
|
||||
m <- d[analysis == "main" & design != "landmark_m5" & variant %in% c("V1_ignore", "V2_cluster", "V3_random", "V4_within") &
|
||||
contrast %in% c("early_none", "A_B1", "now_not")]
|
||||
u <- c("world", "scenario", "design", "variant", "rep")
|
||||
flag <- function(x) x$lo > 0 | x$hi < 0
|
||||
nc <- m[outcome == "limb", c(..u, "est", "own")][, err := est - own]
|
||||
nc <- merge(nc, m[outcome == "nco1", c(..u)][, nco1 := flag(m[outcome == "nco1"])], by = u, all.x = TRUE)
|
||||
nc <- merge(nc, m[outcome == "nco2", c(..u)][, nco2 := flag(m[outcome == "nco2"])], by = u, all.x = TRUE)
|
||||
ne <- d[analysis == "nce" & outcome == "limb" & contrast %in% c("early_none", "A_B1")]
|
||||
nc <- merge(nc, ne[, c(..u)][, nce := flag(ne)], by = u, all.x = TRUE)
|
||||
nc[, any_nc := nco1 | nco2 | fifelse(is.na(nce), FALSE, nce)]
|
||||
nc[, cell_bias := mean(err), by = .(world, scenario, design, variant)]
|
||||
nc[, `:=`(biased_rep = abs(err) > 0.01, biased_cell = abs(cell_bias) > 0.01)]
|
||||
fwrite(nc, file.path(outdir, "nc_reps.csv"))
|
||||
|
||||
acc <- function(x, truth) {
|
||||
ok <- !is.na(x)
|
||||
list(sens = mean(x[ok & truth]), spec = mean(!x[ok & !truth]),
|
||||
n_biased = sum(ok & truth), n_clean = sum(ok & !truth))
|
||||
}
|
||||
ncs <- rbindlist(lapply(c("nco1", "nco2", "nce", "any_nc"), function(k) rbind(
|
||||
nc[, c(control = k, level = "replicate", acc(get(k), biased_rep)), by = world],
|
||||
nc[, c(control = k, level = "cell", acc(get(k), biased_cell)), by = world],
|
||||
nc[, c(world = "all", control = k, level = "replicate", acc(get(k), biased_rep))],
|
||||
nc[, c(world = "all", control = k, level = "cell", acc(get(k), biased_cell))])))
|
||||
hide <- nc[any_nc == FALSE, .(control = "any_nc", med_absbias_clean = median(abs(cell_bias), na.rm = TRUE),
|
||||
p90_absbias_clean = quantile(abs(cell_bias), 0.9, na.rm = TRUE)), by = world]
|
||||
ncs <- merge(ncs, hide, by = c("world", "control"), all.x = TRUE)
|
||||
fwrite(ncs, file.path(outdir, "nc_summary.csv"))
|
||||
cat("wrote", outdir, "\n")
|
||||
@@ -0,0 +1,35 @@
|
||||
# Physician-cluster bootstrap check of the model-based SEs (D28, reduced in D53).
|
||||
# Usage: Rscript simulations/05_bootstrap.R [reps] [B] [cores]
|
||||
# Writes results/bootstrap/<world>_<scenario>_rep<k>.csv: the estimate and the
|
||||
# bootstrap SD for each design, contrast and outcome (V1 point estimates).
|
||||
|
||||
pkgload::load_all("pvisim", quiet = TRUE)
|
||||
suppressPackageStartupMessages(library(data.table))
|
||||
a <- commandArgs(TRUE)
|
||||
reps <- if (length(a) >= 1) as.integer(a[1]) else 20L
|
||||
B <- if (length(a) >= 2) as.integer(a[2]) else 50L
|
||||
cores <- if (length(a) >= 3) as.integer(a[3]) else 11L
|
||||
cfg <- yaml::read_yaml("simulations/config/experiments.yml")
|
||||
n <- cfg$scales$draft$n_patients
|
||||
outdir <- "results/bootstrap"; dir.create(outdir, showWarnings = FALSE, recursive = TRUE)
|
||||
designs <- c("naive", "landmark", "landmark_m5", "ccw", "seq")
|
||||
outs <- c(limb = "t_limb", male = "t_male", disease = "t_disease")
|
||||
|
||||
jobs <- CJ(world = c("W0", "WR"), scenario = c("null", "harm"), rep = seq_len(reps), sorted = FALSE)
|
||||
jobs[, file := file.path(outdir, sprintf("%s_%s_rep%03d.csv", world, scenario, rep))]
|
||||
todo <- jobs[!file.exists(file)]
|
||||
cat(sprintf("bootstrap: %d jobs, %d to do, B = %d, %d cores\n", nrow(jobs), nrow(todo), B, cores))
|
||||
res <- parallel::mclapply(seq_len(nrow(todo)), function(i) tryCatch({
|
||||
j <- todo[i]
|
||||
par <- load_params(j$world, j$scenario)
|
||||
rec <- simulate_records(par, rep = j$rep, n = n) # the same data as the main run's replicate
|
||||
bs <- rbindlist(lapply(seq_len(B), function(b)
|
||||
cbind(b = b, run_designs(bootstrap_records(rec, seed = 1e6 * j$rep + b), designs, "V1_ignore", outs))), fill = TRUE)
|
||||
out <- bs[, .(boot_se = sd(est, na.rm = TRUE), boot_mean = mean(est, na.rm = TRUE), B = sum(!is.na(est))),
|
||||
by = .(design, contrast, outcome)]
|
||||
fwrite(out[, `:=`(world = j$world, scenario = j$scenario, rep = j$rep)], j$file)
|
||||
NULL
|
||||
}, error = function(e) sprintf("%s/%s/%d: %s", todo$world[i], todo$scenario[i], todo$rep[i], conditionMessage(e))),
|
||||
mc.cores = cores, mc.preschedule = FALSE)
|
||||
err <- unlist(res[!vapply(res, is.null, TRUE)])
|
||||
cat(sprintf("done; %d errors\n", length(err))); if (length(err)) cat(head(err, 3), sep = "\n")
|
||||
@@ -0,0 +1,117 @@
|
||||
W0:
|
||||
treatment:
|
||||
window:
|
||||
intercept: -6.3552
|
||||
deferred:
|
||||
intercept: -7.7172
|
||||
outcomes:
|
||||
repeat_revasc:
|
||||
intercept: -5.6375
|
||||
clti:
|
||||
intercept: -7.4903
|
||||
death:
|
||||
intercept: -5.6504
|
||||
W4:
|
||||
treatment:
|
||||
window:
|
||||
intercept: -6.3535
|
||||
deferred:
|
||||
intercept: -6.7744
|
||||
outcomes:
|
||||
repeat_revasc:
|
||||
intercept: -5.6429
|
||||
clti:
|
||||
intercept: -7.4903
|
||||
death:
|
||||
intercept: -5.6542
|
||||
W1:
|
||||
treatment:
|
||||
window:
|
||||
intercept: -6.3552
|
||||
deferred:
|
||||
intercept: -7.7172
|
||||
outcomes:
|
||||
repeat_revasc:
|
||||
intercept: -5.7951
|
||||
clti:
|
||||
intercept: -7.4903
|
||||
death:
|
||||
intercept: -5.6504
|
||||
W1_rho0:
|
||||
treatment:
|
||||
window:
|
||||
intercept: -6.3552
|
||||
deferred:
|
||||
intercept: -7.7172
|
||||
outcomes:
|
||||
repeat_revasc:
|
||||
intercept: -5.7187
|
||||
clti:
|
||||
intercept: -7.4903
|
||||
death:
|
||||
intercept: -5.6504
|
||||
W2:
|
||||
treatment:
|
||||
window:
|
||||
intercept: -7.3097
|
||||
deferred:
|
||||
intercept: -8.5615
|
||||
outcomes:
|
||||
repeat_revasc:
|
||||
intercept: -5.5855
|
||||
clti:
|
||||
intercept: -7.4913
|
||||
death:
|
||||
intercept: -5.6505
|
||||
W3:
|
||||
treatment:
|
||||
window:
|
||||
intercept: -7.3134
|
||||
deferred:
|
||||
intercept: -8.5848
|
||||
outcomes:
|
||||
repeat_revasc:
|
||||
intercept: -6.4032
|
||||
clti:
|
||||
intercept: -7.5232
|
||||
death:
|
||||
intercept: -5.649
|
||||
W5:
|
||||
treatment:
|
||||
window:
|
||||
intercept: -6.3552
|
||||
deferred:
|
||||
intercept: -7.7172
|
||||
outcomes:
|
||||
repeat_revasc:
|
||||
intercept: -5.6375
|
||||
clti:
|
||||
intercept: -7.4903
|
||||
death:
|
||||
intercept: -5.6504
|
||||
WR:
|
||||
treatment:
|
||||
window:
|
||||
intercept: -7.3131
|
||||
deferred:
|
||||
intercept: -9.598
|
||||
outcomes:
|
||||
repeat_revasc:
|
||||
intercept: -6.8121
|
||||
clti:
|
||||
intercept: -7.5233
|
||||
death:
|
||||
intercept: -5.6489
|
||||
WE:
|
||||
treatment:
|
||||
window:
|
||||
intercept: -6.3552
|
||||
deferred:
|
||||
intercept: -7.7172
|
||||
outcomes:
|
||||
repeat_revasc:
|
||||
intercept: -5.6375
|
||||
clti:
|
||||
intercept: -7.4903
|
||||
death:
|
||||
intercept: -5.6504
|
||||
@@ -0,0 +1,16 @@
|
||||
# Experiment grids (plan sections 5.2, 7, 9). Run with simulations/run_experiment.R <grid> [--scale draft|final]
|
||||
main:
|
||||
worlds: [W0, W1, W1_rho0, W2, W3, W4, W5, WE, WR] # WE: EHR/registry-like measurement (D55)
|
||||
scenarios: ["null", harm, benefit_harm]
|
||||
designs: [naive, landmark, landmark_m5, ccw, seq] # landmark_m5: also adjusts for month-5 severity (supplement, D53)
|
||||
variants: [V1_ignore, V3_random, V4_within] # V1 also yields V2 (cluster-robust SE)
|
||||
nce: true # negative-control exposure (landmark, CCW), D32/D36
|
||||
# trims (D57): supplement-only or uninformative cells are skipped to save time
|
||||
v1_only_designs: [landmark_m5] # month-5 landmark (supplement) only without physician terms
|
||||
nce_variants: [V1_ignore, V4_within] # NCE without the random intercept
|
||||
skip_variant: {V3_random: [W0, W4]} # worlds where the physician plays no role by design
|
||||
rct: [realistic, ideal]
|
||||
scales:
|
||||
smoke: {n_patients: 20000, reps: 1}
|
||||
draft: {n_patients: 50000, reps: 100, dir: n50k_v2} # v2: D53 (landmark baseline covariates, NCO-1 severity)
|
||||
final: {n_patients: 50000, reps: 200, dir: n50k_v2} # extends the draft's replicates 1-100 to 200 (D51)
|
||||
@@ -0,0 +1,17 @@
|
||||
# Explore effect sizes: RD(A - B1) by month for MALE and the disease composite in W0.
|
||||
pkgload::load_all("pvisim", quiet = TRUE)
|
||||
suppressMessages(library(data.table))
|
||||
args <- commandArgs(TRUE)
|
||||
grid <- if (length(args)) eval(parse(text = args[1])) else
|
||||
list(c(0, 0), c(0.3, 0.03), c(1.2, 0.04))
|
||||
par0 <- load_params("W0", "null")
|
||||
pop <- make_population(par0, n = 2e5, seed = 9L)
|
||||
ci <- function(t, h) mean(!is.na(t) & t < h)
|
||||
for (g in grid) {
|
||||
par <- load_params("W0", "null", overrides = list(effect = list(relief = g[1], kappa = g[2])))
|
||||
ev <- lapply(c("A", "B1"), function(p) simulate_course(pop, par, policy(p), seed = 10L, record = FALSE)$events)
|
||||
rd <- function(col, h) 100 * (ci(ev[[1]][[col]], h) - ci(ev[[2]][[col]], h))
|
||||
cat(sprintf("relief %.2f kappa %.3f | disease RD @12 %+.1f @24 %+.1f @60 %+.1f | MALE RD @60 %+.1f (A %.1f%% B1 %.1f%%) | death RD %+.2f\n",
|
||||
g[1], g[2], rd("t_disease", 12), rd("t_disease", 24), rd("t_disease", 60), rd("t_male", 60),
|
||||
100 * ci(ev[[1]]$t_male, 60), 100 * ci(ev[[2]]$t_male, 60), rd("t_death", 60)))
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
# Quick look: the natural course of each world against the calibration targets (plan 5.4).
|
||||
pkgload::load_all("pvisim", quiet = TRUE)
|
||||
library(data.table)
|
||||
args <- commandArgs(TRUE); worlds <- if (length(args)) args else c("W0", "WR")
|
||||
for (w in worlds) {
|
||||
par <- load_params(w, "harm")
|
||||
t0 <- Sys.time()
|
||||
pop <- make_population(par, seed = 1L)
|
||||
ev <- simulate_course(pop, par, policy("natural"), seed = 2L, record = FALSE)$events
|
||||
ev <- merge(ev, pop$pat[, .(id, phys_id)], by = "id")
|
||||
early <- !is.na(ev$t_pvi) & ev$t_pvi < 6 & ev$pvi_elective %in% TRUE
|
||||
h <- 53 # Sorber median follow-up about 4.4 years
|
||||
late <- !is.na(ev$t_pvi) & ev$t_pvi >= 6 | !is.na(ev$t_rr) & ev$t_rr >= 6
|
||||
late <- late & pmin(ev$t_pvi * NA^(ev$t_pvi < 6), ifelse(!is.na(ev$t_rr) & ev$t_rr >= 6, ev$t_rr, NA), na.rm = TRUE) < h
|
||||
clti <- !is.na(ev$t_clti_code) & ev$t_clti_code < h
|
||||
phys <- ev[, .(n = .N, rate = mean(!is.na(t_pvi) & t_pvi < 6 & pvi_elective %in% TRUE)), by = phys_id]
|
||||
sdr <- sd(phys$rate); out <- mean(phys$rate >= mean(phys$rate) + 2 * sdr)
|
||||
cat(sprintf("\n%s (%.0fs)\n early PVI %.1f%% (target 3.2)\n physician rate: median %.1f%%, max %.0f%%, >=2SD outliers %.1f%% (0, 58, 5.6)\n",
|
||||
w, as.numeric(Sys.time() - t0, units = "secs"), 100 * mean(early),
|
||||
100 * median(phys$rate), 100 * max(phys$rate), 100 * out))
|
||||
cat(sprintf(" late PVI by 4.4y: early %.1f%% vs not %.1f%% (22.5 vs 3.6)\n", 100 * mean(late[early]), 100 * mean(late[!early])))
|
||||
cat(sprintf(" coded CLTI by 4.4y: early %.1f%% vs not %.1f%% (16.4 vs 7.8)\n", 100 * mean(clti[early]), 100 * mean(clti[!early])))
|
||||
cat(sprintf(" death by 5y %.1f%% (about 35) MALE by 5y %.1f%% amputation %.2f%% disease composite %.1f%%\n",
|
||||
100 * cum_inc(ev$t_death), 100 * cum_inc(ev$t_male), 100 * cum_inc(ev$t_amp), 100 * cum_inc(ev$t_disease)))
|
||||
cat(sprintf(" NCO1 %.1f%% NCO2 %.1f%% NCE %.1f%%\n", 100 * cum_inc(ev$t_nco1), 100 * cum_inc(ev$t_nco2), 100 * cum_inc(ev$t_nce)))
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
# Run an experiment grid; one CSV per world x scenario x replicate, so runs are resumable.
|
||||
# Usage: Rscript simulations/run_experiment.R main [--scale draft] [--cores 10] [--worlds W0,W2]
|
||||
pkgload::load_all("pvisim", quiet = TRUE)
|
||||
suppressPackageStartupMessages(library(data.table))
|
||||
# one thread per worker: parallelism comes from mclapply across replicates
|
||||
setDTthreads(1); fixest::setFixest_nthreads(1)
|
||||
a <- commandArgs(TRUE)
|
||||
opt <- function(flag, default) { i <- match(flag, a); if (is.na(i)) default else a[i + 1] }
|
||||
grid_name <- a[1]
|
||||
scale <- opt("--scale", "draft")
|
||||
cores <- as.integer(opt("--cores", "10"))
|
||||
cfg <- yaml::read_yaml("simulations/config/experiments.yml")
|
||||
g <- cfg[[grid_name]]; sc <- cfg$scales[[scale]]
|
||||
worlds <- if (is.null(opt("--worlds", NULL))) g$worlds else strsplit(opt("--worlds", NULL), ",")[[1]]
|
||||
outdir <- file.path("results", grid_name, if (is.null(sc$dir)) scale else sc$dir)
|
||||
jobs <- CJ(world = worlds, scenario = unlist(g$scenarios), rep = seq_len(sc$reps), sorted = FALSE)
|
||||
jobs[, file := file.path(outdir, world, scenario, sprintf("rep%03d.csv", rep))]
|
||||
todo <- jobs[!file.exists(file)]
|
||||
cat(sprintf("%s/%s: %d jobs, %d to do, %d cores\n", grid_name, scale, nrow(jobs), nrow(todo), cores))
|
||||
|
||||
one_job <- function(i) {
|
||||
j <- todo[i]
|
||||
par <- load_params(j$world, j$scenario)
|
||||
rec <- simulate_records(par, rep = j$rep, n = sc$n_patients)
|
||||
out <- list()
|
||||
with_v2 <- function(r) if (r$variant[1] == "V1_ignore") # same fit, two SEs
|
||||
rbind(r, copy(r)[, `:=`(variant = "V2_cluster", se = se_cluster)]) else r
|
||||
nce_out <- c(limb = "t_limb", male = "t_male", disease = "t_disease")
|
||||
for (v in g$variants) {
|
||||
if (j$world %in% unlist(g$skip_variant[[v]])) next
|
||||
des <- unlist(g$designs)
|
||||
if (v != "V1_ignore") des <- setdiff(des, unlist(g$v1_only_designs))
|
||||
out[[length(out) + 1]] <- with_v2(run_designs(rec, des, v))[, analysis := "main"]
|
||||
if (isTRUE(g$nce) && (is.null(g$nce_variants) || v %in% unlist(g$nce_variants)))
|
||||
out[[length(out) + 1]] <- with_v2(run_designs(nce_records(rec), c("landmark", "ccw"), v, nce_out))[, analysis := "nce"]
|
||||
}
|
||||
# unadjusted NCE (supplement, D36), V1/V2 only
|
||||
out[[length(out) + 1]] <- with_v2(run_designs(nce_records(rec, adjust_pvi = FALSE), c("landmark", "ccw"),
|
||||
"V1_ignore", nce_out))[, analysis := "nce_unadjusted"]
|
||||
out[[length(out) + 1]] <- run_oracle(rec)[, analysis := "main"] # own estimand of landmark / naive
|
||||
for (dz in g$rct) {
|
||||
r <- run_rct(par, dz, rep = j$rep)
|
||||
out[[length(out) + 1]] <- r[, `:=`(contrast = analysis, analysis = "rct", variant = "none", se = se_patient)]
|
||||
}
|
||||
res <- rbindlist(out, fill = TRUE)[, `:=`(world = j$world, scenario = j$scenario, rep = j$rep)]
|
||||
dir.create(dirname(j$file), recursive = TRUE, showWarnings = FALSE)
|
||||
fwrite(res, j$file)
|
||||
NULL
|
||||
}
|
||||
t0 <- Sys.time()
|
||||
res <- parallel::mclapply(seq_len(nrow(todo)), function(i) tryCatch(one_job(i), error = function(e)
|
||||
sprintf("%s/%s/%d: %s", todo$world[i], todo$scenario[i], todo$rep[i], conditionMessage(e))),
|
||||
mc.cores = cores, mc.preschedule = FALSE)
|
||||
err <- unlist(res[!vapply(res, is.null, TRUE)])
|
||||
cat(sprintf("done in %.1f h; %d errors\n", as.numeric(Sys.time() - t0, units = "hours"), length(err)))
|
||||
dir.create(outdir, recursive = TRUE, showWarnings = FALSE)
|
||||
if (length(err)) { writeLines(err, file.path(outdir, sprintf("errors_%s.txt", format(Sys.time(), "%Y%m%d_%H%M")))); cat(head(err, 3), sep = "\n") }
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,973 @@
|
||||
world,scenario,outcome,estimand,value,mcse,n
|
||||
W0,null,limb,rct_realistic_itt_h12,0,0,138782
|
||||
W0,null,limb,rct_realistic_itt_h24,0,0,138782
|
||||
W0,null,limb,rct_realistic_itt,0,0,138782
|
||||
W0,null,male,rct_realistic_itt_h12,0.0316896791667746,0.000489980259881347,138782
|
||||
W0,null,male,rct_realistic_itt_h24,0.066135680620639,0.000696255343077188,138782
|
||||
W0,null,male,rct_realistic_itt,0.140661174578609,0.00098072361759117,138782
|
||||
W0,null,disease,rct_realistic_itt_h12,0,0,138782
|
||||
W0,null,disease,rct_realistic_itt_h24,0,0,138782
|
||||
W0,null,disease,rct_realistic_itt,0,0,138782
|
||||
W0,null,rr,rct_realistic_itt_h12,0.0329190178576472,0.000498508344475691,138782
|
||||
W0,null,rr,rct_realistic_itt_h24,0.0717591137320835,0.000722074502261802,138782
|
||||
W0,null,rr,rct_realistic_itt,0.176524278977813,0.00107249504657992,138782
|
||||
W0,null,nco1,rct_realistic_itt_h12,0,0,138782
|
||||
W0,null,nco1,rct_realistic_itt_h24,0,0,138782
|
||||
W0,null,nco1,rct_realistic_itt,0,0,138782
|
||||
W0,null,nco2,rct_realistic_itt_h12,0,0,138782
|
||||
W0,null,nco2,rct_realistic_itt_h24,0,0,138782
|
||||
W0,null,nco2,rct_realistic_itt,0,0,138782
|
||||
W0,null,limb,rct_ideal_itt_h12,0,0,1000000
|
||||
W0,null,limb,rct_ideal_itt_h24,0,0,1000000
|
||||
W0,null,limb,rct_ideal_itt,0,0,1000000
|
||||
W0,null,male,rct_ideal_itt_h12,0.0313360000000107,0.000174379055808226,1000000
|
||||
W0,null,male,rct_ideal_itt_h24,0.0641560000000435,0.000245132632800575,1000000
|
||||
W0,null,male,rct_ideal_itt,0.132398000000112,0.000338976060504823,1000000
|
||||
W0,null,disease,rct_ideal_itt_h12,0,0,1000000
|
||||
W0,null,disease,rct_ideal_itt_h24,0,0,1000000
|
||||
W0,null,disease,rct_ideal_itt,0,0,1000000
|
||||
W0,null,rr,rct_ideal_itt_h12,0.0327740000000121,0.000178207364955272,1000000
|
||||
W0,null,rr,rct_ideal_itt_h24,0.0702760000000496,0.000255728926453558,1000000
|
||||
W0,null,rr,rct_ideal_itt,0.166642000000146,0.000372733743888863,1000000
|
||||
W0,null,nco1,rct_ideal_itt_h12,0,0,1000000
|
||||
W0,null,nco1,rct_ideal_itt_h24,0,0,1000000
|
||||
W0,null,nco1,rct_ideal_itt,0,0,1000000
|
||||
W0,null,nco2,rct_ideal_itt_h12,0,0,1000000
|
||||
W0,null,nco2,rct_ideal_itt_h24,0,0,1000000
|
||||
W0,null,nco2,rct_ideal_itt,0,0,1000000
|
||||
W0,harm,limb,rct_realistic_itt_h12,-0.000634779123119918,8.59030281036993e-05,138782
|
||||
W0,harm,limb,rct_realistic_itt_h24,0.00357723457388988,0.000197648179331725,138782
|
||||
W0,harm,limb,rct_realistic_itt,0.0657009909762577,0.000715048507757777,138782
|
||||
W0,harm,male,rct_realistic_itt_h12,0.0305563168266289,0.000494042770555967,138782
|
||||
W0,harm,male,rct_realistic_itt_h24,0.0712904014827441,0.000728275405042644,138782
|
||||
W0,harm,male,rct_realistic_itt,0.209013847241733,0.00115467602996962,138782
|
||||
W0,harm,disease,rct_realistic_itt_h12,-0.000492563430128351,7.42094536253605e-05,138782
|
||||
W0,harm,disease,rct_realistic_itt_h24,0.00248801890157732,0.000166343673945743,138782
|
||||
W0,harm,disease,rct_realistic_itt,0.0539520228689354,0.000648343714778322,138782
|
||||
W0,harm,rr,rct_realistic_itt_h12,0.032397552203542,0.000494895030835064,138782
|
||||
W0,harm,rr,rct_realistic_itt_h24,0.0738721748970789,0.000732229204166303,138782
|
||||
W0,harm,rr,rct_realistic_itt,0.206069107976931,0.00113935408855565,138782
|
||||
W0,harm,nco1,rct_realistic_itt_h12,6.53400738957156e-06,6.53399013392965e-06,138782
|
||||
W0,harm,nco1,rct_realistic_itt_h24,-8.45156676626482e-06,1.58783112120704e-05,138782
|
||||
W0,harm,nco1,rct_realistic_itt,-0.00117248163179654,8.93642930544493e-05,138782
|
||||
W0,harm,nco2,rct_realistic_itt_h12,1.43513048739779e-05,1.01883274755683e-05,138782
|
||||
W0,harm,nco2,rct_realistic_itt_h24,1.97093822894091e-05,1.15113069317871e-05,138782
|
||||
W0,harm,nco2,rct_realistic_itt,-0.000146622386963238,3.5567497711738e-05,138782
|
||||
W0,harm,limb,rct_ideal_itt_h12,-0.00069000000000001,3.32193302160264e-05,1000000
|
||||
W0,harm,limb,rct_ideal_itt_h24,0.00369700000000017,7.40090007424914e-05,1000000
|
||||
W0,harm,limb,rct_ideal_itt,0.0625470000000419,0.000250125713973842,1000000
|
||||
W0,harm,male,rct_ideal_itt_h12,0.0302030000000096,0.000176195285949148,1000000
|
||||
W0,harm,male,rct_ideal_itt_h24,0.0694110000000488,0.000257408455725467,1000000
|
||||
W0,harm,male,rct_ideal_itt,0.196825000000176,0.000401807067352472,1000000
|
||||
W0,harm,disease,rct_ideal_itt_h12,-0.000527000000000006,2.88569276083573e-05,1000000
|
||||
W0,harm,disease,rct_ideal_itt_h24,0.00283300000000005,6.47531783851892e-05,1000000
|
||||
W0,harm,disease,rct_ideal_itt,0.0515180000000309,0.00022818390757615,1000000
|
||||
W0,harm,rr,rct_ideal_itt_h12,0.0322370000000116,0.000176793031060403,1000000
|
||||
W0,harm,rr,rct_ideal_itt_h24,0.0723530000000517,0.000259225854016017,1000000
|
||||
W0,harm,rr,rct_ideal_itt,0.193524000000173,0.000395447166412248,1000000
|
||||
W0,harm,nco1,rct_ideal_itt_h12,1e-05,3.1622618487401e-06,1000000
|
||||
W0,harm,nco1,rct_ideal_itt_h24,-5e-06,7.93725235838367e-06,1000000
|
||||
W0,harm,nco1,rct_ideal_itt,-0.001143,3.57448395016914e-05,1000000
|
||||
W0,harm,nco2,rct_ideal_itt_h12,1e-06,9.9999950002108e-07,1000000
|
||||
W0,harm,nco2,rct_ideal_itt_h24,-2e-06,5.47722520992005e-06,1000000
|
||||
W0,harm,nco2,rct_ideal_itt,-0.000447000000000004,2.24232065279446e-05,1000000
|
||||
W0,benefit_harm,limb,rct_realistic_itt_h12,-0.00936046865284354,0.000264129599237624,138782
|
||||
W0,benefit_harm,limb,rct_realistic_itt_h24,-0.0156253817310153,0.000346005778226494,138782
|
||||
W0,benefit_harm,limb,rct_realistic_itt,0.0476375959324454,0.000747755930004121,138782
|
||||
W0,benefit_harm,male,rct_realistic_itt_h12,0.0138456909190784,0.000496906834236644,138782
|
||||
W0,benefit_harm,male,rct_realistic_itt_h24,0.0389507656400573,0.000721058908850233,138782
|
||||
W0,benefit_harm,male,rct_realistic_itt,0.183585931781821,0.00116128147313179,138782
|
||||
W0,benefit_harm,disease,rct_realistic_itt_h12,-0.00701853844622746,0.000227256392216162,138782
|
||||
W0,benefit_harm,disease,rct_realistic_itt_h24,-0.0118343056375335,0.000299104065798181,138782
|
||||
W0,benefit_harm,disease,rct_realistic_itt,0.0400446368325273,0.000676156316495598,138782
|
||||
W0,benefit_harm,rr,rct_realistic_itt_h12,0.0237399250833049,0.000426256751172818,138782
|
||||
W0,benefit_harm,rr,rct_realistic_itt_h24,0.0573123691206079,0.000651693144083697,138782
|
||||
W0,benefit_harm,rr,rct_realistic_itt,0.188126682049733,0.00110432772565977,138782
|
||||
W0,benefit_harm,nco1,rct_realistic_itt_h12,7.59973402766115e-05,2.05244409832568e-05,138782
|
||||
W0,benefit_harm,nco1,rct_realistic_itt_h24,0.000304283539237408,3.99652071948251e-05,138782
|
||||
W0,benefit_harm,nco1,rct_realistic_itt,0.000464506673192971,0.000102228623751637,138782
|
||||
W0,benefit_harm,nco2,rct_realistic_itt_h12,2.13829664316019e-05,1.23792028465136e-05,138782
|
||||
W0,benefit_harm,nco2,rct_realistic_itt_h24,0.000101840077489285,2.47064296907095e-05,138782
|
||||
W0,benefit_harm,nco2,rct_realistic_itt,0.000204351440100052,4.71903593479565e-05,138782
|
||||
W0,benefit_harm,limb,rct_ideal_itt_h12,-0.00960499999999939,9.84110968081316e-05,1000000
|
||||
W0,benefit_harm,limb,rct_ideal_itt_h24,-0.0157609999999951,0.000127720753516602,1000000
|
||||
W0,benefit_harm,limb,rct_ideal_itt,0.0460120000000254,0.000266065585628749,1000000
|
||||
W0,benefit_harm,male,rct_ideal_itt_h12,0.0137869999999963,0.000179958102432602,1000000
|
||||
W0,benefit_harm,male,rct_ideal_itt_h24,0.037648000000017,0.000257220971335908,1000000
|
||||
W0,benefit_harm,male,rct_ideal_itt,0.173512000000153,0.000406166943334289,1000000
|
||||
W0,benefit_harm,disease,rct_ideal_itt_h12,-0.00713100000000062,8.4936145655476e-05,1000000
|
||||
W0,benefit_harm,disease,rct_ideal_itt_h24,-0.0117879999999978,0.000110837913441629,1000000
|
||||
W0,benefit_harm,disease,rct_ideal_itt,0.038659000000018,0.00024133064811169,1000000
|
||||
W0,benefit_harm,rr,rct_ideal_itt_h12,0.0239360000000033,0.00015314394504567,1000000
|
||||
W0,benefit_harm,rr,rct_ideal_itt_h24,0.0565390000000359,0.000231508836720838,1000000
|
||||
W0,benefit_harm,rr,rct_ideal_itt,0.177170000000157,0.000383445942867072,1000000
|
||||
W0,benefit_harm,nco1,rct_ideal_itt_h12,7.79999999999999e-05,8.8314164209386e-06,1000000
|
||||
W0,benefit_harm,nco1,rct_ideal_itt_h24,0.000407000000000003,2.02196525936012e-05,1000000
|
||||
W0,benefit_harm,nco1,rct_ideal_itt,0.000797000000000013,4.44788128322003e-05,1000000
|
||||
W0,benefit_harm,nco2,rct_ideal_itt_h12,3.5e-05,5.91597625080754e-06,1000000
|
||||
W0,benefit_harm,nco2,rct_ideal_itt_h24,0.000195,1.40343141977502e-05,1000000
|
||||
W0,benefit_harm,nco2,rct_ideal_itt,0.000396000000000003,2.91177468907165e-05,1000000
|
||||
W1,null,limb,rct_realistic_itt_h12,0,0,138782
|
||||
W1,null,limb,rct_realistic_itt_h24,0,0,138782
|
||||
W1,null,limb,rct_realistic_itt,0,0,138782
|
||||
W1,null,male,rct_realistic_itt_h12,0.0144235047445299,0.000334145736094463,138782
|
||||
W1,null,male,rct_realistic_itt_h24,0.0306225886076548,0.000483525366728361,138782
|
||||
W1,null,male,rct_realistic_itt,0.0682952261879501,0.000712862931193441,138782
|
||||
W1,null,disease,rct_realistic_itt_h12,0,0,138782
|
||||
W1,null,disease,rct_realistic_itt_h24,0,0,138782
|
||||
W1,null,disease,rct_realistic_itt,0,0,138782
|
||||
W1,null,rr,rct_realistic_itt_h12,0.0153147512364479,0.000343373916484683,138782
|
||||
W1,null,rr,rct_realistic_itt_h24,0.034504584650443,0.000510787639451681,138782
|
||||
W1,null,rr,rct_realistic_itt,0.0920330534174895,0.000813787097961872,138782
|
||||
W1,null,nco1,rct_realistic_itt_h12,0,0,138782
|
||||
W1,null,nco1,rct_realistic_itt_h24,0,0,138782
|
||||
W1,null,nco1,rct_realistic_itt,0,0,138782
|
||||
W1,null,nco2,rct_realistic_itt_h12,0,0,138782
|
||||
W1,null,nco2,rct_realistic_itt_h24,0,0,138782
|
||||
W1,null,nco2,rct_realistic_itt,0,0,138782
|
||||
W1,null,limb,rct_ideal_itt_h12,0,0,1000000
|
||||
W1,null,limb,rct_ideal_itt_h24,0,0,1000000
|
||||
W1,null,limb,rct_ideal_itt,0,0,1000000
|
||||
W1,null,male,rct_ideal_itt_h12,0.0315560000000109,0.000174986339076103,1000000
|
||||
W1,null,male,rct_ideal_itt_h24,0.0641110000000435,0.000245056686666286,1000000
|
||||
W1,null,male,rct_ideal_itt,0.128842000000108,0.000335081690094554,1000000
|
||||
W1,null,disease,rct_ideal_itt_h12,0,0,1000000
|
||||
W1,null,disease,rct_ideal_itt_h24,0,0,1000000
|
||||
W1,null,disease,rct_ideal_itt,0,0,1000000
|
||||
W1,null,rr,rct_ideal_itt_h12,0.0329870000000123,0.000178775999035469,1000000
|
||||
W1,null,rr,rct_ideal_itt_h24,0.0701960000000495,0.000255594447481916,1000000
|
||||
W1,null,rr,rct_ideal_itt,0.162295000000142,0.000368797143395813,1000000
|
||||
W1,null,nco1,rct_ideal_itt_h12,0,0,1000000
|
||||
W1,null,nco1,rct_ideal_itt_h24,0,0,1000000
|
||||
W1,null,nco1,rct_ideal_itt,0,0,1000000
|
||||
W1,null,nco2,rct_ideal_itt_h12,0,0,1000000
|
||||
W1,null,nco2,rct_ideal_itt_h24,0,0,1000000
|
||||
W1,null,nco2,rct_ideal_itt,0,0,1000000
|
||||
W1,harm,limb,rct_realistic_itt_h12,-0.001373599630385,0.000102517735138512,138782
|
||||
W1,harm,limb,rct_realistic_itt_h24,-0.000471771207073629,0.000138484846910791,138782
|
||||
W1,harm,limb,rct_realistic_itt,0.0265679752263632,0.000478240124365267,138782
|
||||
W1,harm,male,rct_realistic_itt_h12,0.012608142874936,0.000344952617431263,138782
|
||||
W1,harm,male,rct_realistic_itt_h24,0.0300850462927523,0.000501966846867299,138782
|
||||
W1,harm,male,rct_realistic_itt,0.0960941863029341,0.000845442821572819,138782
|
||||
W1,harm,disease,rct_realistic_itt_h12,-0.00102941131727323,8.84473969363332e-05,138782
|
||||
W1,harm,disease,rct_realistic_itt_h24,-0.00045693020065716,0.000118343791967768,138782
|
||||
W1,harm,disease,rct_realistic_itt,0.0213147720063486,0.000428165705068078,138782
|
||||
W1,harm,rr,rct_realistic_itt_h12,0.0148059979448982,0.000337744336761176,138782
|
||||
W1,harm,rr,rct_realistic_itt_h24,0.0343427964413466,0.000509697497507762,138782
|
||||
W1,harm,rr,rct_realistic_itt,0.100249385026741,0.000846286929385081,138782
|
||||
W1,harm,nco1,rct_realistic_itt_h12,6.53400738957156e-06,6.53399013392965e-06,138782
|
||||
W1,harm,nco1,rct_realistic_itt_h24,3.24449150747099e-05,1.67253770199362e-05,138782
|
||||
W1,harm,nco1,rct_realistic_itt,-0.000404092312836292,5.94467383510869e-05,138782
|
||||
W1,harm,nco2,rct_realistic_itt_h12,1.43513048739779e-05,1.01883274755683e-05,138782
|
||||
W1,harm,nco2,rct_realistic_itt_h24,2.39525242738712e-05,1.2268410518513e-05,138782
|
||||
W1,harm,nco2,rct_realistic_itt,-3.21786274097218e-05,2.67252013050681e-05,138782
|
||||
W1,harm,limb,rct_ideal_itt_h12,-0.000350000000000002,3.90368735943352e-05,1000000
|
||||
W1,harm,limb,rct_ideal_itt_h24,0.00576900000000044,8.80438449813041e-05,1000000
|
||||
W1,harm,limb,rct_ideal_itt,0.084697000000064,0.000286488076877455,1000000
|
||||
W1,harm,male,rct_ideal_itt_h12,0.0314140000000108,0.000179747491229717,1000000
|
||||
W1,harm,male,rct_ideal_itt_h24,0.0733340000000527,0.000264363621635138,1000000
|
||||
W1,harm,male,rct_ideal_itt,0.204205000000184,0.000407803038213413,1000000
|
||||
W1,harm,disease,rct_ideal_itt_h12,-0.000278,3.39988634513169e-05,1000000
|
||||
W1,harm,disease,rct_ideal_itt_h24,0.00446900000000027,7.75566118324939e-05,1000000
|
||||
W1,harm,disease,rct_ideal_itt,0.072661000000052,0.00026665591888863,1000000
|
||||
W1,harm,rr,rct_ideal_itt_h12,0.0331900000000125,0.000179338852175699,1000000
|
||||
W1,harm,rr,rct_ideal_itt_h24,0.0751360000000545,0.000263800268203047,1000000
|
||||
W1,harm,rr,rct_ideal_itt,0.198015000000177,0.000399055208928203,1000000
|
||||
W1,harm,nco1,rct_ideal_itt_h12,1e-05,3.46408718135054e-06,1000000
|
||||
W1,harm,nco1,rct_ideal_itt_h24,-2.4e-05,9.27358743966557e-06,1000000
|
||||
W1,harm,nco1,rct_ideal_itt,-0.00149599999999997,4.06664724801911e-05,1000000
|
||||
W1,harm,nco2,rct_ideal_itt_h12,1e-06,9.9999950002108e-07,1000000
|
||||
W1,harm,nco2,rct_ideal_itt_h24,-1.1e-05,5.56775349674996e-06,1000000
|
||||
W1,harm,nco2,rct_ideal_itt,-0.00067000000000001,2.67123772807722e-05,1000000
|
||||
W1,benefit_harm,limb,rct_realistic_itt_h12,-0.0100687707503497,0.000273979275962129,138782
|
||||
W1,benefit_harm,limb,rct_realistic_itt_h24,-0.0201671873778362,0.000389225369172784,138782
|
||||
W1,benefit_harm,limb,rct_realistic_itt,-0.0179862610172264,0.000543775907644638,138782
|
||||
W1,benefit_harm,male,rct_realistic_itt_h12,0.000421249459074619,0.000395540209152943,138782
|
||||
W1,benefit_harm,male,rct_realistic_itt_h24,0.00384220084773466,0.000577500168129415,138782
|
||||
W1,benefit_harm,male,rct_realistic_itt,0.0456265915942134,0.000868756295164852,138782
|
||||
W1,benefit_harm,disease,rct_realistic_itt_h12,-0.00755786823382364,0.000235610498666142,138782
|
||||
W1,benefit_harm,disease,rct_realistic_itt_h24,-0.015261928209796,0.00033704582876464,138782
|
||||
W1,benefit_harm,disease,rct_realistic_itt,-0.0134227825916772,0.000480003954380111,138782
|
||||
W1,benefit_harm,rr,rct_realistic_itt_h12,0.010850111951166,0.000289795992225478,138782
|
||||
W1,benefit_harm,rr,rct_realistic_itt_h24,0.0255359971692079,0.000443045073074274,138782
|
||||
W1,benefit_harm,rr,rct_realistic_itt,0.0812230409215406,0.000778945174193417,138782
|
||||
W1,benefit_harm,nco1,rct_realistic_itt_h12,7.59973402766115e-05,2.05244409832568e-05,138782
|
||||
W1,benefit_harm,nco1,rct_realistic_itt_h24,0.000345328509197363,4.23578338108543e-05,138782
|
||||
W1,benefit_harm,nco1,rct_realistic_itt,0.00167760516470874,0.000102598875198916,138782
|
||||
W1,benefit_harm,nco2,rct_realistic_itt_h12,2.13829664316019e-05,1.23792028465136e-05,138782
|
||||
W1,benefit_harm,nco2,rct_realistic_itt_h24,0.000107801730973639,2.54154238646691e-05,138782
|
||||
W1,benefit_harm,nco2,rct_realistic_itt,0.000341525175221389,4.76676185289784e-05,138782
|
||||
W1,benefit_harm,limb,rct_ideal_itt_h12,-0.00922399999999968,9.70304994525534e-05,1000000
|
||||
W1,benefit_harm,limb,rct_ideal_itt_h24,-0.0120759999999976,0.000135735663050895,1000000
|
||||
W1,benefit_harm,limb,rct_ideal_itt,0.0938970000000732,0.000343127313676257,1000000
|
||||
W1,benefit_harm,male,rct_ideal_itt_h12,0.0154609999999951,0.000182794850799761,1000000
|
||||
W1,benefit_harm,male,rct_ideal_itt_h24,0.0452080000000246,0.000268142194991403,1000000
|
||||
W1,benefit_harm,male,rct_ideal_itt,0.194898000000174,0.00043004507855924,1000000
|
||||
W1,benefit_harm,disease,rct_ideal_itt_h12,-0.00684900000000059,8.37979188230937e-05,1000000
|
||||
W1,benefit_harm,disease,rct_ideal_itt_h24,-0.00885899999999994,0.000118247698154484,1000000
|
||||
W1,benefit_harm,disease,rct_ideal_itt,0.0869100000000663,0.000323926923700412,1000000
|
||||
W1,benefit_harm,rr,rct_ideal_itt_h12,0.0252970000000046,0.000157280201523023,1000000
|
||||
W1,benefit_harm,rr,rct_ideal_itt_h24,0.0618910000000412,0.000241492244430772,1000000
|
||||
W1,benefit_harm,rr,rct_ideal_itt,0.192663000000172,0.000396386135515462,1000000
|
||||
W1,benefit_harm,nco1,rct_ideal_itt_h12,7.79999999999999e-05,8.83141642093955e-06,1000000
|
||||
W1,benefit_harm,nco1,rct_ideal_itt_h24,0.000370000000000003,1.98459844803684e-05,1000000
|
||||
W1,benefit_harm,nco1,rct_ideal_itt,4.3e-05,5.36562964714672e-05,1000000
|
||||
W1,benefit_harm,nco2,rct_ideal_itt_h12,3.5e-05,5.91597625080754e-06,1000000
|
||||
W1,benefit_harm,nco2,rct_ideal_itt_h24,0.000184,1.39987908050369e-05,1000000
|
||||
W1,benefit_harm,nco2,rct_ideal_itt,-7.19999999999999e-05,3.59443293997484e-05,1000000
|
||||
W1_rho0,null,limb,rct_realistic_itt_h12,0,0,106170
|
||||
W1_rho0,null,limb,rct_realistic_itt_h24,0,0,106170
|
||||
W1_rho0,null,limb,rct_realistic_itt,0,0,106170
|
||||
W1_rho0,null,male,rct_realistic_itt_h12,0.0162623427800871,0.000403467759713899,106170
|
||||
W1_rho0,null,male,rct_realistic_itt_h24,0.034257765877929,0.000584347981441931,106170
|
||||
W1_rho0,null,male,rct_realistic_itt,0.0745038473435866,0.000847319247700144,106170
|
||||
W1_rho0,null,disease,rct_realistic_itt_h12,0,0,106170
|
||||
W1_rho0,null,disease,rct_realistic_itt_h24,0,0,106170
|
||||
W1_rho0,null,disease,rct_realistic_itt,0,0,106170
|
||||
W1_rho0,null,rr,rct_realistic_itt_h12,0.0174208354954014,0.000416968225183115,106170
|
||||
W1_rho0,null,rr,rct_realistic_itt_h24,0.0388602323683751,0.000619689726151177,106170
|
||||
W1_rho0,null,rr,rct_realistic_itt,0.0999419634753957,0.000964602112897118,106170
|
||||
W1_rho0,null,nco1,rct_realistic_itt_h12,0,0,106170
|
||||
W1_rho0,null,nco1,rct_realistic_itt_h24,0,0,106170
|
||||
W1_rho0,null,nco1,rct_realistic_itt,0,0,106170
|
||||
W1_rho0,null,nco2,rct_realistic_itt_h12,0,0,106170
|
||||
W1_rho0,null,nco2,rct_realistic_itt_h24,0,0,106170
|
||||
W1_rho0,null,nco2,rct_realistic_itt,0,0,106170
|
||||
W1_rho0,null,limb,rct_ideal_itt_h12,0,0,1000000
|
||||
W1_rho0,null,limb,rct_ideal_itt_h24,0,0,1000000
|
||||
W1_rho0,null,limb,rct_ideal_itt,0,0,1000000
|
||||
W1_rho0,null,male,rct_ideal_itt_h12,0.0340940000000134,0.000181630391631942,1000000
|
||||
W1_rho0,null,male,rct_ideal_itt_h24,0.0690780000000484,0.00025368135507928,1000000
|
||||
W1_rho0,null,male,rct_ideal_itt,0.137794000000117,0.000344730058978436,1000000
|
||||
W1_rho0,null,disease,rct_ideal_itt_h12,0,0,1000000
|
||||
W1_rho0,null,disease,rct_ideal_itt_h24,0,0,1000000
|
||||
W1_rho0,null,disease,rct_ideal_itt,0,0,1000000
|
||||
W1_rho0,null,rr,rct_ideal_itt_h12,0.0355950000000149,0.000185445398905871,1000000
|
||||
W1_rho0,null,rr,rct_ideal_itt_h24,0.0754360000000548,0.000264203349532408,1000000
|
||||
W1_rho0,null,rr,rct_ideal_itt,0.172761000000152,0.000378109292240791,1000000
|
||||
W1_rho0,null,nco1,rct_ideal_itt_h12,0,0,1000000
|
||||
W1_rho0,null,nco1,rct_ideal_itt_h24,0,0,1000000
|
||||
W1_rho0,null,nco1,rct_ideal_itt,0,0,1000000
|
||||
W1_rho0,null,nco2,rct_ideal_itt_h12,0,0,1000000
|
||||
W1_rho0,null,nco2,rct_ideal_itt_h24,0,0,1000000
|
||||
W1_rho0,null,nco2,rct_ideal_itt,0,0,1000000
|
||||
W1_rho0,harm,limb,rct_realistic_itt_h12,-0.0013361050646777,0.000117190542881897,106170
|
||||
W1_rho0,harm,limb,rct_realistic_itt_h24,-0.000199473659602554,0.00016501219063867,106170
|
||||
W1_rho0,harm,limb,rct_realistic_itt,0.0279237552752964,0.00055718382017812,106170
|
||||
W1_rho0,harm,male,rct_realistic_itt_h12,0.0144587886735274,0.000414584056428262,106170
|
||||
W1_rho0,harm,male,rct_realistic_itt_h24,0.0340510024625777,0.000606763317824294,106170
|
||||
W1_rho0,harm,male,rct_realistic_itt,0.103495525655754,0.000995986223038789,106170
|
||||
W1_rho0,harm,disease,rct_realistic_itt_h12,-0.000966024324043659,9.89895229423779e-05,106170
|
||||
W1_rho0,harm,disease,rct_realistic_itt_h24,-8.31239638047122e-05,0.000141161203711099,106170
|
||||
W1_rho0,harm,disease,rct_realistic_itt,0.0228166798980784,0.000502703874328023,106170
|
||||
W1_rho0,harm,rr,rct_realistic_itt_h12,0.0168849054515258,0.00041055143195152,106170
|
||||
W1_rho0,harm,rr,rct_realistic_itt_h24,0.0387736230445496,0.000619157819043317,106170
|
||||
W1_rho0,harm,rr,rct_realistic_itt,0.108816624683441,0.00100267246838486,106170
|
||||
W1_rho0,harm,nco1,rct_realistic_itt_h12,0,0,106170
|
||||
W1_rho0,harm,nco1,rct_realistic_itt_h24,8.30510581723947e-06,8.30507910279427e-06,106170
|
||||
W1_rho0,harm,nco1,rct_realistic_itt,-0.000319062365139344,6.37576618224476e-05,106170
|
||||
W1_rho0,harm,nco2,rct_realistic_itt_h12,0,0,106170
|
||||
W1_rho0,harm,nco2,rct_realistic_itt_h24,3.50615336313888e-06,3.50615891102087e-06,106170
|
||||
W1_rho0,harm,nco2,rct_realistic_itt,-5.26083778885331e-05,3.06349260265071e-05,106170
|
||||
W1_rho0,harm,limb,rct_ideal_itt_h12,-0.000334000000000002,3.93686225820893e-05,1000000
|
||||
W1_rho0,harm,limb,rct_ideal_itt_h24,0.00581800000000045,8.84768380765336e-05,1000000
|
||||
W1_rho0,harm,limb,rct_ideal_itt,0.0849540000000643,0.000286860275893109,1000000
|
||||
W1_rho0,harm,male,rct_ideal_itt_h12,0.0340030000000134,0.000186324437451083,1000000
|
||||
W1_rho0,harm,male,rct_ideal_itt_h24,0.0784980000000578,0.000272536353533258,1000000
|
||||
W1_rho0,harm,male,rct_ideal_itt,0.212454000000192,0.000413610079528282,1000000
|
||||
W1_rho0,harm,disease,rct_ideal_itt_h12,-0.000266,3.43209738206234e-05,1000000
|
||||
W1_rho0,harm,disease,rct_ideal_itt_h24,0.00447300000000027,7.77624091120047e-05,1000000
|
||||
W1_rho0,harm,disease,rct_ideal_itt,0.0728310000000522,0.000266920672557173,1000000
|
||||
W1_rho0,harm,rr,rct_ideal_itt_h12,0.0358380000000152,0.000186042032231041,1000000
|
||||
W1_rho0,harm,rr,rct_ideal_itt_h24,0.08067200000006,0.000272477574151182,1000000
|
||||
W1_rho0,harm,rr,rct_ideal_itt,0.209534000000189,0.000407491721200427,1000000
|
||||
W1_rho0,harm,nco1,rct_ideal_itt_h12,8e-06,3.46409237750559e-06,1000000
|
||||
W1_rho0,harm,nco1,rct_ideal_itt_h24,-2.3e-05,8.99997061103611e-06,1000000
|
||||
W1_rho0,harm,nco1,rct_ideal_itt,-0.00147299999999998,4.02843675763495e-05,1000000
|
||||
W1_rho0,harm,nco2,rct_ideal_itt_h12,1e-06,9.9999950002108e-07,1000000
|
||||
W1_rho0,harm,nco2,rct_ideal_itt_h24,-7e-06,5.38516025765599e-06,1000000
|
||||
W1_rho0,harm,nco2,rct_ideal_itt,-0.000546000000000007,2.43659985224613e-05,1000000
|
||||
W1_rho0,benefit_harm,limb,rct_realistic_itt_h12,-0.00978245408702786,0.000311474250813258,106170
|
||||
W1_rho0,benefit_harm,limb,rct_realistic_itt_h24,-0.0192033040015254,0.000438243377013365,106170
|
||||
W1_rho0,benefit_harm,limb,rct_realistic_itt,-0.0153924928954827,0.000617493172659435,106170
|
||||
W1_rho0,benefit_harm,male,rct_realistic_itt_h12,0.00177395900211174,0.000460458555015892,106170
|
||||
W1_rho0,benefit_harm,male,rct_realistic_itt_h24,0.00730717553832012,0.000671928525462793,106170
|
||||
W1_rho0,benefit_harm,male,rct_realistic_itt,0.0538636617312581,0.00100963757693059,106170
|
||||
W1_rho0,benefit_harm,disease,rct_realistic_itt_h12,-0.00714369696164247,0.000263232069347728,106170
|
||||
W1_rho0,benefit_harm,disease,rct_realistic_itt_h24,-0.0139297664193082,0.000369962763837187,106170
|
||||
W1_rho0,benefit_harm,disease,rct_realistic_itt,-0.0108484078699601,0.000534054620194249,106170
|
||||
W1_rho0,benefit_harm,rr,rct_realistic_itt_h12,0.0119936567519674,0.000347437677675633,106170
|
||||
W1_rho0,benefit_harm,rr,rct_realistic_itt_h24,0.0285024847698048,0.000534906512758664,106170
|
||||
W1_rho0,benefit_harm,rr,rct_realistic_itt,0.088924625651879,0.000926453186350461,106170
|
||||
W1_rho0,benefit_harm,nco1,rct_realistic_itt_h12,0.000100116370211912,2.57359968375756e-05,106170
|
||||
W1_rho0,benefit_harm,nco1,rct_realistic_itt_h24,0.000372922895699211,5.46968380429417e-05,106170
|
||||
W1_rho0,benefit_harm,nco1,rct_realistic_itt,0.00176038422237092,0.000127814346255105,106170
|
||||
W1_rho0,benefit_harm,nco2,rct_realistic_itt_h12,1.14055757904836e-05,8.09474394691263e-06,106170
|
||||
W1_rho0,benefit_harm,nco2,rct_realistic_itt_h24,6.31004145073422e-05,2.26778385327308e-05,106170
|
||||
W1_rho0,benefit_harm,nco2,rct_realistic_itt,0.000279311699475823,4.96074893035325e-05,106170
|
||||
W1_rho0,benefit_harm,limb,rct_ideal_itt_h12,-0.0091969999999997,9.67802417397742e-05,1000000
|
||||
W1_rho0,benefit_harm,limb,rct_ideal_itt_h24,-0.0119569999999977,0.000135654082691168,1000000
|
||||
W1_rho0,benefit_harm,limb,rct_ideal_itt,0.0951390000000745,0.000343915063173529,1000000
|
||||
W1_rho0,benefit_harm,male,rct_ideal_itt_h12,0.0174999999999968,0.000187775797163343,1000000
|
||||
W1_rho0,benefit_harm,male,rct_ideal_itt_h24,0.0498540000000292,0.000275438883753021,1000000
|
||||
W1_rho0,benefit_harm,male,rct_ideal_itt,0.202967000000182,0.000434913091212629,1000000
|
||||
W1_rho0,benefit_harm,disease,rct_ideal_itt_h12,-0.00682700000000058,8.35726753857821e-05,1000000
|
||||
W1_rho0,benefit_harm,disease,rct_ideal_itt_h24,-0.008782,0.000118156152933778,1000000
|
||||
W1_rho0,benefit_harm,disease,rct_ideal_itt,0.0881540000000675,0.000324972110009958,1000000
|
||||
W1_rho0,benefit_harm,rr,rct_ideal_itt_h12,0.0273460000000067,0.000163316246233535,1000000
|
||||
W1_rho0,benefit_harm,rr,rct_ideal_itt_h24,0.0667490000000461,0.000250107119050684,1000000
|
||||
W1_rho0,benefit_harm,rr,rct_ideal_itt,0.203860000000183,0.000404745723139152,1000000
|
||||
W1_rho0,benefit_harm,nco1,rct_ideal_itt_h12,7.49999999999999e-05,8.65992927222551e-06,1000000
|
||||
W1_rho0,benefit_harm,nco1,rct_ideal_itt_h24,0.000373000000000003,1.97195555475079e-05,1000000
|
||||
W1_rho0,benefit_harm,nco1,rct_ideal_itt,2.4e-05,5.39999946668951e-05,1000000
|
||||
W1_rho0,benefit_harm,nco2,rct_ideal_itt_h12,3.6e-05,5.99989199903001e-06,1000000
|
||||
W1_rho0,benefit_harm,nco2,rct_ideal_itt_h24,0.00019,1.41408592383115e-05,1000000
|
||||
W1_rho0,benefit_harm,nco2,rct_ideal_itt,0.000117,3.49998044424634e-05,1000000
|
||||
W2,null,limb,rct_realistic_itt_h12,0,0,138782
|
||||
W2,null,limb,rct_realistic_itt_h24,0,0,138782
|
||||
W2,null,limb,rct_realistic_itt,0,0,138782
|
||||
W2,null,male,rct_realistic_itt_h12,0.0337810807940289,0.00050535197517823,138782
|
||||
W2,null,male,rct_realistic_itt_h24,0.0706124607788977,0.000718022668609694,138782
|
||||
W2,null,male,rct_realistic_itt,0.15039557347479,0.00100866830147449,138782
|
||||
W2,null,disease,rct_realistic_itt_h12,0,0,138782
|
||||
W2,null,disease,rct_realistic_itt_h24,0,0,138782
|
||||
W2,null,disease,rct_realistic_itt,0,0,138782
|
||||
W2,null,rr,rct_realistic_itt_h12,0.0350866714353561,0.000514159961326907,138782
|
||||
W2,null,rr,rct_realistic_itt_h24,0.0765060586504421,0.000743918029824463,138782
|
||||
W2,null,rr,rct_realistic_itt,0.188204683561983,0.00109988933258071,138782
|
||||
W2,null,nco1,rct_realistic_itt_h12,0,0,138782
|
||||
W2,null,nco1,rct_realistic_itt_h24,0,0,138782
|
||||
W2,null,nco1,rct_realistic_itt,0,0,138782
|
||||
W2,null,nco2,rct_realistic_itt_h12,0,0,138782
|
||||
W2,null,nco2,rct_realistic_itt_h24,0,0,138782
|
||||
W2,null,nco2,rct_realistic_itt,0,0,138782
|
||||
W2,null,limb,rct_ideal_itt_h12,0,0,1000000
|
||||
W2,null,limb,rct_ideal_itt_h24,0,0,1000000
|
||||
W2,null,limb,rct_ideal_itt,0,0,1000000
|
||||
W2,null,male,rct_ideal_itt_h12,0.0329770000000123,0.00017876665648551,1000000
|
||||
W2,null,male,rct_ideal_itt_h24,0.0673800000000467,0.000250802582918274,1000000
|
||||
W2,null,male,rct_ideal_itt,0.138373000000118,0.000345366345885349,1000000
|
||||
W2,null,disease,rct_ideal_itt_h12,0,0,1000000
|
||||
W2,null,disease,rct_ideal_itt_h24,0,0,1000000
|
||||
W2,null,disease,rct_ideal_itt,0,0,1000000
|
||||
W2,null,rr,rct_ideal_itt_h12,0.0344580000000138,0.000182605164868548,1000000
|
||||
W2,null,rr,rct_ideal_itt_h24,0.073691000000053,0.000261416595724307,1000000
|
||||
W2,null,rr,rct_ideal_itt,0.173625000000153,0.000378897557888212,1000000
|
||||
W2,null,nco1,rct_ideal_itt_h12,0,0,1000000
|
||||
W2,null,nco1,rct_ideal_itt_h24,0,0,1000000
|
||||
W2,null,nco1,rct_ideal_itt,0,0,1000000
|
||||
W2,null,nco2,rct_ideal_itt_h12,0,0,1000000
|
||||
W2,null,nco2,rct_ideal_itt_h24,0,0,1000000
|
||||
W2,null,nco2,rct_ideal_itt,0,0,1000000
|
||||
W2,harm,limb,rct_realistic_itt_h12,-0.000647095963942255,8.58503040561573e-05,138782
|
||||
W2,harm,limb,rct_realistic_itt_h24,0.0036405242539962,0.000199173446527444,138782
|
||||
W2,harm,limb,rct_realistic_itt,0.0669917748050983,0.000721672135709116,138782
|
||||
W2,harm,male,rct_realistic_itt_h12,0.032542405057567,0.000508352078311317,138782
|
||||
W2,harm,male,rct_realistic_itt_h24,0.0757891246498343,0.000748628637494755,138782
|
||||
W2,harm,male,rct_realistic_itt,0.220119775713179,0.00117638221802888,138782
|
||||
W2,harm,disease,rct_realistic_itt_h12,-0.000504880270950688,7.41485576314562e-05,138782
|
||||
W2,harm,disease,rct_realistic_itt_h24,0.00254566205020426,0.000168064607656196,138782
|
||||
W2,harm,disease,rct_realistic_itt,0.0549723983558079,0.000654344898207341,138782
|
||||
W2,harm,rr,rct_realistic_itt_h12,0.034452698741701,0.00050960970223207,138782
|
||||
W2,harm,rr,rct_realistic_itt_h24,0.0786394813495808,0.000753394477334654,138782
|
||||
W2,harm,rr,rct_realistic_itt,0.219344001226149,0.00116567947137748,138782
|
||||
W2,harm,nco1,rct_realistic_itt_h12,6.53400738957156e-06,6.53399013392965e-06,138782
|
||||
W2,harm,nco1,rct_realistic_itt_h24,-8.45156676626482e-06,1.58783112120704e-05,138782
|
||||
W2,harm,nco1,rct_realistic_itt,-0.00116717312537871,8.95220971394273e-05,138782
|
||||
W2,harm,nco2,rct_realistic_itt_h12,1.43513048739779e-05,1.01883274755683e-05,138782
|
||||
W2,harm,nco2,rct_realistic_itt_h24,1.97093822894091e-05,1.15113069317871e-05,138782
|
||||
W2,harm,nco2,rct_realistic_itt,-0.000159798329295093,3.67672360388556e-05,138782
|
||||
W2,harm,limb,rct_ideal_itt_h12,-0.00069000000000001,3.30986993702438e-05,1000000
|
||||
W2,harm,limb,rct_ideal_itt_h24,0.00370400000000017,7.39883800605641e-05,1000000
|
||||
W2,harm,limb,rct_ideal_itt,0.0625790000000419,0.00025018167150836,1000000
|
||||
W2,harm,male,rct_ideal_itt_h12,0.0317890000000111,0.000180384199638679,1000000
|
||||
W2,harm,male,rct_ideal_itt_h24,0.072679000000052,0.000262794145596412,1000000
|
||||
W2,harm,male,rct_ideal_itt,0.202625000000182,0.000406091257445419,1000000
|
||||
W2,harm,disease,rct_ideal_itt_h12,-0.000530000000000006,2.87353284301963e-05,1000000
|
||||
W2,harm,disease,rct_ideal_itt_h24,0.00283200000000005,6.47300531131894e-05,1000000
|
||||
W2,harm,disease,rct_ideal_itt,0.0515850000000309,0.0002283111643668,1000000
|
||||
W2,harm,rr,rct_ideal_itt_h12,0.0338660000000132,0.000181088635878571,1000000
|
||||
W2,harm,rr,rct_ideal_itt_h24,0.0758370000000552,0.000264918382582814,1000000
|
||||
W2,harm,rr,rct_ideal_itt,0.201188000000181,0.00040129713262865,1000000
|
||||
W2,harm,nco1,rct_ideal_itt_h12,1e-05,3.1622618487401e-06,1000000
|
||||
W2,harm,nco1,rct_ideal_itt_h24,-7e-06,7.81024653897016e-06,1000000
|
||||
W2,harm,nco1,rct_ideal_itt,-0.001151,3.58284132919553e-05,1000000
|
||||
W2,harm,nco2,rct_ideal_itt_h12,1e-06,9.9999950002108e-07,1000000
|
||||
W2,harm,nco2,rct_ideal_itt_h24,-2e-06,5.47722520992009e-06,1000000
|
||||
W2,harm,nco2,rct_ideal_itt,-0.000430000000000004,2.21317667617172e-05,1000000
|
||||
W2,benefit_harm,limb,rct_realistic_itt_h12,-0.00956931188534672,0.000266722694787558,138782
|
||||
W2,benefit_harm,limb,rct_realistic_itt_h24,-0.015997189555154,0.000349278093848526,138782
|
||||
W2,benefit_harm,limb,rct_realistic_itt,0.0482680667089776,0.000753022714736988,138782
|
||||
W2,benefit_harm,male,rct_realistic_itt_h12,0.0152903234806801,0.000510562769863247,138782
|
||||
W2,benefit_harm,male,rct_realistic_itt_h24,0.0426586754708397,0.000742059131594603,138782
|
||||
W2,benefit_harm,male,rct_realistic_itt,0.19384749564067,0.00118274574590801,138782
|
||||
W2,benefit_harm,disease,rct_realistic_itt_h12,-0.00717623900129959,0.000229450691329312,138782
|
||||
W2,benefit_harm,disease,rct_realistic_itt_h24,-0.0121039723747459,0.000302167336843336,138782
|
||||
W2,benefit_harm,disease,rct_realistic_itt,0.0405570112816336,0.000680988232258797,138782
|
||||
W2,benefit_harm,rr,rct_realistic_itt_h12,0.0254104353872551,0.000440262059614362,138782
|
||||
W2,benefit_harm,rr,rct_realistic_itt_h24,0.0615396754787447,0.00067385273770769,138782
|
||||
W2,benefit_harm,rr,rct_realistic_itt,0.200788058510693,0.00113184883009162,138782
|
||||
W2,benefit_harm,nco1,rct_realistic_itt_h12,7.59973402766115e-05,2.05244409832568e-05,138782
|
||||
W2,benefit_harm,nco1,rct_realistic_itt_h24,0.000304283539237408,3.99652071948251e-05,138782
|
||||
W2,benefit_harm,nco1,rct_realistic_itt,0.000482621722916582,0.000102446486450744,138782
|
||||
W2,benefit_harm,nco2,rct_realistic_itt_h12,2.13829664316019e-05,1.23792028465136e-05,138782
|
||||
W2,benefit_harm,nco2,rct_realistic_itt_h24,0.000101840077489285,2.47064296907095e-05,138782
|
||||
W2,benefit_harm,nco2,rct_realistic_itt,0.000204351440100052,4.71903593479565e-05,138782
|
||||
W2,benefit_harm,limb,rct_ideal_itt_h12,-0.00962099999999938,9.85212482625226e-05,1000000
|
||||
W2,benefit_harm,limb,rct_ideal_itt_h24,-0.0157609999999951,0.00012769726261298,1000000
|
||||
W2,benefit_harm,limb,rct_ideal_itt,0.0460580000000254,0.000266170360176093,1000000
|
||||
W2,benefit_harm,male,rct_ideal_itt_h12,0.0149959999999954,0.000183191484472827,1000000
|
||||
W2,benefit_harm,male,rct_ideal_itt_h24,0.0403690000000197,0.000261765818697534,1000000
|
||||
W2,benefit_harm,male,rct_ideal_itt,0.179103000000158,0.000410337806437503,1000000
|
||||
W2,benefit_harm,disease,rct_ideal_itt_h12,-0.00714100000000062,8.50294426588907e-05,1000000
|
||||
W2,benefit_harm,disease,rct_ideal_itt_h24,-0.0117909999999978,0.000110824060199874,1000000
|
||||
W2,benefit_harm,disease,rct_ideal_itt,0.0387480000000181,0.000241562812732483,1000000
|
||||
W2,benefit_harm,rr,rct_ideal_itt_h12,0.0251730000000045,0.00015694368439448,1000000
|
||||
W2,benefit_harm,rr,rct_ideal_itt_h24,0.0593490000000387,0.00023683896680697,1000000
|
||||
W2,benefit_harm,rr,rct_ideal_itt,0.184437000000164,0.000389463724920822,1000000
|
||||
W2,benefit_harm,nco1,rct_ideal_itt_h12,7.79999999999999e-05,8.8314164209376e-06,1000000
|
||||
W2,benefit_harm,nco1,rct_ideal_itt_h24,0.000404000000000003,2.0145391135619e-05,1000000
|
||||
W2,benefit_harm,nco1,rct_ideal_itt,0.000787000000000013,4.45014677399414e-05,1000000
|
||||
W2,benefit_harm,nco2,rct_ideal_itt_h12,3.3e-05,5.74446786046983e-06,1000000
|
||||
W2,benefit_harm,nco2,rct_ideal_itt_h24,0.000187,1.37464552158657e-05,1000000
|
||||
W2,benefit_harm,nco2,rct_ideal_itt,0.000370000000000003,2.87378339478796e-05,1000000
|
||||
W3,null,limb,rct_realistic_itt_h12,0,0,138782
|
||||
W3,null,limb,rct_realistic_itt_h24,0,0,138782
|
||||
W3,null,limb,rct_realistic_itt,0,0,138782
|
||||
W3,null,male,rct_realistic_itt_h12,0.00903030984809614,0.000265285956694921,138782
|
||||
W3,null,male,rct_realistic_itt_h24,0.0193240465029616,0.000386258154997006,138782
|
||||
W3,null,male,rct_realistic_itt,0.0440196832365507,0.000580334073905548,138782
|
||||
W3,null,disease,rct_realistic_itt_h12,0,0,138782
|
||||
W3,null,disease,rct_realistic_itt_h24,0,0,138782
|
||||
W3,null,disease,rct_realistic_itt,0,0,138782
|
||||
W3,null,rr,rct_realistic_itt_h12,0.00980558532326791,0.000275302372913034,138782
|
||||
W3,null,rr,rct_realistic_itt_h24,0.0224939568930911,0.000414150028887632,138782
|
||||
W3,null,rr,rct_realistic_itt,0.0632610041136034,0.000684900741084694,138782
|
||||
W3,null,nco1,rct_realistic_itt_h12,0,0,138782
|
||||
W3,null,nco1,rct_realistic_itt_h24,0,0,138782
|
||||
W3,null,nco1,rct_realistic_itt,0,0,138782
|
||||
W3,null,nco2,rct_realistic_itt_h12,0,0,138782
|
||||
W3,null,nco2,rct_realistic_itt_h24,0,0,138782
|
||||
W3,null,nco2,rct_realistic_itt,0,0,138782
|
||||
W3,null,limb,rct_ideal_itt_h12,0,0,1000000
|
||||
W3,null,limb,rct_ideal_itt_h24,0,0,1000000
|
||||
W3,null,limb,rct_ideal_itt,0,0,1000000
|
||||
W3,null,male,rct_ideal_itt_h12,0.0168199999999962,0.000128899525213432,1000000
|
||||
W3,null,male,rct_ideal_itt_h24,0.0345620000000139,0.000182837272338099,1000000
|
||||
W3,null,male,rct_ideal_itt,0.0722660000000516,0.000259020511240493,1000000
|
||||
W3,null,disease,rct_ideal_itt_h12,0,0,1000000
|
||||
W3,null,disease,rct_ideal_itt_h24,0,0,1000000
|
||||
W3,null,disease,rct_ideal_itt,0,0,1000000
|
||||
W3,null,rr,rct_ideal_itt_h12,0.0178909999999972,0.000132879314112807,1000000
|
||||
W3,null,rr,rct_ideal_itt_h24,0.0389820000000183,0.000193758622198589,1000000
|
||||
W3,null,rr,rct_ideal_itt,0.0965230000000759,0.000295442567130136,1000000
|
||||
W3,null,nco1,rct_ideal_itt_h12,0,0,1000000
|
||||
W3,null,nco1,rct_ideal_itt_h24,0,0,1000000
|
||||
W3,null,nco1,rct_ideal_itt,0,0,1000000
|
||||
W3,null,nco2,rct_ideal_itt_h12,0,0,1000000
|
||||
W3,null,nco2,rct_ideal_itt_h24,0,0,1000000
|
||||
W3,null,nco2,rct_ideal_itt,0,0,1000000
|
||||
W3,harm,limb,rct_realistic_itt_h12,-0.000588249907037527,8.24840115132283e-05,138782
|
||||
W3,harm,limb,rct_realistic_itt_h24,0.00351844644831374,0.000194074633264242,138782
|
||||
W3,harm,limb,rct_realistic_itt,0.0660614353198847,0.000716019815307085,138782
|
||||
W3,harm,male,rct_realistic_itt_h12,0.00834947836101997,0.000276711896017007,138782
|
||||
W3,harm,male,rct_realistic_itt_h24,0.0235198547064754,0.000436716645646618,138782
|
||||
W3,harm,male,rct_realistic_itt,0.111320688064598,0.000900182566646114,138782
|
||||
W3,harm,disease,rct_realistic_itt_h12,-0.00044603421404596,7.02226079620628e-05,138782
|
||||
W3,harm,disease,rct_realistic_itt_h24,0.00243844263822334,0.000162308204945514,138782
|
||||
W3,harm,disease,rct_realistic_itt,0.0537864738547063,0.000646605565247129,138782
|
||||
W3,harm,rr,rct_realistic_itt_h12,0.009669523269744,0.000273479054555858,138782
|
||||
W3,harm,rr,rct_realistic_itt_h24,0.0233429831683389,0.000422018922865152,138782
|
||||
W3,harm,rr,rct_realistic_itt,0.0778699410977019,0.000755464856924967,138782
|
||||
W3,harm,nco1,rct_realistic_itt_h12,6.53400738957156e-06,6.53399013392965e-06,138782
|
||||
W3,harm,nco1,rct_realistic_itt_h24,-8.45156676626482e-06,1.58783112120704e-05,138782
|
||||
W3,harm,nco1,rct_realistic_itt,-0.00114175384665679,8.83389863805806e-05,138782
|
||||
W3,harm,nco2,rct_realistic_itt_h12,1.43513048739779e-05,1.01883274755683e-05,138782
|
||||
W3,harm,nco2,rct_realistic_itt_h24,1.97093822894091e-05,1.15113069317871e-05,138782
|
||||
W3,harm,nco2,rct_realistic_itt,-0.000159798329295093,3.67672360388556e-05,138782
|
||||
W3,harm,limb,rct_ideal_itt_h12,-0.000649000000000009,3.24742790372589e-05,1000000
|
||||
W3,harm,limb,rct_ideal_itt_h24,0.00364500000000016,7.30596603814059e-05,1000000
|
||||
W3,harm,limb,rct_ideal_itt,0.0614270000000408,0.00024811232067428,1000000
|
||||
W3,harm,male,rct_ideal_itt_h12,0.0159559999999953,0.000132081058687124,1000000
|
||||
W3,harm,male,rct_ideal_itt_h24,0.0390860000000184,0.000198131987835618,1000000
|
||||
W3,harm,male,rct_ideal_itt,0.135530000000115,0.000347591166603257,1000000
|
||||
W3,harm,disease,rct_ideal_itt_h12,-0.000490000000000005,2.80670607653485e-05,1000000
|
||||
W3,harm,disease,rct_ideal_itt_h24,0.00277000000000004,6.3673598139629e-05,1000000
|
||||
W3,harm,disease,rct_ideal_itt,0.0503310000000297,0.000225725032816082,1000000
|
||||
W3,harm,rr,rct_ideal_itt_h12,0.017620999999997,0.000131910956173239,1000000
|
||||
W3,harm,rr,rct_ideal_itt_h24,0.0401920000000195,0.000196663680264494,1000000
|
||||
W3,harm,rr,rct_ideal_itt,0.114807000000094,0.000319321707298835,1000000
|
||||
W3,harm,nco1,rct_ideal_itt_h12,1.1e-05,3.31660654886121e-06,1000000
|
||||
W3,harm,nco1,rct_ideal_itt_h24,-7e-06,7.93725084644526e-06,1000000
|
||||
W3,harm,nco1,rct_ideal_itt,-0.001143,3.57448395016908e-05,1000000
|
||||
W3,harm,nco2,rct_ideal_itt_h12,1e-06,9.9999950002108e-07,1000000
|
||||
W3,harm,nco2,rct_ideal_itt_h24,4.2351647362715e-22,5.29150262212918e-06,1000000
|
||||
W3,harm,nco2,rct_ideal_itt,-0.000428000000000004,2.21318055297241e-05,1000000
|
||||
W3,benefit_harm,limb,rct_realistic_itt_h12,-0.00940372301744105,0.00026426342050986,138782
|
||||
W3,benefit_harm,limb,rct_realistic_itt_h24,-0.0157683072512824,0.00034646646266276,138782
|
||||
W3,benefit_harm,limb,rct_realistic_itt,0.0476437098675707,0.000749080677845296,138782
|
||||
W3,benefit_harm,male,rct_realistic_itt_h12,-0.00244380704260416,0.000352039283668794,138782
|
||||
W3,benefit_harm,male,rct_realistic_itt_h24,0.000480156976458699,0.00049420426918914,138782
|
||||
W3,benefit_harm,male,rct_realistic_itt,0.0904566992351862,0.000918464245987336,138782
|
||||
W3,benefit_harm,disease,rct_realistic_itt_h12,-0.00701065013339392,0.000226573633546181,138782
|
||||
W3,benefit_harm,disease,rct_realistic_itt_h24,-0.0118546838717681,0.000298550065502498,138782
|
||||
W3,benefit_harm,disease,rct_realistic_itt,0.039773855046475,0.000674822314350528,138782
|
||||
W3,benefit_harm,rr,rct_realistic_itt_h12,0.0073096552975273,0.000238753234295454,138782
|
||||
W3,benefit_harm,rr,rct_realistic_itt_h24,0.0177366927987561,0.000370980734206821,138782
|
||||
W3,benefit_harm,rr,rct_realistic_itt,0.0682386327080767,0.000716941848502967,138782
|
||||
W3,benefit_harm,nco1,rct_realistic_itt_h12,7.59973402766115e-05,2.05244409832568e-05,138782
|
||||
W3,benefit_harm,nco1,rct_realistic_itt_h24,0.000304283539237408,3.99652071948251e-05,138782
|
||||
W3,benefit_harm,nco1,rct_realistic_itt,0.000500793138392635,0.000102032165589858,138782
|
||||
W3,benefit_harm,nco2,rct_realistic_itt_h12,2.13829664316019e-05,1.23792028465136e-05,138782
|
||||
W3,benefit_harm,nco2,rct_realistic_itt_h24,0.000101840077489285,2.47064296907095e-05,138782
|
||||
W3,benefit_harm,nco2,rct_realistic_itt,0.000217072611910802,4.63066004091942e-05,138782
|
||||
W3,benefit_harm,limb,rct_ideal_itt_h12,-0.00939399999999955,9.73537506412589e-05,1000000
|
||||
W3,benefit_harm,limb,rct_ideal_itt_h24,-0.0154449999999951,0.000126421722717641,1000000
|
||||
W3,benefit_harm,limb,rct_ideal_itt,0.0451160000000245,0.000263800201942329,1000000
|
||||
W3,benefit_harm,male,rct_ideal_itt_h12,0.0031670000000001,0.000147895132141516,1000000
|
||||
W3,benefit_harm,male,rct_ideal_itt_h24,0.0132119999999967,0.00020868503313997,1000000
|
||||
W3,benefit_harm,male,rct_ideal_itt,0.115277000000095,0.000354632504534453,1000000
|
||||
W3,benefit_harm,disease,rct_ideal_itt_h12,-0.00691300000000059,8.36612839434453e-05,1000000
|
||||
W3,benefit_harm,disease,rct_ideal_itt_h24,-0.011476999999998,0.00010934934142963,1000000
|
||||
W3,benefit_harm,disease,rct_ideal_itt,0.0377310000000171,0.000238623912544438,1000000
|
||||
W3,benefit_harm,rr,rct_ideal_itt_h12,0.0130149999999969,0.000113717236050071,1000000
|
||||
W3,benefit_harm,rr,rct_ideal_itt_h24,0.0309500000000103,0.000173902551733555,1000000
|
||||
W3,benefit_harm,rr,rct_ideal_itt,0.103537000000083,0.000306961055563444,1000000
|
||||
W3,benefit_harm,nco1,rct_ideal_itt_h12,7.79999999999999e-05,8.83141642093734e-06,1000000
|
||||
W3,benefit_harm,nco1,rct_ideal_itt_h24,0.000402000000000003,2.00957307904308e-05,1000000
|
||||
W3,benefit_harm,nco1,rct_ideal_itt,0.000793000000000013,4.43888629161159e-05,1000000
|
||||
W3,benefit_harm,nco2,rct_ideal_itt_h12,3.3e-05,5.74446786046983e-06,1000000
|
||||
W3,benefit_harm,nco2,rct_ideal_itt_h24,0.000188,1.37827666309731e-05,1000000
|
||||
W3,benefit_harm,nco2,rct_ideal_itt,0.000366000000000002,2.86682061525463e-05,1000000
|
||||
W4,null,limb,rct_realistic_itt_h12,0,0,138782
|
||||
W4,null,limb,rct_realistic_itt_h24,0,0,138782
|
||||
W4,null,limb,rct_realistic_itt,0,0,138782
|
||||
W4,null,male,rct_realistic_itt_h12,0.0314301517471955,0.000488135642992868,138782
|
||||
W4,null,male,rct_realistic_itt_h24,0.0651870494834334,0.000691691002028623,138782
|
||||
W4,null,male,rct_realistic_itt,0.133995787257383,0.000960963787695265,138782
|
||||
W4,null,disease,rct_realistic_itt_h12,0,0,138782
|
||||
W4,null,disease,rct_realistic_itt_h24,0,0,138782
|
||||
W4,null,disease,rct_realistic_itt,0,0,138782
|
||||
W4,null,rr,rct_realistic_itt_h12,0.0326495807952769,0.000496607482977814,138782
|
||||
W4,null,rr,rct_realistic_itt_h24,0.0706865829537568,0.00071714466087967,138782
|
||||
W4,null,rr,rct_realistic_itt,0.167725642698684,0.00105114817642789,138782
|
||||
W4,null,nco1,rct_realistic_itt_h12,0,0,138782
|
||||
W4,null,nco1,rct_realistic_itt_h24,0,0,138782
|
||||
W4,null,nco1,rct_realistic_itt,0,0,138782
|
||||
W4,null,nco2,rct_realistic_itt_h12,0,0,138782
|
||||
W4,null,nco2,rct_realistic_itt_h24,0,0,138782
|
||||
W4,null,nco2,rct_realistic_itt,0,0,138782
|
||||
W4,null,limb,rct_ideal_itt_h12,0,0,1000000
|
||||
W4,null,limb,rct_ideal_itt_h24,0,0,1000000
|
||||
W4,null,limb,rct_ideal_itt,0,0,1000000
|
||||
W4,null,male,rct_ideal_itt_h12,0.0310470000000104,0.000173600356539896,1000000
|
||||
W4,null,male,rct_ideal_itt_h24,0.0625880000000419,0.000242323631239363,1000000
|
||||
W4,null,male,rct_ideal_itt,0.121135000000101,0.000326342322989634,1000000
|
||||
W4,null,disease,rct_ideal_itt_h12,0,0,1000000
|
||||
W4,null,disease,rct_ideal_itt_h24,0,0,1000000
|
||||
W4,null,disease,rct_ideal_itt,0,0,1000000
|
||||
W4,null,rr,rct_ideal_itt_h12,0.0324740000000118,0.000177418824603335,1000000
|
||||
W4,null,rr,rct_ideal_itt_h24,0.0685640000000479,0.000252829938703752,1000000
|
||||
W4,null,rr,rct_ideal_itt,0.152055000000131,0.000359157732727129,1000000
|
||||
W4,null,nco1,rct_ideal_itt_h12,0,0,1000000
|
||||
W4,null,nco1,rct_ideal_itt_h24,0,0,1000000
|
||||
W4,null,nco1,rct_ideal_itt,0,0,1000000
|
||||
W4,null,nco2,rct_ideal_itt_h12,0,0,1000000
|
||||
W4,null,nco2,rct_ideal_itt_h24,0,0,1000000
|
||||
W4,null,nco2,rct_ideal_itt,0,0,1000000
|
||||
W4,harm,limb,rct_realistic_itt_h12,-0.000627062213012661,8.62493618988047e-05,138782
|
||||
W4,harm,limb,rct_realistic_itt_h24,0.00355735445447908,0.000197263010075687,138782
|
||||
W4,harm,limb,rct_realistic_itt,0.064209964660219,0.000707931355134836,138782
|
||||
W4,harm,male,rct_realistic_itt_h12,0.0302959762106384,0.000492102326285693,138782
|
||||
W4,harm,male,rct_realistic_itt_h24,0.0702653620366293,0.000723483267829581,138782
|
||||
W4,harm,male,rct_realistic_itt,0.200766870513184,0.00113862740540386,138782
|
||||
W4,harm,disease,rct_realistic_itt_h12,-0.000484846520021094,7.46099874018648e-05,138782
|
||||
W4,harm,disease,rct_realistic_itt_h24,0.00248636075716954,0.000166080110011518,138782
|
||||
W4,harm,disease,rct_realistic_itt,0.0525976621937731,0.000641176211394885,138782
|
||||
W4,harm,rr,rct_realistic_itt_h12,0.0321195850346531,0.00049281544942983,138782
|
||||
W4,harm,rr,rct_realistic_itt_h24,0.0727389540473738,0.000726968108529196,138782
|
||||
W4,harm,rr,rct_realistic_itt,0.196531826803881,0.00112111646334625,138782
|
||||
W4,harm,nco1,rct_realistic_itt_h12,6.53400738957156e-06,6.53399013392965e-06,138782
|
||||
W4,harm,nco1,rct_realistic_itt_h24,-7.15379975309977e-06,1.90454145071415e-05,138782
|
||||
W4,harm,nco1,rct_realistic_itt,-0.00113926334365766,8.83869459965305e-05,138782
|
||||
W4,harm,nco2,rct_realistic_itt_h12,6.53400738957156e-06,6.53399013392965e-06,138782
|
||||
W4,harm,nco2,rct_realistic_itt_h24,1.18920848050027e-05,8.44994964807204e-06,138782
|
||||
W4,harm,nco2,rct_realistic_itt,-0.000142201617028631,3.35877497802296e-05,138782
|
||||
W4,harm,limb,rct_ideal_itt_h12,-0.00068600000000001,3.327956436009e-05,1000000
|
||||
W4,harm,limb,rct_ideal_itt_h24,0.00373100000000017,7.41692634389976e-05,1000000
|
||||
W4,harm,limb,rct_ideal_itt,0.059637000000039,0.000245006996288818,1000000
|
||||
W4,harm,male,rct_ideal_itt_h12,0.0299240000000093,0.000175449577441567,1000000
|
||||
W4,harm,male,rct_ideal_itt_h24,0.0678620000000472,0.000254795504193282,1000000
|
||||
W4,harm,male,rct_ideal_itt,0.182678000000162,0.000391084068095295,1000000
|
||||
W4,harm,disease,rct_ideal_itt_h12,-0.000525000000000006,2.88915969614512e-05,1000000
|
||||
W4,harm,disease,rct_ideal_itt_h24,0.00286600000000006,6.49290847306103e-05,1000000
|
||||
W4,harm,disease,rct_ideal_itt,0.0490340000000284,0.00022328382575656,1000000
|
||||
W4,harm,rr,rct_ideal_itt_h12,0.0319460000000113,0.000176021172259444,1000000
|
||||
W4,harm,rr,rct_ideal_itt_h24,0.07063500000005,0.000256381935350656,1000000
|
||||
W4,harm,rr,rct_ideal_itt,0.177898000000157,0.000383622603083671,1000000
|
||||
W4,harm,nco1,rct_ideal_itt_h12,9e-06,2.99998649996762e-06,1000000
|
||||
W4,harm,nco1,rct_ideal_itt_h24,-5e-06,7.81024807547441e-06,1000000
|
||||
W4,harm,nco1,rct_ideal_itt,-0.00109800000000001,3.52816438955922e-05,1000000
|
||||
W4,harm,nco2,rct_ideal_itt_h12,3e-06,1.73204820949881e-06,1000000
|
||||
W4,harm,nco2,rct_ideal_itt_h24,-1e-06,5.19615232638288e-06,1000000
|
||||
W4,harm,nco2,rct_ideal_itt,-0.000439000000000004,2.21992630281702e-05,1000000
|
||||
W4,benefit_harm,limb,rct_realistic_itt_h12,-0.00929276007926325,0.000263562909258414,138782
|
||||
W4,benefit_harm,limb,rct_realistic_itt_h24,-0.0152956796657341,0.000345323877379886,138782
|
||||
W4,benefit_harm,limb,rct_realistic_itt,0.0491758811390879,0.000753483858758792,138782
|
||||
W4,benefit_harm,male,rct_realistic_itt_h12,0.0137017389425249,0.00049518558763673,138782
|
||||
W4,benefit_harm,male,rct_realistic_itt_h24,0.0384161956441744,0.000716619811699705,138782
|
||||
W4,benefit_harm,male,rct_realistic_itt,0.178788067867868,0.0011506403550177,138782
|
||||
W4,benefit_harm,disease,rct_realistic_itt_h12,-0.00695082987264717,0.000226591955285682,138782
|
||||
W4,benefit_harm,disease,rct_realistic_itt_h24,-0.0115624044636492,0.000298942839282845,138782
|
||||
W4,benefit_harm,disease,rct_realistic_itt,0.0411974385194535,0.000680812020253607,138782
|
||||
W4,benefit_harm,rr,rct_realistic_itt_h12,0.0235282645331711,0.000424435802539338,138782
|
||||
W4,benefit_harm,rr,rct_realistic_itt_h24,0.0564045829754239,0.000647006977339569,138782
|
||||
W4,benefit_harm,rr,rct_realistic_itt,0.179864727156537,0.00108685035753337,138782
|
||||
W4,benefit_harm,nco1,rct_realistic_itt_h12,7.59973402766115e-05,2.05244409832568e-05,138782
|
||||
W4,benefit_harm,nco1,rct_realistic_itt_h24,0.000310392406001315,4.03922202687285e-05,138782
|
||||
W4,benefit_harm,nco1,rct_realistic_itt,0.000397348624478565,0.000105014903563497,138782
|
||||
W4,benefit_harm,nco2,rct_realistic_itt_h12,1.35656689471956e-05,9.59877335354946e-06,138782
|
||||
W4,benefit_harm,nco2,rct_realistic_itt_h24,8.38179845360632e-05,2.38086068010171e-05,138782
|
||||
W4,benefit_harm,nco2,rct_realistic_itt,0.000187238134616088,4.6073264402126e-05,138782
|
||||
W4,benefit_harm,limb,rct_ideal_itt_h12,-0.00954899999999944,9.81825676947545e-05,1000000
|
||||
W4,benefit_harm,limb,rct_ideal_itt_h24,-0.0151279999999953,0.000127574071096906,1000000
|
||||
W4,benefit_harm,limb,rct_ideal_itt,0.0485780000000279,0.000269766895514397,1000000
|
||||
W4,benefit_harm,male,rct_ideal_itt_h12,0.0136269999999964,0.000179263227883511,1000000
|
||||
W4,benefit_harm,male,rct_ideal_itt_h24,0.0370390000000164,0.000254921777177018,1000000
|
||||
W4,benefit_harm,male,rct_ideal_itt,0.165850000000145,0.000400004721849139,1000000
|
||||
W4,benefit_harm,disease,rct_ideal_itt_h12,-0.00708100000000062,8.47163469414974e-05,1000000
|
||||
W4,benefit_harm,disease,rct_ideal_itt_h24,-0.0113059999999981,0.00011062628242945,1000000
|
||||
W4,benefit_harm,disease,rct_ideal_itt,0.0405780000000199,0.000244469683023112,1000000
|
||||
W4,benefit_harm,rr,rct_ideal_itt_h12,0.0237180000000031,0.000152458048249918,1000000
|
||||
W4,benefit_harm,rr,rct_ideal_itt_h24,0.0552410000000346,0.00022899657621463,1000000
|
||||
W4,benefit_harm,rr,rct_ideal_itt,0.164206000000144,0.000372889245706426,1000000
|
||||
W4,benefit_harm,nco1,rct_ideal_itt_h12,7.79999999999999e-05,8.83141642094399e-06,1000000
|
||||
W4,benefit_harm,nco1,rct_ideal_itt_h24,0.000387000000000003,2.00212444919833e-05,1000000
|
||||
W4,benefit_harm,nco1,rct_ideal_itt,0.00067800000000001,4.5338066963389e-05,1000000
|
||||
W4,benefit_harm,nco2,rct_ideal_itt_h12,3.6e-05,5.99989199902816e-06,1000000
|
||||
W4,benefit_harm,nco2,rct_ideal_itt_h24,0.000191,1.39629337532708e-05,1000000
|
||||
W4,benefit_harm,nco2,rct_ideal_itt,0.000335000000000002,2.97470633006767e-05,1000000
|
||||
W5,null,limb,rct_realistic_itt_h12,0,0,138782
|
||||
W5,null,limb,rct_realistic_itt_h24,0,0,138782
|
||||
W5,null,limb,rct_realistic_itt,0,0,138782
|
||||
W5,null,male,rct_realistic_itt_h12,0.0316896791667746,0.000489980259881347,138782
|
||||
W5,null,male,rct_realistic_itt_h24,0.066135680620639,0.000696255343077188,138782
|
||||
W5,null,male,rct_realistic_itt,0.140661174578609,0.00098072361759117,138782
|
||||
W5,null,disease,rct_realistic_itt_h12,0,0,138782
|
||||
W5,null,disease,rct_realistic_itt_h24,0,0,138782
|
||||
W5,null,disease,rct_realistic_itt,0,0,138782
|
||||
W5,null,rr,rct_realistic_itt_h12,0.0329190178576472,0.000498508344475691,138782
|
||||
W5,null,rr,rct_realistic_itt_h24,0.0717591137320835,0.000722074502261802,138782
|
||||
W5,null,rr,rct_realistic_itt,0.176524278977813,0.00107249504657992,138782
|
||||
W5,null,nco1,rct_realistic_itt_h12,0,0,138782
|
||||
W5,null,nco1,rct_realistic_itt_h24,0,0,138782
|
||||
W5,null,nco1,rct_realistic_itt,0,0,138782
|
||||
W5,null,nco2,rct_realistic_itt_h12,0,0,138782
|
||||
W5,null,nco2,rct_realistic_itt_h24,0,0,138782
|
||||
W5,null,nco2,rct_realistic_itt,0,0,138782
|
||||
W5,null,limb,rct_ideal_itt_h12,0,0,1000000
|
||||
W5,null,limb,rct_ideal_itt_h24,0,0,1000000
|
||||
W5,null,limb,rct_ideal_itt,0,0,1000000
|
||||
W5,null,male,rct_ideal_itt_h12,0.0313360000000107,0.000174379055808226,1000000
|
||||
W5,null,male,rct_ideal_itt_h24,0.0641560000000435,0.000245132632800575,1000000
|
||||
W5,null,male,rct_ideal_itt,0.132398000000112,0.000338976060504823,1000000
|
||||
W5,null,disease,rct_ideal_itt_h12,0,0,1000000
|
||||
W5,null,disease,rct_ideal_itt_h24,0,0,1000000
|
||||
W5,null,disease,rct_ideal_itt,0,0,1000000
|
||||
W5,null,rr,rct_ideal_itt_h12,0.0327740000000121,0.000178207364955272,1000000
|
||||
W5,null,rr,rct_ideal_itt_h24,0.0702760000000496,0.000255728926453558,1000000
|
||||
W5,null,rr,rct_ideal_itt,0.166642000000146,0.000372733743888863,1000000
|
||||
W5,null,nco1,rct_ideal_itt_h12,0,0,1000000
|
||||
W5,null,nco1,rct_ideal_itt_h24,0,0,1000000
|
||||
W5,null,nco1,rct_ideal_itt,0,0,1000000
|
||||
W5,null,nco2,rct_ideal_itt_h12,0,0,1000000
|
||||
W5,null,nco2,rct_ideal_itt_h24,0,0,1000000
|
||||
W5,null,nco2,rct_ideal_itt,0,0,1000000
|
||||
W5,harm,limb,rct_realistic_itt_h12,-0.000634779123119918,8.59030281036993e-05,138782
|
||||
W5,harm,limb,rct_realistic_itt_h24,0.00357723457388988,0.000197648179331725,138782
|
||||
W5,harm,limb,rct_realistic_itt,0.0657009909762577,0.000715048507757777,138782
|
||||
W5,harm,male,rct_realistic_itt_h12,0.0305563168266289,0.000494042770555967,138782
|
||||
W5,harm,male,rct_realistic_itt_h24,0.0712904014827441,0.000728275405042644,138782
|
||||
W5,harm,male,rct_realistic_itt,0.209013847241733,0.00115467602996962,138782
|
||||
W5,harm,disease,rct_realistic_itt_h12,-0.000492563430128351,7.42094536253605e-05,138782
|
||||
W5,harm,disease,rct_realistic_itt_h24,0.00248801890157732,0.000166343673945743,138782
|
||||
W5,harm,disease,rct_realistic_itt,0.0539520228689354,0.000648343714778322,138782
|
||||
W5,harm,rr,rct_realistic_itt_h12,0.032397552203542,0.000494895030835064,138782
|
||||
W5,harm,rr,rct_realistic_itt_h24,0.0738721748970789,0.000732229204166303,138782
|
||||
W5,harm,rr,rct_realistic_itt,0.206069107976931,0.00113935408855565,138782
|
||||
W5,harm,nco1,rct_realistic_itt_h12,6.53400738957156e-06,6.53399013392965e-06,138782
|
||||
W5,harm,nco1,rct_realistic_itt_h24,-8.45156676626482e-06,1.58783112120704e-05,138782
|
||||
W5,harm,nco1,rct_realistic_itt,-0.00117248163179654,8.93642930544493e-05,138782
|
||||
W5,harm,nco2,rct_realistic_itt_h12,1.43513048739779e-05,1.01883274755683e-05,138782
|
||||
W5,harm,nco2,rct_realistic_itt_h24,1.97093822894091e-05,1.15113069317871e-05,138782
|
||||
W5,harm,nco2,rct_realistic_itt,-0.000146622386963238,3.5567497711738e-05,138782
|
||||
W5,harm,limb,rct_ideal_itt_h12,-0.00069000000000001,3.32193302160264e-05,1000000
|
||||
W5,harm,limb,rct_ideal_itt_h24,0.00369700000000017,7.40090007424914e-05,1000000
|
||||
W5,harm,limb,rct_ideal_itt,0.0625470000000419,0.000250125713973842,1000000
|
||||
W5,harm,male,rct_ideal_itt_h12,0.0302030000000096,0.000176195285949148,1000000
|
||||
W5,harm,male,rct_ideal_itt_h24,0.0694110000000488,0.000257408455725467,1000000
|
||||
W5,harm,male,rct_ideal_itt,0.196825000000176,0.000401807067352472,1000000
|
||||
W5,harm,disease,rct_ideal_itt_h12,-0.000527000000000006,2.88569276083573e-05,1000000
|
||||
W5,harm,disease,rct_ideal_itt_h24,0.00283300000000005,6.47531783851892e-05,1000000
|
||||
W5,harm,disease,rct_ideal_itt,0.0515180000000309,0.00022818390757615,1000000
|
||||
W5,harm,rr,rct_ideal_itt_h12,0.0322370000000116,0.000176793031060403,1000000
|
||||
W5,harm,rr,rct_ideal_itt_h24,0.0723530000000517,0.000259225854016017,1000000
|
||||
W5,harm,rr,rct_ideal_itt,0.193524000000173,0.000395447166412248,1000000
|
||||
W5,harm,nco1,rct_ideal_itt_h12,1e-05,3.1622618487401e-06,1000000
|
||||
W5,harm,nco1,rct_ideal_itt_h24,-5e-06,7.93725235838367e-06,1000000
|
||||
W5,harm,nco1,rct_ideal_itt,-0.001143,3.57448395016914e-05,1000000
|
||||
W5,harm,nco2,rct_ideal_itt_h12,1e-06,9.9999950002108e-07,1000000
|
||||
W5,harm,nco2,rct_ideal_itt_h24,-2e-06,5.47722520992005e-06,1000000
|
||||
W5,harm,nco2,rct_ideal_itt,-0.000447000000000004,2.24232065279446e-05,1000000
|
||||
W5,benefit_harm,limb,rct_realistic_itt_h12,-0.00936046865284354,0.000264129599237624,138782
|
||||
W5,benefit_harm,limb,rct_realistic_itt_h24,-0.0156253817310153,0.000346005778226494,138782
|
||||
W5,benefit_harm,limb,rct_realistic_itt,0.0476375959324454,0.000747755930004121,138782
|
||||
W5,benefit_harm,male,rct_realistic_itt_h12,0.0138456909190784,0.000496906834236644,138782
|
||||
W5,benefit_harm,male,rct_realistic_itt_h24,0.0389507656400573,0.000721058908850233,138782
|
||||
W5,benefit_harm,male,rct_realistic_itt,0.183585931781821,0.00116128147313179,138782
|
||||
W5,benefit_harm,disease,rct_realistic_itt_h12,-0.00701853844622746,0.000227256392216162,138782
|
||||
W5,benefit_harm,disease,rct_realistic_itt_h24,-0.0118343056375335,0.000299104065798181,138782
|
||||
W5,benefit_harm,disease,rct_realistic_itt,0.0400446368325273,0.000676156316495598,138782
|
||||
W5,benefit_harm,rr,rct_realistic_itt_h12,0.0237399250833049,0.000426256751172818,138782
|
||||
W5,benefit_harm,rr,rct_realistic_itt_h24,0.0573123691206079,0.000651693144083697,138782
|
||||
W5,benefit_harm,rr,rct_realistic_itt,0.188126682049733,0.00110432772565977,138782
|
||||
W5,benefit_harm,nco1,rct_realistic_itt_h12,7.59973402766115e-05,2.05244409832568e-05,138782
|
||||
W5,benefit_harm,nco1,rct_realistic_itt_h24,0.000304283539237408,3.99652071948251e-05,138782
|
||||
W5,benefit_harm,nco1,rct_realistic_itt,0.000464506673192971,0.000102228623751637,138782
|
||||
W5,benefit_harm,nco2,rct_realistic_itt_h12,2.13829664316019e-05,1.23792028465136e-05,138782
|
||||
W5,benefit_harm,nco2,rct_realistic_itt_h24,0.000101840077489285,2.47064296907095e-05,138782
|
||||
W5,benefit_harm,nco2,rct_realistic_itt,0.000204351440100052,4.71903593479565e-05,138782
|
||||
W5,benefit_harm,limb,rct_ideal_itt_h12,-0.00960499999999939,9.84110968081316e-05,1000000
|
||||
W5,benefit_harm,limb,rct_ideal_itt_h24,-0.0157609999999951,0.000127720753516602,1000000
|
||||
W5,benefit_harm,limb,rct_ideal_itt,0.0460120000000254,0.000266065585628749,1000000
|
||||
W5,benefit_harm,male,rct_ideal_itt_h12,0.0137869999999963,0.000179958102432602,1000000
|
||||
W5,benefit_harm,male,rct_ideal_itt_h24,0.037648000000017,0.000257220971335908,1000000
|
||||
W5,benefit_harm,male,rct_ideal_itt,0.173512000000153,0.000406166943334289,1000000
|
||||
W5,benefit_harm,disease,rct_ideal_itt_h12,-0.00713100000000062,8.4936145655476e-05,1000000
|
||||
W5,benefit_harm,disease,rct_ideal_itt_h24,-0.0117879999999978,0.000110837913441629,1000000
|
||||
W5,benefit_harm,disease,rct_ideal_itt,0.038659000000018,0.00024133064811169,1000000
|
||||
W5,benefit_harm,rr,rct_ideal_itt_h12,0.0239360000000033,0.00015314394504567,1000000
|
||||
W5,benefit_harm,rr,rct_ideal_itt_h24,0.0565390000000359,0.000231508836720838,1000000
|
||||
W5,benefit_harm,rr,rct_ideal_itt,0.177170000000157,0.000383445942867072,1000000
|
||||
W5,benefit_harm,nco1,rct_ideal_itt_h12,7.79999999999999e-05,8.8314164209386e-06,1000000
|
||||
W5,benefit_harm,nco1,rct_ideal_itt_h24,0.000407000000000003,2.02196525936012e-05,1000000
|
||||
W5,benefit_harm,nco1,rct_ideal_itt,0.000797000000000013,4.44788128322003e-05,1000000
|
||||
W5,benefit_harm,nco2,rct_ideal_itt_h12,3.5e-05,5.91597625080754e-06,1000000
|
||||
W5,benefit_harm,nco2,rct_ideal_itt_h24,0.000195,1.40343141977502e-05,1000000
|
||||
W5,benefit_harm,nco2,rct_ideal_itt,0.000396000000000003,2.91177468907165e-05,1000000
|
||||
WE,null,limb,rct_realistic_itt_h12,0,0,138782
|
||||
WE,null,limb,rct_realistic_itt_h24,0,0,138782
|
||||
WE,null,limb,rct_realistic_itt,0,0,138782
|
||||
WE,null,male,rct_realistic_itt_h12,0.0316896791667746,0.000489980259881347,138782
|
||||
WE,null,male,rct_realistic_itt_h24,0.066135680620639,0.000696255343077188,138782
|
||||
WE,null,male,rct_realistic_itt,0.140661174578609,0.00098072361759117,138782
|
||||
WE,null,disease,rct_realistic_itt_h12,0,0,138782
|
||||
WE,null,disease,rct_realistic_itt_h24,0,0,138782
|
||||
WE,null,disease,rct_realistic_itt,0,0,138782
|
||||
WE,null,rr,rct_realistic_itt_h12,0.0329190178576472,0.000498508344475691,138782
|
||||
WE,null,rr,rct_realistic_itt_h24,0.0717591137320835,0.000722074502261802,138782
|
||||
WE,null,rr,rct_realistic_itt,0.176524278977813,0.00107249504657992,138782
|
||||
WE,null,nco1,rct_realistic_itt_h12,0,0,138782
|
||||
WE,null,nco1,rct_realistic_itt_h24,0,0,138782
|
||||
WE,null,nco1,rct_realistic_itt,0,0,138782
|
||||
WE,null,nco2,rct_realistic_itt_h12,0,0,138782
|
||||
WE,null,nco2,rct_realistic_itt_h24,0,0,138782
|
||||
WE,null,nco2,rct_realistic_itt,0,0,138782
|
||||
WE,null,limb,rct_ideal_itt_h12,0,0,1000000
|
||||
WE,null,limb,rct_ideal_itt_h24,0,0,1000000
|
||||
WE,null,limb,rct_ideal_itt,0,0,1000000
|
||||
WE,null,male,rct_ideal_itt_h12,0.0313360000000107,0.000174379055808226,1000000
|
||||
WE,null,male,rct_ideal_itt_h24,0.0641560000000435,0.000245132632800575,1000000
|
||||
WE,null,male,rct_ideal_itt,0.132398000000112,0.000338976060504823,1000000
|
||||
WE,null,disease,rct_ideal_itt_h12,0,0,1000000
|
||||
WE,null,disease,rct_ideal_itt_h24,0,0,1000000
|
||||
WE,null,disease,rct_ideal_itt,0,0,1000000
|
||||
WE,null,rr,rct_ideal_itt_h12,0.0327740000000121,0.000178207364955272,1000000
|
||||
WE,null,rr,rct_ideal_itt_h24,0.0702760000000496,0.000255728926453558,1000000
|
||||
WE,null,rr,rct_ideal_itt,0.166642000000146,0.000372733743888863,1000000
|
||||
WE,null,nco1,rct_ideal_itt_h12,0,0,1000000
|
||||
WE,null,nco1,rct_ideal_itt_h24,0,0,1000000
|
||||
WE,null,nco1,rct_ideal_itt,0,0,1000000
|
||||
WE,null,nco2,rct_ideal_itt_h12,0,0,1000000
|
||||
WE,null,nco2,rct_ideal_itt_h24,0,0,1000000
|
||||
WE,null,nco2,rct_ideal_itt,0,0,1000000
|
||||
WE,harm,limb,rct_realistic_itt_h12,-0.000634779123119918,8.59030281036993e-05,138782
|
||||
WE,harm,limb,rct_realistic_itt_h24,0.00357723457388988,0.000197648179331725,138782
|
||||
WE,harm,limb,rct_realistic_itt,0.0657009909762577,0.000715048507757777,138782
|
||||
WE,harm,male,rct_realistic_itt_h12,0.0305563168266289,0.000494042770555967,138782
|
||||
WE,harm,male,rct_realistic_itt_h24,0.0712904014827441,0.000728275405042644,138782
|
||||
WE,harm,male,rct_realistic_itt,0.209013847241733,0.00115467602996962,138782
|
||||
WE,harm,disease,rct_realistic_itt_h12,-0.000492563430128351,7.42094536253605e-05,138782
|
||||
WE,harm,disease,rct_realistic_itt_h24,0.00248801890157732,0.000166343673945743,138782
|
||||
WE,harm,disease,rct_realistic_itt,0.0539520228689354,0.000648343714778322,138782
|
||||
WE,harm,rr,rct_realistic_itt_h12,0.032397552203542,0.000494895030835064,138782
|
||||
WE,harm,rr,rct_realistic_itt_h24,0.0738721748970789,0.000732229204166303,138782
|
||||
WE,harm,rr,rct_realistic_itt,0.206069107976931,0.00113935408855565,138782
|
||||
WE,harm,nco1,rct_realistic_itt_h12,6.53400738957156e-06,6.53399013392965e-06,138782
|
||||
WE,harm,nco1,rct_realistic_itt_h24,-8.45156676626482e-06,1.58783112120704e-05,138782
|
||||
WE,harm,nco1,rct_realistic_itt,-0.00117248163179654,8.93642930544493e-05,138782
|
||||
WE,harm,nco2,rct_realistic_itt_h12,1.43513048739779e-05,1.01883274755683e-05,138782
|
||||
WE,harm,nco2,rct_realistic_itt_h24,1.97093822894091e-05,1.15113069317871e-05,138782
|
||||
WE,harm,nco2,rct_realistic_itt,-0.000146622386963238,3.5567497711738e-05,138782
|
||||
WE,harm,limb,rct_ideal_itt_h12,-0.00069000000000001,3.32193302160264e-05,1000000
|
||||
WE,harm,limb,rct_ideal_itt_h24,0.00369700000000017,7.40090007424914e-05,1000000
|
||||
WE,harm,limb,rct_ideal_itt,0.0625470000000419,0.000250125713973842,1000000
|
||||
WE,harm,male,rct_ideal_itt_h12,0.0302030000000096,0.000176195285949148,1000000
|
||||
WE,harm,male,rct_ideal_itt_h24,0.0694110000000488,0.000257408455725467,1000000
|
||||
WE,harm,male,rct_ideal_itt,0.196825000000176,0.000401807067352472,1000000
|
||||
WE,harm,disease,rct_ideal_itt_h12,-0.000527000000000006,2.88569276083573e-05,1000000
|
||||
WE,harm,disease,rct_ideal_itt_h24,0.00283300000000005,6.47531783851892e-05,1000000
|
||||
WE,harm,disease,rct_ideal_itt,0.0515180000000309,0.00022818390757615,1000000
|
||||
WE,harm,rr,rct_ideal_itt_h12,0.0322370000000116,0.000176793031060403,1000000
|
||||
WE,harm,rr,rct_ideal_itt_h24,0.0723530000000517,0.000259225854016017,1000000
|
||||
WE,harm,rr,rct_ideal_itt,0.193524000000173,0.000395447166412248,1000000
|
||||
WE,harm,nco1,rct_ideal_itt_h12,1e-05,3.1622618487401e-06,1000000
|
||||
WE,harm,nco1,rct_ideal_itt_h24,-5e-06,7.93725235838367e-06,1000000
|
||||
WE,harm,nco1,rct_ideal_itt,-0.001143,3.57448395016914e-05,1000000
|
||||
WE,harm,nco2,rct_ideal_itt_h12,1e-06,9.9999950002108e-07,1000000
|
||||
WE,harm,nco2,rct_ideal_itt_h24,-2e-06,5.47722520992005e-06,1000000
|
||||
WE,harm,nco2,rct_ideal_itt,-0.000447000000000004,2.24232065279446e-05,1000000
|
||||
WE,benefit_harm,limb,rct_realistic_itt_h12,-0.00936046865284354,0.000264129599237624,138782
|
||||
WE,benefit_harm,limb,rct_realistic_itt_h24,-0.0156253817310153,0.000346005778226494,138782
|
||||
WE,benefit_harm,limb,rct_realistic_itt,0.0476375959324454,0.000747755930004121,138782
|
||||
WE,benefit_harm,male,rct_realistic_itt_h12,0.0138456909190784,0.000496906834236644,138782
|
||||
WE,benefit_harm,male,rct_realistic_itt_h24,0.0389507656400573,0.000721058908850233,138782
|
||||
WE,benefit_harm,male,rct_realistic_itt,0.183585931781821,0.00116128147313179,138782
|
||||
WE,benefit_harm,disease,rct_realistic_itt_h12,-0.00701853844622746,0.000227256392216162,138782
|
||||
WE,benefit_harm,disease,rct_realistic_itt_h24,-0.0118343056375335,0.000299104065798181,138782
|
||||
WE,benefit_harm,disease,rct_realistic_itt,0.0400446368325273,0.000676156316495598,138782
|
||||
WE,benefit_harm,rr,rct_realistic_itt_h12,0.0237399250833049,0.000426256751172818,138782
|
||||
WE,benefit_harm,rr,rct_realistic_itt_h24,0.0573123691206079,0.000651693144083697,138782
|
||||
WE,benefit_harm,rr,rct_realistic_itt,0.188126682049733,0.00110432772565977,138782
|
||||
WE,benefit_harm,nco1,rct_realistic_itt_h12,7.59973402766115e-05,2.05244409832568e-05,138782
|
||||
WE,benefit_harm,nco1,rct_realistic_itt_h24,0.000304283539237408,3.99652071948251e-05,138782
|
||||
WE,benefit_harm,nco1,rct_realistic_itt,0.000464506673192971,0.000102228623751637,138782
|
||||
WE,benefit_harm,nco2,rct_realistic_itt_h12,2.13829664316019e-05,1.23792028465136e-05,138782
|
||||
WE,benefit_harm,nco2,rct_realistic_itt_h24,0.000101840077489285,2.47064296907095e-05,138782
|
||||
WE,benefit_harm,nco2,rct_realistic_itt,0.000204351440100052,4.71903593479565e-05,138782
|
||||
WE,benefit_harm,limb,rct_ideal_itt_h12,-0.00960499999999939,9.84110968081316e-05,1000000
|
||||
WE,benefit_harm,limb,rct_ideal_itt_h24,-0.0157609999999951,0.000127720753516602,1000000
|
||||
WE,benefit_harm,limb,rct_ideal_itt,0.0460120000000254,0.000266065585628749,1000000
|
||||
WE,benefit_harm,male,rct_ideal_itt_h12,0.0137869999999963,0.000179958102432602,1000000
|
||||
WE,benefit_harm,male,rct_ideal_itt_h24,0.037648000000017,0.000257220971335908,1000000
|
||||
WE,benefit_harm,male,rct_ideal_itt,0.173512000000153,0.000406166943334289,1000000
|
||||
WE,benefit_harm,disease,rct_ideal_itt_h12,-0.00713100000000062,8.4936145655476e-05,1000000
|
||||
WE,benefit_harm,disease,rct_ideal_itt_h24,-0.0117879999999978,0.000110837913441629,1000000
|
||||
WE,benefit_harm,disease,rct_ideal_itt,0.038659000000018,0.00024133064811169,1000000
|
||||
WE,benefit_harm,rr,rct_ideal_itt_h12,0.0239360000000033,0.00015314394504567,1000000
|
||||
WE,benefit_harm,rr,rct_ideal_itt_h24,0.0565390000000359,0.000231508836720838,1000000
|
||||
WE,benefit_harm,rr,rct_ideal_itt,0.177170000000157,0.000383445942867072,1000000
|
||||
WE,benefit_harm,nco1,rct_ideal_itt_h12,7.79999999999999e-05,8.8314164209386e-06,1000000
|
||||
WE,benefit_harm,nco1,rct_ideal_itt_h24,0.000407000000000003,2.02196525936012e-05,1000000
|
||||
WE,benefit_harm,nco1,rct_ideal_itt,0.000797000000000013,4.44788128322003e-05,1000000
|
||||
WE,benefit_harm,nco2,rct_ideal_itt_h12,3.5e-05,5.91597625080754e-06,1000000
|
||||
WE,benefit_harm,nco2,rct_ideal_itt_h24,0.000195,1.40343141977502e-05,1000000
|
||||
WE,benefit_harm,nco2,rct_ideal_itt,0.000396000000000003,2.91177468907165e-05,1000000
|
||||
WR,null,limb,rct_realistic_itt_h12,0,0,138782
|
||||
WR,null,limb,rct_realistic_itt_h24,0,0,138782
|
||||
WR,null,limb,rct_realistic_itt,0,0,138782
|
||||
WR,null,male,rct_realistic_itt_h12,0.00326265613441116,0.000161011731454756,138782
|
||||
WR,null,male,rct_realistic_itt_h24,0.00708260461688506,0.000234991461002943,138782
|
||||
WR,null,male,rct_realistic_itt,0.0159597525547126,0.000354917769662178,138782
|
||||
WR,null,disease,rct_realistic_itt_h12,0,0,138782
|
||||
WR,null,disease,rct_realistic_itt_h24,0,0,138782
|
||||
WR,null,disease,rct_realistic_itt,0,0,138782
|
||||
WR,null,rr,rct_realistic_itt_h12,0.00390068897873103,0.000174372283650843,138782
|
||||
WR,null,rr,rct_realistic_itt_h24,0.00963484905166457,0.000271376362156474,138782
|
||||
WR,null,rr,rct_realistic_itt,0.030981137329801,0.00048593194676983,138782
|
||||
WR,null,nco1,rct_realistic_itt_h12,0,0,138782
|
||||
WR,null,nco1,rct_realistic_itt_h24,0,0,138782
|
||||
WR,null,nco1,rct_realistic_itt,0,0,138782
|
||||
WR,null,nco2,rct_realistic_itt_h12,0,0,138782
|
||||
WR,null,nco2,rct_realistic_itt_h24,0,0,138782
|
||||
WR,null,nco2,rct_realistic_itt,0,0,138782
|
||||
WR,null,limb,rct_ideal_itt_h12,0,0,1000000
|
||||
WR,null,limb,rct_ideal_itt_h24,0,0,1000000
|
||||
WR,null,limb,rct_ideal_itt,0,0,1000000
|
||||
WR,null,male,rct_ideal_itt_h12,0.0144499999999958,0.000119646134497257,1000000
|
||||
WR,null,male,rct_ideal_itt_h24,0.0295130000000089,0.000169416595498972,1000000
|
||||
WR,null,male,rct_ideal_itt,0.0604990000000398,0.000238509687434567,1000000
|
||||
WR,null,disease,rct_ideal_itt_h12,0,0,1000000
|
||||
WR,null,disease,rct_ideal_itt_h24,0,0,1000000
|
||||
WR,null,disease,rct_ideal_itt,0,0,1000000
|
||||
WR,null,rr,rct_ideal_itt_h12,0.0154799999999951,0.000123775480609506,1000000
|
||||
WR,null,rr,rct_ideal_itt_h24,0.033686000000013,0.000180630156408789,1000000
|
||||
WR,null,rr,rct_ideal_itt,0.0827870000000621,0.000275694237569834,1000000
|
||||
WR,null,nco1,rct_ideal_itt_h12,0,0,1000000
|
||||
WR,null,nco1,rct_ideal_itt_h24,0,0,1000000
|
||||
WR,null,nco1,rct_ideal_itt,0,0,1000000
|
||||
WR,null,nco2,rct_ideal_itt_h12,0,0,1000000
|
||||
WR,null,nco2,rct_ideal_itt_h24,0,0,1000000
|
||||
WR,null,nco2,rct_ideal_itt,0,0,1000000
|
||||
WR,harm,limb,rct_realistic_itt_h12,-0.00132523127022743,9.87182615298501e-05,138782
|
||||
WR,harm,limb,rct_realistic_itt_h24,-0.000391612509628908,0.000137328320143579,138782
|
||||
WR,harm,limb,rct_realistic_itt,0.0268614851094127,0.000479680170586925,138782
|
||||
WR,harm,male,rct_realistic_itt_h12,0.00183086191430631,0.000186524882607493,138782
|
||||
WR,harm,male,rct_realistic_itt_h24,0.00660876178106249,0.000270497788471577,138782
|
||||
WR,harm,male,rct_realistic_itt,0.0430522807826039,0.000592228316505743,138782
|
||||
WR,harm,disease,rct_realistic_itt_h12,-0.000978475276970814,8.43708423896255e-05,138782
|
||||
WR,harm,disease,rct_realistic_itt_h24,-0.000368508831361038,0.000117786559586914,138782
|
||||
WR,harm,disease,rct_realistic_itt,0.0214007067446433,0.00042827954516037,138782
|
||||
WR,harm,rr,rct_realistic_itt_h12,0.0037420432221117,0.000170623602771209,138782
|
||||
WR,harm,rr,rct_realistic_itt_h24,0.00945306147737351,0.000269083492149388,138782
|
||||
WR,harm,rr,rct_realistic_itt,0.0346859729178713,0.000513920148878995,138782
|
||||
WR,harm,nco1,rct_realistic_itt_h12,1.05565276719135e-05,7.67290992256411e-06,138782
|
||||
WR,harm,nco1,rct_realistic_itt_h24,3.64674353570518e-05,1.720227729913e-05,138782
|
||||
WR,harm,nco1,rct_realistic_itt,-0.000362712154403391,5.78770255563114e-05,138782
|
||||
WR,harm,nco2,rct_realistic_itt_h12,1.43513048739779e-05,1.01883274755683e-05,138782
|
||||
WR,harm,nco2,rct_realistic_itt_h24,2.39525242738712e-05,1.2268410518513e-05,138782
|
||||
WR,harm,nco2,rct_realistic_itt,-3.21786274097218e-05,2.67252013050681e-05,138782
|
||||
WR,harm,limb,rct_ideal_itt_h12,-0.000337000000000002,3.86637612112804e-05,1000000
|
||||
WR,harm,limb,rct_ideal_itt_h24,0.00560400000000042,8.67559518652397e-05,1000000
|
||||
WR,harm,limb,rct_ideal_itt,0.0822680000000616,0.000282962852999299,1000000
|
||||
WR,harm,male,rct_ideal_itt_h12,0.014272999999996,0.000126353794842551,1000000
|
||||
WR,harm,male,rct_ideal_itt_h24,0.0369670000000163,0.000193861911966953,1000000
|
||||
WR,harm,male,rct_ideal_itt,0.138377000000118,0.000351378436835139,1000000
|
||||
WR,harm,disease,rct_ideal_itt_h12,-0.000265,3.35101443597238e-05,1000000
|
||||
WR,harm,disease,rct_ideal_itt_h24,0.00431000000000025,7.6114544602554e-05,1000000
|
||||
WR,harm,disease,rct_ideal_itt,0.0701190000000495,0.000262564898334477,1000000
|
||||
WR,harm,rr,rct_ideal_itt_h12,0.015637999999995,0.000124424486963914,1000000
|
||||
WR,harm,rr,rct_ideal_itt_h24,0.0364470000000158,0.000187703532708246,1000000
|
||||
WR,harm,rr,rct_ideal_itt,0.107241000000087,0.000310245657376612,1000000
|
||||
WR,harm,nco1,rct_ideal_itt_h12,9e-06,3.31661257912739e-06,1000000
|
||||
WR,harm,nco1,rct_ideal_itt_h24,-2.2e-05,8.9442448535487e-06,1000000
|
||||
WR,harm,nco1,rct_ideal_itt,-0.00146099999999998,4.0110665401783e-05,1000000
|
||||
WR,harm,nco2,rct_ideal_itt_h12,1e-06,9.9999950002108e-07,1000000
|
||||
WR,harm,nco2,rct_ideal_itt_h24,-1e-05,5.47721644638051e-06,1000000
|
||||
WR,harm,nco2,rct_ideal_itt,-0.000626000000000009,2.60309070911949e-05,1000000
|
||||
WR,benefit_harm,limb,rct_realistic_itt_h12,-0.0100692054330358,0.000273838728306849,138782
|
||||
WR,benefit_harm,limb,rct_realistic_itt_h24,-0.0202388549769331,0.000389934711939864,138782
|
||||
WR,benefit_harm,limb,rct_realistic_itt,-0.0182894827099266,0.0005456625670121,138782
|
||||
WR,benefit_harm,male,rct_realistic_itt_h12,-0.00774914797382966,0.000305314608664021,138782
|
||||
WR,benefit_harm,male,rct_realistic_itt_h24,-0.0149271553871344,0.000439529129829332,138782
|
||||
WR,benefit_harm,male,rct_realistic_itt,-0.00351118384900574,0.000640293782628295,138782
|
||||
WR,benefit_harm,disease,rct_realistic_itt_h12,-0.00750716023907874,0.000234635092877723,138782
|
||||
WR,benefit_harm,disease,rct_realistic_itt_h24,-0.015204708121949,0.000336574207932989,138782
|
||||
WR,benefit_harm,disease,rct_realistic_itt,-0.0135974833244941,0.000479920260541957,138782
|
||||
WR,benefit_harm,rr,rct_realistic_itt_h12,0.00257621420842002,0.000142259721726994,138782
|
||||
WR,benefit_harm,rr,rct_realistic_itt_h24,0.00629731243916255,0.000223283735341535,138782
|
||||
WR,benefit_harm,rr,rct_realistic_itt,0.0245015631438965,0.000452100073874658,138782
|
||||
WR,benefit_harm,nco1,rct_realistic_itt_h12,8.00198605589534e-05,2.09148811829732e-05,138782
|
||||
WR,benefit_harm,nco1,rct_realistic_itt_h24,0.000349351029479705,4.25483580158384e-05,138782
|
||||
WR,benefit_harm,nco1,rct_realistic_itt,0.00169299260303213,0.000103132274584187,138782
|
||||
WR,benefit_harm,nco2,rct_realistic_itt_h12,2.13829664316019e-05,1.23792028465136e-05,138782
|
||||
WR,benefit_harm,nco2,rct_realistic_itt_h24,0.000107801730973639,2.54154238646691e-05,138782
|
||||
WR,benefit_harm,nco2,rct_realistic_itt,0.000341525175221389,4.76676185289784e-05,138782
|
||||
WR,benefit_harm,limb,rct_ideal_itt_h12,-0.0090509999999998,9.60993204924743e-05,1000000
|
||||
WR,benefit_harm,limb,rct_ideal_itt_h24,-0.0119589999999977,0.000134439511749023,1000000
|
||||
WR,benefit_harm,limb,rct_ideal_itt,0.0913770000000707,0.00033987533578012,1000000
|
||||
WR,benefit_harm,male,rct_ideal_itt_h12,0.00228599999999998,0.000143118042902288,1000000
|
||||
WR,benefit_harm,male,rct_ideal_itt_h24,0.0147399999999956,0.000209691994123175,1000000
|
||||
WR,benefit_harm,male,rct_ideal_itt,0.137433000000117,0.000386244961794193,1000000
|
||||
WR,benefit_harm,disease,rct_ideal_itt_h12,-0.00666600000000056,8.26653763315159e-05,1000000
|
||||
WR,benefit_harm,disease,rct_ideal_itt_h24,-0.00873600000000003,0.000116771924297951,1000000
|
||||
WR,benefit_harm,disease,rct_ideal_itt,0.0842650000000636,0.00032011312028207,1000000
|
||||
WR,benefit_harm,rr,rct_ideal_itt_h12,0.0118289999999978,0.000108632751778034,1000000
|
||||
WR,benefit_harm,rr,rct_ideal_itt_h24,0.0298510000000092,0.000170990987478453,1000000
|
||||
WR,benefit_harm,rr,rct_ideal_itt,0.106172000000086,0.000310949363104946,1000000
|
||||
WR,benefit_harm,nco1,rct_ideal_itt_h12,7.69999999999999e-05,8.77462654469781e-06,1000000
|
||||
WR,benefit_harm,nco1,rct_ideal_itt_h24,0.000369000000000003,1.98711811175825e-05,1000000
|
||||
WR,benefit_harm,nco1,rct_ideal_itt,7.89999999999999e-05,5.33572278051064e-05,1000000
|
||||
WR,benefit_harm,nco2,rct_ideal_itt_h12,3.3e-05,5.74446786046983e-06,1000000
|
||||
WR,benefit_harm,nco2,rct_ideal_itt_h24,0.000179,1.38191157096822e-05,1000000
|
||||
WR,benefit_harm,nco2,rct_ideal_itt,-6.99999999999999e-05,3.54400211626608e-05,1000000
|
||||
|
@@ -0,0 +1,34 @@
|
||||
# Validation (plan section 15): in a clean world, each design averaged over replicates
|
||||
# should match its own truth within Monte Carlo error.
|
||||
# Usage: Rscript simulations/validate_designs.R [world] [scenario] [reps] [cores] [n]
|
||||
|
||||
pkgload::load_all("pvisim", quiet = TRUE)
|
||||
suppressPackageStartupMessages(library(data.table))
|
||||
a <- commandArgs(TRUE)
|
||||
world <- if (length(a) >= 1) a[1] else "W0"
|
||||
scenario <- if (length(a) >= 2) a[2] else "harm"
|
||||
reps <- if (length(a) >= 3) as.integer(a[3]) else 40L
|
||||
cores <- if (length(a) >= 4) as.integer(a[4]) else 10L
|
||||
n <- if (length(a) >= 5) as.numeric(a[5]) else 5e4
|
||||
|
||||
par <- load_params(world, scenario)
|
||||
res <- rbindlist(parallel::mclapply(seq_len(reps), function(r) {
|
||||
rec <- simulate_records(par, rep = 1000L + r, n = n)
|
||||
rbindlist(lapply(c("naive", "landmark", "ccw", "seq"), function(d)
|
||||
cbind(rep = r, run_designs(rec, d, "V1_ignore", outcomes = c(male = "t_male", disease = "t_disease", nco1 = "t_nco1")))),
|
||||
fill = TRUE)
|
||||
}, mc.cores = cores), fill = TRUE)
|
||||
|
||||
w_ <- world; s_ <- scenario
|
||||
tr <- fread("simulations/truths/truths.csv")[world == w_ & scenario == s_]
|
||||
map <- data.table(design = c("ccw", "ccw", "seq"), contrast = c("A_B1", "A_B2", "now_not"),
|
||||
estimand = c("rd_A_B1", "rd_A_B2", "seq_itt"))
|
||||
s <- res[, .(mean_est = mean(est), emp_se = sd(est), mean_se = mean(se), reps = .N),
|
||||
by = .(design, contrast, outcome)]
|
||||
s <- merge(s, map, by = c("design", "contrast"), all.x = TRUE)
|
||||
s <- merge(s, tr[, .(outcome, estimand, truth = value)], by = c("outcome", "estimand"), all.x = TRUE)
|
||||
s[, `:=`(bias = mean_est - truth, mcse_bias = emp_se / sqrt(reps))]
|
||||
s[, z := bias / mcse_bias]
|
||||
print(s[order(design, outcome), .(design, contrast, outcome, truth = round(truth, 4), mean_est = round(mean_est, 4),
|
||||
bias = round(bias, 4), z = round(z, 1), emp_se = round(emp_se, 4), mean_se = round(mean_se, 4))])
|
||||
fwrite(res, sprintf("results/validate_%s_%s.csv", world, scenario))
|
||||
Reference in New Issue
Block a user