tte-surgery/docs/literature/01-surgical-rct-limitations.md

# Literature review 1: limitations of RCTs in surgery, mapped to TTE on claims data

All citations were checked against PubMed through NCBI E-utilities, 2026-10-27. The numbers come from PubMed abstracts; full texts were not read. Papers with no PubMed abstract are described only in general terms.

**Corrections and caveats**
- Chapman 2014 was published in *BMJ*, not *Surgery*.
- The page numbers for McCulloch 2002 are from memory, not from PubMed.
- BASIL-3 recruited 481 of the planned 861 patients. That the paclitaxel signal caused the shortfall is a guess, not a verified finding.

---

## 1. Classic critiques and the IDEAL framework

| Paper | PMID | Takeaway |
|---|---|---|
| McCulloch P et al. 2002, *BMJ* "Randomised trials in surgery: problems and possible solutions" | 12065273 | The founding critique. Barriers: equipoise, learning curve, quality control, blinding, surgeon preference. |
| McLeod RS 1996, *Surgery* 119:483 | 8619200 | Problems specific to RCTs in surgery. |
| Solomon MJ, McLeod RS 1995, *Surgery* | 7652679 | Many surgical questions could be randomized but are not. |
| Solomon MJ, McLeod RS 1998, *Med J Aust* | 9803252 | Overview of RCTs in surgery: past, present and future. |
| Barkun JS et al. 2009, *Lancet* 374:1089 | 19782874 | IDEAL series, paper 1: the stages of surgical innovation. |
| Ergina PL et al. 2009, *Lancet* 374:1097 | 19782875 | Few of the challenges are unique to surgery, but in surgery they occur together: complexity, surgeon factors, and the range of outcomes. |
| McCulloch P et al. 2009, *Lancet* 374:1105 | 19782876 | IDEAL recommendations: registries and prospective databases, with RCTs "whenever possible". |
| McCulloch 2013 / Ergina 2013 / Cook 2013, *BMJ* (IDEAL series) | 23778427, 23778426, 23778425 | Observational studies in the Exploration and Assessment stages; RCTs and long-term registry evaluation. |
| Hirst A et al. 2019, *Ann Surg* | 29697448 | IDEAL update: adds a Pre-IDEAL stage and covers devices. |
| Cook JA 2009, *Trials* 10:9 | 19200379 | Checklist for the design, conduct and analysis of surgical RCTs. |
| Pronk AJM et al. 2023, *Br J Surg* 110:1300 | 37379487 | Of 1,188 surgical RCTs, only **22.1%** published in 2019 were at low risk of bias (14.7% in 2009). |
| Ahmed Ali U et al. 2018, *J Surg Res* 228:1 | 29907196 | Of 228 negative surgical RCTs, the median sample size was about 76–80, so most were underpowered. |

## 2. Recruitment failure, discontinuation, and equipoise

| Paper | PMID | Takeaway |
|---|---|---|
| **Chapman SJ et al. 2014, *BMJ* 349:g6870** | 25491195 | Of 395 surgical RCTs, **21% stopped early**. The main reason was poor recruitment (44% of stopped trials). Of completed trials, 34% were unpublished at a median of 4.9 years. |
| **Rosenthal R et al. 2015, *Ann Surg* 262:68** | 24979608 | Discontinued: **43%** of surgical vs **27%** of medical RCTs. Stopped for slow recruitment: 18% vs 11%. Discontinuation predicted non-publication (OR 4.18). |
| Kasenda B et al. 2014, *JAMA* 311:1045 | 24618966 | Across all fields, **24.9%** of RCTs were discontinued, most often for poor recruitment. |
| Cook JA et al. 2008, *Contemp Clin Trials* 29:631 | 18397843 | **Trials of surgery vs medical treatment recruited about 2× slower** than trials of surgery vs surgery (median 68 vs 35 days per recruit). This applies directly to PVI vs exercise. |
| Donovan J et al. 2002, *BMJ* 325:766 (ProtecT) | 12364308 | Qualitative work on how recruiters presented the options made an "impossible" trial feasible. |
| Donovan JL et al. 2014, *J Clin Epidemiol* 67:912 | 24811157 | Clinicians' "hunches" undermined equipoise, and **surgeons felt this most intensely**. |
| Rooshenas L et al. 2016, *PLoS Med* 13:e1002147 | 27755555 | In recorded consultations, the equipoise clinicians stated differed from how they actually presented the options. |
| Donovan JL et al. 2022, *Trials* 23:258 (QRI-Two) | 35379301 | A rapid recruitment intervention for trials that are failing to recruit. |
| Hamdy FC et al. 2023, *NEJM* 388:1547 (ProtecT at 15 years) | 36912538 | A monitor-vs-intervene trial can be completed. |

## 3. Surgeon skill, learning curve, expertise-based trials, and clustering

| Paper | PMID | Takeaway |
|---|---|---|
| **Devereaux PJ et al. 2005, *BMJ* 330:88** | 15637373 | Proposes expertise-based RCTs: each surgeon performs only the procedure they are expert in, to avoid differential-expertise bias. |
| Cook JA et al. 2015, *Trials* 16:241 | 26025450 | Review of 43 expertise-based RCTs: 92% compliance, but expertise criteria were poorly reported. |
| Cook JA et al. 2004, *Clin Trials* 1:421 | 16279280 | Statistical methods for learning curves in surgical trials. |
| Ramsay CR et al. 2000 / 2001 | 11155830, 11319991 | Assessing learning curves in health technology. |
| **Lee KJ, Thompson SG 2005, *BMJ* 330:142**; *Clin Trials* 2:163 | 15649931, 16279138 | Outcomes cluster by clinician even under individual randomization. Ignoring it makes the estimates look more precise than they are. |
| Roberts C, Roberts SA 2005, *Clin Trials* 2:152 | 16279137 | Clustering induced by treatment, which can differ between arms (partially nested designs). |
| Walters SJ 2010, *J Clin Nurs* 19:1102 | 20492055 | Therapist effects: cluster-level, random-effects, and marginal models compared. |
| **Birkmeyer JD et al. 2013, *NEJM* 369:1434** | 24106936 | Peer ratings of surgical skill from video: the bottom quartile vs the top had **14.5% vs 5.2%** complications and **0.26% vs 0.05%** mortality. |
| Birkmeyer JD et al. 2003, *NEJM* 349:2117 | 14645640 | Surgeon volume and mortality. |

## 4. Blinding and sham surgery

| Paper | PMID | Takeaway |
|---|---|---|
| **Wartolowska K et al. 2014, *BMJ* 348:g3253** | 24850821 | Across 53 placebo-controlled surgical RCTs, the placebo arm improved in 74%. Surgery was **no better than placebo in 51%**. |
| Wartolowska K et al. 2016, *BMJ Open* | 27008687 | Placebo-controlled surgical trials are feasible but recruit slowly. |
| Boutron I et al. 2017, *Ann Intern Med* 167:40 | 28630973 | CONSORT extension for non-pharmacologic treatments: expertise, blinding, and standardization. |

## 5. Standardization, crossover, and contamination

| Paper | PMID | Takeaway |
|---|---|---|
| **Garas G et al. 2018, *Ann Surg Oncol* 25:221** | 29110271 | Across 40 RCTs of minimally invasive vs open surgery, crossover averaged **12.6% (0–45%)** and worsened the ITT results. |
| **Weinstein JN et al. 2006, *JAMA* 296:2441 (SPORT)** | 17119140 | **Only 50%** of the surgery arm had surgery, and **30%** of the nonoperative arm did, so the ITT comparison could not be interpreted. |
| Hernán MA, Robins JM 2017, *NEJM* 377:1391 | 28976864 | Per-protocol effects in pragmatic trials. |

## 6. Generalizability and external validity

| Paper | PMID | Takeaway |
|---|---|---|
| **Rothwell PM 2005, *Lancet* 365:82** | 15639683 | "To whom do the results of this trial apply?" |
| Rothwell PM 2006, *PLoS Clin Trials* | 16871331 | Factors that affect external validity. |
| Cutteridge J et al. 2025, *BJS Open* | 40105904 | Patients in 307 vascular RCTs were **younger** than those in the UK National Vascular Registry. |
| Jones HG et al. 2026, *EJVES* | 42526742 | Endovascular lower-limb revascularization is under-researched relative to how often it is done. Its trials are 72% efficacy trials and 39% industry-funded. |
| Loudon K et al. 2015, *BMJ* (PRECIS-2) | 25956159 | Tool for rating how pragmatic or explanatory a trial is. |

## 7. Trials of timing, "when to operate" (the key analogue)

| Paper | PMID | Takeaway |
|---|---|---|
| **Fitzgibbons RJ et al. 2006, *JAMA* 295:285** | 16418463 | Hernia watchful waiting: 23% of the waiting arm crossed over to surgery, and 17% of the surgery arm crossed the other way. |
| **Fitzgibbons RJ et al. 2013, *Ann Surg* 258:508** | 24022443 | Long-term crossover was **68%** (79% in men over 65). "Watchful waiting" works out as "delayed surgery". |
| O'Dwyer PJ et al. 2006, *Ann Surg* | 16858177 | UK hernia trial: crossover was higher than predicted. |
| **UK Small Aneurysm Trial 1998, *Lancet*** | 9853436 | HR 0.94. Operative mortality of 5.8% caused an early deficit in survival. |
| **Lederle FA et al. 2002, *NEJM* (ADAM)** | 12000813 | **61.6% of the surveillance arm were eventually repaired.** Surveillance is a dynamic strategy, not "no treatment". |
| Cao P et al. 2011, *EJVES* (CAESAR) | 20869890 | Delayed repair under surveillance: 59.7% by 36 months and 84.5% by 54 months. One in six aneurysms became unsuitable for EVAR, a harm of delay that the ITT effect on mortality does not capture. |
| Ouriel K et al. 2010, *JVS* (PIVOTAL) | 20304589 | 31% of the surveillance arm were repaired by 20 months. HR 1.01. |
| Ulug P et al. 2020, *Cochrane* | 32609382 | Early repair of small AAAs has no advantage. |

## 8. Pragmatic trials, registry trials, and TTE

| Paper | PMID | Takeaway |
|---|---|---|
| **Hernán MA, Robins JM 2016, *AJE* 183:758** | 26994063 | The foundational TTE paper. |
| **Hernán MA et al. 2016, *J Clin Epidemiol* 79:70** | 27237061 | Specifying the target trial avoids self-inflicted biases such as immortal time. |
| Hernán MA et al. 2022, *JAMA* 328:2446 | 36508210 | JAMA Guide to TTE. |
| Cain LE et al. 2010, *Int J Biostat* | 21972433 | "When to start treatment?": dynamic regimes. |
| **Maringe C et al. 2020, *Int J Epidemiol* 49:1719** | 32386426 | CCW tutorial on surgery in older patients with lung cancer. The naive 22% benefit fell to 11%. |
| Wang SV et al. 2023, *JAMA* 329:1376 (RCT-DUPLICATE) | 37097356 | Of 32 RCTs emulated in claims data, agreement was r = 0.93 when the emulation was close and r = 0.53 when it was not. All were drug trials. |
| Cashin AG et al. 2025, *JAMA* / *BMJ* (TARGET) | 40899949 / 40903028 | Reporting guideline for TTE. |
| Scola G et al. 2023, *BMC Med Res Methodol* | 37587484 | Only 16% of TTEs used cloning to handle immortal time. |
| Katsoulis M et al. 2024, *Epidemiology* 35:730 | 39024012 | TTE of bariatric surgery. |
| Hoskins W et al. 2026, *JBJS Rev* | 42896011 | Orthopaedic primer. TTE **does not** address unmeasured confounding. |
| Lauer MS, D'Agostino RB 2013, *NEJM* | 23991657 | The randomized registry trial. |
| Zwarenstein M et al. 2008, *BMJ* | 19001484 | CONSORT extension for pragmatic trials. |
| Brookhart MA et al. 2006, *Epidemiology* 17:268 | 16617275 | Physician preference as an instrumental variable. |

**Gap:** we found no surgery-specific systematic review of TTE methods (JAMA Surg / Ann Surg, 2022–2026). Applied examples: Sakowitz 2025, *Ann Surg* (TCAR vs CEA), PMID 40607703; Talathi 2026, *Surgery* (STRASS 2 emulated), PMID 41936769.

## 9. Vascular surgery

| Paper | PMID | Takeaway |
|---|---|---|
| Farber A et al. 2022, *NEJM* (BEST-CLI) | 36342173 | Cohort 1: MALE or death 42.6% with surgery vs 57.4% with endovascular (HR 0.68). Whether a vein conduit was available decided which cohort a patient entered. |
| Bradbury AW et al. 2023, *Lancet* (BASIL-2) | 37116524 | n = 345. Vein bypass first was worse (aHR 1.35), which conflicts with BEST-CLI. |
| Hunt 2017 *Trials*; Bradbury 2025 *BMJ* (BASIL-3) | 28526046, 39993822 | **481 of 861 planned** randomized. No benefit from drug-coated balloons or drug-eluting stents. |
| **Katsanos K et al. 2018, *JAHA*** | 30561254 | Paclitaxel: 5-year mortality 14.7% vs 8.1% (RR 1.93). Only 3 RCTs had 5-year data. |
| Rocha-Singh KJ et al. 2020, *Circulation* | 32370548 | Individual patient data: **23–24% lost to follow-up or withdrew**. After recovering vital status, HR 1.38 → 1.27. |
| Secemsky EA et al. 2019, *JAMA Cardiol* | 30747949 | Medicare: no excess mortality. |
| Freisinger E et al. 2020, *Eur Heart J* | 31593987 | German claims data: no excess mortality. |
| Nordanstig J et al. 2020, *NEJM* (SWEDEPAD, interim) | 33296560 | Registry RCT: HR 1.06. |
| **Nordanstig J et al. 2025, *Lancet* (SWEDEPAD 2, claudication)** | 40902614 | No benefit in quality of life. 5-year mortality HR 1.47 (1.09–1.98). |
| Falkenberg M et al. 2025, *Lancet* (SWEDEPAD 1, CLTI) | 40902617 | The CLTI companion trial. |
| Fakhry F et al. 2015, *JAMA* (ERASE) | 26547465 | Revascularization plus exercise beat exercise alone at 12 months. |
| Murphy TP et al. 2012, *Circulation* (CLEVER) | 22090168 | Exercise was better for walking time and stenting for quality of life. n = 111. |
| Nordanstig 2014, *Circulation*; Djerf 2020, *Circ CV Interv* (IRONIC) | 25095886, 31937137 | The quality-of-life benefit seen at 1 year was **lost by 5 years**, at about twice the cost. |
| **Sorber R et al. 2023, *JVS*** | 37276171 | Medicare. Early PVI 3.2%. Late PVI 22.5% vs 3.6% (aHR 6.89). High-use physicians. |
| Dun C et al. 2025, *Ann Vasc Surg* 110:424 | 39419322 | Market competition predicts early PVI. |
| Weaver ML et al. 2023, *JVS* | 37141948 | Market competition and claudication management in the VQI. |
| Madabhushi V et al. 2021, *JVS* | 33775749 | Revascularizing claudicants: CLTI HR 2.9, amputation HR 4.5. |
| Conte MS et al. 2025, *JVS* (SVS focused update) | 40316185 | Major evidence gaps for claudication. |

---

## Taxonomy: RCT limitations in surgery → what TTE on claims can do

IV = internal validity, EV = external validity, F = feasibility.

| # | Limitation | Threatens | Can TTE on claims fix it? | Can it go in the simulation DGP? |
|---|---|---|---|---|
| 1 | Recruitment failure and discontinuation | F (and EV) | **Yes.** No consent is needed. The price is confounding instead of randomization. | Indirectly: compare a "would-consent" subset with the whole population. |
| 2 | Patient and surgeon equipoise and preference | F, EV, IV | **Partly.** Preference is what makes treatment vary in practice, and can serve as an IV (high-use physicians). Patient preference is unmeasured. | **Yes.** A latent physician "intervention propensity". |
| 3 | Surgeon skill and learning curve | IV, EV | **Partly.** Skill is not in claims, but the operator NPI is, so random or fixed effects are possible. Skill and preference cannot be separated if they are correlated. | **Yes.** Latent skill, correlated with preference. |
| 4 | Clustering and provider heterogeneity | IV (precision), EV | **Mostly.** Mixed models, cluster-robust SEs. | **Yes.** Random intercepts and random slopes by physician. |
| 5 | Blinding and sham | IV | **No.** No patient-reported outcomes in claims, and outcome ascertainment is not blinded. | Optional: differential ascertainment of CLTI. |
| 6 | Standardization and co-interventions | IV, EV | **Partly.** Procedure type and Part D fills are visible. Exercise, smoking, and adherence are not. | **Yes.** Unmeasured exercise uptake. |
| 7 | Crossover and deferral | IV | **Yes, TTE's strongest case**, using dynamic strategies with CCW or sequential trials. It needs the triggers for crossover to be measured. | **Yes, central.** |
| 8 | Immortal time and misaligned time zero | IV | **Yes, by design.** | **Yes.** The naive design as a comparator. |
| 9 | Selected surgeons, centers, and patients | EV | **Yes, largely.** Population-based data, plus transportability. | **Yes.** A "trial of high-skill operators" vs the whole population. |
| 10 | Long horizon, rare outcomes, changing technology | F, IV | **Partly.** Follow-up is long and complete, but device codes are coarse and calendar time confounds. | **Yes.** Time-varying effect (early benefit, late harm). |

**In one line:** TTE fixes feasibility, generalizability, and (with dynamic strategies) crossover. It cannot fix unmeasured confounding by indication or the lack of patient-reported outcomes. Surgeon skill sits in between: it cannot be measured, but its structure can be modelled through the NPI.

## Read first
1. Sorber 2023 *JVS* (37276171)
2. ADAM 2002 *NEJM* (12000813) and CAESAR 2011 (20869890)
3. Fitzgibbons 2006 *JAMA* and 2013 *Ann Surg* (16418463, 24022443)
4. Hernán & Robins 2016 *AJE*; Hernán 2016 *J Clin Epidemiol* (26994063, 27237061)
5. Maringe 2020 *Int J Epidemiol* (32386426)
6. McCulloch 2002 *BMJ*; Ergina 2009 *Lancet* (12065273, 19782875)
7. Chapman 2014 *BMJ*; Rosenthal 2015 *Ann Surg* (25491195, 24979608)
8. Katsanos 2018; Nordanstig 2020 and 2025 (30561254, 33296560, 40902614)