tte-surgery/docs/literature/01-surgical-rct-limitations.md

Literature review 1: limitations of RCTs in surgery, mapped to TTE on claims data

All citations were checked against PubMed through NCBI E-utilities, 2026-10-27. The numbers come from PubMed abstracts; full texts were not read. Papers with no PubMed abstract are described only in general terms.

Corrections and caveats

• Chapman 2014 was published in BMJ, not Surgery.
• The page numbers for McCulloch 2002 are from memory, not from PubMed.
• BASIL-3 recruited 481 of the planned 861 patients. That the paclitaxel signal caused the shortfall is a guess, not a verified finding.

---

1. Classic critiques and the IDEAL framework

Paper	PMID	Takeaway
McCulloch P et al. 2002, BMJ "Randomised trials in surgery: problems and possible solutions"	12065273	The founding critique. Barriers: equipoise, learning curve, quality control, blinding, surgeon preference.
McLeod RS 1996, Surgery 119:483	8619200	Problems specific to RCTs in surgery.
Solomon MJ, McLeod RS 1995, Surgery	7652679	Many surgical questions could be randomized but are not.
Solomon MJ, McLeod RS 1998, Med J Aust	9803252	Overview of RCTs in surgery: past, present and future.
Barkun JS et al. 2009, Lancet 374:1089	19782874	IDEAL series, paper 1: the stages of surgical innovation.
Ergina PL et al. 2009, Lancet 374:1097	19782875	Few of the challenges are unique to surgery, but in surgery they occur together: complexity, surgeon factors, and the range of outcomes.
McCulloch P et al. 2009, Lancet 374:1105	19782876	IDEAL recommendations: registries and prospective databases, with RCTs "whenever possible".
McCulloch 2013 / Ergina 2013 / Cook 2013, BMJ (IDEAL series)	23778427, 23778426, 23778425	Observational studies in the Exploration and Assessment stages; RCTs and long-term registry evaluation.
Hirst A et al. 2019, Ann Surg	29697448	IDEAL update: adds a Pre-IDEAL stage and covers devices.
Cook JA 2009, Trials 10:9	19200379	Checklist for the design, conduct and analysis of surgical RCTs.
Pronk AJM et al. 2023, Br J Surg 110:1300	37379487	Of 1,188 surgical RCTs, only 22.1% published in 2019 were at low risk of bias (14.7% in 2009).
Ahmed Ali U et al. 2018, J Surg Res 228:1	29907196	Of 228 negative surgical RCTs, the median sample size was about 76–80, so most were underpowered.

2. Recruitment failure, discontinuation, and equipoise

Paper	PMID	Takeaway
Chapman SJ et al. 2014, BMJ 349:g6870	25491195	Of 395 surgical RCTs, 21% stopped early. The main reason was poor recruitment (44% of stopped trials). Of completed trials, 34% were unpublished at a median of 4.9 years.
Rosenthal R et al. 2015, Ann Surg 262:68	24979608	Discontinued: 43% of surgical vs 27% of medical RCTs. Stopped for slow recruitment: 18% vs 11%. Discontinuation predicted non-publication (OR 4.18).
Kasenda B et al. 2014, JAMA 311:1045	24618966	Across all fields, 24.9% of RCTs were discontinued, most often for poor recruitment.
Cook JA et al. 2008, Contemp Clin Trials 29:631	18397843	Trials of surgery vs medical treatment recruited about 2× slower than trials of surgery vs surgery (median 68 vs 35 days per recruit). This applies directly to PVI vs exercise.
Donovan J et al. 2002, BMJ 325:766 (ProtecT)	12364308	Qualitative work on how recruiters presented the options made an "impossible" trial feasible.
Donovan JL et al. 2014, J Clin Epidemiol 67:912	24811157	Clinicians' "hunches" undermined equipoise, and surgeons felt this most intensely.
Rooshenas L et al. 2016, PLoS Med 13:e1002147	27755555	In recorded consultations, the equipoise clinicians stated differed from how they actually presented the options.
Donovan JL et al. 2022, Trials 23:258 (QRI-Two)	35379301	A rapid recruitment intervention for trials that are failing to recruit.
Hamdy FC et al. 2023, NEJM 388:1547 (ProtecT at 15 years)	36912538	A monitor-vs-intervene trial can be completed.

3. Surgeon skill, learning curve, expertise-based trials, and clustering

Paper	PMID	Takeaway
Devereaux PJ et al. 2005, BMJ 330:88	15637373	Proposes expertise-based RCTs: each surgeon performs only the procedure they are expert in, to avoid differential-expertise bias.
Cook JA et al. 2015, Trials 16:241	26025450	Review of 43 expertise-based RCTs: 92% compliance, but expertise criteria were poorly reported.
Cook JA et al. 2004, Clin Trials 1:421	16279280	Statistical methods for learning curves in surgical trials.
Ramsay CR et al. 2000 / 2001	11155830, 11319991	Assessing learning curves in health technology.
Lee KJ, Thompson SG 2005, BMJ 330:142; Clin Trials 2:163	15649931, 16279138	Outcomes cluster by clinician even under individual randomization. Ignoring it makes the estimates look more precise than they are.
Roberts C, Roberts SA 2005, Clin Trials 2:152	16279137	Clustering induced by treatment, which can differ between arms (partially nested designs).
Walters SJ 2010, J Clin Nurs 19:1102	20492055	Therapist effects: cluster-level, random-effects, and marginal models compared.
Birkmeyer JD et al. 2013, NEJM 369:1434	24106936	Peer ratings of surgical skill from video: the bottom quartile vs the top had 14.5% vs 5.2% complications and 0.26% vs 0.05% mortality.
Birkmeyer JD et al. 2003, NEJM 349:2117	14645640	Surgeon volume and mortality.

4. Blinding and sham surgery

Paper	PMID	Takeaway
Wartolowska K et al. 2014, BMJ 348:g3253	24850821	Across 53 placebo-controlled surgical RCTs, the placebo arm improved in 74%. Surgery was no better than placebo in 51%.
Wartolowska K et al. 2016, BMJ Open	27008687	Placebo-controlled surgical trials are feasible but recruit slowly.
Boutron I et al. 2017, Ann Intern Med 167:40	28630973	CONSORT extension for non-pharmacologic treatments: expertise, blinding, and standardization.

5. Standardization, crossover, and contamination

Paper	PMID	Takeaway
Garas G et al. 2018, Ann Surg Oncol 25:221	29110271	Across 40 RCTs of minimally invasive vs open surgery, crossover averaged 12.6% (0–45%) and worsened the ITT results.
Weinstein JN et al. 2006, JAMA 296:2441 (SPORT)	17119140	Only 50% of the surgery arm had surgery, and 30% of the nonoperative arm did, so the ITT comparison could not be interpreted.
Hernán MA, Robins JM 2017, NEJM 377:1391	28976864	Per-protocol effects in pragmatic trials.

6. Generalizability and external validity

Paper	PMID	Takeaway
Rothwell PM 2005, Lancet 365:82	15639683	"To whom do the results of this trial apply?"
Rothwell PM 2006, PLoS Clin Trials	16871331	Factors that affect external validity.
Cutteridge J et al. 2025, BJS Open	40105904	Patients in 307 vascular RCTs were younger than those in the UK National Vascular Registry.
Jones HG et al. 2026, EJVES	42526742	Endovascular lower-limb revascularization is under-researched relative to how often it is done. Its trials are 72% efficacy trials and 39% industry-funded.
Loudon K et al. 2015, BMJ (PRECIS-2)	25956159	Tool for rating how pragmatic or explanatory a trial is.

7. Trials of timing, "when to operate" (the key analogue)

Paper	PMID	Takeaway
Fitzgibbons RJ et al. 2006, JAMA 295:285	16418463	Hernia watchful waiting: 23% of the waiting arm crossed over to surgery, and 17% of the surgery arm crossed the other way.
Fitzgibbons RJ et al. 2013, Ann Surg 258:508	24022443	Long-term crossover was 68% (79% in men over 65). "Watchful waiting" works out as "delayed surgery".
O'Dwyer PJ et al. 2006, Ann Surg	16858177	UK hernia trial: crossover was higher than predicted.
UK Small Aneurysm Trial 1998, Lancet	9853436	HR 0.94. Operative mortality of 5.8% caused an early deficit in survival.
Lederle FA et al. 2002, NEJM (ADAM)	12000813	61.6% of the surveillance arm were eventually repaired. Surveillance is a dynamic strategy, not "no treatment".
Cao P et al. 2011, EJVES (CAESAR)	20869890	Delayed repair under surveillance: 59.7% by 36 months and 84.5% by 54 months. One in six aneurysms became unsuitable for EVAR, a harm of delay that the ITT effect on mortality does not capture.
Ouriel K et al. 2010, JVS (PIVOTAL)	20304589	31% of the surveillance arm were repaired by 20 months. HR 1.01.
Ulug P et al. 2020, Cochrane	32609382	Early repair of small AAAs has no advantage.

8. Pragmatic trials, registry trials, and TTE

Paper	PMID	Takeaway
Hernán MA, Robins JM 2016, AJE 183:758	26994063	The foundational TTE paper.
Hernán MA et al. 2016, J Clin Epidemiol 79:70	27237061	Specifying the target trial avoids self-inflicted biases such as immortal time.
Hernán MA et al. 2022, JAMA 328:2446	36508210	JAMA Guide to TTE.
Cain LE et al. 2010, Int J Biostat	21972433	"When to start treatment?": dynamic regimes.
Maringe C et al. 2020, Int J Epidemiol 49:1719	32386426	CCW tutorial on surgery in older patients with lung cancer. The naive 22% benefit fell to 11%.
Wang SV et al. 2023, JAMA 329:1376 (RCT-DUPLICATE)	37097356	Of 32 RCTs emulated in claims data, agreement was r = 0.93 when the emulation was close and r = 0.53 when it was not. All were drug trials.
Cashin AG et al. 2025, JAMA / BMJ (TARGET)	40899949 / 40903028	Reporting guideline for TTE.
Scola G et al. 2023, BMC Med Res Methodol	37587484	Only 16% of TTEs used cloning to handle immortal time.
Katsoulis M et al. 2024, Epidemiology 35:730	39024012	TTE of bariatric surgery.
Hoskins W et al. 2026, JBJS Rev	42896011	Orthopaedic primer. TTE does not address unmeasured confounding.
Lauer MS, D'Agostino RB 2013, NEJM	23991657	The randomized registry trial.
Zwarenstein M et al. 2008, BMJ	19001484	CONSORT extension for pragmatic trials.
Brookhart MA et al. 2006, Epidemiology 17:268	16617275	Physician preference as an instrumental variable.

Gap: we found no surgery-specific systematic review of TTE methods (JAMA Surg / Ann Surg, 2022–2026). Applied examples: Sakowitz 2025, Ann Surg (TCAR vs CEA), PMID 40607703; Talathi 2026, Surgery (STRASS 2 emulated), PMID 41936769.

9. Vascular surgery

Paper	PMID	Takeaway
Farber A et al. 2022, NEJM (BEST-CLI)	36342173	Cohort 1: MALE or death 42.6% with surgery vs 57.4% with endovascular (HR 0.68). Whether a vein conduit was available decided which cohort a patient entered.
Bradbury AW et al. 2023, Lancet (BASIL-2)	37116524	n = 345. Vein bypass first was worse (aHR 1.35), which conflicts with BEST-CLI.
Hunt 2017 Trials; Bradbury 2025 BMJ (BASIL-3)	28526046, 39993822	481 of 861 planned randomized. No benefit from drug-coated balloons or drug-eluting stents.
Katsanos K et al. 2018, JAHA	30561254	Paclitaxel: 5-year mortality 14.7% vs 8.1% (RR 1.93). Only 3 RCTs had 5-year data.
Rocha-Singh KJ et al. 2020, Circulation	32370548	Individual patient data: 23–24% lost to follow-up or withdrew. After recovering vital status, HR 1.38 → 1.27.
Secemsky EA et al. 2019, JAMA Cardiol	30747949	Medicare: no excess mortality.
Freisinger E et al. 2020, Eur Heart J	31593987	German claims data: no excess mortality.
Nordanstig J et al. 2020, NEJM (SWEDEPAD, interim)	33296560	Registry RCT: HR 1.06.
Nordanstig J et al. 2025, Lancet (SWEDEPAD 2, claudication)	40902614	No benefit in quality of life. 5-year mortality HR 1.47 (1.09–1.98).
Falkenberg M et al. 2025, Lancet (SWEDEPAD 1, CLTI)	40902617	The CLTI companion trial.
Fakhry F et al. 2015, JAMA (ERASE)	26547465	Revascularization plus exercise beat exercise alone at 12 months.
Murphy TP et al. 2012, Circulation (CLEVER)	22090168	Exercise was better for walking time and stenting for quality of life. n = 111.
Nordanstig 2014, Circulation; Djerf 2020, Circ CV Interv (IRONIC)	25095886, 31937137	The quality-of-life benefit seen at 1 year was lost by 5 years, at about twice the cost.
Sorber R et al. 2023, JVS	37276171	Medicare. Early PVI 3.2%. Late PVI 22.5% vs 3.6% (aHR 6.89). High-use physicians.
Dun C et al. 2025, Ann Vasc Surg 110:424	39419322	Market competition predicts early PVI.
Weaver ML et al. 2023, JVS	37141948	Market competition and claudication management in the VQI.
Madabhushi V et al. 2021, JVS	33775749	Revascularizing claudicants: CLTI HR 2.9, amputation HR 4.5.
Conte MS et al. 2025, JVS (SVS focused update)	40316185	Major evidence gaps for claudication.

---

Taxonomy: RCT limitations in surgery → what TTE on claims can do

IV = internal validity, EV = external validity, F = feasibility.

#	Limitation	Threatens	Can TTE on claims fix it?	Can it go in the simulation DGP?
1	Recruitment failure and discontinuation	F (and EV)	Yes. No consent is needed. The price is confounding instead of randomization.	Indirectly: compare a "would-consent" subset with the whole population.
2	Patient and surgeon equipoise and preference	F, EV, IV	Partly. Preference is what makes treatment vary in practice, and can serve as an IV (high-use physicians). Patient preference is unmeasured.	Yes. A latent physician "intervention propensity".
3	Surgeon skill and learning curve	IV, EV	Partly. Skill is not in claims, but the operator NPI is, so random or fixed effects are possible. Skill and preference cannot be separated if they are correlated.	Yes. Latent skill, correlated with preference.
4	Clustering and provider heterogeneity	IV (precision), EV	Mostly. Mixed models, cluster-robust SEs.	Yes. Random intercepts and random slopes by physician.
5	Blinding and sham	IV	No. No patient-reported outcomes in claims, and outcome ascertainment is not blinded.	Optional: differential ascertainment of CLTI.
6	Standardization and co-interventions	IV, EV	Partly. Procedure type and Part D fills are visible. Exercise, smoking, and adherence are not.	Yes. Unmeasured exercise uptake.
7	Crossover and deferral	IV	Yes, TTE's strongest case, using dynamic strategies with CCW or sequential trials. It needs the triggers for crossover to be measured.	Yes, central.
8	Immortal time and misaligned time zero	IV	Yes, by design.	Yes. The naive design as a comparator.
9	Selected surgeons, centers, and patients	EV	Yes, largely. Population-based data, plus transportability.	Yes. A "trial of high-skill operators" vs the whole population.
10	Long horizon, rare outcomes, changing technology	F, IV	Partly. Follow-up is long and complete, but device codes are coarse and calendar time confounds.	Yes. Time-varying effect (early benefit, late harm).

In one line: TTE fixes feasibility, generalizability, and (with dynamic strategies) crossover. It cannot fix unmeasured confounding by indication or the lack of patient-reported outcomes. Surgeon skill sits in between: it cannot be measured, but its structure can be modelled through the NPI.

Read first

1. Sorber 2023 JVS (37276171)
2. ADAM 2002 NEJM (12000813) and CAESAR 2011 (20869890)
3. Fitzgibbons 2006 JAMA and 2013 Ann Surg (16418463, 24022443)
4. Hernán & Robins 2016 AJE; Hernán 2016 J Clin Epidemiol (26994063, 27237061)
5. Maringe 2020 Int J Epidemiol (32386426)
6. McCulloch 2002 BMJ; Ergina 2009 Lancet (12065273, 19782875)
7. Chapman 2014 BMJ; Rosenthal 2015 Ann Surg (25491195, 24979608)
8. Katsanos 2018; Nordanstig 2020 and 2025 (30561254, 33296560, 40902614)