16 KiB
16 KiB
Literature review 1: limitations of RCTs in surgery, mapped to TTE on claims data
All citations were checked against PubMed through NCBI E-utilities, 2026-10-27. The numbers come from PubMed abstracts; full texts were not read. Papers with no PubMed abstract are described only in general terms.
Corrections and caveats
- Chapman 2014 was published in BMJ, not Surgery.
- The page numbers for McCulloch 2002 are from memory, not from PubMed.
- BASIL-3 recruited 481 of the planned 861 patients. That the paclitaxel signal caused the shortfall is a guess, not a verified finding.
1. Classic critiques and the IDEAL framework
| Paper | PMID | Takeaway |
|---|---|---|
| McCulloch P et al. 2002, BMJ "Randomised trials in surgery: problems and possible solutions" | 12065273 | The founding critique. Barriers: equipoise, learning curve, quality control, blinding, surgeon preference. |
| McLeod RS 1996, Surgery 119:483 | 8619200 | Problems specific to RCTs in surgery. |
| Solomon MJ, McLeod RS 1995, Surgery | 7652679 | Many surgical questions could be randomized but are not. |
| Solomon MJ, McLeod RS 1998, Med J Aust | 9803252 | Overview of RCTs in surgery: past, present and future. |
| Barkun JS et al. 2009, Lancet 374:1089 | 19782874 | IDEAL series, paper 1: the stages of surgical innovation. |
| Ergina PL et al. 2009, Lancet 374:1097 | 19782875 | Few of the challenges are unique to surgery, but in surgery they occur together: complexity, surgeon factors, and the range of outcomes. |
| McCulloch P et al. 2009, Lancet 374:1105 | 19782876 | IDEAL recommendations: registries and prospective databases, with RCTs "whenever possible". |
| McCulloch 2013 / Ergina 2013 / Cook 2013, BMJ (IDEAL series) | 23778427, 23778426, 23778425 | Observational studies in the Exploration and Assessment stages; RCTs and long-term registry evaluation. |
| Hirst A et al. 2019, Ann Surg | 29697448 | IDEAL update: adds a Pre-IDEAL stage and covers devices. |
| Cook JA 2009, Trials 10:9 | 19200379 | Checklist for the design, conduct and analysis of surgical RCTs. |
| Pronk AJM et al. 2023, Br J Surg 110:1300 | 37379487 | Of 1,188 surgical RCTs, only 22.1% published in 2019 were at low risk of bias (14.7% in 2009). |
| Ahmed Ali U et al. 2018, J Surg Res 228:1 | 29907196 | Of 228 negative surgical RCTs, the median sample size was about 76–80, so most were underpowered. |
2. Recruitment failure, discontinuation, and equipoise
| Paper | PMID | Takeaway |
|---|---|---|
| Chapman SJ et al. 2014, BMJ 349:g6870 | 25491195 | Of 395 surgical RCTs, 21% stopped early. The main reason was poor recruitment (44% of stopped trials). Of completed trials, 34% were unpublished at a median of 4.9 years. |
| Rosenthal R et al. 2015, Ann Surg 262:68 | 24979608 | Discontinued: 43% of surgical vs 27% of medical RCTs. Stopped for slow recruitment: 18% vs 11%. Discontinuation predicted non-publication (OR 4.18). |
| Kasenda B et al. 2014, JAMA 311:1045 | 24618966 | Across all fields, 24.9% of RCTs were discontinued, most often for poor recruitment. |
| Cook JA et al. 2008, Contemp Clin Trials 29:631 | 18397843 | Trials of surgery vs medical treatment recruited about 2× slower than trials of surgery vs surgery (median 68 vs 35 days per recruit). This applies directly to PVI vs exercise. |
| Donovan J et al. 2002, BMJ 325:766 (ProtecT) | 12364308 | Qualitative work on how recruiters presented the options made an "impossible" trial feasible. |
| Donovan JL et al. 2014, J Clin Epidemiol 67:912 | 24811157 | Clinicians' "hunches" undermined equipoise, and surgeons felt this most intensely. |
| Rooshenas L et al. 2016, PLoS Med 13:e1002147 | 27755555 | In recorded consultations, the equipoise clinicians stated differed from how they actually presented the options. |
| Donovan JL et al. 2022, Trials 23:258 (QRI-Two) | 35379301 | A rapid recruitment intervention for trials that are failing to recruit. |
| Hamdy FC et al. 2023, NEJM 388:1547 (ProtecT at 15 years) | 36912538 | A monitor-vs-intervene trial can be completed. |
3. Surgeon skill, learning curve, expertise-based trials, and clustering
| Paper | PMID | Takeaway |
|---|---|---|
| Devereaux PJ et al. 2005, BMJ 330:88 | 15637373 | Proposes expertise-based RCTs: each surgeon performs only the procedure they are expert in, to avoid differential-expertise bias. |
| Cook JA et al. 2015, Trials 16:241 | 26025450 | Review of 43 expertise-based RCTs: 92% compliance, but expertise criteria were poorly reported. |
| Cook JA et al. 2004, Clin Trials 1:421 | 16279280 | Statistical methods for learning curves in surgical trials. |
| Ramsay CR et al. 2000 / 2001 | 11155830, 11319991 | Assessing learning curves in health technology. |
| Lee KJ, Thompson SG 2005, BMJ 330:142; Clin Trials 2:163 | 15649931, 16279138 | Outcomes cluster by clinician even under individual randomization. Ignoring it makes the estimates look more precise than they are. |
| Roberts C, Roberts SA 2005, Clin Trials 2:152 | 16279137 | Clustering induced by treatment, which can differ between arms (partially nested designs). |
| Walters SJ 2010, J Clin Nurs 19:1102 | 20492055 | Therapist effects: cluster-level, random-effects, and marginal models compared. |
| Birkmeyer JD et al. 2013, NEJM 369:1434 | 24106936 | Peer ratings of surgical skill from video: the bottom quartile vs the top had 14.5% vs 5.2% complications and 0.26% vs 0.05% mortality. |
| Birkmeyer JD et al. 2003, NEJM 349:2117 | 14645640 | Surgeon volume and mortality. |
4. Blinding and sham surgery
| Paper | PMID | Takeaway |
|---|---|---|
| Wartolowska K et al. 2014, BMJ 348:g3253 | 24850821 | Across 53 placebo-controlled surgical RCTs, the placebo arm improved in 74%. Surgery was no better than placebo in 51%. |
| Wartolowska K et al. 2016, BMJ Open | 27008687 | Placebo-controlled surgical trials are feasible but recruit slowly. |
| Boutron I et al. 2017, Ann Intern Med 167:40 | 28630973 | CONSORT extension for non-pharmacologic treatments: expertise, blinding, and standardization. |
5. Standardization, crossover, and contamination
| Paper | PMID | Takeaway |
|---|---|---|
| Garas G et al. 2018, Ann Surg Oncol 25:221 | 29110271 | Across 40 RCTs of minimally invasive vs open surgery, crossover averaged 12.6% (0–45%) and worsened the ITT results. |
| Weinstein JN et al. 2006, JAMA 296:2441 (SPORT) | 17119140 | Only 50% of the surgery arm had surgery, and 30% of the nonoperative arm did, so the ITT comparison could not be interpreted. |
| Hernán MA, Robins JM 2017, NEJM 377:1391 | 28976864 | Per-protocol effects in pragmatic trials. |
6. Generalizability and external validity
| Paper | PMID | Takeaway |
|---|---|---|
| Rothwell PM 2005, Lancet 365:82 | 15639683 | "To whom do the results of this trial apply?" |
| Rothwell PM 2006, PLoS Clin Trials | 16871331 | Factors that affect external validity. |
| Cutteridge J et al. 2025, BJS Open | 40105904 | Patients in 307 vascular RCTs were younger than those in the UK National Vascular Registry. |
| Jones HG et al. 2026, EJVES | 42526742 | Endovascular lower-limb revascularization is under-researched relative to how often it is done. Its trials are 72% efficacy trials and 39% industry-funded. |
| Loudon K et al. 2015, BMJ (PRECIS-2) | 25956159 | Tool for rating how pragmatic or explanatory a trial is. |
7. Trials of timing, "when to operate" (the key analogue)
| Paper | PMID | Takeaway |
|---|---|---|
| Fitzgibbons RJ et al. 2006, JAMA 295:285 | 16418463 | Hernia watchful waiting: 23% of the waiting arm crossed over to surgery, and 17% of the surgery arm crossed the other way. |
| Fitzgibbons RJ et al. 2013, Ann Surg 258:508 | 24022443 | Long-term crossover was 68% (79% in men over 65). "Watchful waiting" works out as "delayed surgery". |
| O'Dwyer PJ et al. 2006, Ann Surg | 16858177 | UK hernia trial: crossover was higher than predicted. |
| UK Small Aneurysm Trial 1998, Lancet | 9853436 | HR 0.94. Operative mortality of 5.8% caused an early deficit in survival. |
| Lederle FA et al. 2002, NEJM (ADAM) | 12000813 | 61.6% of the surveillance arm were eventually repaired. Surveillance is a dynamic strategy, not "no treatment". |
| Cao P et al. 2011, EJVES (CAESAR) | 20869890 | Delayed repair under surveillance: 59.7% by 36 months and 84.5% by 54 months. One in six aneurysms became unsuitable for EVAR, a harm of delay that the ITT effect on mortality does not capture. |
| Ouriel K et al. 2010, JVS (PIVOTAL) | 20304589 | 31% of the surveillance arm were repaired by 20 months. HR 1.01. |
| Ulug P et al. 2020, Cochrane | 32609382 | Early repair of small AAAs has no advantage. |
8. Pragmatic trials, registry trials, and TTE
| Paper | PMID | Takeaway |
|---|---|---|
| Hernán MA, Robins JM 2016, AJE 183:758 | 26994063 | The foundational TTE paper. |
| Hernán MA et al. 2016, J Clin Epidemiol 79:70 | 27237061 | Specifying the target trial avoids self-inflicted biases such as immortal time. |
| Hernán MA et al. 2022, JAMA 328:2446 | 36508210 | JAMA Guide to TTE. |
| Cain LE et al. 2010, Int J Biostat | 21972433 | "When to start treatment?": dynamic regimes. |
| Maringe C et al. 2020, Int J Epidemiol 49:1719 | 32386426 | CCW tutorial on surgery in older patients with lung cancer. The naive 22% benefit fell to 11%. |
| Wang SV et al. 2023, JAMA 329:1376 (RCT-DUPLICATE) | 37097356 | Of 32 RCTs emulated in claims data, agreement was r = 0.93 when the emulation was close and r = 0.53 when it was not. All were drug trials. |
| Cashin AG et al. 2025, JAMA / BMJ (TARGET) | 40899949 / 40903028 | Reporting guideline for TTE. |
| Scola G et al. 2023, BMC Med Res Methodol | 37587484 | Only 16% of TTEs used cloning to handle immortal time. |
| Katsoulis M et al. 2024, Epidemiology 35:730 | 39024012 | TTE of bariatric surgery. |
| Hoskins W et al. 2026, JBJS Rev | 42896011 | Orthopaedic primer. TTE does not address unmeasured confounding. |
| Lauer MS, D'Agostino RB 2013, NEJM | 23991657 | The randomized registry trial. |
| Zwarenstein M et al. 2008, BMJ | 19001484 | CONSORT extension for pragmatic trials. |
| Brookhart MA et al. 2006, Epidemiology 17:268 | 16617275 | Physician preference as an instrumental variable. |
Gap: we found no surgery-specific systematic review of TTE methods (JAMA Surg / Ann Surg, 2022–2026). Applied examples: Sakowitz 2025, Ann Surg (TCAR vs CEA), PMID 40607703; Talathi 2026, Surgery (STRASS 2 emulated), PMID 41936769.
9. Vascular surgery
| Paper | PMID | Takeaway |
|---|---|---|
| Farber A et al. 2022, NEJM (BEST-CLI) | 36342173 | Cohort 1: MALE or death 42.6% with surgery vs 57.4% with endovascular (HR 0.68). Whether a vein conduit was available decided which cohort a patient entered. |
| Bradbury AW et al. 2023, Lancet (BASIL-2) | 37116524 | n = 345. Vein bypass first was worse (aHR 1.35), which conflicts with BEST-CLI. |
| Hunt 2017 Trials; Bradbury 2025 BMJ (BASIL-3) | 28526046, 39993822 | 481 of 861 planned randomized. No benefit from drug-coated balloons or drug-eluting stents. |
| Katsanos K et al. 2018, JAHA | 30561254 | Paclitaxel: 5-year mortality 14.7% vs 8.1% (RR 1.93). Only 3 RCTs had 5-year data. |
| Rocha-Singh KJ et al. 2020, Circulation | 32370548 | Individual patient data: 23–24% lost to follow-up or withdrew. After recovering vital status, HR 1.38 → 1.27. |
| Secemsky EA et al. 2019, JAMA Cardiol | 30747949 | Medicare: no excess mortality. |
| Freisinger E et al. 2020, Eur Heart J | 31593987 | German claims data: no excess mortality. |
| Nordanstig J et al. 2020, NEJM (SWEDEPAD, interim) | 33296560 | Registry RCT: HR 1.06. |
| Nordanstig J et al. 2025, Lancet (SWEDEPAD 2, claudication) | 40902614 | No benefit in quality of life. 5-year mortality HR 1.47 (1.09–1.98). |
| Falkenberg M et al. 2025, Lancet (SWEDEPAD 1, CLTI) | 40902617 | The CLTI companion trial. |
| Fakhry F et al. 2015, JAMA (ERASE) | 26547465 | Revascularization plus exercise beat exercise alone at 12 months. |
| Murphy TP et al. 2012, Circulation (CLEVER) | 22090168 | Exercise was better for walking time and stenting for quality of life. n = 111. |
| Nordanstig 2014, Circulation; Djerf 2020, Circ CV Interv (IRONIC) | 25095886, 31937137 | The quality-of-life benefit seen at 1 year was lost by 5 years, at about twice the cost. |
| Sorber R et al. 2023, JVS | 37276171 | Medicare. Early PVI 3.2%. Late PVI 22.5% vs 3.6% (aHR 6.89). High-use physicians. |
| Dun C et al. 2025, Ann Vasc Surg 110:424 | 39419322 | Market competition predicts early PVI. |
| Weaver ML et al. 2023, JVS | 37141948 | Market competition and claudication management in the VQI. |
| Madabhushi V et al. 2021, JVS | 33775749 | Revascularizing claudicants: CLTI HR 2.9, amputation HR 4.5. |
| Conte MS et al. 2025, JVS (SVS focused update) | 40316185 | Major evidence gaps for claudication. |
Taxonomy: RCT limitations in surgery → what TTE on claims can do
IV = internal validity, EV = external validity, F = feasibility.
| # | Limitation | Threatens | Can TTE on claims fix it? | Can it go in the simulation DGP? |
|---|---|---|---|---|
| 1 | Recruitment failure and discontinuation | F (and EV) | Yes. No consent is needed. The price is confounding instead of randomization. | Indirectly: compare a "would-consent" subset with the whole population. |
| 2 | Patient and surgeon equipoise and preference | F, EV, IV | Partly. Preference is what makes treatment vary in practice, and can serve as an IV (high-use physicians). Patient preference is unmeasured. | Yes. A latent physician "intervention propensity". |
| 3 | Surgeon skill and learning curve | IV, EV | Partly. Skill is not in claims, but the operator NPI is, so random or fixed effects are possible. Skill and preference cannot be separated if they are correlated. | Yes. Latent skill, correlated with preference. |
| 4 | Clustering and provider heterogeneity | IV (precision), EV | Mostly. Mixed models, cluster-robust SEs. | Yes. Random intercepts and random slopes by physician. |
| 5 | Blinding and sham | IV | No. No patient-reported outcomes in claims, and outcome ascertainment is not blinded. | Optional: differential ascertainment of CLTI. |
| 6 | Standardization and co-interventions | IV, EV | Partly. Procedure type and Part D fills are visible. Exercise, smoking, and adherence are not. | Yes. Unmeasured exercise uptake. |
| 7 | Crossover and deferral | IV | Yes, TTE's strongest case, using dynamic strategies with CCW or sequential trials. It needs the triggers for crossover to be measured. | Yes, central. |
| 8 | Immortal time and misaligned time zero | IV | Yes, by design. | Yes. The naive design as a comparator. |
| 9 | Selected surgeons, centers, and patients | EV | Yes, largely. Population-based data, plus transportability. | Yes. A "trial of high-skill operators" vs the whole population. |
| 10 | Long horizon, rare outcomes, changing technology | F, IV | Partly. Follow-up is long and complete, but device codes are coarse and calendar time confounds. | Yes. Time-varying effect (early benefit, late harm). |
In one line: TTE fixes feasibility, generalizability, and (with dynamic strategies) crossover. It cannot fix unmeasured confounding by indication or the lack of patient-reported outcomes. Surgeon skill sits in between: it cannot be measured, but its structure can be modelled through the NPI.
Read first
- Sorber 2023 JVS (37276171)
- ADAM 2002 NEJM (12000813) and CAESAR 2011 (20869890)
- Fitzgibbons 2006 JAMA and 2013 Ann Surg (16418463, 24022443)
- Hernán & Robins 2016 AJE; Hernán 2016 J Clin Epidemiol (26994063, 27237061)
- Maringe 2020 Int J Epidemiol (32386426)
- McCulloch 2002 BMJ; Ergina 2009 Lancet (12065273, 19782875)
- Chapman 2014 BMJ; Rosenthal 2015 Ann Surg (25491195, 24979608)
- Katsanos 2018; Nordanstig 2020 and 2025 (30561254, 33296560, 40902614)