Elizabeth Hofheinz, M.P.H., M.Ed.
While level of evidence (LOE) is now a widely used measure, until now, there has been no research on variation in LOE based on subspecialty. But a multicenter team has taken up the challenge. Their work, “Research methodologic quality varies significantly by subspecialty: An analysis of AAOS meeting abstracts,” is published in the April 2021 edition of the Journal of Clinical Orthopaedics and Trauma.
Commenting to OSN was co-author Sean T. Campbell, M.D., an orthopedic trauma surgeon at Hospital for Special Surgery in New York, who noted, “We developed the idea for this study after observing that papers accepted for presentation at the AAOS annual meeting were highly variable with regard to level of evidence. For example, some small retrospective series were included alongside large, randomized trials. We wanted to understand more about this, and in particular we wanted to determine if this level of evidence discrepancy was related to orthopaedic subspecialty.” 
For this work, two independent raters assessed all abstracts for podium and poster presentations from the 2017 annual AAOS meeting (clinical research only). Level of evidence (I to V) was assesses using the latest Journal of Bone and Joint Surgery guidelines.
The final analysis included 1,083 abstracts, 710 of which were podium presentations while 373 were poster presentations. Country of origin was as follows: United States (70.3%), Japan (4.7%), and South Korea (4.4%).
“Overall,” said the authors, “7.9% of studies were level I, 8.0% level II, 52.5% level III, 31.4% level IV, and 0.2% level V. There was a significant difference in LOE of abstracts by subspecialty. The subspecialties with the highest percentage of level I and II studies were shoulder & elbow (22.6%), adult reconstruction knee (21.6%), hand & wrist (20.5%), and sports (18.5%). Foot & ankle (7.9%), pediatrics (6.2%), and oncology (0%) had the lowest percentage of level I and II studies. Overall, 59.2% of studies were therapeutic, 33.1% prognostic, 5.9% diagnostic, and 1.9% economic/decision analysis. Type of study also differed significantly by subspecialty, although therapeutic studies accounted for the majority of study types among all subspecialties.”
The team found a significant difference between LOE of US versus non-US abstracts. Among the US studies, 11.6% were level I and II, compared to 26.1% for abstracts from other countries. “There was no difference in LOE between podium versus poster presentations,” said the authors. There was a trend toward higher percentage of level I and II evidence among subspecialties that had a higher total number of accepted clinical studies.”
Dr. Campbell: “The most important results of the study were that methodologic quality and level of evidence did have a relationship with subspecialty category. We found that papers accepted in the shoulder and elbow, adult knee reconstruction, hand and wrist, and sports categories had the highest mean level of evidence. Trauma, spine, and hip reconstruction were somewhere in the middle. The foot and ankle, pediatrics, and orthopaedic oncology categories had the lowest mean level of evidence.”
“One of the reasons we were interested in this topic is because we wanted to determine if those of us submitting papers in subspecialty fields that were in the middle, or on the lower end of the list could learn anything from our colleagues who were engaging in studies with more rigorous methodologic quality. This led us to think on the potential reasons for these discrepancies.”
“One of the obvious potential reasons is that procedures or disease processes that are very common are easier to study in a methodologically robust way, such as a randomized trial; but that rare injuries or diseases may be more suited to a case series or comparative study. In fact, waiting to enroll enough patients for a very rare problem could even unnecessarily delay good information and science from reaching surgeons (and patients)! Future work might include identifying modifiable factors that could lead to increasing the number of studies with high level of evidence in the fields producing less, when appropriate.”
“Our overall message is that some of our colleagues are doing a very large number of level 1 and 2 studies with robust methodologic quality, and we all probably have something to learn from them! We acknowledge that rarer injuries and diseases may actually best be studied and reported on with retrospective work.”
