Original Research

---

Looking Under the Hood: Factors that Drive Successful Study Group Participation and Publications in Pediatric Spine Programs

Sonya B. Levine¹; Bradley T. Hammoor, MS¹; Abby Morris, MHA¹; Sushrut Arora¹; Afrain Z. Boby¹; Hiroko Matsumoto, PhD^1,2; Michael W. Fields¹; Adam N. Fano¹; Matthew Oetgen, MD, MBA^3,4; Tricia St. Hilaire, MPH³, Michael G. Vitale, MD, MPH^1,3; David Skaggs, MD, MMM^3,5; Pediatric Spine Study Group³

¹Division of Pediatric Orthopedic Surgery, Department of Orthopaedic Surgery, Columbia University Irving Medical Center, New York, NY; ²Department of Epidemiology, Mailman School of Public Health, Columbia University, New York, NY; ³Pediatric Spine Study Group; ⁴Division of Orthopedic Surgery, Children’s National Hospital, Washington DC; ⁵Children’s Orthopedic Center, Children’s Hospital of Los Angeles, Los Angeles, CA

Correspondence to: Hiroko Matsumoto, PhD; Division of Pediatric Orthopaedic Surgery, Columbia University Medical Center, 3959 Broadway, 8 North, New York, NY 10032, E-mail: [email protected]

Received: January 2, 2022; Accepted: January 2, 2022; Published: February 1, 2022

DOI: 10.55275/JPOSNA-2022-0009

Volume 4, Number 1, February 2022

Abstract:

Introduction: Multicenter clinical research is critically important in a constantly evolving landscape, particularly in early onset scoliosis (EOS). There is variability from center to center with regard to the “quality of participation” in such research efforts as well as publications on EOS. The purpose of this study was to examine factors associated with “high-performing” centers.

Methods: This was a multicenter retrospective cohort study of 21 academic medical centers participating in an EOS registry. Factors examined include research personnel, spine research focus, and regular participation of faculty in research, collected through an eight-question survey. Outcomes included the quality of participation in the study group, derived from quarterly site reports generated by the registry, and average annual publication volume of each institution obtained from a PubMed search. Univariable analyses were utilized to investigate institutional factors associated with the quality of performance and publication volume.

Results: All 21 sites completed the survey. Centers with full-time spine research staff had higher average quality of participation scores (90 vs. 60, p=0.026) as did centers with dedicated spine research meetings (90 vs. 70, p=0.074) than those that lacked these features. Additionally, centers with higher average publication volumes were more likely to have full-time research staff (8.0 vs. 3.5, p=0.115), a research team focused on spine (8.9 vs. 4.8, 0.067), and a dedicated physician assistant (PA) or nurse with >95% focus on spine (9.0 vs. 5.4, p=0.107).

Conclusion: The single most important and significant factor in quality of participation in the study group was a center having full-time research staff, with research meetings dedicated to spine being nearly significant. Factors most predictive of publications was a research team focused primarily on spine and a spine dedicated PA or nurse. If institutions want to improve spine research, we recommend investing in these factors.

Level of Evidence: Level IV: Case series (no historical or control group)

Key Concepts:

• Full-time research staff is associated with higher site score in this registry, with dedicated spine research meetings being g nearly significant.
• Having a research team as well as a PA or nurse focused on spine are most associated with a higher volume of spine publications.
• Institutions should consider investing in dedicated spine staff and research meetings if they want to ensure greater registry data quality and research productivity.

Introduction

Research study group participation has been on the rise within the academic spine community over the past decade and serves as an important step for institutions that would like to procure “a seat at the table” of cutting-edge spine innovations, procedures, and techniques. In addition to study group involvement, scholarly productivity is often used as a metric for assessing a spine division’s academic success. Hence, understanding the organizational factors and processes that allow leading institutions in spine surgery to successfully participate in study groups and to consistently produce high-quality research is of significant interest to spine surgeons seeking to improve their academic performance.

Few studies in orthopaedic surgery have investigated the institutional strategies and policies employed by “high-performing centers,” that is, centers with outstanding registry participation and significant scholarly productivity. However, a number of studies outside of orthopaedics have investigated reasons for poor-quality registry data and best practices for research registry upkeep.^1–6 Despite this research, there remains a gap in the literature regarding how registry participation and data quality are influenced by the protocols and procedures of institutions that maintain them. With regards to scholarly productivity, Buccheit et al. identified research time allocation of faculty and teaching experience prior to becoming faculty as two important factors predicting research output.⁷ Similarly, White et al. found that high-performing researchers place a greater personal emphasis on research and accordingly devote more of their time to research.⁸ Other studies have identified research mentorship, program size, and grant-based funding as factors associated with increased research productivity.^9–13 Finally, Brocato et al. found that low research output participants had increased nonresearch clinical and academic demands that reduced the likelihood of engagement in research.¹⁴

To our knowledge, there are no studies investigating the relationship between a spine division’s institutional characteristics/protocols and its academic performance as measured by study group participation and scholarly productivity. Therefore, the aim of this study was to identify organizational standards and processes that are conducive to generating consistent, high-quality participation in a pediatric spinal deformity registry and significant research productivity. It was hypothesized that several key institutional and organizational factors (e.g., dedicated research time, dedicated research teams) are associated with greater academic success. Insights gained from the findings of this study can help guide academic orthopaedic surgeons in developing quality improvement programs aimed at implementing policies and practices appropriate for generating high-quality research.

Materials and Methods

Study Design and Setting

This is a multicenter retrospective cohort study of 21 academic medical centers participating in the Pediatric Spine Study Group (PSSG) early onset scoliosis (EOS) registry. PSSG is a leading organization in the field of immature spinal deformity, accommodating a total of 147 clinician members and data composed of over 8,000 patients with scoliosis.

Institutional Factors

Institutional factors and attributes of research teams were assessed via an eight-question survey developed based on the seven themes in the 2015 Taylor et al. study.¹ Eight questions were designed to elucidate the availability of full-time research personnel, sources of research funding, accessibility of spine-focused research efforts, the incorporation of designated research meetings with surgeon attendance, multidisciplinary spine conferences, and the involvement of predominantly spine-focused clinical support staff such as physician assistants (PAs) and nurses. The survey was completed by lead pediatric orthopaedic surgeons who are members of PSSG.

Primary Outcome: Quality of Performance

Quality of site performance was determined using quarterly Site Scores generated by the PSSG EOS registry. All participating centers provided permission to retrieve and analyze their site performance reports. The report includes data on consistent registry participation (as assessed by monthly user activity in the online registry portal), data query fulfillment (a measure of data completeness and willingness to participate in quality assurance efforts), baseline data entry (the quality and completeness of data for newly entered patients), and patient follow-up (a measure of site willingness to continuously update patient data at their subsequent visits). These four factors are each assessed as a percentage out of 100, converted to a 0–1 scale, and summed to attain an overall “Site Score,” which allows comparison to other sites in order to provide a ranking. Site Score is provided on a scale of 1–4, which was converted to a percentage out of 100 for the purposes of this study.

Secondary Outcome: Volume of Publication

A PubMed search was conducted to gather information regarding the average number of publications per year authored by each participating surgeon within the last 5 years. This measure was used as a proxy for annual research productivity at the site.

Statistical Analysis

Associations between institutional factors and site scores/scholarly productivity were investigated using chi-squared or Fisher’s exact tests for categorical variables and independent t-tests or analysis of variance (ANOVA) for continuous variables. All analyses were two-tailed and a p-value of less than 0.05 was considered statistically significant. Statistical calculations were conducted using IBM SPSS Statistics Version 26.

Results

Survey results were obtained from 21 (100%) of the 21 solicited orthopaedic surgeons. Organizational factors and their associations with registry site score are enumerated in Table 1. A significant association was observed between the presence of full-time research personnel and Site Score. Centers employing full-time research personnel (N=17) had a higher mean Site Score than those that did not (N=3) (90±10 vs. 60±5, p=0.026). Additionally, centers with dedicated spine research meetings (N=10) had a higher mean Site Score than those without (N=11) (90±20 vs. 70±30, p=0.074); however, this difference was not statistically significant.

Institutional factors and their association with yearly average publication volume are presented in Table 2. Sites with full-time research personnel had an average publication volume of 8.0±5.3 per year, whereas sites without full-time research personnel had an average of 3.5±2.2 per year (p=0.115). Sites with dedicated spine research teams (N=12) had an average publication volume of 8.9±5.6 per year, whereas those without such a team (N=9) had an average of 4.8±3.6 per year (p=0.067). Sites with multidisciplinary conferences during which spine cases are discussed had an average publication volume of 8.1±5.3, whereas those without such conferences had an average of 4.6±3.9 (p=0.161). Finally, practices with a PA or nurse who predominantly sees spine patients had a mean publication volume of 9.0±5.2, whereas those without had an average of 5.4±4.7 (p=0.107).

There was no evidence of association between number of patients enrolled in the registry and Site Score or publication volume.

Discussion

This is the first study of its kind to investigate the factors that connote high and low performing centers in terms of research output and explore the institutional factors that influence registry quality of performance. Previous studies have investigated how research productivity is affected by different institutional realities and hierarchical structures,^10–14 and still others have pursued data quality and the factors by which it is influenced.^1–6 However, this study fills a gap in the literature by combining these two ideas into one, and proving that there is a need to pursue and understand how the institutional factors and structure of an organization can influence the quality and maintenance of registry data as well as its overall implications and contributions to the field at large.

Of the few studies that have been conducted examining the features that determine increased research productivity in the health sciences, Mills et al. examined factors associated with research productivity among family medicine residency programs. Research mentorship, direct faculty involvement in research, and larger program size were identified as independent factors driving productivity.¹² A 2005 study of academic radiology research productivity found NIH funding, large resident program size, and the presence of fellows were significantly associated with research output.¹³ Additionally, Brocato et al. found that participants with low research output tended to have increased nonresearch clinical and academic demands that reduced the likelihood of them engaging in research.¹⁴

The results of the present study suggest that the employment of full-time research personnel may play a significant role in achieving high-quality performance in study groups. Though prior studies identified research mentorship and a robust fellowship program as contributors to strong output,^12,13 none have expressly identified full-time research personnel as the leading contributor to high-quality participation.^12,13 In order to facilitate high performance in study groups and the contribution of high-quality data, investing in hiring full-time research personnel may be important. Higher-quality data likely translates into higher-quality research, leading to more impactful findings that can positively affect patient care. With the results of this study in mind, surgeons can feel more comfortable informing their administrations that the employment of full-time research staff may allow them to contribute more meaningfully to study groups. Furthermore, as national/international research collaboration is now being considered by the U.S. News and World Report when establishing rankings, this finding may help convince administrators to invest in research support. That is, if institutions are interested in their national rankings, investing in research support may be of great benefit to their evaluation. Finally, study groups may consider employment of full-time research staff as a desirable quality of a collaborating institution. Our results suggest that a participating program with this particular characteristic could contribute to the overall caliber of the group. Further, the presence of research meetings devoted to spine may also be associated with higher-quality participation. When meetings like this take place, it likely allows for protected time to discuss registry involvement and the quality of data being contributed.

The findings of this study also suggest that having a dedicated, full-time researcher or team of researchers may be associated with increased research output. Though this association did not reach statistical significance, it is intuitive, and we feel that the result would prove statistically significant with a larger sample of participants. In like with this idea, Brocato et al. proved that the varied demands of clinicians can contribute to low productivity in research; providing research teams with a singular, directed goal will likely aid research endeavors and outcomes.¹⁴ Additionally, this study found that the presence of research personnel focused primarily on spine may also increase the likelihood of scholarly productivity and quality. Taking these findings together, employing full-time research personnel with a primary focus on spine may allow for better organization of tasks, more frequent communication, more efficient and productive use of time when executing studies, and more robust data analysis. Full-time research personnel elevate the quality of a spine department, which in turn could help contribute to designation as a high-performing center. It is notable that the volume of patients enrolled in the study group is not related to quality of study group participation or number of research publications. This is encouraging to lower-volume programs as strategic investment in dedicated staff and meetings should still be associated with improved research.

There are certainly limitations to this study. First, the 21 sites that were included in the analysis were all members of one study group. Since members of PSSG are generally influential surgeons who are employed by renowned institutions, there is a possibility of selection bias. Second, the survey was limited in scope and focused primarily on processes and protocols surrounding research. It could be beneficial to perform a secondary study that focuses more broadly on hospital protocols, such as how departments are organized and how funding is allocated throughout the institution; there may be other important factors in predicting high performance that are not included in this study. Third, the study group in question is specifically an EOS registry, so the subgroup is finite and specific; conducting similar research using study groups in other subspecialties would serve as a valuable complement to the present study. Finally, there remains questions on what exactly defines a high-performing center. Though this study attempted to help demonstrate which characteristics of a center can contribute to this quality study group participation and publication volume, there are surely other factors to consider when making this distinction.

This study showed that there is precedent to investigate how the organizational and institutional factors of a department can influence data collection/maintenance and research productivity. The results of this survey and study group performance review indicated that employing full-time research personnel to aid in spine research efforts may directly impact the quality and quantity of research performed at each institution. This finding supports funding to employ research personnel, as it may assist their rankings and elevate their contribution to the field of spine surgery. Investing in research personnel will provide a mutually beneficial outcome, providing clinicians with more bandwidth to focus on their surgical and clinical responsibilities and researchers devoted to pursuing a variety of hypotheses that will contribute meaningfully to the field. These findings warrant future study to interrogate the impact of other larger-scale institutional factors on research and quality of performance.

References

1. Taylor N, Clay-Williams R, Hogden E, et al. High performing hospitals: a qualitative systematic review of associated factors and practical strategies for improvement Quality, performance, safety and outcomes. BMC Health Serv Res. 2015;15(1):244. doi: https://doi.org/10.1186/s12913-015-0879-z.
2. Arts D, De Keizer N, Scheffer GJ, et al. Quality of data collected for severity of illness scores in the Dutch National Intensive Care Evaluation (NICE) registry. Intensive Care Med. 2002;28(5):656–659. doi: https://doi.org/10.1007/s00134-002-1272-z.
3. Cnattingius S, Ericson A, Gunnarskog J, et al. A quality study of a medical birth registry. Scand J Soc Med. 1990;18(2):143-148. doi: https://doi.org/10.1177/140349489001800209.
4. Bray F, Parkin DM. Evaluation of data quality in the cancer registry: principles and methods. Part I: Comparability, validity and timeliness. Eur J Cancer. 2009;45(5):747-755. doi: https://doi.org/10.1016/j.ejca.2008.11.032.
5. Marriott J, Noonan VK, Donner E, et al. Neurological registry quality control and quality assurance. Can J Neurol Sci. 2013;40(S2):S47-S50. doi: https://doi.org/10.1017/s0317167100017170.
6. Teppo L, Pukkala E, Lehtonen M. Data quality and quality control of a population-based cancer registry: experience in Finland. Acta Oncol (Madr). 1994;33(4):365-369. doi: https://doi.org/10.3109/02841869409098430.
7. Buchheit S, Allison B, Collins DLC. Intra-institutional factors that influence accounting research productivity. J Appl Bus Res. 2001;17(2):17-32.
8. White CS, James K, Burke LA, et al. What makes a “research star”? Factors influencing the research productivity of business faculty. Int J Product Perform Manag. 2012;61(6):584-602. doi: https://doi.org/10.1108/17410401211249175.
9. Baird LL. What characterizes a productive research department? Res High Educ. 1986;25(3):211-225. doi: https://doi.org/10.1007/BF00991787.
10. Dundar H, Lewis DR. Determinants of research productivity in higher education. Res High Educ. 1998;39(6):607-631. doi: https://doi.org/10.1023/A:1018705823763.
11. Levitan AS, Ray R. Personal and institutional characteristics affecting research productivity of academic accountants. J Educ Bus. 1992;67(6):335-341. doi: https://doi.org/10.1080/08832323.1992. 10117569.
12. Mills OF, Zyzanski SJ, Flocke S. Factors associated with research productivity in family practice residencies. Fam Med. 1995;27(3): 188-193. http://europepmc.org/abstract/MED/7774779.
13. Itagaki MW, Pile-Spellman J. Factors associated with academic radiology research productivity. Radiology. 2005;237(3):774-780. doi: https://doi.org/10.1148/radiol.2373041508.
14. Brocato JJ, Mavis B. The research productivity of faculty in Family Medicine Departments at U.S. Medical Schools: A National Study. Acad Med. 2005; 80(3):244-252. https://journals.lww.com/academicmedicine/Fulltext/2005/03000/The_Research_Productivity_of_Faculty_in_Family.8.aspx.