Original Research

---

QI/PI: POSNA Safe Surgery Program (PSSP)—First-Year Results Implementing Quality Metrics

Kali Tileston, MD¹; Maike van Niekerk, PhD¹; Michael Vitale, MD²; Robert Cho, MD³; Brian Brighton, MD⁴; Abhishek Tippabhatla, BS¹; Matthew Rohde, BS¹; Kevin Shea, MD¹

¹Stanford School of Medicine, Pediatric Orthopedic Surgery, Stanford University, Stanford, CA; ²Department of Pediatric Orthopedic Surgery, Morgan Stanley Children’s Hospital of New York Presbyterian, Columbia University Medical Center, New York, NY; ³Department of Orthopedic Surgery, Shriners for Children Medical Center-Los Angeles, Pasadena, CA; ⁴Department of Orthopedic Surgery, Carolinas Healthcare System/Levine Children’s Hospital, Charlotte, NC

Correspondence: Kali Tileston, MD, Stanford School of Medicine, Pediatric Orthopedic Surgery, Stanford University, 453 Quarry Road, Stanford, CA 94305. E-mail: [email protected]

Received: July 30, 2022; Accepted: January 13, 2023; Published: May 1, 2023

DOI: 10.55275/JPOSNA-2023-544

Volume 5, Number 2, May 2023

Abstract

Background: Ranking of orthopaedic programs by external organizations is expanding to pediatric orthopaedics. These external organizations rarely consult pediatric orthopaedists themselves; therefore, POSNA members strongly supported the creation of a new performance evaluation. As a result, POSNA developed a member-driven process for driving quality improvements in pediatric orthopaedics: the POSNA Safe Surgery Program (PSSP). The PSSP aims to develop key quality metrics that members believe improve outcomes in pediatric orthopaedics. This paper aims to summarize the first year of implementing the PSSP quality metrics.

Methods: The POSNA Quality, Safety, and Value Initiative (QSVI) Council developed 20 PSSP quality metrics for five domains: sports medicine, trauma, spine, hip/lower extremity (LE), and hand/upper extremity (UE). The quality metrics were integrated into five online surveys (one per domain) and distributed to POSNA member orthopaedic centers across North America.

Results: Thirty-three POSNA member orthopaedic centers responded to at least one domain-specific survey. Spine had the highest response rate (88%), while hand/UE and hip/LE had the lowest (70% and 73%, respectively). Centers meeting each quality metric ranged from 65% to 92% in sports medicine, 62% to 100% in trauma, 79% to 100% in spine, 83% to 96% in hip/LE, and 83% to 100% in hand/UE. Large and very large centers, as well as specialized children’s hospitals, provided more detailed protocols and procedures, likely due to greater resources and specialization. There was nearly 100% agreement between centers on protocols with well-known, easy-to-follow checklists for fulfilling a quality metric.

Conclusions: The primary goal of the PSSP is to create internally developed, surgeon-driven quality metrics that determine high-quality care. By using these quality metrics and reports, we hope surgeons can gain institutional resources to drive improvements in their centers. In its first year, the PSSP demonstrated that these quality metrics can be successfully distributed and reviewed by POSNA members. Our future work will focus on expanding the PSSP to more pediatric orthopaedic centers, iteratively evaluating and modifying the metrics, and adding metrics for additional domains.

Level of Evidence: Level IV

Key Concepts

• Ranking of orthopaedic programs by external organizations is expanding to pediatric orthopaedics which will have important clinical and financial implications.
• Pediatric orthopaedists are rarely consulted in quality evaluations by external organizations; therefore, POSNA created a member-driven performance evaluation system—the POSNA Safe Surgery Program (PSSP)—to develop surgeon-driven quality metrics to improve high-quality care.
• Twenty PSSP quality metrics were developed by the POSNA Quality, Safety, and Value Initiative (QSVI) Council across five domains: sports medicine, trauma, spine, hip/lower extremity (LE), and hand/upper extremity (UE).
• Our first year distributing the PSSP quality metrics to POSNA member centers revealed (1) high engagement rates and (2) diverse ranges of quality metric fulfillment by centers.
• In the future, we will expand the PSSP to more pediatric orthopaedic centers, iteratively evaluate and modify the PSSP quality metrics, and develop metrics for additional domains.

Introduction

A national healthcare quality improvement campaign was launched in the United States after the 1998 Advisory Commission on Consumer Protection and Quality in the Health Care Industry. During this time period, the science of quality measurement was still in its infancy: quality metrics were not widely available for many care settings and clinical conditions; measurement initiatives failed to routinely involve healthcare professionals or patients; and metrics were frequently developed and promulgated by individual healthcare organizations, leading to competing measures and poor multi-stakeholder engagement.¹ It was during this campaign that the National Quality Forum (NQF) was established, which developed a standardized set of criteria for desirable characteristics of quality metrics.

Since this time, hospital rankings have become much more ubiquitous and widely utilized by the general population.² Such rankings are typically done by external organizations, including the U.S. News & World Report (USNWR), the National Surgical Quality Improvement Program (NSQIP), Healthgrades, and the Centers for Medicare and Medicaid Services (CMS). They are intended to improve patients’ quality of care, as well as provide them with the information needed to choose an optimal provider.^3,4 However, they also have additional consequences: besides affecting patients’ perceptions of providers’ and centers’ expertise, they have also been incorporated into orthopaedic reimbursement schemes, with higher quality centers receiving greater payments than lower quality centers.^5–8 Although pay-for-performance has not yet reached pediatric orthopaedics, external organizations do create quality metrics that affect patients’ perceptions of an institution and its providers’ competence. Given the important implications of these metrics, it is crucial that they reliably measure quality performance.

Discussions between the POSNA Quality, Safety, and Value Initiative (QSVI) Council, its Board of Directors, and its membership revealed several concerns with existing quality metrics. In particular, concerns were voiced about many national quality metrics not aligning with quality improvement objectives, not leading to improvements in care, and not meeting NQF standards (Appendix). Additionally, concerns were raised about the limited involvement of POSNA content experts in the development of metrics used by external organizations. Accordingly, members voiced support for the creation of quality metrics led by POSNA members who could apply their expertise in specific domains of pediatric orthopaedics to develop sound metrics informed by NQF standards.

Based on the feedback of POSNA members, the POSNA QSVI Council created a member-driven program to drive quality improvements in pediatric orthopaedics in a manner applicable to POSNA member centers of all sizes: the POSNA Safe Surgery Program (PSSP). The primary focus of the PSSP is to create internally developed, surgeon-driven quality metrics that determine high-quality care. In this paper, we aim to summarize the first year of experience implementing the PSSP quality metrics.

Materials and Methods

PSSP Quality Metrics

Informed by NQF standards, the POSNA QSVI Council developed 20 PSSP quality metrics for the five domains of (1) sports medicine, (2) trauma, (3) spine, (4) hip/lower extremity (LE), and (5) hand/upper extremity (UE) (Figure A1, Appendix, Table 1). The quality metrics for each of the five domains were initially developed by their respective QSVI Council Committee in sports medicine, trauma, spine, hip/LE, or hand/UE. Each committee was asked to create their own metrics to drive high-quality and safe orthopaedic care for their respective domain. After each committee developed their metrics, they were reviewed, edited, and endorsed by all QSVI Council Chairs, and secondarily reviewed by the POSNA Board of Directors.

PSSP Quality Metric Surveys

The 20 PSSP quality metrics were subsequently incorporated into five online surveys (one per domain; each included relevant domain-specific quality metrics). They were then electronically distributed to POSNA member orthopaedic centers across North America. The initial trial was limited to centers with physicians on the QSVI Council and Board of Directors. Physicians were asked whether or not their center fulfilled each quality metric (responding “yes” or “no”). Where applicable, they were then prompted to describe how their center met the metric in a free-text response and to submit any relevant protocols. The survey was distributed to a total of 35 academic and private centers, which were classified as being small (≤ 4), medium (5 to 8), large (9 to 14), or very large (≥ 15) based on their number of pediatric orthopaedists on site.

Data Analyses

From the first-year pilot, the response rates and ability of each center to fulfill quality metrics were reviewed and presented as proportions. Respondents’ free-text responses and submitted protocols were analyzed to understand the resources available at each center and whether certain metrics were biased towards particular facilities (e.g., those with more resources). If a pediatric orthopaedist responded more than once to a survey, or multiple pediatric orthopaedists from the same center responded to a survey, we included the most recent survey response in our analyses. We did this to control for overrepresentation of responses and capture the most up-to-date response from each institution.

Results

Overview

33 of the 35 invited POSNA member orthopaedic centers responded to at least one domain-specific survey (Table 2). The majority of centers were large (42%) or medium (30%) in size, with 9% of centers being classified as small. All centers were located in North America (including five in California; four in Texas; three in Ohio; two in New York, Pennsylvania, and Tennessee; one in Colorado, Connecticut, Delaware, Florida, Georgia, Kentucky, Massachusetts, Maryland, Michigan, Minnesota, North Carolina, Oregon, and Utah), with two being in Canada (one in British Columbia and Ontario each).

PSSP Quality Metric Survey

Among the domains surveyed, spine had the highest response rate (88%) while hand/UE and hip/LE had the lowest (70% and 73%, respectively). In sports medicine, 65% to 92% of centers were found to fulfill each quality metric, 62% to 100% in trauma, 79% to 100% in spine, 83% to 96% in hip/LE, and 83% to 100% in hand/UE (Figure 1). Detailed technical protocols and procedures were more common in large and very large centers as well as specialized children’s hospitals, likely due to greater resource availability and specialization.

Free-text Responses and Submitted Protocols

Oftentimes, the free-text responses were more detailed and specific when a pediatric orthopaedist specializing in the domain of interest completed the survey (e.g., a fellowship-trained pediatric spine surgeon completed the spine survey). Specific protocols illustrating how a center met a particular quality metric were more commonly submitted by large and very large centers. Smaller centers, by contrast, tended to report having contracts with affiliated adult hospitals or transfer agreements with larger regional hospitals to meet their quality goals. Interestingly, centers had nearly 100% consensus on protocols when they had well-known, easy-to-follow checklists available for fulfilling a quality metric. In areas where no such consensus protocols exist, there was increased variation, creativity of solutions, and difficulty meeting quality metrics. In a similar vein, if large registries were available in a certain domain, the quality improvement metric was more likely to be fulfilled (examples of checklists and registries are outlined in Table 3).

Sports Medicine

Twenty-eight respondents from 26 centers completed the sports medicine-specific survey, which included four quality metrics on VTE prophylaxis pathways, multi-modal pain management protocols, return to play guidelines, and institutional contributions to quality improvement initiatives or registries (Table A1, Appendix).

Around half of centers reported having pre-determined guidelines for VTE prophylaxis. The majority of centers had multi-modal pain management protocols, employing strategies such as orthopaedic, anesthesiology, and pain management team collaborations; pre- and postoperative pain evaluations; and pain management education on ice, elevation, and anti-inflammatory medication. Most centers made concerted efforts to minimize the use of narcotics. Almost all centers had return to play guidelines, with some having very detailed protocols including illustrations along with the guidelines for specific sports. Guidelines were also made available in patients’ charts in certain centers so that families and physical therapists could easily access them. The majority of centers contributed to quality improvement initiatives or registries, particularly morbidity and mortality (M&M) conferences, Sports Cohort Outcomes REgistry (SCORE), and Research in OsteoChondritis Dissecans of the Knee (ROCK).

Trauma

Twenty-nine respondents from 26 centers completed the trauma-specific survey, which included five quality metrics: minimizing after-hours trauma cases, managing dysvascular limbs and polytrauma, reviewing complications, verifying trauma-specific Continuing Medical Education (CME) training, and antibiotic protocols for open fractures (Table A2, Appendix).

All centers had a protocol for managing urgent trauma cases. Larger centers often used a “bump” or dedicated trauma room to treat these cases, whereas smaller centers tended to share time with adult trauma rooms or only reviewed cases delayed more than 24 hours. Most centers also had plans in place for managing dysvascular limbs and polytrauma, including multi-disciplinary collaborations with on-call vascular surgery or plastic surgery teams and/or transfer agreements with other adult or tertiary care facilities. Complications were almost universally reviewed and discussed at M&M conferences. Finally, centers typically reported that they had formal protocols in place for treating open fractures with antibiotics. Most treated over 50% of patients with antibiotics within an hour of presentation though reported rates ranged from 24% to 100%. Among the quality metrics measured, centers had the most difficulty meeting trauma-specific CME requirements for physicians taking orthopaedic trauma calls, likely because this is only required at level one trauma centers.

Spine

Thirty-four respondents from 29 centers completed the spine-specific survey, which included four quality metrics on preoperative multi-disciplinary conferences, intraoperative protocols for surgical site infection control, consistent neuromonitoring in the operating room, and institutional participation in spine deformity quality dashboards (Table A3, Appendix).

Quality metrics were generally met in this domain (range from 79% to 100%), as there are available protocols that have been widely disseminated and accepted by POSNA as best practice, including for intraoperative surgical site infection control, neuromonitoring, and quality dashboards. The majority of centers with spine specialists also included the following in their preoperative plans: patient risk stratification, patient education classes, as well as conferences with nursing, physical therapy, and operating room staff.

Hip/Lower Extremity

Twenty-seven respondents from 24 centers completed the hip/LE-specific survey, which included four quality metrics on multi-disciplinary communications, protocols for timely access to care, institutional contributions to quality improvement initiatives, and VTE prophylaxis pathways (Table A4, Appendix).

The majority of centers conducted multi-disciplinary conferences of hip/LE patients with all team members to better prepare for upcoming surgeries. Additionally, some centers held case conferences with the adult orthopaedic team to discuss more complex deformity cases. In order to ensure timely access to care to address hip/LE differences, most centers had a scheduling algorithm so that patients could be seen within 1 week of referral. Centers typically had VTE prophylaxis pathways in place and contributed to quality improvement initiatives or registries, particularly via M&M conferences and Academic Network of Conservational Hip Outcomes Research (ANCHOR).

Hand/Upper Extremity

Twenty-seven respondents from 23 centers completed the hand/UE-specific survey, which included three quality metrics on replant/revascularization systems, comprehensive evaluations of congenital hand differences, and accessibility of hand therapists (Table A5, Appendix).

Like spine and hip/LE, quality metrics were generally met in this domain (range from 83% to 100%). Most centers had a replant/revascularization system in place, although not all had their own replant teams. Before surgery, almost all centers had methods for comprehensively evaluating patients with congenital hand differences. Every center reported having access to hand therapists; most reported having clinics dedicated to congenital hand disorders with therapists available onsite and referral options available to other subspecialties for conditions that may be genetic or hematologic in etiology.

Discussion

Overview

In this paper, we aimed to summarize the first year of experience implementing the PSSP quality metrics. We found relatively high rates of first-year engagement, with the vast majority of centers completing at least one domain-specific survey. Amongst the domains surveyed, spine had the highest response rate while hand/UE and hip/LE had the lowest. Centers meeting each metric ranged from 65% to 92% in sports medicine, 62% to 100% in trauma, 79% to 100% in spine, 83% to 96% in hip/LE, and 83% to 100% in hand/UE. We found near 100% agreement on protocols that had well-known, easy-to-follow checklists readily available for assessing a quality metric (such as those for antibiotic administration or neuromonitoring) which hospitals had in place prior to the implementation of PSSP quality metrics.

Future Clinical and Research Implications

Expansion of Centers

As the PSSP enters into its third year, work will be focused on expanding its reach to an increasingly diverse group of pediatric orthopaedic centers. We plan to distribute the quality metrics to all POSNA members, allowing us to evaluate them across centers of all sizes and various payor mixes. Our goal is to ensure that all pediatric orthopaedic centers as well as adult orthopaedic centers that include a pediatric practice can participate in the PSSP process if they wish to do so. The success of our first 2 years of piloting has already allowed us to expand our reach to a much larger number of centers, with the ultimate goal of expanding to all centers providing pediatric orthopaedic care across North America, including adult centers treating pediatric patients.

Iterative Evaluation and Improvement of Quality Metrics

We also aim to ensure that the metrics we develop are well-described, comprehensive, measurable, and easy-to-use. Using a feedforward system, we will assess metrics regularly, yielding feedback via surveys in order to improve their relevance and quality. Doing so is critical because inadequately designed metrics can result in confusion and poor managerial decisions that are at odds with the objectives of high-quality care.⁹ The metrics will evolve based upon member feedback, metric performance, and new research suggesting additional metrics that can be used to improve outcomes.

Development of Quality Metrics for Additional Domains

Furthermore, we aim to develop quality metrics for additional domains. An example of an additional domain is neuromuscular conditions, where possible quality metrics might include multi-disciplinary communications for patients with chronic neuromuscular differences, multi-modal pain management models, protocol(s) for neuromuscular hip surveillance, and soft tissue breakdown/ulceration management programs. Another example is the inclusion of a quality metric on the presence of child abuse teams and resources for the trauma-specific survey. As with the original 20 PSSP quality metrics, new metrics will be developed by the POSNA QSVI Council. Feedback for the development of new metrics from the entire POSNA community will be elicited.

Continuous Information Sharing

Building and fostering a culture of engagement and continuous improvement is crucial to improving the care provided to pediatric orthopaedic patients.¹⁰ Accordingly, the PSSP plans to disseminate resources and references that will enable POSNA members to advocate for changes within their institutions that will promote quality pediatric care. By creating a framework for quality improvement activities, we hope the PSSP will facilitate the development of quality improvement activities by individual centers. Additionally, we plan to disseminate personalized feedback to individual centers to illustrate their performance, setting 80% as a target for most quality metrics and 90% for high-performance metrics.

Strengths and Limitations

Strengths

The PSSP quality metrics are unique because they were developed primarily by content experts as opposed to metrics often used by external agencies. The PSSP allows POSNA members to take an active role in the quality assessment of children’s hospitals and pediatric orthopaedic programs by defining metrics that they consider crucial for providing exceptional routine and complex orthopaedic surgical care.¹¹ A strength of the current metrics has been the broad participation of POSNA committees, councils, and members in metric development and distribution. By creating relevant quality metrics considered to be central to the practice of POSNA members, we hope to provide patients, families, hospitals, and credentialing programs with impactful information to guide care in a truly meaningful way.

Furthermore, the free-text responses can be easily disseminated among the entire community. They can then be utilized as a roadmap by institutions striving to meet a certain metric. Just as a rising tide lifts all boats, we believe that cooperation throughout all of our centers across North America will improve pediatric orthopaedic care broadly for all of our patients.

Limitations

One of the potential criticisms of the PSSP is that it does not rank centers, unlike other external agencies. However, this is purposeful. We hope that our pediatric orthopaedic institutions utilize the PSSP as an impetus for their own quality improvement. Furthermore, the PSSP quality metrics are not static. The POSNA QSVI Council will continue to evaluate the PSSP quality metrics to ensure they are relevant to POSNA members and improve care for patients and their families. Feedback to improve the PSSP from the POSNA community is encouraged and welcomed. The POSNA QSVI Council appreciates the PSSP will require continuous maintenance and evaluation. Another reason for not ranking centers is the limitations for accessing the patient factors that contribute to risk of complications. Risk adjustment for patient populations is very difficult to do well and requires significant resources that are well beyond those available at most centers.

The ability to rapidly compile responses and give feedback to participants is another area of growth for the PSSP team. The POSNA QSVI Council is creating a robust database that is more nimble than the current platform. It will allow physicians to save their responses and return to them at a later time for completion, will flag multiple responses from the same institution, and will allow for more immediate feedback to the responding centers.

Conclusions

The primary goal of the PSSP is to create internally developed, surgeon-driven metrics that determine high-quality care. Surgeons can gain institutional resources and support by using the metrics and quality reports to drive quality improvement in their centers. In its first year, the PSSP demonstrated that these metrics can be successfully distributed and reviewed by POSNA members. Our future work will focus on expanding the PSSP to more pediatric orthopaedic centers, iteratively evaluating and modifying the metrics, and adding metrics for additional domains.

Additional Links

A sample of quality improvement checklists and registries is outlined in Table 3.

Disclaimer

The authors have no conflicts of interest to report related to this manuscript.

References

1. Burstin H, Leatherman S, Goldmann D. The evolution of healthcare quality measurement in the United States. J Intern Med. 2016;279(2):154-159.
2. Hanauer DA, Zheng K, Singer DC, et al. Public awareness, perception, and use of online physician rating sites. JAMA. 2014;311(7):734-735.
3. Berwick DM, James B, Coye MJ. Connections between quality measurement and improvement. Med Care. 2003;41(1 Suppl):I30-I38.
4. Teng K, Margolius D, Boulanger B. Why compensation tied to group quality metrics makes sense. Perm J. 2019;23:18.266.
5. Yaraghi N, Wang W, Gao GG, et al. How online quality ratings influence patients’ choice of medical providers: Controlled experimental survey study. J Med Internet Res. 2018;20(3):e99.
6. Huesch MD, Currid-Halkett E, Doctor JN. Public hospital quality report awareness: evidence from National and Californian internet searches and social media mentions, 2012. BMJ Open. 2014;4(3):e004417.
7. Bhattacharyya T, Freiberg AA, Mehta P, et al. Measuring the report card: the validity of pay-for-performance metrics in orthopedic surgery. Health Aff (Millwood). 2009;28(2):526-532.
8. Centers for Medicare & Medicaid Services. The Hospital Value-Based Purchasing (VBP) Program. Published 2021. Available at: https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/Value-Based-Programs/HVBP/Hospital-Value-Based-Purchasing. Accessed July 16, 2022.
9. Torabi E, Cayirli T, Froehle CM, et al. FASStR: A framework for ensuring high-quality operational metrics in health care. Am J Manag Care. 2020;26(6):e172-e178.
10. Azzolini E, Ricciardi W, Gray M. Healthcare organizational performance: why changing the culture really matters. Commentary. Ann Ist Super Sanita. 2018;54(1):6-8.
11. Shea K, McCarthy J, Brighton B. The History of POSNA Quality, Safety, Value Initiative Committee/Council – 2011-2020. JPOSNA^®. 2020;2(2). Available at: https://www.jposna.org/ojs/index.php/jposna/article/view/120. Accessed November 13, 2021.

Appendix