Article Text
PDF
Abstract
Objective The aim of this exploratory study was to compare the performance of carotid artery stenting (CAS) best practices between Intersocietal Accreditation Commission (IAC) accredited facilities and non-accredited facilities certified by the Centers for Medicare and Medicaid Services (CMS).
Methods A random, anonymous survey was sent to CMS and IAC accredited facilities querying facility routine performance of 16 CAS procedure components found in published guidelines and utilised during clinical trials.
Results There were 28 responses (response rate=17%). Significant differences were found between the CMS and the IAC facilities for four of 16 procedure measures: determination of modified Rankin Scale score prior to stenting (p=0.012, 95% CI 20% to 80%), accurate measurement of per cent stenosis using electronic callipers (p=0.005, 95% CI 24% to 84%), confirmation of anticoagulation with activated clotting time greater than 250 s prior to crossing the lesion (p=0.03, 95% CI 7% to 69%), and comparison of facility outcomes to accepted benchmarks for stroke and death (p=0.03, 95% CI 7% to 69%). Overall, IAC facilities performed all 16 procedures more frequently (97%) than CMS facilities (66%) (p<0.001, 95% CI 24% to 36%).
Conclusions Although the sample size was small, the results demonstrated IAC accredited facilities are more likely to follow best practices, to use quantitative tools to select appropriate patients, and quantitively measure patient-centred clinical outcomes compared with CMS certified facilities. The findings raise the question as to the value of CMS certification versus IAC accreditation as a requirement for reimbursement.
- quality measurement
- clinical practice guidelines
- accreditation
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
In 2005, the Centers for Medicare and Medicaid Services (CMS) instituted coverage for carotid artery stenting (CAS) procedures for high-risk surgical patients. At that time, there were no recognised multispecialty organisations evaluating CAS procedures. Therefore, CMS established a certification mechanism for evaluating CAS facilities as a condition of reimbursement for CAS procedures.1
For initial CMS certification, a facility must meet structural criteria of either having participated in a clinical trial leading to Food and Drug Administration approval of a stenting device or self-attest that it meets minimum standards for equipment, device inventory, staffing and infrastructure. The certification duration is 2 years. To be recertified, facilities submit a log of CAS procedures performed. The log includes patient selection information related to high surgical risk criteria, symptomatic status, modified Rankin Scale (mRS) score if the patient had a history of stroke, per cent diameter stenosis of the artery, and whether there were complications during the hospital stay.
Since the establishment of CMS’ certification requirements, stakeholders from professional societies gathered to create standards and develop the Intersocietal Accreditation Commission (IAC) Carotid Artery Stenting accreditation programme based on published guidelines and expert opinion.2 The IAC Standards for Carotid Stenting Accreditation define minimum levels of quality based on both processes and outcomes related to the performance of CAS procedures.3 The IAC began accrediting CAS facilities in 2011.
The IAC CAS accreditation process requires submission of case logs that include clinical outcomes. IAC staff select a random sample of cases for detailed analysis of the relevant medical records and procedure images. Trained IAC staff also perform a mandatory site visit. The quality of facility operation is determined based on compliance with the Standards. Expert physicians on the IAC CAS Board review the submitted materials and site visit findings to make accreditation decisions. Major deficiencies such as failure to perform neurological assessments must be addressed before accreditation is awarded.
Whether there are differences in adherence to published guidelines and best practices between IAC accredited and CMS non-accredited CAS facilities is not known.2 Therefore, this exploratory survey was designed to assess and compare the performance of CAS best practices between IAC accredited and non-accredited facilities.
Methods
An anonymous survey was conducted in 2017 to compare CAS practices between IAC accredited and non-accredited facilities (online supplementary file A). Non-IAC accredited facilities were randomly selected from a pool of CMS certified facilities. The IAC and CMS facilities selected were mutually exclusive. The same survey was used for both groups, but different survey media was utilised due to differences in the contact information available.
Supplemental material
Centers for Medicare and Medicaid Services
A list of CMS certified facilities was obtained from the CMS.gov website.4 At the time the file was retrieved, there were 1366 certified facilities. The data available included facility name, address, provider number, and the effective date of certification.
The facility name and address were verified on the internet for a consecutive sample of 100 facilities to test the integrity of the list. Out of those 100 facilities, one facility was no longer in business. Thus, it was assumed that the information was incorrect for 1% of the list. Therefore, at least a 10% random sample plus 1% (n=152) was selected from the 1366 CMS certified facilities. The selection was made using a random number generator without replacement. A paper survey was mailed to the selected CMS facilities on at least three separate occasions in January, February and March 2017.
Intersocietal Accreditation Commission
The same survey was sent electronically to the technical directors of all accredited IAC facilities twice in August 2017; 2 weeks apart.
Survey
The survey consisted of a combination of dichotomous, checkbox, multiple-choice and free text questions. There were two categories of questions: demographic items (five) and procedure metrics (16). Respondents were asked to indicate if they routinely performed 16 procedure components (table 1). These components were selected from protocols found in the literature, required for CMS certification, utilised during CAS clinical trials, and included in documentation and procedure quality variables assessed in the IAC accreditation process.2 5 6
Performance of self-reported best practice procedure components
For categorical variables, the frequency and percentage were calculated. For continuous variables, the total, median and range were reported. A summed score and percentage by metric and group were calculated. Comparisons between groups were made using Fisher’s exact test. The significance level was set at 0.05, with the 95% CI reported. Statistical analyses were performed using IBM SPSS V.22.0.
As the list of CMS certified facilities included facilities that potentially might have participated in the Carotid Revascularization and Medical Management for Asymptomatic Carotid Stenosis Study Trial (CREST-2), a further subanalysis was performed to compare the best practices of CMS facilities that indicated participation in the CREST-2 and those that did not participate in CREST-2.7 This evaluation was done for exploratory purpose, and as there was only a small number of facilities, a statistical analysis was not performed. The frequency and percentage were reported for each group and overall.
A typical response rate for email surveys is 2%–5%, for generic postal surveys 7.5%, and for personalised postal surveys 10.5%.8 Therefore, a minimum response rate of 10.5% was established for all surveys due to the mixed media survey methods used.
The research protocol was reviewed by an institutional review board and deemed to be exempt as the survey was voluntary, anonymous, and did not involve the collection of protected health information. An explanation was provided at the beginning of the survey, and all respondents indicated their willingness to participate. As an incentive to participate in the survey, a $100 charitable donation was offered.
Patient and public involvement statement
Patients were not involved in the design or data collection for this study.
Results
Paper surveys were sent to 152 randomly selected CMS facilities, and electronic surveys of the same questions were sent to 6 IAC facilities. There were 28 responses (overall response rate=17%) to the survey (CMS=22 (15%) and IAC=6 (100%)).
Demographic data
Most of the responding facilities were medium-sized with between 100 and 500 beds (n=23, 82%). The rest of the facilities were almost equally distributed between small facilities, less than 100 beds (n=2, 7%), and large facilities, more than 500 beds (n=3, 11%). The median number of procedures performed per facility annually was 13 (range 3–90).
The median number of physicians performing CAS per facility was 2 (range 1–8). Most facilities had a combination of physician specialties performing procedures. Of the total reported, vascular surgeons (n=18, 36%) and interventional cardiologists (n=13, 26%) were the most frequent followed by interventional radiologists (n=10, 20%). Interventional neurologists (n=5, 10%) and neurosurgeons (n=4, 8%) represented the smallest numbers.
Most facilities participated in at least one registry or clinical study (n=20, 71%). The American College of Cardiology’s National Cardiovascular Data Registry-Peripheral Vascular Intervention (NCDR-PVI™) was the most common registry (n=10, 36%) followed by the CREST-2 registry (n=8, 29%). Facilities also participated in the following registries: the Society for Vascular Surgery Vascular Quality Initiative (SVS-VQI) (n=6, 21%); the Safety and Efficacy Study for Reverse Flow Used During Carotid Artery Stenting Procedure (ROADSTER) (n=2, 7%); and the Carotid Stent Clinical Study for the treatment of carotid artery stenosis in patients at increased risk for adverse events from carotid endarterectomy (SCAFFOLD) (n=3, 11%). Almost one-third of responding facilities (n=8, 29%) did not participate in registries. We found that CMS certified facilities were less likely to engage in registries (64%) compared with IAC facilities (100%), although the difference was not significant (p=0.14, 95% CI −6% to 57%).
Procedure metrics overall
Utilisation of an embolic protection device during the stenting procedure was most frequently utilised (n=26, 93%) by the facilities. Thirty-day post-stenting neurological assessment (n=24, 86%), 30-day mRS score (n=20, 71%), and 30-day National Institute of Health Stroke Scale (NIHSS) score (n=20, 71%) were also frequently performed (table 1).
The least performed metric was the determination of the degree of stenosis from the catheter angiogram using electronic callipers (n=13, 46%). Other less frequently performed metrics included mRS assessment prior to stenting (n=14, 50%), facility outcomes benchmarking for stroke and death (n=17, 61%), and non-invasive diagnostic imaging 30-days post-stent (n=19, 68%).
Overall, the average performance of the 16 procedure metrics by the respondent facilities was 73%.
Procedure metric comparison by accreditation status
Significant differences were found between the CMS and IAC facilities for four of the 16 procedure measures with the IAC accredited facilities reporting more frequent metric performance (table 1): determination of mRS score prior to stenting (p=0.012, 95% CI 20% to 80%), accurate measurement of per cent stenosis using electronic callipers (p=0.005, 95% CI 24% to 84%), confirmation of anticoagulation with activated clotting time greater than 250 s prior to crossing the lesion (p=0.03, 95% CI 7% to 69%), and comparison of facility outcomes to accepted benchmarks for stroke and death (p=0.03, 95% CI 7% to 69%).
Overall, IAC facilities performed all 16 metrics more frequently (97%) than CMS facilities (66%) (p<0.001, 95% CI 24% to 36%).
The exploratory analysis evaluating CMS facilities participating in the CREST-2 trial demonstrated that those facilities (n=3) utilised almost all of the 16 best practices when performing CAS with the exception of two practices (table 2): mRS pre-stent (67%) and 30-day non-invasive imaging (67%). The omitted best practices for the two variables represented different facilities.
Performance of self-reported best practice procedure components for CMS facilities (n=22)
Discussion
The use of CAS to reduce the risk of stroke and death among patients 70 years or older with carotid artery disease increased significantly from 2007 to 2014.9 The American College of Cardiology and American Heart Association guidelines recommend outcomes for periprocedural stroke or mortality of <6% for symptomatic patients and <3% for asymptomatic patients.10 However, favourable results of CAS are highly dependent on the incidence of periprocedural complications.11 Periprocedural events are contingent on an operator’s skill and experience, patient selection and procedure technique.
Multiple studies have documented the relationship between patient outcomes and competence of the operating team and facility.12–14 There is a steep learning curve for CAS with an inverse relationship between operator or facility volume and event rates. Operator and facility volumes were the most significant determinant of periprocedural outcomes.14 Recommendations for training criteria and volume requirements vary between professional societies.15 The experience and training of physicians performing routine CAS are not captured in the CMS data.13
CAS is associated with potential serious complications, and patients should not be subjected to that risk if they are not likely to benefit from the procedure.16 In general, symptomatic patients that have >50% carotid stenosis and asymptomatic patients with >80% carotid stenosis are suitable candidates for CAS.5 6 There is ongoing controversy about indications and ongoing research. However, outside of approved clinical trials, CMS limits reimbursement to symptomatic, high surgical risk patients with stenosis >70%.1
The intent of the CMS certification programme is to confirm that facilities receiving payment for CAS procedures are qualified to perform CAS and achieve acceptable clinical outcomes.7 Indeed, the CMS payment memorandum states “facilities and providers that routinely and repeatedly perform this procedure and follow patients for long periods of aftercare have a greater chance of successful outcomes.”1 However, the current structural focus of the CMS self-certification process does not require facilities to measure outcomes.1 A study by Epstein et al found that risk-standardised CAS outcomes fluctuate markedly across the USA; implying that facilities with both high and low adverse events are certified by CMS.17
The IAC Standards are based on published guidelines and expert consensus.3 They are designed to create best practices with the expectation that these practices will maximise CAS procedure quality and clinical outcomes. The Standards also provide metrics that can be used to measure clinical outcomes accurately. The results of this study show that IAC facilities are significantly more compliant with recommended best practices than non-IAC accredited facilities. It is reasonable to assume that this is likely to lead to better clinical outcomes and appropriately selected patients.
One of those best practices is electronic calliper measurement of per cent stenosis. In fact, the CMS memorandum for CAS states that if the degree of stenosis is less than 70% by angiography at the start of the procedure, CAS should not proceed. However, the memorandum does not suggest a method of ascertaining the degree of stenosis. Per cent stenosis may be determined by visual estimate or measured with electronic callipers. Determination of stenosis by electronic calliper has superior accuracy. Visual estimate of per cent stenosis has been demonstrated to misclassify and overestimate the degree of stenosis, especially in the presence of 50%–80% stenosis.18 Overestimation of stenosis may lead to unnecessary interventions. In this study, only 32% of CMS certified facilities indicated they used electronic callipers to determine per cent stenosis compared with 100% of IAC accredited facilities.
Assessment of peri-CAS and post-CAS stroke is necessary to determine clinical outcome for a procedure performed to prevent stroke. This requires both a follow-up visit and an objective measure of neurological deficit, such as the NIHSS. While major neurological deficits will be obvious, minor deficits may be overlooked but may still have a large impact on quality of life.19 This evaluation is critical in comparing outcomes to national benchmarks.
The findings from this study suggest the utility of accreditation in promoting adherence to best practices of patient selection and assessment of patient outcomes. The comparison of results from CMS facilities to IAC accredited facilities implies that CMS facilities may not necessarily comply with process measures such as the metrics assessed in this study despite CMS requirements. The results indicate it is helpful to have an external entity that audits and provides oversight to ensure best practices such as appropriate patient selection, assessment of clinical outcomes, and comparison of clinical outcomes to nationally accepted benchmarks of patient benefit.
Further, this exploratory examination of a limited number of CMS facilities that also participated in the CREST-2 trial, although notable but not statistically significant, showed they were for the most part very similar to the IAC accredited facilities. This likewise suggests that an external mechanism of accountability, whether in the form of accreditation or participation in a clinical trial, contributes to improving adherence to best practices to ensure quality patient care. Accreditation potentially allows the quality present during a clinical trial to be generalised outside of the clinical trial environment. This is an area for further investigation.
Limitations
This study is limited by the absence of actual outcomes determination and reporting by non-IAC accredited facilities. Although outcomes data might be available from clinical registries, many of the CMS facilities indicated they did not participate in registries. The study is further limited in that the selected best practices were also not verified directly but self-reported with likely response bias. However, compliance with the selected best practices by IAC facilities was confirmed at the time of the accreditation process site visit. It was assumed that the selected procedure best practices were a determinant of better patient clinical outcomes. It was also assumed that the IAC accreditation process was an accurate assessment of facility quality and best practice adherence.
The study is further limited by the small number of facilities that have sought specialty CAS accreditation by the IAC or the other accreditation organisation; Accreditation for Cardiovascular Excellence (ACE). ACE’s accreditation requirements are relatively similar to the IAC’s accreditation programme.20 The response rate for the survey was low but in line with response rates of other internet surveys.8 Finally, inherent selection bias is possible in that the type of individual likely to respond to a survey might also be more likely to adhere to performance guidelines.
Conclusions
Our evaluation of guideline adherence and best practices among facilities performing CAS procedures found that despite the small number of facilities, IAC accredited facilities are more likely to follow best practices compared with CMS certified facilities. The most relevant quality measures are appropriate patient selection and reduced risk of postprocedure stroke or death. IAC accredited facilities are more likely than CMS accredited facilities to use quantitative tools to select appropriate patients and quantitatively measure patient-centred clinical outcomes. The results raise the question as to the value of CMS certification as a requirement for reimbursement.
Footnotes
Contributors DS: conception, design, analysis, interpretation, manuscript drafting and revision, and final approval. MBF: conception, design, analysis, interpretation, manuscript drafting and revision, and final approval. BTK: conception, design, interpretation, manuscript revision, and final approval. ML: conception, design, manuscript revision, and final approval. JSM: analysis, interpretation, manuscript drafting and revision, and final approval. NM: conception, design, manuscript revision and final approval.
Funding The Intersocietal Accreditation Commission paid for employees MBF, ML and NM participation. Authors DS, BK and JSM are members of the Intersocietal Accreditation Commission Board of Directors.
Competing interests Authors MBF, ML and NM are employees of the Intersocietal Accreditation Commission. Authors DS, BK and JSM are members of the Intersocietal Accreditation Commission Carotid Stenting Board of Directors.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data are available upon reasonable request. All data relevant to the study are included in the article or uploaded as supplementary information.