Jump to: Page Content, Site Navigation, Site Search,
You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.
Published 3 October 2008, doi:10.1136/bmj.a1607
Cite this as: BMJ 2008;337:a1607
Salomeh Keyhani, assistant professor1, Lawrence C Kleinman, associate professor1, Michael Rothschild, clinical professor2, Joseph M Bernstein, assistant professor3, Rebecca Anderson, former project manager1, Mark Chassin, president4
1 Department of Health Policy, Mount Sinai School of Medicine, One Gustave L Levy Place, Box 1077, New York, NY 10029, USA, 2 Departments of Otolaryngology and Pediatrics, Mount Sinai School of Medicine, 3 Departments of Otolaryngology and Pediatrics, New York University School of Medicine, New York, 4 The Joint Commission, Oakbrook Terrace, IL, USA
Correspondence to: S Keyhani Salomeh.Keyhani{at}mountsinai.org
Design Retrospective cohort study.
Setting New York metropolitan area practices associated with five diverse hospitals.
Participants 682 of 1046 children who received tympanostomy tubes in the five hospitals for whom charts from the hospital, primary care physician, and otolaryngologist could be accessed.
Results The mean age was 3.8 years. On average, children with acute otitis media had fewer than four infections in the year before surgery. Children with otitis media with effusion had less than 30 consecutive days of effusion at the time of surgery. Concordance with recommendations was very low: 30.3% (n=207) of all tympanostomies were concordant with the explicit criteria developed for this study and 7.5% (n=13) with the 1994 guideline from the American Academy of Pediatrics, American Academy of Family Medicine, and American Academy of Otolaryngology—Head and Neck Surgery. Children who had previously had tympanostomy tube surgery, who were having a concomitant procedure, or who had "at risk conditions" were more likely to be discordant.
Conclusions A significant majority of tympanostomy tube insertions in the largest and most populous metropolitan area in the United States were inappropriate according to the explicit criteria and not recommended according to both guidelines. Regardless of whether current practice represents a substantial overuse of surgery or the guidelines are overly restrictive, the persistent discrepancy between guidelines and practice cannot be good for children or for people interested in improving their health care.
Although many studies have assessed the degree to which the use of procedures in adults is concordant with guidelines,22 23 24 studies in children are less common. Only one study has examined the appropriateness of insertion of tympanostomy tubes in practice.4 5 This study, published in 1994, reported that less than half of surgeries among children in the United States were appropriate. Since then, several guidelines on the management of otitis media have been published.2 25 26 In this paper, we compare the clinical characteristics of the children in our study with the recommended indications for surgery as codified by the prevailing guideline at that time (the 1994 guideline on otitis media with effusion developed by the American Academy of Pediatrics, American Academy of Family Medicine, and American Academy of Otolaryngology—Head and Neck Surgery2 26) and a set of explicit criteria that we developed in 2000 specifically for this study as an update to the 1994 guidelines by using the RAND appropriateness method. To our knowledge, this is the first study to examine the appropriateness of insertion of tympanostomy tubes with data collected by independent audits of the records.
Clinical data abstracted from the chart included dates of service, otoscopic findings, previous history of otitis media, treatment with antibiotics, clinicians diagnosis of acute otitis media or otitis media with effusion, documented hearing loss, notation about the impact of otitis media on family life, and the presence of conditions that may be considered to put the child "at risk" for worse outcomes (autism, developmental delay, Downs syndrome, craniofacial syndromes that include cognitive, speech or language delay, or visual impairment).27 When hearing loss was present, we dichotomised it as mild (20-35 dB loss in the best ear) or moderate to severe (>35 dB loss in the best ear).27 For the rare (1%) children for whom the physician documented hearing loss but a formal assessment was absent from all records, we considered hearing loss to be present. Similarly, we considered the documentation of a parents or physicians assertion of speech or language delay without a formal assessment to be sufficient evidence, unless a subsequent assessment documented normal speech before surgery.27 We considered severe disruptions of family life to be present on the basis of documentation indicating missed school or work, a comment about an excessive number of physicians appointments, serious disturbances in the familys usual affairs, or considerable anxiety about the impact of ear disease any time in the three months leading up to surgery.27 We made two explicit assumptions to guide our interpretation of the data from medical records: unless otherwise documented, we postulated that otitis media with effusion persisted for 60 days after any documentation and that otoscopic findings did not return to normal for 28 days after acute otitis media. For example, we considered a child who had otitis media with effusion documented on day 1 and on day 50 to have had an effusion for 110 consecutive days (50 plus 60) if no other examinations were documented. We identified surgeries or procedures that were done concurrently with insertion of tympanostomy tubes from hospital administrative data.
Development of explicit criteria
The RAND appropriateness method uses a two round modified Delphi process to integrate literature with expert opinion into explicit criteria,28 in this case rating the appropriateness of tympanostomy tubes for children under 18 years old. We convened an expert panel of four otolaryngologists, four paediatricians, and one family physician and provided them with a detailed literature review. The panel identified relevant clinical factors that we organised into an exhaustive and mutually exclusive list of potential clinical scenarios to represent the range of circumstances that might present to a clinician. The panellists then rated each scenario on a scale of 1 to 9, with 1 meaning very inappropriate and 9 very appropriate (round 1). Appropriate is defined to mean that the likely benefits exceed the likely risks by a sufficient margin that the procedure is worth doing. The experts then met in a face to face meeting in March 2000. At this meeting, the scenarios were discussed and modified as needed and finally reassessed on the same nine point scale (round 2).
Even though the overwhelming majority of candidates for tubes present with acute otitis media or otitis media with effusion, the inclusion of detailed clinical factors such as age, duration and laterality of effusion, frequency of acute otitis media, extent of hearing loss, otoscopic findings, speech delay, and presence of significant disruption of family life resulted in 2268 permutations, each of which was rated as described above. The results of the second round are reported as the panels findings, with the median score representing the overall finding. We interpreted scores of 1, 2, and 3 as inappropriate; 7, 8, and 9 as appropriate; and 4, 5, and 6 as of equivocal or uncertain appropriateness. For this study, we considered three or more panellists rating a scenario 7-9 and three or more rating it 1-3 to represent significant disagreement and interpreted tubes for children who presented with these scenarios as of uncertain appropriateness.
The 1994 guideline was developed independently of this study by the three clinical societies and was published as a clinical practice guideline by the US Agency for Health Care Policy and Research.25 The 1994 guideline was limited to "healthy" children from their first birthday until before they turn 4 and suggests that insertion of a tympanostomy tube is optional after three months of persistent effusion with bilateral hearing loss and is recommended after four to six months of bilateral effusion.
Analysis
We mapped each child to the detailed clinical scenario rated by the panel that was consistent with the details of the clinical history. We mapped children with both acute otitis media and otitis media with effusion to two clinical scenarios, one for which acute otitis media predominates and one for which otitis media with effusion predominates. The scenario with the higher (more appropriate) rating yielded the appropriateness rating for that particular childs tympanostomy. We considered surgeries identified as appropriate or uncertain to be concordant with the explicit criteria; others were discordant.
We also compared practice with the 1994 academy guideline, which was the guideline in force at the time of our data collection. As no national guidelines on the surgical management of acute otitis media exist, we limited our analysis to children with otitis media with effusion. We considered surgeries that were either recommended or optional to be concordant with the academy guideline.
We also examined how alternatives to our expert panels judgment would affect the appropriateness ratings for the treatment of children with acute otitis media. Finally, we looked at the relation to appropriateness of the presence of conditions that would place a child at risk for poor outcomes, a history of tympanostomy tube surgery, and other procedures done at the time of tympanostomy tube surgery.5
We used
2 tests to examine differences in appropriateness ratings between hospitals and subpopulations of children. Statistical analyses used Stata Statistical Software version 9.2.
|
|
|
|
|
Cases classified with acute otitis media—Overall, 48% of the cases were not concordant with the explicit criteria developed for acute otitis media (table 5
). Low frequency of infection was the most common reason why the cases were not concordant with the explicit criteria. The expert panel believed that the benefit of delaying surgery until after a failure of antibiotic prophylaxis for recurrent acute otitis media outweighed concerns about the development of antimicrobial resistance. Evolving views on the use of antibiotics suggested that we should also present our analysis as if the expert panel had reversed its judgment,29 and we reanalysed the data considering frequently recurrent acute otitis media without a trial of prophylaxis to be an appropriate indication for insertion of a tympanostomy tube rather than an uncertain one. This increased the overall proportion of appropriate cases from 7% to 22.1%; the proportion concordant remained unchanged (table 5
).
Analysis based on academy guidelines
The 1994 guideline was concerned with healthy children aged 1-3 years with otitis media with effusion: 172 children in our sample met these criteria. Among these 172 cases, 7.5% of tympanostomy tube insertions were concordant with the guideline and 92.5% were not. If we expanded the sample to include all 533 healthy children older than 1 year, then 5.6% of tympanostomies were concordant with the guideline (table 5
). Again, the main reason for discordance with the guideline was short duration of effusion.
Additional analyses
In an additional analysis, we excluded all children with potentially extenuating circumstances, such as a history of previous tube insertion, another surgery/procedure at the time of tube insertion (for which the tubes do not carry a marginal risk of anaesthesia), and the presence of various conditions that would place a child at risk of a poor developmental outcomes from the sample. We found that the explicit criteria would consider 9.1% of the surgeries to be appropriate, 30.5% to be of uncertain appropriateness, and 60.4% to be inappropriate. Thus even in a liberal review, more than 60% of cases were not concordant with the panels findings (table 5
). Not surprisingly, concordance with the criteria also was lower for children in each of these three subpopulations than for the population as a whole (table 6
).
|
Implications of findings
The finding that 69% of cases deviated from the practices specifically developed for this study—or that more than 92% of surgeries would have been "not recommended" according to the guideline in force at the time of the surgery—suggests considerable overuse of this procedure. Our data suggest that children often receive tympanostomy tubes for effusions of short duration in a manner that is inconsistent with expert judgment. The experts who developed the academy guideline and the explicit criteria explicitly sought to balance the risks and benefits of the procedure. Historically, the major benefit of tympanostomy tubes discussed in the literature pertains to speech and language development. Recent research provides strong evidence that delay in the insertion of tympanostomy tubes is not associated with worse behavioural or developmental outcomes.30 31 However, these findings do not imply that tubes should be avoided or that there are not health systems for which tubes may be underused.
What if the major benefit of tubes is not in terms of promoting enhanced development but in terms of improving functional status or quality of life? Limited evidence shows that tubes improve disease specific quality of life.32 33 34 If tubes do make children feel better or otherwise improve the quality of their lives in the short term, then the emphasis on long term outcomes and development that has predominated in the guideline may not be sufficient.
Our data also show that otolaryngologists treat children differently if they have one or more of three specific circumstances—a history of previously having had tympanostomy tubes, the scheduling of a concomitant surgery, and the presence of one or more of the conditions we identified as putting a child "at risk." In these three situations, children receive tympanostomy tubes with less current disease. About half of our sample had one or more of these circumstances present. The extent to which these circumstances should be extenuating is questionable.
The assessment of the use of tubes in children with recurrent acute otitis media is not covered by clinical society guidelines, and the research evidence of efficacy is sparse. The explicit criteria developed by the panel of experts we convened generally indicate that tympanostomy tubes should be reserved for children with at least six episodes of recurrent acute otitis media in 12 months, who had had at least one infection that broke through antibiotic prophylaxis. The use of prophylactic antibiotics has fallen out of favour because of concerns about antibiotic resistance rather than controversy about its effectiveness. When we reanalysed the data, dropping the panels requirement for a failure of antibiotic prophylaxis, a substantial majority still failed to meet the standard for sufficient frequency and were still considered inappropriate.
Strengths and limitations
In this study, we focused only on children who had received tubes; we did not consider the possible underuse of tympanostomy in some populations. We restricted our analyses to those children for whom we had complete data (a strength), but our timing during implementation of a US federal privacy rule restricted our access to charts for about one third of children (a weakness). However, children with complete data were similar to those without complete data in terms of sociodemographic characteristics available in the hospital medical record.27 The finding that the insertion of tympanostomy tubes is often inappropriate is robust to the extent of missing data in our study. Even considering the most generous assumption that insertion of a tympanostomy tube was appropriate for every child with data missing, more than 40% of cases would still be considered inappropriate. Therefore, the missing data would not change the overall conclusions.
To make assessments about the course of otitis media we needed to translate the intermittent clinical assessments available from the charts into the continuous variables (days of effusion) that we used in our analysis. The need to impute findings is an unavoidable limitation; however, as we described in the methods, we made generous clinical assumptions that would favour a longer duration of effusion and concordance with the explicit criteria and the academy guideline. The data came from medical record notes with all the limitations therein. We recognise that variables such as "severe disruption of family life" that rely on a physicians notation of the impact of the disease on the family may not be regularly documented in the medical record. However, changes in the frequency with which this variable was observed would probably not alter the conclusions; the duration of effusion was too short in the vast majority of the cases.
A significant strength of this research is that we used expert recommendations from two sources to examine the concordance between practice and guidelines. To our knowledge this is the only study on appropriateness of tympanostomy tube insertion that has been done in the past decade.4 Although this study was local in nature, it focused on the most populous metropolitan area in the United States and is probably representative of other urban areas in the country. In addition, this study is the first to use data independently collected from community physicians medical records4; the cost of doing a national study of this scope would have been prohibitive.
Conclusions
Regardless of whether current practice represents a substantial overuse of surgery or the guidelines are overly restrictive, the persistent discrepancy between guidelines and clinical practice cannot be good either for children or for those interested in improving their health. Substantial overuse would expose children to risk and consume resources that could be better applied to otherwise improving the health of children. Erroneous guidelines could lead clinicians, policy makers, and researchers to ill advised interventions and undermine the value of guidelines in general. Given the ubiquity of this disease and its surgical treatment, resolution of these issues should represent an urgent priority. The UK experience may prove a useful resource for policy makers in the United States as they take on these challenges.
|
Cite this as: BMJ 2008;337:a1607
Contributors: MC and MR were responsible for the conception and design of the project. SK, MC, and RA were responsible for data collection. SK, LCK, MC, and RA were responsible for data analysis. All authors were involved in interpretation of the data. SK, LCK, MR, JMB, and MC were involved in drafting and revising the manuscript. SK is the guarantor.
Funding: Agency for Health Care Research and Quality (R01 HS 10302). The funding agency had no role in the design or implementation of the study; all views presented are the authors own and may not reflect the views of AHRQ.
Competing interests: MC was the principal investigator on this study before he left his position as chair of the Department of Health Policy to become the president of the Joint Commission.
Ethical approval: Institutional review boards of all five hospitals. This is a retrospective medical record review and no contact with patients was permitted.
This work was presented at the June 2007 and 2008 annual AcademyHealth meetings.
Provenance and peer review: Not commissioned; externally peer reviewed.
![]()
CiteULike
Complore
Connotea
Del.icio.us
Digg
Reddit
StumbleUpon
Technorati
Twitter What's this?
Read all Rapid Responses