Background: The enormous volume of Orthodontic literature available makes it a daunting task for specialists to summarise all available evidence. Hence the need, for Systematic and Narrative reviews of literature. To ensure that evidence from such reviews is reliable, valid and beneficial to the local population, there is a need for critical appraisal of the review process. This paper aims to inform Orthodontists on how to do a critical appraisal of in the field of Orthodontics.

Method: A critical appraisal tool, (CASP tool) the Critical Appraisal Skill Program (CASP), Public Health Resource Unit, and Institute of Health Oxford was used to appraise in broad terms the review process of Systematic and Narrative reviews as it applies to field of orthodontic literature.

Results: The CASP tool can be used for the critical appraisal of Systematic and Narrative review of literature in the field of orthodontics; allowing orthodontists to base their clinical decisions upon reliable and valid evidence beneficial for the intended local population.

With the advancement in the digital world today, more dentists are likely to rely on information available on the websites such as ‘Google’ and ‘Yahoo’ to update their knowledge. Clinical decisions are however more reliable if they are research-based. The practise of orthodontics is constantly changing, requiring evidence to prove the authenticity of clinical decisions.¹ Most specialists in orthodontists base their decisions on evidence available from journal publications and other print literature. The large volume of literature available in orthodontics connotes the need for it to be synthesised and reduced into a manageable format, aiming to identify all relevant research and to provide valid and reliable evidence. Hence, the need to review the process of arriving at such evidence. The integrity of any review is important, and should aim at minimising bias in reporting so as to avoid casting a doubt on the evidence produced.¹⁻⁴ Appraisals of systematic review process and that of other research papers is important so that reliable and valid evidence is produced, this is known as critical appraisal.⁴ Critical appraisal of the Systematic review process of orthodontic literature aim at promoting the best available evidence upon which clinicians can base their treatment decisions upon. The Conchrane collaboration is an international organisation that produces and distributes systematic reviews in the field of Orthodontics. Several topics that cover virtually all aspects of clinical orthodontics can be downloaded from the Conchrane library. The aim of this paper is to inform orthodontists on how to do a critical appraisal of systematic and narrative reviews of literature in the field of orthodontics.

A critical appraisal tool, (CASP tool) the Critical Appraisal Skill Program (CASP), Public Health Resource Unit, and Institute of Health Oxford1 was applied in general terms to appraise Systematic and Narrative reviews in the field of Orthodontics that were downloaded from the conchrane library.3 Generally speaking, CASP tool thrives on the basis of asking relevant questions to appraise the review process. Broadly speaking there are three questions to be asked when appraising systematic reviews: Question 1; what are the main results/evidence provided from the review and are they valid?; Question 2; Are the results/evidence provided from the review reliable? Question 3; would the results/evidence provided from the review be useful to the intended local population? Each broad question is further subdivided into smaller questions to guide the process.

Question 1: are the results /evidence provided from the review valid?

If the conclusion of the systematic review is to be reckoned with, its results must be valid and as such applicable to both local and general orthodontic population. The validity of the review may require combining results of several studies, taking into account the outcome of each study which needs to be clearly displayed using a data extraction form. Thereafter a meta-analysis of all similar results would be done, so that similarity and differences in results are explained. A meta‐analysis is the statistical combination of at least 2 studies to produce a single estimate of the effect of the health care intervention under consideration. The meta-analyses of the results contribute further to the validity of the systematic review. Further questions to guide the appraisal process are asked before making a conclusion.

Did the systematic review have a research question?

Firstly it is important to know if the review addressed a clear research question viz the population studied; intervention given; and outcome considered, Authors of orthodontic literature should ask clear and focused research questions. A good research question lays the foundation for any good research as it allows you to identify what you want to know as you start by ‘knowing less’ to ‘knowing more’.⁷ Having a vague research question often leads to an unfocused research. The research question must be asked from an appropriate research orthodontic population, this would lead to further questions;

Was the research population appropriate?

The research population studied should be appropriate; all papers included in the review should address the research question. For systematic reviews in orthodontics, it is appropriate to search papers from an orthodontic research population such as the Conchrane collaboration group⁴ as well as orthodontic literature and journals. Studies where there have been some form of intervention should be reported as Evidence Based (EBD) examples are Randomised Clinical Trials or/and Prospective Clinical Trials studies (RCTs and PCCTs). Outcome of such studies should be well defined. However not all studies on the subject matter may be included in the review, so there must be a criteria for selection. This brings us to the next question.

What databases were searched and how was the study identified?

To collate all necessary studies for systematic reviews in orthodontics, a wide range of databases should be searched such as MEDLINE, EMBASE, Cochrane databases of systematic reviews, Cochrane central register of controlled trials. These are databases for health sciences; hence searching them would be appropriate since orthodontics is considered as health science. The importance of databases in health care is to allow useful information about management and treatment of diseases to be shared by specialists as well as collection of relevant and useful information in an organised manner. It also useful to do manual and hand searches of grey (unpublished literature), scientific abstracts and conference proceedings ,coupled with manual searching of the reference lists of articles that were not identified by electronic search, authors of published abstracts and principal investigators of on-going trials may also be contacted for outcome on trials. This obviously allows maximum contribution to the number of relevant papers that can be retrieved. Putting it in lay terms, it is helpful to search for as much evidence as possible. In some cases, searching for non-English papers may contribute to the generation of a more robust literature, allowing for more evidence from a wider geographical representation of the population to be studied. This would allow a high number of original studies to be retrieved .Subsequently the selectivity of the papers and the rigorous scientific evidence required for systematic review studies allows for significant reduction in number of papers that would be eventually included in the review. The inclusion criteria determines the quality of the papers that will be reviewed, other factors that can introduce bias to the results such as the sample size, protocol and experience of orthodontists are then taken into consideration.

Further questions are asked;

Were the searches terms appropriate?

Search terms are important and can contribute to the selection of the right papers, they should include the keywords. For example, if the focus of the systematic review was to provide evidence on the best timing for orthodontic treatment, then search terms should include the following “ORTHODONTIC TREATMENT‘ ”AGE/ORTHODONTIC TREATMENT’ and TIMING/ORTHODONTIC TREATMENT’ used either as single term or as combination). What about the quality of studies reviewed?

Quality of studies reviewed

The reviewer can assess the quality of the included studies viz the scoring system (CASP) and by using more than one assessor? Usually two independent reviewers should assess the quality of the paper using a scoring scale of high, medium or low. Critical appraisal of the methodological quality of primary research is an essential feature of systematic review⁵ as lack of adherence to the defined validity criteria may be the reason why primary studies on the same topic may provide different results.⁶

Was there an exclusion and inclusion criteria for the studies selected?

The review should include the right type of study that addressed the research question and the appropriate study design? It is helpful if the research design of the systematic review had an inclusion and exclusion criteria of papers that were included in the review. For example we can decide that inclusion criteria would be mainly Randomised Control Trials (RCTs) and Prospective Controlled Clinical Trials (PCCTs) that had controls, so that studies without controls would be excluded. RCTs and PCCTs are evidence based studies that are appropriate for determining if a cause and effect relationship exists between treatment and outcome.

Question 2: what is the main result of the systematic review?

The main result of the review is the overall outcome that would be reported, for example in orthodontic literature, if best timing for Orthodontic treatment were to be reported, the main result of the systematic review should inform clinicians of both the physical and developmental age at which starting orthodontic treatment would be most beneficial. If the results of such review are to be accepted as the best evidence for the research topic, then there is the need to ensure results are not due to chance This is where statistical inference, such as point estimate 95% confidence interval and p-values come in, graphical representation such as forest plot can help to show the mean differences in results. The advantage of statistical inference is that results can be generalised to a larger population, for example the results from a systematic review on best timing for orthodontic treatment may be generally applicable to treatment timing in an orthodontic population. The number of primary studies eventually reviewed would determine if the statistical power of the results would be able to detect an effect. The higher the number of primary studies reviewed the more likely it is to have an increase in statistical power. The results of the systematic review should be clearly stated; successful interpretation of systematic review results allows for quality health care decisions and requires consideration of their quality and judgement. This is especially important if participants and interventions assessed are to be applied to a different setting.⁷ Lastly the question as to whether the review would be useful for a local orthodontic population should then be asked.

Question 3: will the results of the review be applicable to a local orthodontic population?

We would then need to find out the relevance of the results, can it be applied locally once the study population is defined, in this case all systemic reviews on the relevant orthodontic literature. It is most likely applicable worldwide to any orthodontic population since the principles of treatment are basically the same. Systematic reviews can provide evidence and answer specific questions especially where there are controversies, for example, timing of Orthodontic treatment can be controversial as there are different schools of thought as to the best timing for orthodontic treatment.⁸ The results and conclusion of the systematic review should be able to resolve such controversies. This would then be a major benefit for conducting such studies, the conclusion can then be provided as evidence to all other orthodontists as well as health policy makers. If the outcome of a systematic review in the field of orthodontics is significant, then it said to be beneficial to the intended population. The rigours associated with conducting such a review certainly outweigh the harms and costs involved. Further advantage is that results can then be applied to the general and local orthodontic population to make cost effective decisions in clinic orthodontic practise.

Critical appraisal of narrative reviews in orthodontics

A Narrative traditional review of literature uses an informal and unsystematic subjective method to collect and interpret information which is often summarised subjectively.⁹ Processes such as searching, quality appraisal and data synthesis are usually not described; therefore they are prone to bias. The main advantage of such reviews are that they are often conducted by ‘experts’ who usually have a thorough knowledge of the field of the research topic, while the disadvantage is that the author may have a preconceived notion or bias and may overestimate the value of some studies.¹⁰

Narrative reviews can be authoritative and informative since they depend on the subjective view of authors and can stimulate discussion and provoke controversy. Advantages of Narrative reviews lie in the fact that they are more readable, relevant and can make clear recommendation for practice. In a narrative review conclusions and recommendations can be based solely on opinion and not in relation to existing evidence, it is this subjective nature of a narrative that makes it more prone to bias. Since Narrative reviews do not describe processes such as searching, quality appraisal and data synthesis, a modification of the CASP (2010)¹¹ tool can be used to appraise the review and a review protocol can be developed by asking the following questions; was there a clear statement of the research aims?; was the methodology appropriate?; was the research design appropriate to address the aims of the research?; was the recruitment strategy appropriate to address the aims of the researcher; was the data collection appropriate for the research?;was the data analysis rigorously sufficient?; was there a clear statement about findings?; How valuable was the research?

Critical appraisal of narrative reviews of orthodontic literature should consider the following;

Was there a broad and over all research aim or question?

The broad research aim is usually a summary of available evidence on the research topic, while the overall review question aims to find out the cause and effect relationship observed within the research. The specific review questions should be clearly stated, to determine the validity of the results of the review, the eligibility criteria PICO (Population Intervention Comparison Outcome) should be used to appraise the review. Tables of comparison would allow papers from primary research to be critically appraised. Inclusion and exclusion criteria of studies to be included in the review should be clearly stated. Any intervention study should be selected for the review, as this would contribute to the quality of papers reviewed.

Data extraction form

A data extraction form is useful when conducting narrative reviews, relevant information, such as author and year of study, main focus of the study, sample size, methods, strength and limitations should be included in the data extraction form. The importance of validity in research is that it allows research findings to be generalised to the population. The study design of the selected papers for the review were examined and revealed that despite a substantial literature, there was very little qualitative research in the field of cleft. The data extraction form had relevant information, such as author and year of study, main focus of the study, sample size, methods, strength and limitations.

Search strategy

The search strategy should be clearly stated, this contributes to the selection of relevant literature for the review. When appraising reviews, it is important to include relevant studies that will help answer the research question, hence the need to search several electronic bibliographic databases in the field of health sciences. In addition, manual and hand searches of grey (unpublished literature), scientific abstracts and conference proceedings, coupled with manual searching of the reference lists of articles that were not identified by electronic search. The keyword used for the search is important and helps to retrieve relevant papers for the review.

Exclusion and Inclusion criteria

The inclusion and exclusion criteria for papers to be reviewed should be clearly stated, for example inclusion and exclusion criteria may include names of author and year of publication.

The results of the narrative reviews should be clearly presented, comparison tables help to highlight results of different authors and their conclusion. The review should be able to summarise findings and provide the best evidence on the subject matter and identify the gap in literature.

None.

The author declares that there is no conflict of interest.

1. Critical Appraisal Skill Program (CASP) Public Health Resource Unit, Institute of Health, and Oxford; 2013.
2. CRANE Project Team 2009. Crane Database. Progress Report September 2007-October 2008. London: Royal College of Surgeons of England, UK.
3. CRD Report. Centre for reviews and Disseminations. New York: University of York; 2001.
4. Egger M, Smith GD, Schneider M, et al. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315:629−634.
5. Hedin A, Kallestal C. Knowledge‐based public health work. Part 2: Handbook for compilation of reviews on interventions in the field of public health. National Institute of Public Health. 2004.
6. Hill A, Spittlehouse C. Critical Appraisal Skill Programme. Oxford; 2001.
7. McNeil CK. Orthodontic procedures in the treatment of congenital cleft lip and palate. Dent Rec. 1950;70(5):126−132.
8. Mossey PA, Little J. Epidemiology of Oral Clefts: An International Perspective. In: Wyszynski DF, editor. Cleft Lip and Palate: From Origin to Treatment. Oxford University Press; 2002:127−158.
9. Moher D, Liberati A, Tetzlaff J, et al. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. Ann Intern Med. 2009;151(4):264−269.
10. Nelson P, Glenny AM, Kirk S, et al. Parents’ experiences of caring for a child with a cleftlip and/or palate: a review of the literature. Child: Care, Health Dev. 2012;38(1):6−20.
11. Georgiade NG, Latham RA. Maxillary arch alignment in bilateral cleft lip and palate infant, using the pinned coaxial screw appliance. Last Reconstruct Surg. 1975;56(1):52−60.
12. Eskild A, Oxman A, Magnus P, et al. Screening for toxoplasmosis in pregnancy: what is the evidence of reducing a health problem? J Med Screen. 1996;3(4):188−194.
13. Klassen TP, Jadad AR, Moher D. Guides for Reading and Interpreting Systematic Reviews. Getting Started. Arch Pediatr Adolesc Med. 1998;152(7):700−704.
14. Liberati A, Altman DG, Tet Zlaff J, et al. The PRISMA statement for reporting systematic reviews and meat analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6(7):e1000100.
15. Oxman AD, Cook DJ, Guyatt GH. Users' guides to the medical literature. VI. How to use an overview. Evidence-Based Medicine Working Group. JAMA. 1994;272(17):1367−1371.
16. Parahoo K. Nursing research, Principles, process and issues. Basingstoke: Macmillan Press Ltd. 1997.
17. Papadopoulos MA, Koumpridou EN, Vakalis ML. et al. Effectiveness of pre-surgical infant orthopedic treatment for cleft lip and palate patients: a systematic review and meta-analysis. Orthod Craniofac Res. 2002;15(4):207–236.
18. Payne G, Payne J. Key Concepts in Social Research. London: Sage; 2004.
19. Richardson WS, Detsky AS. Users' guides to the medical literature. VII. How to use a clinical decision analysis. A. Are the results of the study valid? Evidence-Based Medicine Working Group. JAMA. 1995;273(16):1292−1295.
20. Ross RB, McNamera MC . Effect of presurgical infant orthopaedics on facial esthetics in complete bilateral cleft lip and palate. Cleft Palate Craniofac J. 1994;31:68−73.
21. Undertaking Systematic Reviews of Research on Effectiveness. CRD’s Guidance for those carrying out or commissioning Reviews. CRD Report Number 4. 2nd edn, NHS Centre for Reviews and Dissemination, University of York; 2001.