- Assessment Principles
- Search Strategy
- Assessment Process
- External Review Process
- Completed DAWG Reviews
The goal of the Drug Assessment Working Group (DAWG) of the Therapeutics Initiative is to clarify the state of scientific evidence regarding effectiveness and safety of drug therapy and to relate that evidence to the care of individual patients.
The DAWG consists of salaried employees (with expertise in systematic review methodology, clinical pharmacology, and epidemiology), Clinical Fellows in pharmacology, and graduate students from a range of medical disciplines. The DAWG systematically reviews and, when appropriate, critically appraises research relevant to new and existing drugs. The purpose of this document is to describe this systematic review process.
The DAWG follows a predefined protocol consisting of a research question, a literature search strategy, and a hierarchy of clinically validated outcome measures.
The research question asks whether the new drug provides a therapeutic advantage over similar drug therapy for a clinical condition (‘indication’) or versus placebo, in the case of the first drug therapy for a clinical condition. The DAWG relies on its own and external clinical experts both to define clinical conditions and to determine similar drug therapy.
The same DAWG members review all drugs. That is, the Working Group does not change as drugs change. And, all of the Working Group are expected to contribute to the design and conclusions of each systematic review, although individual members or groups take the lead for each project.
None of the DAWG members receives any funding directly or indirectly from the drug industry, nor do they or their families have any stock in any of these companies, except through unassigned mutual funds found in University pensions.
The Drug Assessment Working Group (DAWG) is committed to analyze scientific evidence on the effectiveness and safety of drug therapies used in Canada. They systematically review and, when appropriate, critically appraise research relevant to new and existing drugs. Often the goal is to discover whether a new drug provides a therapeutic advantage over existing similar drug therapy for a clinical condition. The Therapeutics Initiative has a dedicated team of researchers that is committed to the highest standards of research.
The drug assessment is guided by the following principles:
- Therapeutic advantage will be determined by utilizing a hierarchical framework. A hierarchical framework means that an assessment proceeds if, and only if, a series of conditions are met. If conditions are not met, then the assessment stops at that point. Therefore, the ordering of the hierarchy is crucial. The DAWG hierarchy:
- Type of study: The study must use the strongest possible design: almost always, the double and triple blind, randomized controlled trial.
- Type of participants: The study must include appropriate participants with an appropriate spectrum of disease from mild to severe; (i.e. a patient population found under normal conditions of clinical care).
- Type of intervention: The study must compare the new drug with an established drug with a similar mechanism of action; or with placebo, for drugs with a new mechanism of action. When the mechanism of action is unknown, comparators must have a similar clinical endpoint.
- Type of outcome measures: The study must use the most clinically valid outcome measures. If possible or practical, this should include final outcomes such as serious morbidity and mortality measures. Intermediate outcomes will be examined, but given less weight because they do not necessarily reflect ultimate morbidity or mortality.
- Generally, drug manufacturers have the obligation of demonstrating that their product provides therapeutic advantage over established therapies.
A predefined search strategy sets minimum inclusion and exclusion criteria based on major study design features listed above, such as patient selection and assignment, therapies chosen, appropriateness of outcome measures, and the degree of ‘blinding’ as to treatment allocation.
The search strategy only includes full clinical trial reports. Abstracts and poster presentations summarizing clinical trials are insufficient substitutes for full trial reports. Abstracts and poster presentations provide an outline of research design and a summary of major findings. However, they do not provide sufficient details to determine the validity/quality of the study or its findings. The latter requires analysis of how the trial was conducted and how the results were analysed.
The search includes trials submitted by the manufacturer and electronic data bases: MEDLINE 1966-present database, Current Contents 1996-present database, EMBASE 1988-present, and Cochrane database. In addition, retrieved clinical trials and some review articles are hand searched for references.
A ‘Search Findings’ section outlines all identified clinical trials and how they are categorized relative to the inclusion and exclusion criteria. Clinical trials and observational studies underway but not yet completed or published are also listed when submitted, but are not considered further in the systematic review unless full trial details are provided by the manufacturer.
The search strategy includes unpublished, full clinical trial reports submitted by the manufacturer to the Federal and Provincial regulatory and funding agencies, but considered proprietary. The TI considers this material in its systematic reviews. However, without publication, the Federal HPB, provincial Drug Benefit Plans, or review groups such as the TI make decisions regarding safety and efficacy, at least in part, on material that is not replicable or refutable. The TI believes that conducting and completing a clinical trial, but not submitting it for publication and providing full disclosure of the study report is unethical.
A full trial report in a peer reviewed journal is the preferred form of publication. Publication is sometimes delayed by journals. In these instances, copies of the articles as they will appear in publication are included in the drug assessment, along with proof of acceptance by the journal.
Another source of data are trials submitted for publication to a journal but refused publication. In this case the unpublished trial reports are given the full weight of a published trial, assuming reasonable proof of journal rejection is provided.
Clinical trial reports of trials comparing the correct treatments, using the best study design, and the most valid outcome measures are subjected to full critical appraisal. Critical appraisal means assessing the scientific validity of a study report following rigorous predefined criteria, most widely known through the Cochrane Collaboration publications. Critical appraisal methodology follows the work of Sackett et al,1 and is patterned after Schechter and LeBlanc.2
Each trial is appraised separately by at least two independent DAWG researchers. Discrepancies regarding assessments of the scientific validity of a clinical trial are resolved through consensus. Each drug assessment is discussed weekly by the full DAWG.
A concluding section of the report summarizes and draws conclusions from the body of available scientific evidence.
Approximately two-thirds of the DAWG reports are sent for external review. DAWG reports are not sent for external review if:
- the review is of a new formulation of an existing drug where the therapeutic advantage is limited to improved convenience and perhaps compliance with dosing regimen.
- the review is an update of a previously reviewed drug sent to reviewers, but no new published RCT evidence is found;
- the review is of a ‘me-too’ drug with an established mechanism of action and indication, and no valid published RCT evidence distinguishing the new drug from other members of a well established therapeutic category, such as beta-andrenergic blockers for hypertension or non-steroidal anti-inflammatory medications for osteo-arthritis.
For external review, the draft DAWG report, including the critical appraisal results, plus retrieved publications are sent to at least three experts (clinical or scientific) for assessment and comments. The reviewers are selected based on their knowledge of systematic review and critical appraisal methodology, their expert clinical or research knowledge of the disease studied, their ability to respond within a relatively short time-period (i.e. approximately 2 -3 weeks), and their willingness to declare any conflict of interest. Whenever possible, reviewers without conflicts of interest are chosen. Reviewers are asked, at minimum, to complete a checklist indicating the completeness of the review (any scientific evidence omitted) and whether they agree with the conclusions regarding critical appraisal.
The DAWG takes account of external reviews in the following ways:
1) New Citations are retrieved, and if they meet the inclusion criteria, are added to the review.
2) Internal Validity Issues. Of central concern to the DAWG, are any disagreements between it and external reviewers regarding the internal validity of the appraised clinical trials, particularly regarding key methodological steps such as randomization, the degree of intention to treat analysis, and clinical validity of outcome measures. These disagreements are discussed within the DAWG, and if the external reviewer’s comments are considered valid, the draft report is revised. If the DAWG does not consider the methodological issue raised by the external reviewer as valid, the external reviewer is contacted directly for discussion. If the issue remains un-resolved, both the DAWG and the external reviewer’s opinions are presented to the SIEC for consideration.
3) External Validity Issues. Interpreting external validity, meaning the extent to which research findings can be extrapolated to different contexts or individuals than those found in clinical trials, is not only difficult, but often the most difficult step in interpreting research findings, especially by individual clinicians. In part because of the inherent difficulty in establishing external validity, there are often disagreements between the DAWG and external reviewers regarding the interpretation of a therapeutic effect. The DAWG tends to be more conservative, while clinical reviewers are often more liberal in their degree of extrapolation to new individuals or contexts.
The DAWG recognizes that, although external validity issues are extremely important to individual clinicians, these concerns do not affect its mandate to establish the extent and strength of therapeutic effect in scientific evidence. Therefore, disagreements regarding external validity are not discussed directly with the external reviewers. Instead, both the DAWG and external reviewer’s opinions are presented to the Scientific Information and Education Committee (SIEC) for discussion.
The reports from the three experts are presented to the SIEC before final conclusions are made. An attempt is made to reach a consensus. The conclusions are voted on and, if passed by a simple majority, are accepted as the position of the TI regarding the scientific evidence. Individual members of the SIEC, who are in a conflict of interest position on a particular drug, must declare this and absent themselves from the relevant discussion and vote.
- Sackett DL, Haynes RB, Guyatt GH, Tugwell P. Clinical Epidemiology: a basic science for clinical medicine. 2nd re.ed. Boston: Little, Brown and Company, 1991.
- Schechter M, and Leblanc FE. Critical appraisal of published literature. In: W. Hanstroidle, O. Spitzer, B. McPeek, D.S. Mulder, and M.F. Mckneally editors, Principles and practice of research: strategies for surgical investigators. New York: Springer-Verlag, 1986:112-17.