Efficiency and contribution of strategies for finding randomized controlled trials: a case study from a systematic review on therapeutic interventions of chronic depression

Hand-searching contents of relevant journals

Hand-searching journals comprises the systematic manual or electronically page-by-page examination of the contents of a journal issue or conference proceedings to identify eligible reports of trials and is described as a useful adjunct to searching electronic databases.^1,2 Relevant journals can be identified through electronic database searches on the basis of identified relevant records. Journals that publish most of the potentially relevant articles found can be considered.

Citation tracking

Forward citation tracking comprises the systematic examination of all references that cite any of the studies already included in the analysis, e.g., by using the Science Citation Index.

Backward citation tracking describes the systematic examination of reference lists of studies already included.

Screening reference lists of related systematic reviews

Because systematic reviews represent the most convenient sources of references to potentially relevant studies, reference lists of included and excluded studies of previous or similar reviews on the topic of interest can be screened for eligibility.¹ Special sources to identify existing reviews are for instance the Cochrane Database of Systematic Reviews, the Database of Abstracts of Reviews of Effects, and the Health Technology Assessment Database. Since reviews are also often indexed with corresponding keywords, they can be identified in electronic databases such as MEDLINE or EMBASE as well.

Searching clinical trial registers

Clinical trial registers constitute electronic databases that contain clinical trials only. Various national and international clinical trial registers exist, such as ClinicalTrials.gov and the International Clinical Trials Registry Platform (ICTRP).

Contacting first authors of included studies

All first authors of primary studies that are already included in the systematic review are contacted with an inquiry for further published or unpublished trials.

Review eligibility criteria

Studies were included if they were conducted on adults with a diagnosis of chronic major depression, dysthymia, double depression, or recurrent depression without complete remission between episodes. Psychotherapeutic, pharmacological and combined interventions were considered. As comparator treatment no intervention, waitlist control groups, psychological or pharmacological placebos or other active interventions were regarded. The only studies included were RCTs that reported at least one outcome regarding the efficacy of the interventions. Further details on the review eligibility criteria are provided in the published study protocol.¹⁴

Electronic database search

First, a sensitive electronic database search targeting the occurrence of any of a large number of free-text terms at any point of the records was conducted in the following databases on January 18, 2010: CENTRAL, MEDLINE, EMBASE, ISI Web of Science, BIOSIS, PsycINFO and CINAHL. A disease component was combined (AND) with a design component focusing on RCTs for all searches. Within both components search items were combined with OR (e.g., MEDLINE): (((chron$ adj3 depress$) or dysthym$ or (double adj1 depress$) or (treatment adj1 resist$ adj1 depress$) or (non adj1 respon$ adj3 depress$) or (recurrent adj3 depress$)).ab,ti,sh.) AND ((random$ or rct).ab,ti. or random$.sh.).

Further detailed information regarding each particular electronic database search strategy is provided in Table 1.

We preferred a sensitive electronic database search in contrast to a less extensive electronic database search to identify preferably all existing evidence. In this context, sensitivity is defined as the ratio of the number of relevant reports identified to the total number of relevant reports in existence.¹ Within the electronic database search, no language restrictions were applied, and all publications, including conference and dissertation abstracts from 1970 forward, were considered.

Additional search strategies

The following additional search strategies were conducted subsequent to the sensitive electronic database search and are listed in the order of their application. Potentially relevant studies as well as the studies that could be included in our systematic review subsequent to a detailed examination for eligibility were added to those already identified through the electronic database search in the order of the application of the additional search strategies.

Statistical analysis

We defined two evaluation criteria to assess the benefits of the different search strategies: i) efficiency=a/b ; ii) contribution=a/c.

a=number of included studies identified by the search strategy

b=number of screened full-text articles identified by the search strategy

c=total number of included studies.

Accordingly, efficiency is defined as the ratio of included studies by the use of a certain search strategy to screened full texts identified through this search strategy. Contribution is defined as the ratio of included studies by the use of a certain search strategy to the total number of included studies in the systematic review.

Within our analyses records or references/papers were considered and counted first. Subsequently, the number of relevant studies was extracted (Figure 1).¹⁸ In order to judge the value of the studies that could be identified through additional search strategies for our systematic review, further analyses regarding the risk of bias were performed applying the Cochrane Risk of Bias Tool,¹ including the following criteria: appropriate generation of allocation sequence, appropriate allocation concealment, appropriate blinding of participants, appropriate handling with incomplete outcome data, no selective outcome reporting, no other systematic errors within the study (Table 2). The global risk of bias was judged to be low when at least four of the six Cochrane criteria were met and there were no further indications of a high risk of bias. When three or more of the Cochrane criteria were not met and at least one of the unmet criteria suggested serious risk of bias, the global risk of bias of the included study was judged to be high. Subsequently, the methodological quality of the included studies identified through the sensitive electronic database search and those identified through additional search strategies was compared.

The impact of the studies identified through additional search strategies on the total results of our systematic review was estimated by comparing the results of the meta-analyses with and without inclusion of these studies. Effectiveness measures including benefit ratios for response rates and odds ratios for dropout rates were calculated for three relevant comparisons: selective serotonin-reuptake inhibitors (SSRI) vs. placebo, tricyclic antidepressants (TCA) vs. placebo, and SSRI vs. TCA. Detailed information on statistical analysis of the regarding systematic review is provided in previous publications.^14,17

Results of the additional search strategies

Within all applied additional search strategies a total number of 27,007 references were identified and screened for potentially relevant studies. The detailed screening lead to 82 potentially relevant studies that were checked for eligibility (Figure 1). From the total number of 27,007 references identified through additional search strategies screening contents of relevant journals provided 19,076 references, forward citation tracking 2979, backward citation tracking 1692 and screening reference lists of related systematic reviews 1191 additional references that needed to be screened for relevance. Searching clinical trial registers and contacting first authors pro vided 2053 and 16 further references (Figure 1).

Efficiency

Out of all full-text articles identified through additional search strategies, 9.8% (8/82) could be included in our systematic review. Among the full-texts assessed through hand-searching contents of relevant journals and those identified through forward and backward citation tracking 6.1% (2/33) and 0.0% [(0/7); (0/15)], respectively, were included in our systematic review (Table 3). Through screening reference lists of related systematic reviews 31.3% (5/16) of all included studies could be identified. Searching clinical trial registers, did not lead to any further study inclusion, indicating an efficiency of 0.0% (0/2). One additional study identified by contacting all first authors of the previously included studies was found to be eligible for our systematic review through screening, which corresponds to an efficiency of 9.1% (1/11) (Table 3).

Contribution

Four per cent (2/50) of the included studies were obtained through hand-searching contents of relevant journals, whereas forward and backward citation tracking did not contribute to the total amount of included studies [0.0% (0/50)]. Screening reference lists of related systematic reviews had the highest contribution to the total amount of included studies [10.0% (5/50)]. Searching clinical trial registers provided no contribution to the total amount of included studies [0.0% (0/50)], but through contacting all first authors of the already included studies, another 2.0% (1/50) could be identified (Table 3).

Evaluation of the methodological quality of the studies

The global risk of bias was evaluated as low for a total number 11 studies, as unclear for 26 studies and 13 studies were judged to have a high risk of bias. Regarding the eight studies resulting from additional search strategies, four were judged to have a high risk of bias and the remaining four studies were evaluated to have an unclear overall risk of bias. Concerning the 42 studies that were identified through the electronic database search nine were evaluated to have a high risk of bias and a total number of 22 studies to have an unclear overall risk of bias. The remaining 11 studies resulting from the electronic database search were judged to have a low risk of bias. The results of the methodological quality assessment are presented for individual studies in Table 2.

Impact of the studies identified through additional search strategies

When comparing the results of the meta-analyses for the three comparisons (SSRI vs. placebo; TCA vs. placebo and SSRI vs. TCA) we found no systematic differences regarding the response and dropout rates after exclusion of the additionally identified studies. In the sequence of the above defined comparisons, response rates altered from 1.49 (1.29-1.99) to 1.49 (1.29-1.72), 1.74 (1.50-2.02) to 1.75 (1.48-2.07), and 1.01 (0.84-1.21) to 0.99 (0.83-1.17) after exclusion of the additional studies and dropout rates changed from 0.81 (0.49-1.33) to 0.83 (0.49-1.39), 1.14 (0.74-1.78) to 1.12 (0.79-1.59), and 0.41 (0.19-0.86) to 0.53 (0.29-0.96), respectively. For none of the comparisons statistical significance was influenced by the exclusion of the additional identified studies.

Limitations

Since our investigation is a single case study of a systematic review on interventions of chronic depression, the generalizability of our findings to other research topics may be limited. Above, the presented search strategies were applied in the research area of therapies that focus on chronic depression so that we could draw on a relatively large number of existing systematic reviews, 11 in total, that were rewarding for our analysis. Accordingly, in areas less researched, this beneficial additional search strategy cannot be applied.²³ Furthermore, we conducted a sensitive electronic database search which might have limited the gain of additional search strategies. In contrast, the use of only one database (e.g., MEDLINE) and a more restrictive search could have resulted in a significantly larger benefit through additional search strategies.

Another possible limitation of our findings may consist in the choice of the journals that were searched for relevant content since this selection may have influenced the number of identified studies through this strategy as well as the relative efficiency of other strategies conducted subsequently.

In addition, we did not consult an expert, e.g. a librarian, for designing the terms for the electronic database search, so we may have missed a few studies. However, it should be noted that several of the authors have extensive training and experience in performing systematic reviews and searches within them.

Above, the contribution of any additional search strategy strongly depends on the order of its application. In this case contents of relevant journals were searched at first. As we did not investigate any alternative sequence of additional search strategies and thus did not calculate the overlap between the applied additional search strategies, the generalizability of our findings may be limited. Although we applied many additional search strategies, the search for grey literature was limited. As the inclusion of unpublished trials in systematic reviews is a controversially discussed issue that is capable of both reducing and introducing bias, we decided not to search for additional unpublished trials.^1,24

Implications for practice

We found that the performance of additional search strategies, especially screening reference lists of related systematic reviews and hand-searching contents of relevant journals, uncovers a substantial amount of relevant RCTs in order to conduct systematic reviews of interventions in the context of chronic depression, even if a sensitive electronic database search is performed. Thus, database searches alone may miss a crucial amount (in this case 16%) of relevant data. On the other hand we found no effect of the additional identified studies on the overall results of our meta-analyses on the effectiveness on different antidepressants for chronic depression. Further, the additional identified studies were more often judged to have a high risk of bias. Thus, when planning a search strategy for a systematic review costs and benefits of additional search strategies need to be carefully weighted depending on specific requirements of the review that is to be undertaken, such as the relevance of identifying preferably all studies or the availability of resources for dealing with methodological weak studies.

Based on our findings, since a great amount of relevant trials can be identified through conducting a sensitive electronic database search, including a search in the database CENTRAL, this search strategy may be the first choice if resources are limited. Since in controversial research fields where the evaluation of a certain treatment strategy may depend on singly studies it is of high relevance to find as much evidence as possible. In this case, we recommend additional search strategies to ensure the completeness of existing evidence. Based on our case study results’ we recommend screening reference lists of related systematic reviews as well as contacting experts as additional search strategies. If resources allow for, hand-searching contents of relevant journals can be applied. Since screening reference lists of related systematic reviews is not exceedingly time-consuming, it is absolutely worthwhile to apply this search strategy to identify relevant trials, even if resources are limited and a pool of potentially relevant systematic reviews can be easily accessed.

Additional search strategies such as hand-searching may be necessary for the conduction of systematic reviews of interventions, especially in research areas with conflicting evidence, because automated electronic literature retrieval systems are likely to be prone to error, either due to technical reasons, due to database content limitations or due to users’ misapplication.

Implications for research

Because the use of a restricted electronic database search in comparison to a sensitive one, as performed in our approach, increases the risk of missing relevant literature right at the beginning of the search,⁴ further research is needed to estimate the benefits of additional search strategies after conducting a more restricted electronic database research. For example, more information is needed on sequence effects when performing different additional search strategies. This could provide crucial information about optimal search strategies in case of limited resources, as required within the field of rapid reviews. Above, to evaluate the benefit of additional search strategies, further research is needed to estimate the relative importance of trials according to their source of identification. The question, therefore, of whether trials reported in journals that are not indexed by electronic databases such as MEDLINE or EMBASE are of smaller power and thus less important for the results of rapid reviews needs to be addressed. These aspects can provide relevant information for the development of methodological standards in the process of conducting rapid reviews.