Elena V. Syurina,1,2 Tobias Schulte in den Bäumen,2 Frans J.M. Feron,1 Angela Brand2
1Department of Social Medicine, School for Public Health and Primary Care (CAPHRI), Faculty of Health, Medicine and Life Sciences, Maastricht University, the Netherlands; 2Institute for Public Health Genomics, School for Public Health and Primary Care (CAPHRI), Cluster of Genetics and Cell Biology, Faculty of Health, Medicine and Life Sciences, Maastricht University, the Netherlands
Key words: public health genomics, genomics, translational research, public health, policy, legislation.
Acknowledgements: the authors thank the Public Health Genomics European Network (PHGEN) for commissioning work on evidence analysis in Public Health Genomics within the European Union. This work was supported by two grants of the European Commission (DG SANCO, Agreement PHGEN I No 2005313 and Agreement PHGEN II No 20081302).
The helpful comments of E. Ambrosino are gratefully acknowledged. J.A. Lal is acknowledged for the help in the initial development of the compilation of evidence.
Contributions: EVS, data initial search, methodology choice, data analysis and interpretation, manuscript drafting; TSB, data initial search, methodology choice advice, data assessment, initial text review; FJMF, article active reviewing, advice on methodology and data integration, final approval; AB, article initial idea and revision.
Conflict of interest: the authors report no conflict of interest.
©Copyright E.V. Syurina et al., 2012
Licensee PAGEPress, Italy
Journal of Public Health Research 2012; 1:e8
doi:10.4081/jphr.2012.e8
Significance for public health
The promises on improvements made by genomics for public health have been considerable in the last decade, however, as we see in practice only a relatively small number of promising innovations make their way into the health application. This can be possibly due to the ineffective mechanisms of translation of discoveries into public policy. The relevance of the present study roots from its ability to potentially reduce the time delay between basic science inventions and their use in the everyday Public Health practice.
In the realm of the Public Health policy-making nowadays there is a need to come up with a strategy that can be utilised for assessing and translating the new knowledge offered by the genome-based information as well as facilitate its translation into practice. And the present work can help identifying the possible weaknesses of the present legislative approach and suggest a framework to look at the policy-making in the field of Public Health Genomics.
AbstractBackground. In the last decades we have seen a constant growth in the fields of science related to the use of genome-based health information. However, there is a gap between basic science research and the Public Health everyday practice. For a successful introduction of genome-based technologies policy actions on the international level are needed. This work represents the initial stage of the PHGEN II (Public Health Genomics European Network II) project. In order to prepare a base for bridging genomics and Public Health, an inventory study of the existing legislative base dealing with controversies of genome-based knowledge was conducted. The work results in the mapping of the most and the least legislatively covered areas and some preliminary conclusions about the existing gaps.
|
Introduction
Problem statement
An exponential growth in the fields of science related to the use of gene and genome based health information for improving the health of the individuals can be noted. The initial boost to the fields was provided by the Human Genome project, the 13-year old initiative which goals, among others, were to identify all genes in human DNA and improve data analysis. Lately the focus of the research is shifting from the highly individual level of dealing with rare genetic diseases towards more improvements for the population in general through inclusion of more genomic components in Public Health practice. This leads to the development of Public Health Genetics, which later evolved into Public Health Genomics. The Public Health Genomics is defined as the responsible and effective translation of genome-based knowledge and technologies into public policy and health services for the benefit of the population health.1
Since the beginning of the decoding of the human genome the possible Ethical, Legal and Social implications were considered to be important. The consultative bodies in the developed countries on all levels (international, national, regional) paid a lot of attention to protection of the rights and the freedoms of individuals who undertake the genetic test, against possible misuse of genetic data such as genetic discrimination. The first case of genetic discrimination that resulted in legal actions was observed in the Unites States of America in 2010.2 Anticipation of such events resulted in 53 legal and advisory acts published by international organisations since 1990 to 2010 (www.humgen.org/int/) which are dealing with problems of information misuse and genetic discrimination.
Despite of the importance of the Ethical, Legal and Social considerations on genome-based information and technology development, a broader application of these technologies in Public Health around the world is potentially possible when the attention of the policy-makers focuses also on other important aspects of translational process. In the literature it is discussed that before the actual impact of these advances could be seen, the existing gap between gene discovery and the application of genomic information in Public Health should be bridged.3,4,5 And one of the potential ways of narrowing such gap would be a wider use of translational research methodologies.
When speaking about translational research the definition formulated by several National Institutes of Health (NIH) is widely used. Translational research is the process of applying ideas, insights and discoveries generated through basic scientific inquiry to the treatment or prevention of human disease.6 The goal of such research should be enhanced transition of the innovations from bench to bedside.7
Translational research is a continuing process and is perceived to consist of four important steps. The first stage implies the transfer of new understandings of disease mechanisms gained in the laboratory into the development of new methods for diagnosis, therapy, and prevention and their first testing in humans,8 the next step includes the translation of results from clinical studies into everyday clinical practice and health decision making,8 third step focuses on searching ways to move these findings into the daily care of patient(s),9 and finally the last phase attempts moving scientific knowledge into the public sector and thereby changing people’s everyday lives.9
Due to the fact that sometimes for personalised genomic interventions minimal clinical benefit is shown during initial steps of assessment of intervention (due to rareness of the disease or too personalised nature of the intervention), enthusiasm of the researchers fades. Instead, the attention is shifted to the new wave of promising therapeutic interventions.10
In order to optimise the process of transition of the genome-based innovation it is important to review the current approach to the organisation of Public Health services and the present practice of translational research should be challenged.11
The present article aims at discussing the present policy approach to the translation of genome-based technologies in Public Health through identification of gaps in existing international evidence in the field. In order to do so the overview of the list of international policy documents compiled during the mapping exercise at initial stage of Public Health Genomics European Network II Project will be presented. The collected evidence will be analysed according to the defined list of important areas for development of Public Health Genomics. Since policy-making is a continuous process rather than a pure list of policy areas, the Public Health Wheel will be used as a benchmark, representing tasks and processes in Public Health12 in order to systematically link genomics and Public Health. The results of this two-step analysis will be discussed, the identified gaps in the international policies will be mentioned, and conclusions about future policy actions in Public Health Genomics and Public Policymaking will be presented.
Significance for Public Health
The promises on improvements made by genomics for Public Health have been considerable in the last decade, however, as we see in practice only a relatively small number of promising innovations make their way into the health application. This can be possibly due to the ineffective mechanisms of translation of discoveries into public policy. The relevance of the present study roots from its ability to potentially reduce the time delay between basic science inventions and their use in the everyday Public Health practice.
In the realm of the Public Health policy-making nowadays there is a need to come up with a strategy that can be utilised for assessing and translating the new knowledge offered by the genome-based information as well as facilitate its translation into practice.13 The present work can help identifying the possible weaknesses of the present legislative approach and suggest a framework to look at the policy-making in the field of Public Health Genomics.
Study design and methodology
To start the process of reconsidering the vision of healthcare and initiating changes in the current Public Health approach, it is important to conduct research on the existing best practices. Addressing these needs, the Public Health Genomics European Network II (PHGEN II) was initiated with support of the European Commission to develop European Best Practice Guidelines for Quality Assurance, Provision and Use of Genome-based Information and Technologies in 2008. These guidelines will assist the EU Member States, Applicant and EFTA-EEA (European Free Trade Agreement – European Economic Area) countries with evidence-based guidance on the timely and responsible integration of genome-based information and technologies into healthcare for the benefit of population health. The project assembles 20 associate partners and 3 collaborating partners from 14 European countries, and convenes top European scientists from different areas linked to Public Health Genomics: Public Health, Public Policy, Law, Medical and Basic sciences, Ethics, Health Technology Assessment and Sociology.
Eligibility criteria
In order to develop the new best-practice guidelines, the inventory of existing evidence was needed. The decision was made within the project to focus on policy guidelines: rules or principles that provide guidance to appropriate behaviour, and thus include non-mandatory international policy documents giving suggestions rather than prescribing actions were selected. This was agreed to due to the aim of the project of constructing the first edition of European Best Practice Guidelines, thus the potentially similar documents were included.
Inclusion criteria were designed in order to achieve the widest coverage and to exclude inappropriate guidelines. It was agreed that the analysis would take the meta-level, including guidelines proposed by the international organisations or scientific and research groups. The attention was paid to the fact that the candidate documents need to touch important Public Health issues within the genomic sciences, to have a wide scope covering the issue from the different angles and to be flexible to possible future changes in the technology. The focus on the more genome-based, not just genetic, information, used in PHGEN II is a relatively new accomplishment, which had been previously promoted in Europe during the first phase of PHGEN (2006-2008). The guidelines adopted after January 1990 and until May 2010 were considered to be fulfilling the time-line criteria, set by the project. However, if two or more documents dealing with the same topic were issued by the same institution, only the latest one was included in the list of compiled evidence.
Information sources and data collection process
The search for the documents was conducted in three directions: expert databases, practice oriented, academic. The search process is depicted in the Table 1.
Results
The final compilation of the existing guidelines consists of 101 documents (January 2011) covering policies regarding genetic services in healthcare being one of the tasks of Public Health Genomics.
Study selection
After the search for documents, a consensus-based assessment and evaluation was done through the expert survey amongst the partners of the PHGEN II project. The compiled list of 150 guidelines was sent to the partners and they were asked to analyse the relevance of the documents according to 6 categories according to their experience and the vision of Public Health Genomics (high, high-medium, medium, medium-low, low relevance and non-relevance). Each category was assigned the numerical value from 5 to 0 (5 – high, 0 – non-relevant), inspired by the Likert scale.14,15 Participating partners were asked to bear the definition of Public Health Genomics in mind as a reference for the relevance assessment: i) is the document applicable to different genomic information and technologies; ii) does the document deal with a present or potential Public Health issue and iii) could information provided in the document be used for the new set of European Guidelines in PHG. After completion of the assessment form by the partners the mean relevance was calculated. The documents which were assessed as non-relevant by the partners were excluded from the final compilation of the guidelines. In addition, partners had an option of adding the documents they considered to be relevant. In such case the procedure was repeated for these documents as well. Based on the input of the experts the list was narrowed to 101 documents.
Summary measures and synthesis of results
Areas of coverage of the collected documents were identified using the list of important policy areas for successful introduction of genome-based technologies into Public Health by Haga and Willard.16 In their article the authors suggest dividing the important policy issues in genome sciences in 5 main categories (research, legal, economic, education and acceptance/implementation issues), which are later subdivided. The division and identification of the areas important for the development of Public Health Genomics is done taking into consideration the natural history of any genome application or advance from discovery into everyday practice and the international and US policy focus regarding the genomics. The division was chosen because of its substantial link to the policy-making process in Public Health Genomics and the in-depth consideration of the current legislation applicable in the field.
The assessments were collected, centrally assessed and allocated to the areas using the majority of the votes, in case of 2 or more dominant choices, several areas were added to the document. If the document was dealing with one group of issues and using the methods of the other, for instance looking at the research issues from the legal point of view, both categories were assigned to the document. In order to identify the gaps in a more reliable way, the mean eminence of the documents included in each area was calculated.
Public health policy is a complicated non-linear process, thus, in order to identify the existing policy gaps, it is important to not only analyse the coverage of the important issues, but also to look at the consequent processes in Public Health and its tasks. Thus it was also decided to analyse the collected evidence in accordance with the blueprint for Public Health Genomics, which depicts an image designed to develop or contribute to general knowledge of the impact of human genetic variation on health and disease.5 The blueprint for Public Health Genomics has the same outline as the Public Health Wheel, developed by the Institute of Medicine (IOM).12 The three core functions of Public Health were initially identified: assessment, policy development and assurance. Later these areas were expanded into the 10 essential Public Health tasks.17 The division of the Public Health Genomics tasks and their description can be found in Table 3.
The allocation of the documents according to the blueprint for Public Health Genomics was done in the similar manner to the previous stage of analysis concerning assessment of relevance of the documents.
On the completion of the two steps of the analysis the areas with the least coverage were identified based on the number of documents covering each task and on the medium relevance of the documents.
Analysis
We observed the following division of scope of action: 47 documents regulate the situation in the European Union, 49 guidelines are adopted on the wider international level and represent positions of such organisations such as the OECD, UN, WHO and others. In 6 cases country specific reports were added to the list because, despite their national focus, the text of the guidelines touched upon important issues and was general enough to be adopted on the international level (topics included genetics of common diseases, pharmacogenomics, tissue use and genetic test evaluation). Forty-three documents were agreed to be of high relevancy for the future development of the guidelines, 6 guidelines were considered to be of high/medium relevancy. Partners rated 42 documents out of 101 as ones of the medium relevancy, while 4 of medium/low relevancy.
The results of the first step of analysis (coverage of the important areas according to S.Haga) are presented in Figure 3 (the median relevance of the documents included in each criteria was assigned in terms of colour in accordance with the legend). From the presented chart we can see that the legal and research areas regarding genomic sciences are quite well covered, both by the number of the documents dealing with the respective issues and by the median relevance of the documents. On the other hand, economic, education, acceptance as well as implementation issues are not high on the policy agenda. The number of the documents is significantly less and the relevance of these documents is on average lower.
However, there is no homogeneity in coverage of the topics even within one group of issues. After the detailed topic-specific analysis of the documents it was found that the most covered topics are: protection of human subjects (including informed consent issues, and ethical considerations on genetic testing); genetic discrimination (including genetic discrimination at work place); privacy and confidentiality (including genetic counselling); provision of the necessary facilities (in respect of ensuring quality of the provided genetic tests and screenings); access to tools, data and research samples (in respect of protection of human samples and organization of biobanks)
Analysis showed that the following important policy issues were virtually not covered in the international guidelines: trade agreements (including defining the role of the Pharmaceutical Companies); reimbursement of health-care providers by insurers and governments; public adoption of genomic technology; behaviour modification in response to genomic results; cultural aspects (except the non-discrimination on the national base)
The majority of the collected evidence is dealing with the genetic screening and genetic testing, i.e. genetic services in healthcare, rather than the introduction of genome-based information and technologies into Public Health, i.e. Public Health Genomics including genetic services. Out of 102 documents 61 were dealing with genetic screening and testing, which was clearly stated in their title. The main concern of the majority of these guidelines was safety of the human subjects: physical and psychological. The importance of the protection and safety of the human subjects cannot be contested. However, from the broader Public Health Genomics perspective a situation in which focussing only on ethical genetic issues and concerns is drawing attention away from other important considerations should be avoided.
In the final list, 22 guidelines represent mainly the provision of medical services in general. These documents do not directly relate to provision of services including genetic/genomic information. Nevertheless, it was admitted by the experts that such guidelines can link the gap between basic science and public policy reality, since these documents deal with patient-centred healthcare and public participation, patient safety, Public Health innovations and property rights.
A lot of attention was paid to clinical validity and clinical utility of genetic screening and genetic testing of human subjects. In other words, many existing documents focus on the decision criteria which define utility and reliability of the proposed genetic intervention. In the present discussion in European health policy-making the main criteria for the policy-makers and decision-makers is the impact of genome-based information on the population health. However, such criteria can be contested in the future due to the current paradigm shift from (stratified) population utility and validity towards more personal utility and validity. Furthermore, one of the most important limitations of the existing evidence is the fact that they focus mainly on the innovations with (mono)genetic focus, leaving aside the high variety of other –omic domains: i.e. pharmacogenomics, epigenomics, proteomics.
In general, a substantial difference in speed can be observed between the development of genome-based basic sciences and Public Health policies. This can partially be explained by the difficulty to assess these health innovations, which are intended to be used in Public Health, and the time needed for taking the necessary actions. The present approach to the introduction of the new technologies into Public Health is based on the translation of one size fits all interventions through proving their positive impact on population health, i.e. to as many as possible citizens. However, the traditional tools for the assessment of technologies do not seem to work in the area of genome-based information and technologies anymore and need to be adopted,18 which can contribute to the slowing down the introduction of genome-based innovations into Public Health.
After the analysis of the collected policy evidence in Public Health Genomics according to the Public Health Wheel (Table 2), the initial hypothesis about the unequal distribution of evidence among the tasks and processes in Public Health could be reconfirmed. Although the priority is rightfully given to the safety and protection of the human subjects, some other potentially important considerations, such as reimbursement strategies, are missing.
The most covered area was Link to/Provide Care, which can be explained by a high number of documents dealing with the issues of protection of human subjects during the process of genetic screening/test provision. Furthermore, the tasks of Diagnose and Investigate together with Inform, Educate and Empower were frequently discussed. However, it should be noted that the medium relevance of the documents dealing with education and empowerment is lower than for the ones describing Link to/Provide Care and Diagnose and Investigate. For instance, in the Inform, Educate and Empower, a lot of attention was paid to the issues of education of the patients, rather than social Health Literacy.
Among the areas in which substantial lack of guidance exists are: mobilisation of community partnerships, which focuses on fostering collaboration between public and private agencies and constituent groups to promote effective and efficient communication and policy making on genome-based information and technologies; process of enforcement of the legal framework, promoting the enforcement of policies and standards enacted to ensure the appropriate use of genomic information the effectiveness, accessibility and quality of genetic tests and services; assurance of competent workforce (including the education and health literacy of health professionals) and evaluation of the translation process; evaluation of the translational process of genome-based information and technologies and its effectiveness, efficiency and accessibility. On the other hand evidence exists on the national levels about the quality of genetic tests and services. However, there is still no unified approach to it; monitoring health status to identify health problems within the community. There is some evidence on the national and international level.19 However, the level of integration of genome-based information into this evidence is very low. Out of 101 documents only 2 dealt with the issue of using the genome-based information and technologies for analyzing the health status of the people.
Discussion
The present article summarizes the results of a mapping of the existing guidelines and best practice recommendations in the field of Public Health Genomics on the international level, which had been carried out by the Public Health Genomics European Network (PHGEN) (www.phgen.eu). The analysis of the areas covered by these guidelines showed that the issue of the introduction of the genome-based information and technologies into the Public Health is unequally covered. There is a considerable lack of guidelines addressing all of the Public Health needs and perspectives. The main attention is paid nowadays to the ethical and legal aspects of the use of genetic services rather than to the translation of the prospective research inventions into Public Health. While privacy and confidentiality issues were mentioned in a vast majority of the analysed documents (which supports the view on protection of the subjects as highest priority), issues such as the allocation of funds, trade agreements, cost-effectiveness, reimbursement of health-care provision by insurers and governments, public adoption of genomic technology are barely covered by any of the extracted documents. Moreover, the existing guidelines focus on the genetic screening and genetic testing aspects of genomics, which represent only a small part of all the opportunities of the use of genome-based information.
Due to the fact that the present research is the first and only research, assessing policy evidence in the field of Public Health Genomics, the initial literature search was of a very broad nature and the assessment of the documents was done using consensus-based method. The initial data set used during the analysis was not included in the publication due to its size. Despite the relatively low number of the experts participating in the assessment (n=35), the study involved a multidisciplinary group of scientists from different fields linked to Public Health Genomics (basic science, Public Health, medicine, law etc.) thus having a wide area of topics covered. It is important to note the absence of coherency in the terminology (i.e. genetic and genomic), which was found during the assessment of the evidence, which could potentially influence the results.
The potential limitations of the study include, firstly, the absence of strict assessment criteria and the use of expert opinions and perceptions of the policy evidence in the field. Lately there is a lot of debate in the literature regarding the validity and reliability of consensus-based assessment for creation of the clinical guidelines in different domains. In this study the consensus-based assessment was not used for the clinical purposes.20 In the researched field of public policy-making in Healthcare, public assessment methods, including experts, are widely used for consultation purposes.21,22,23 Secondly, the research misses the qualities of the systematic literature review, due to the fact that the aim of the exercise was to provide initial mapping of the areas presently covered by the international policy evidence, rather than establishing the state of the art in the area of PHG. In the present article the combination of the heterogeneous team and high level of agreement in assessments given by the participants shows the value of such practice-oriented evaluation for the future discussion on the policy and regulation in the field of Public Health Genomics.
Based on the analysis of the collected evidence from the Public Health perspective and its tasks and processes, it was revealed that there is a considerable bias in the coverage. Because the process of Public Health policy-making is ongoing, complex and interconnected, such bias harms the evidence-base, quality and speed of the translational policy-making in the field of Public Health Genomics. In order to create an integrative model for the translation of the outcome of genome-based research into everyday Public Health practice a more combinatory approach with elements of Public Health and public policy is needed. The attention of policy-makers should cover different topics within the field of Public Health Genomics (including health literacy, insurance difficulties, trade and intellectual property issues etc.) and some approaches from the social and public policy sciences can be beneficial. One of the solutions might be cross-integration of the Public Health Wheel (covering the sequence of tasks of Public Health) and the blueprint for Public Health Genomics (identifying the area specific points of focus). Such combination can be used as a map of important areas that are to be cover legislatively in order to facilitate the assessment and translation of genome innovations in the Public Health.
More research is needed focussing not only on the translation of genome-based innovations into clinical practice, but also on the translation from bench to Public Health policy and back in order to secure the continuous feedback loop to ensure quality. We suggest that a more in-depth assessment of the quality of existing policy evidence should be executed followed by the research on the needs of different stakeholders that are to be incorporated in the policy decisions.
| Table 1. Information sources and the process of evidence search. |
| Table 2. Policy issues in the genome sciences. |
| Table 3. Division of tasks in Public Health Genomics (adapted from Beskow et al., 20015). |
| Figure 1. Division of the documents by their relevance to Public Health Genomics (n=101). |
| Figure 2. Study selection. Identification and selection of the documents. |
| Figure 3. Coverage of important areas of genomic sciences for healthcare by international policy evidence. |
| Figure 4. Division of the existing international policy evidence in Public Health Genomics, according to the major public health tasks. |
References
1. Burke W, Khoury MJ, Stewart A, Zimmern RL, Bellagio Group The path from genome-based research to population health: development of an international public health genomics network. Genet Med 2006;8:451-8. [CrossRef][PubMed]
2. US Hears First Case of Genetic Discrimination. 2010 [Cited 15 June 2010]; Available from: http://legal-news.ashbycohen.co.uk/ discrimination/us-hears-first-case-of-genetic-discrimination/
3. Brand A. Public health genomics-public health goes personalized? Eur J Public Health 2011;21:2-3. [CrossRef][PubMed]
4. Khoury MJ, Gwinn M, Burke W, Bowen S, Zimmern R. Will genomics widen or help heal the schism between medicine and public health? Am J Prev Med 2007;33:310-7. [CrossRef][PubMed]
5. Beskow LM, Khoury MJ, Baker TG, Thrasher JF. The integration of genomics into public health research, policy and practice in the United States. Community Genetics 2001;4:2-11. [CrossRef][PubMed]
6. [Anonymous]. Lost in clinical translation. Nat Med 2004;10:879. [CrossRef][PubMed]
7. Weinshilboum R, Wang L. Pharmacogenomics: bench to bedside. Nat Rev Drug Discov 2004;3:739-48. [CrossRef][PubMed]
8. Woolf SH. The meaning of translational research and why it matters. JAMA 2008;299:211-3. [CrossRef][PubMed]
9. Kon AA. The Clinical and Translational Science Award (CTSA) Consortium and the translational research model. Am J Bioeth 2008;8:58-60; discussion W1-3.
10. Marincola FM. Translational medicine: a two-way road. J Transl Med 2003;1:1. [CrossRef][PubMed]
11. Brand A, Schröder P, Brand H, Zimmern R. Getting ready for the future: integration of genomics into public health research, policy and practice in Europe and globally. Community Genet 2006;9:67-71. [CrossRef][PubMed]
12. Institute of Medicine of the National Academies. The Future of Public Health. 1988, National Academy Press, Washington, DC, USA.
13. Brand A, Brand H, Schulte in den Baumen T. The impact of genetics and genomics on public health. Eur J Hum Genet 2007;16:5-13. [CrossRef][PubMed]
14. Galbraith C, Ehrlich S, DeNoble A. Predicting Technology success: identifying key predictors and assessing expert evaluation for advanced technologies. J Technol Transfer 2006;31:673-84. [CrossRef]
15. Jones R, Flaherty EG, Binns HJ, Price LL, Slora E, Abney D, et al. Clinicians description of factors influencing their reporting of suspected child abuse: report of the child abuse reporting experience study research group. Pediatrics 2008;122:259-66. [CrossRef][PubMed]
16. Haga SB, Willard HF. Defining the spectrum of genome policy. Nat Rev Genet 2006;7:966-72. [CrossRef][PubMed]
17. Public Health in America. May 01, 2008. [Cited 05 November 2010]. Available from: http://www.health.gov/phfunctions/public.htm
18. in den Baumen TS. Genome-based knowledge and public health: the vision of tomorrow and the challenge of today. Eur J Public Health 2011;21:4-5. [CrossRef][PubMed]
19. Nicklas W. International harmonization of health monitoring. ILAR J 2008;49:338-46.[PubMed]
20. Pilling S. History, context, process, and rationale for the development of clinical guidelines. Psychol Psychother 2008;81(Pt 4):331-50. [CrossRef][PubMed]
21. Aurora RN, Morgenthaler TI. On the goodness of recommendations: the changing face of practice parameters. Sleep 2010;33:1273-6. [PubMed]
22. Van den Broucke S, Dargent G, Pletschette M. Development and assessment of criteria to select projects for funding in the EU health programme. Eur J Public Health. 2011 Jun 20 (Epub ahead of print). [PubMed]
23. Lahtinen E, Koskinen-Ollonqvist P, Rouvinen-Wilenius P, Tuominen P, Mittelmark MB. The development of quality criteria for research: a Finnish approach. Health promotion international 2005;20:306-15. [CrossRef][PubMed]