Compilation and indexing terms, Copyright 2016 Elsevier Inc.
PB - Springer Verlag PY - 2014 SN - 03029743 SP - 263-268 ST - Using ensemble of Bayesian classifying algorithms for medical systematic reviews T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) TI - Using ensemble of Bayesian classifying algorithms for medical systematic reviews UR - http://dx.doi.org/10.1007/978-3-319-06483-3_23 VL - 8436 LNAI ID - 1590 ER - TY - JOUR AB - BACKGROUND: There is increasing interest in innovative methods to carry out systematic reviews of complex interventions. Theory-based approaches, such as logic models, have been suggested as a means of providing additional insights beyond that obtained via conventional review methods. METHODS: This paper reports the use of an innovative method which combines systematic review processes with logic model techniques to synthesise a broad range of literature. The potential value of the model produced was explored with stakeholders. RESULTS: The review identified 295 papers that met the inclusion criteria. The papers consisted of 141 intervention studies and 154 non-intervention quantitative and qualitative articles. A logic model was systematically built from these studies. The model outlines interventions, short term outcomes, moderating and mediating factors and long term demand management outcomes and impacts. Interventions were grouped into typologies of practitioner education, process change, system change, and patient intervention. Short-term outcomes identified that may result from these interventions were changed physician or patient knowledge, beliefs or attitudes and also interventions related to changed doctor-patient interaction. A range of factors which may influence whether these outcomes lead to long term change were detailed. Demand management outcomes and intended impacts included content of referral, rate of referral, and doctor or patient satisfaction. CONCLUSIONS: The logic model details evidence and assumptions underpinning the complex pathway from interventions to demand management impact. The method offers a useful addition to systematic review methodologies. TRIAL REGISTRATION NUMBER: PROSPERO registration number: CRD42013004037. AU - Baxter, Susan K. AU - Blank, Lindsay AU - Woods, Helen Buckley AU - Payne, Nick AU - Rimmer, Melanie AU - Goyder, Elizabeth DA - 2014 DO - 10.1186/1471-2288-14-62 J2 - BMC Med Res Methodol KW - *Data Mining *Disease Management *Referral and Consultation Health Knowledge, Attitudes, Practice Humans Models, Theoretical Patient Satisfaction Physician-Patient Relations L1 - internal-pdf://3182432947/Baxter-2014-Using logic model methods in syste.pdf LA - eng PY - 2014 SN - 1471-2288 1471-2288 SP - 62 ST - Using logic model methods in systematic review synthesis: describing complex pathways in referral management interventions T2 - BMC medical research methodology TI - Using logic model methods in systematic review synthesis: describing complex pathways in referral management interventions UR - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4028001/pdf/1471-2288-14-62.pdf VL - 14 ID - 104 ER - TY - JOUR AU - Bekhuis, Tanja AU - Demner-Fushman, Dina DA - 2012 DP - Google Scholar IS - 3 L1 - internal-pdf://0238962600/Bekhuis-2012-Screening nonrandomized studies f.pdf PY - 2012 SP - 197-207 ST - Screening nonrandomized studies for medical systematic reviews T2 - Artificial intelligence in medicine TI - Screening nonrandomized studies for medical systematic reviews: a comparative study of classifiers UR - http://www.sciencedirect.com/science/article/pii/S0933365712000620 http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3393813/ https://secure.jbs.elsevierhealth.com/action/consumeSsoCookie?redirectUri=http%3A%2F%2Fwww.aiimjournal.com%2Faction%2FconsumeSharedSessionAction%3FSERVER%3DWZ6myaEXBLFhx%252B6Ws3Nrug%253D%253D%26MAID%3D%252B%252BdV6uYg46%252FLaBL1Y5Tbqw%253D%253D%26JSESSIONID%3DaaaIGiDfjivzKaJygmwDv%26ORIGIN%3D831040708%26RD%3DRD&acw=&utt= https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3393813/pdf/nihms378928.pdf VL - 55 Y2 - 2016/09/24/15:33:48 ID - 2324 ER - TY - JOUR AB - Preparing a systematic review can take hundreds of hours to complete, but the process of reconciling different results from multiple studies is the bedrock of evidence-based medicine. We introduce a two-step approach to automatically extract three facets - two entities (the agent and object) and the way in which the entities are compared (the endpoint) - from direct comparative sentences in full-text articles. The system does not require a user to predefine entities in advance and thus can be used in domains where entity recognition is difficult or unavailable. As with a systematic review, the tabular summary produced using the automatically extracted facets shows how experimental results differ between studies. Experiments were conducted using a collection of more than 2million sentences from three journals Diabetes, Carcinogenesis and Endocrinology and two machine learning algorithms, support vector machines (SVM) and a general linear model (GLM). Finf1/inf and accuracy measures for the SVM and GLM differed by only 0.01 across all three comparison facets in a randomly selected set of test sentences. The system achieved the best performance of 92% for objects, whereas the accuracy for both agent and endpoints was 73%. Finf1/inf scores were higher for objects (0.77) than for endpoints (0.51) or agents (0.47). A situated evaluation of Metformin, a drug to treat diabetes, showed system accuracy of 95%, 83% and 79% for the object, endpoint and agent respectively. The situated evaluation had higher Finf1/inf scores of 0.88, 0.64 and 0.62 for object, endpoint, and agent respectively. On average, only 5.31% of the sentences in a full-text article are direct comparisons, but the tabular summaries suggest that these sentences provide a rich source of currently underutilized information that can be used to accelerate the systematic review process and identify gaps where future research should be focused. 2015 Elsevier Inc.. AU - Blake, Catherine AU - Lucic, Ana DA - 2015 DO - 10.1016/j.jbi.2015.05.004 J2 - Journal of Biomedical Informatics KW - artificial intelligence data mining information retrieval Learning algorithms Learning systems Support Vector Machines L1 - internal-pdf://2379086070/Blake-2015-Automatic endpoint detection to sup.pdf N1 -Compilation and indexing terms, Copyright 2016 Elsevier Inc.
PY - 2015 SN - 15320464 SP - 42-56 ST - Automatic endpoint detection to support the systematic review process T2 - Journal of Biomedical Informatics TI - Automatic endpoint detection to support the systematic review process UR - http://dx.doi.org/10.1016/j.jbi.2015.05.004 http://ac.els-cdn.com/S1532046415000830/1-s2.0-S1532046415000830-main.pdf?_tid=300e5366-832e-11e6-b3d1-00000aacb35d&acdnat=1474814688_f53dfcb0bf9a79471087125e43c910d0 VL - 56 ID - 1408 ER - TY - CONF AB - In this paper, we describe the construction of a test collection for evaluating clinical information retrieval. The purpose of this test collection is to provide a basis for researchers to experiment with PECO-structured queries. Systematic reviews are used as a starting point for generating queries and relevance judgments. We give some details on the difficulties encountered in building this resource and report the results achieved by current state-of-the-art approaches. 2010 ACM. AU - Boudin, Florian AU - Nie, Jian-Yun AU - Dawes, Martin C3 - 4th International Workshop on Data and Text Mining in Biomedical Informatics, DTMBIO'10, Co-located with 19th International Conference on Information and Knowledge Management, CIKM'10, October 26, 2010 - October 30, 2010 DA - 2010 DO - 10.1145/1871871.1871882 KW - bioinformatics data mining information retrieval Knowledge management Natural language processing systems Standardization L1 - internal-pdf://2381148771/p57-boudin.pdf N1 -Compilation and indexing terms, Copyright 2016 Elsevier Inc.
PB - Association for Computing Machinery PY - 2010 SP - 57-60 ST - Deriving a test collection for clinical information retrieval from systematic reviews T3 - International Conference on Information and Knowledge Management, Proceedings TI - Deriving a test collection for clinical information retrieval from systematic reviews UR - http://dx.doi.org/10.1145/1871871.1871882 http://dl.acm.org/citation.cfm?doid=1871871.1871882 ID - 1413 ER - TY - CONF AB - High quality, cost-effective medical care requires consideration of the best available, most appropriate evidence in the care of each patient, a practice known as Evidence-based Medicine (EBM). EBM is dependent upon the wide availability and coverage of accurate, objective syntheses called evidence reports (also called systematic reviews). These are compiled by a time and resource-intensive process that is largely manual, and that has not taken advantage of many of the advances in information processing technologies that have assisted other textual domains. We propose a specific text-mining based pipeline to support the creation and updating of evidence reports that provides support for the literature collection, collation, and triage steps of the systematic review process. The pipeline includes a metasearch engine that covers both bibliographic databases and selected "grey" literature; a module that classifies articles according to study type; a module for grouping studies that are closely related (e.g. that derive from the same underlying clinical trial or same study cohort); and an automated system that ranks publications according to the likelihood that they will meet inclusion criteria for the report. The proposed pipeline will also enable groups performing systematic review to reuse tools and models created by other groups, and will provide a test-bed for further informatics research to develop improved tools in the future. Ultimately, this should increase the rate that high-quality systematic reviews and meta-analyses can be generated, accessed and utilized by clinicians, patients, care-givers, and policymakers, resulting in better and more cost-effective care. 2010 ACM. AU - Cohen, Aaron M. AU - Adams, Clive E. AU - Davis, John M. AU - Yu, Clement AU - Yu, Philip S. AU - Meng, Weiyi AU - Duggan, Lorna AU - McDonagh, Marian AU - Smalheiser, Neil R. C3 - 1st ACM International Health Informatics Symposium, IHI'10, November 11, 2010 - November 12, 2010 DA - 2010 DO - 10.1145/1882992.1883046 KW - automation Cost Benefit Analysis cost effectiveness data mining Data processing information retrieval Information services Pipelines L1 - internal-pdf://3130390643/p376-cohen.pdf N1 -Compilation and indexing terms, Copyright 2016 Elsevier Inc.
PB - Association for Computing Machinery PY - 2010 SP - 376-380 ST - Evidence-based medicine, the essential role of systematic reviews, and the need for automated text mining tools T3 - IHI'10 - Proceedings of the 1st ACM International Health Informatics Symposium TI - Evidence-based medicine, the essential role of systematic reviews, and the need for automated text mining tools UR - http://dx.doi.org/10.1145/1882992.1883046 http://dl.acm.org/citation.cfm?doid=1882992.1883046 ID - 1816 ER - TY - JOUR AU - Cohen, Aaron M. AU - Ambert, Kyle AU - McDonagh, Marian DA - 2009 DP - Google Scholar IS - 5 L1 - http://skynet.ohsu.edu/~cohenaa/cohen-crosstopic-jamia2009.pdf internal-pdf://2033505236/Cohen-2009-Cross-topic learning for work prior.pdf PY - 2009 SP - 690-704 ST - Cross-topic learning for work prioritization in systematic review creation and update T2 - Journal of the American Medical Informatics Association TI - Cross-topic learning for work prioritization in systematic review creation and update UR - http://jamia.oxfordjournals.org/content/16/5/690.short https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2744720/pdf/690.S1067502709001224.main.pdf VL - 16 Y2 - 2016/09/24/15:35:34 ID - 2338 ER - TY - JOUR AB - Objective: To determine whether automated classification of document citations can be useful in reducing the time spent by experts reviewing journal articles for inclusion in updating systematic reviews of drug class efficacy for treatment of disease. Design: A test collection was built using the annotated reference files from 15 systematic drug class reviews. A voting perceptron-based automated citation classification system was constructed to classify each article as containing highquality, drug class–specific evidence or not. Cross-validation experiments were performed to evaluate performance. Measurements: Precision, recall, and F-measure were evaluated at a range of sample weightings. Work saved over sampling at 95% recall was used as the measure of value to the review process. Results: A reduction in the number of articles needing manual review was found for 11 of the 15 drug review topics studied. For three of the topics, the reduction was 50% or greater. Conclusion: Automated document citation classification could be a useful tool in maintaining systematic reviews of the efficacy of drug therapy. Further work is needed to refine the classification system and determine the best manner to integrate the system into the production of systematic reviews. AU - Cohen, Aaron M. AU - Hersh, William R. AU - Peterson, K. AU - Yen, Po-Yin DA - 2006 DP - Google Scholar IS - 2 L1 - https://www.researchgate.net/profile/Po_Yin_Yen/publication/7413557_Reducing_Workload_in_Systematic_Review_Preparation_Using_Automated_Citation_Classification/links/54ac331e0cf23c69a2b778b2.pdf internal-pdf://3139294211/Cohen-2006-Reducing workload in systematic rev.pdf PY - 2006 SP - 206-219 ST - Reducing workload in systematic review preparation using automated citation classification T2 - Journal of the American Medical Informatics Association TI - Reducing workload in systematic review preparation using automated citation classification UR - http://jamia.oxfordjournals.org/content/13/2/206.short https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1447545/pdf/206.pdf VL - 13 Y2 - 2016/09/24/15:32:42 ID - 2315 ER - TY - JOUR AB - OBJECTIVE: For many literature review tasks, including systematic review (SR) and other aspects of evidence-based medicine, it is important to know whether an article describes a randomized controlled trial (RCT). Current manual annotation is not complete or flexible enough for the SR process. In this work, highly accurate machine learning predictive models were built that include confidence predictions of whether an article is an RCT. MATERIALS AND METHODS: The LibSVM classifier was used with forward selection of potential feature sets on a large human-related subset of MEDLINE to create a classification model requiring only the citation, abstract, and MeSH terms for each article. RESULTS: The model achieved an area under the receiver operating characteristic curve of 0.973 and mean squared error of 0.013 on the held out year 2011 data. Accurate confidence estimates were confirmed on a manually reviewed set of test articles. A second model not requiring MeSH terms was also created, and performs almost as well. DISCUSSION: Both models accurately rank and predict article RCT confidence. Using the model and the manually reviewed samples, it is estimated that about 8000 (3%) additional RCTs can be identified in MEDLINE, and that 5% of articles tagged as RCTs in Medline may not be identified. CONCLUSION: Retagging human-related studies with a continuously valued RCT confidence is potentially more useful for article ranking and review than a simple yes/no prediction. The automated RCT tagging tool should offer significant savings of time and effort during the process of writing SRs, and is a key component of a multistep text mining pipeline that we are building to streamline SR workflow. In addition, the model may be useful for identifying errors in MEDLINE publication types. The RCT confidence predictions described here have been made available to users as a web service with a user query form front end at: http://arrowsmith.psych.uic.edu/cgi-bin/arrowsmith_uic/RCT_Tagger.cgi. AU - Cohen, Aaron M. AU - Smalheiser, Neil R. AU - McDonagh, Marian S. AU - Yu, Clement AU - Adams, Clive E. AU - Davis, John M. AU - Yu, Philip S. DA - 2015/05//undefined DO - 10.1093/jamia/ocu025 IS - 3 J2 - J Am Med Inform Assoc KW - *Artificial Intelligence *Randomized Controlled Trials as Topic *Review Literature as Topic *Support Vector Machine Evidence-Based Medicine Humans information retrieval Information Storage and Retrieval/*methods Medline natural language processing Randomized Controlled Trials as Topic ROC Curve Support Vector Machines systematic reviews L1 - internal-pdf://1319041805/Cohen-2015-Automated confidence ranked classif.pdf LA - eng PY - 2015 SN - 1527-974X 1067-5027 SP - 707-717 ST - Automated confidence ranked classification of randomized controlled trial articles: an aid to evidence-based medicine T2 - Journal of the American Medical Informatics Association : JAMIA TI - Automated confidence ranked classification of randomized controlled trial articles: an aid to evidence-based medicine UR - http://jamia.oxfordjournals.org/content/jaminfo/22/3/707.full.pdf VL - 22 ID - 229 ER - TY - JOUR AB - Julian Elliott and colleagues discuss how the current inability to keep systematic reviews up-to-date hampers the translation of knowledge into action. They propose living systematic reviews as a contribution to evidence synthesis to enhance the accuracy and utility of health evidence. AU - Elliott, Julian H. AU - Turner, Tari AU - Clavisi, Ornella AU - Thomas, James AU - Higgins, Julian P. T. AU - Mavergames, Chris AU - Gruen, Russell L. DA - 2014/02/18/ DO - 10.1371/journal.pmed.1001603 DP - PLoS Journals IS - 2 J2 - PLOS Med KW - Data management Ecosystems Health Services Research Health systems strengthening Meta-analysis Research validity Statistical data Systematic reviews L1 - http://journals.plos.org/plosmedicine/article/asset?id=10.1371/journal.pmed.1001603.PDF internal-pdf://0620089368/Elliott-2014-Living Systematic Reviews_ An Eme.pdf PY - 2014 SN - 1549-1676 SP - e1001603 ST - Living Systematic Reviews T2 - PLOS Med TI - Living Systematic Reviews: An Emerging Opportunity to Narrow the Evidence-Practice Gap UR - http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1001603 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3928029/pdf/pmed.1001603.pdf VL - 11 Y2 - 2016/09/24/20:51:50 ID - 2504 ER - TY - CONF AB - Systematic Literature Review (SR) and Systematic Mappings (SM) are scientific literature review techniques that follow well-defined stages, according to a protocol previously elaborated. The goal is helping in finding evidence about a particular research topic and mapping a research area, respectively. Their steps are laborious and a computational support is essential to improve the quality of their conduction. Aiming to offer computational support to these types of reviews, the StArt (State of the Art through Systematic Review) tool was developed. Besides the expected functionalities, StArt generates studies score, uses information visualization and text mining techniques to facilitate the research area mapping and to identify the studies relevance. StArt has been developed through an incremental process by academics who adopt SR and SM. As the expectation is to have a tool that really aids the conduction of these types of reviews, new ideas are always investigated and make StArt different from other alternatives. Visualization and text mining techniques seems to be a powerful resource for facilitating data abstraction in the context of SRs and SMs, allowing the improvement of the review and the conclusions about it. AU - Fabbri, Sandra AU - Hernandes, Elis AU - Di Thommazo, Andre AU - Belgamo, Anderson AU - Zamboni, Augusto AU - Silva, Cleiton C3 - 14th International Conference on Enterprise Information Systems, ICEIS 2012, June 28, 2012 - July 1, 2012 DA - 2012 KW - data mining Flow visualization Information systems Mapping Research TOOLS visualization N1 -Compilation and indexing terms, Copyright 2016 Elsevier Inc.
PB - International Conference on Enterprise Information PY - 2012 SP - 36-45 ST - Managing Literature reviews information through visualization T3 - ICEIS 2012 - Proceedings of the 14th International Conference on Enterprise Information Systems TI - Managing Literature reviews information through visualization VL - 2 ISAS ID - 1026 ER - TY - CHAP A2 - Cordeiro, J. A2 - Maciaszek, L. A. A2 - Filipe, J. AB - Systematic Literature Review (SLR or SR) and Systematic Mapping (SM) are scientific literature review techniques that follow well-defined stages, according to a protocol previously elaborated. Besides systematizing the search for relevant studies, the SR predicts the organization and the analysis of the obtained results. However, the SR application is laborious because there are many steps to be followed. Aiming to offer computational support to SR and SM, the StArt (State of the Art through Systematic Review) tool was developed. Besides helping the steps of SR or SM, the StArt tool has implemented visualization and text mining techniques to support the conduction and the reporting of the SR or SM. A comparative analysis was carried out in relation to StArt and other similar tools. AU - Fabbri, Sandra AU - Hernandes, Elis AU - Di Thommazo, Andre AU - Belgamo, Anderson AU - Zamboni, Augusto AU - Silva, Cleiton L1 - internal-pdf://1433925857/chp%253A10.1007%252F978-3-642-40654-6_15.pdf PY - 2013 SN - 978-3-642-40654-6 978-3-642-40653-9 SP - 243-256 ST - Using Information Visualization and Text Mining to Facilitate the Conduction of Systematic Literature Reviews T2 - Enterprise Information Systems, Iceis 2012 TI - Using Information Visualization and Text Mining to Facilitate the Conduction of Systematic Literature Reviews VL - 141 ID - 2208 ER - TY - JOUR AB - Context: Systematic Literature Reviews (SLRs) are an important component to identify and aggregate research evidence from different empirical studies. Despite its relevance, most of the process is conducted manually, implying additional effort when the Selection Review task is performed and leading to reading all studies under analysis more than once. Objective: We propose an approach based on Visual Text Mining (VTM) techniques to assist the Selection Review task in SLR. It is implemented into a VTM tool (Revis), which is freely available for use. Method: We have selected and implemented appropriate visualization techniques into our approach and validated and demonstrated its usefulness in performing real SLRs. Results: The results have shown that employment of VTM techniques can successfully assist in the Selection Review task, speeding up the entire SLR process in comparison to the conventional approach. Conclusion: VTM techniques are valuable tools to be used in the context of selecting studies in the SLR process, prone to speed up some stages of SLRs. 2012 Elsevier B.V. All rights reserved. AU - Felizardo, Katia R. AU - Andery, Gabriel F. AU - Paulovich, Fernando V. AU - Minghim, Rosane AU - Maldonado, Jose C. DA - 2012 DO - 10.1016/j.infsof.2012.04.003 IS - 10 J2 - Information and Software Technology KW - Information systems software engineering L1 - internal-pdf://1119399015/Felizardo-2012-A visual analysis approach to v.pdf N1 -Compilation and indexing terms, Copyright 2016 Elsevier Inc.
PY - 2012 SN - 09505849 SP - 1079-1091 ST - A visual analysis approach to validate the selection review of primary studies in systematic reviews T2 - Information and Software Technology TI - A visual analysis approach to validate the selection review of primary studies in systematic reviews UR - http://dx.doi.org/10.1016/j.infsof.2012.04.003 http://ac.els-cdn.com.ezproxy.lib.vt.edu/S0950584912000742/1-s2.0-S0950584912000742-main.pdf?_tid=934d5512-8333-11e6-b584-00000aacb362&acdnat=1474817002_ee53571ece3b3276144ae465be9aa5bb VL - 54 ID - 1278 ER - TY - CONF AB - One of the activities associated with the systematic literature review (SLR) process is the selection of primary studies. When the researcher faces large volumes of primary studies to be analysed, the process used to select studies can be arduous, specially when the selection review activity is performed and all studies under analysis are read more than once. An experiment was conducted as a pilot test to compare the performance and accuracy of graduate students in conducting the selection review activity manually and using visual text mining (VTM) techniques. This paper describes a replication study that used the same experimental design and materials of the original experiment. The results have confirmed the outcomes of the original experiment, i.e., VTM is promising and can improve the performance of the selection review of primary studies. There is a positive relationship between the use of VTM techniques and the time spent to conduct the selection review activity. Copyright 2013 by Knowledge Systems Institute Graduate School. AU - Felizardo, Katia Romero AU - Barbosa, Ellen Francine AU - Maldonado, Jose Carlos C3 - 25th International Conference on Software Engineering and Knowledge Engineering, SEKE 2013, June 27, 2013 - June 29, 2013 DA - 2013 KW - data mining Design of experiments knowledge engineering software engineering Students N1 -Compilation and indexing terms, Copyright 2016 Elsevier Inc.
PB - Knowledge Systems Institute Graduate School PY - 2013 SN - 23259000 SP - 141-146 ST - A visual approach to validate the selection review of primary studies in systematic reviews: A replication study T3 - Proceedings of the International Conference on Software Engineering and Knowledge Engineering, SEKE TI - A visual approach to validate the selection review of primary studies in systematic reviews: A replication study VL - 2013-January ID - 1482 ER - TY - CONF AB - Context: In order to preserve the value of Systematic Reviews (SRs), they should be frequently updated considering new evidence that has been produced since the completion of the previous version of the reviews. However, the update of an SR is a time consuming, manual task. Thus, many SRs have not been updated as they should be and, therefore, they are currently outdated. Objective: The main contribution of this paper is to support the update of SRs. Method: We propose USR-VTM, an approach based on Visual Text Mining (VTM) techniques, to support selection of new evidence in the form of primary studies. We then present a tool, named Revis, which supports our approach. Finally, we evaluate our approach through a comparison of outcomes achieved using USR-VTM versus the traditional (manual) approach. Results: Our results show that USR-VTM increases the number of studies correctly included compared to the traditional approach. Conclusions: USR-VTM effectively supports the update of SRs. Copyright 2014 ACM. AU - Felizardo, Katia Romero AU - Nakagawa, Elisa Yumi AU - MacDonell, Stephen G. AU - Maldonado, Jose Carlos C3 - 18th International Conference on Evaluation and Assessment in Software Engineering, EASE 2014, May 12, 2014 - May 14, 2014 DA - 2014 DO - 10.1145/2601248.2601252 KW - data mining software engineering L1 - internal-pdf://1159126505/a4-felizardo.pdf N1 -Compilation and indexing terms, Copyright 2016 Elsevier Inc.
PB - Association for Computing Machinery PY - 2014 SP - Brunel-University ST - A visual analysis approach to update systematic reviews T3 - ACM International Conference Proceeding Series TI - A visual analysis approach to update systematic reviews UR - http://dx.doi.org/10.1145/2601248.2601252 http://dl.acm.org/citation.cfm?doid=2601248.2601252 ID - 1540 ER - TY - CONF AU - Felizardo, Katia R. AU - Salleh, Norsaremah AU - Martins, Rafael M. AU - Mendes, Emilia AU - MacDonell, Stephen G. AU - Maldonado, Jose C. C3 - 2011 International Symposium on Empirical Software Engineering and Measurement DA - 2011 DP - Google Scholar L1 - http://aut.researchgateway.ac.nz/bitstream/handle/10292/3470/Felizardo,%20Salleh,%20Martins,%20Mendes,%20MacDonell%20and%20Maldonado%20(2011)%20ESEM.pdf?sequence=2 PB - IEEE PY - 2011 SP - 77-86 ST - Using visual text mining to support the study selection activity in systematic literature reviews TI - Using visual text mining to support the study selection activity in systematic literature reviews UR - http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6092556 http://ieeexplore.ieee.org/document/6092556/ Y2 - 2016/09/24/16:06:07 ID - 2426 ER - TY - JOUR AB - OBJECTIVE: To determine whether the automatic classification of documents can be useful in systematic reviews on medical topics, and specifically if the performance of the automatic classification can be enhanced by using the particular protocol of questions employed by the human reviewers to create multiple classifiers. METHODS AND MATERIALS: The test collection is the data used in large-scale systematic review on the topic of the dissemination strategy of health care services for elderly people. From a group of 47,274 abstracts marked by human reviewers to be included in or excluded from further screening, we randomly selected 20,000 as a training set, with the remaining 27,274 becoming a separate test set. As a machine learning algorithm we used complement naive Bayes. We tested both a global classification method, where a single classifier is trained on instances of abstracts and their classification (i.e., included or excluded), and a novel per-question classification method that trains multiple classifiers for each abstract, exploiting the specific protocol (questions) of the systematic review. For the per-question method we tested four ways of combining the results of the classifiers trained for the individual questions. As evaluation measures, we calculated precision and recall for several settings of the two methods. It is most important not to exclude any relevant documents (i.e., to attain high recall for the class of interest) but also desirable to exclude most of the non-relevant documents (i.e., to attain high precision on the class of interest) in order to reduce human workload. RESULTS: For the global method, the highest recall was 67.8% and the highest precision was 37.9%. For the per-question method, the highest recall was 99.2%, and the highest precision was 63%. The human-machine workflow proposed in this paper achieved a recall value of 99.6%, and a precision value of 17.8%. CONCLUSION: The per-question method that combines classifiers following the specific protocol of the review leads to better results than the global method in terms of recall. Because neither method is efficient enough to classify abstracts reliably by itself, the technology should be applied in a semi-automatic way, with a human expert still involved. When the workflow includes one human expert and the trained automatic classifier, recall improves to an acceptable level, showing that automatic classification techniques can reduce the human workload in the process of building a systematic review. AU - Frunza, Oana AU - Inkpen, Diana AU - Matwin, Stan AU - Klement, William AU - O'Blenis, Peter DA - 2011/01//undefined DO - 10.1016/j.artmed.2010.10.005 IS - 1 J2 - Artif Intell Med KW - *Abstracting and Indexing as Topic *Artificial Intelligence *Bibliometrics *Databases, Bibliographic *Data Mining *Review Literature as Topic Aged Aged, 80 and over Algorithms Evidence-Based Medicine Health Services for the Aged Humans Pattern Recognition, Automated Publications/*classification Workflow L1 - internal-pdf://4204148803/Frunza-2011-Exploiting the systematic review p.pdf LA - eng PY - 2011 SN - 1873-2860 0933-3657 SP - 17-25 ST - Exploiting the systematic review protocol for classification of medical abstracts T2 - Artificial intelligence in medicine TI - Exploiting the systematic review protocol for classification of medical abstracts UR - http://ac.els-cdn.com.ezproxy.lib.vt.edu/S0933365710001247/1-s2.0-S0933365710001247-main.pdf?_tid=96876766-8335-11e6-a702-00000aacb360&acdnat=1474817866_f8561bb67a0e758daf8dbeec3cf2c28a VL - 51 ID - 184 ER - TY - JOUR AB - Medical systematic reviews answer particular questions within a very specific domain of expertise by selecting and analysing the current pertinent literature. As part of this process, the phase of screening articles usually requires a long time and significant effort as it involves a group of domain experts evaluating thousands of articles in order to find the relevant instances. Our goal is to support this process through automatic tools. There is a recent trend of applying text classification methods to semi-automate the screening phase by providing decision support to the group of experts, hence helping reduce the required time and effort. In this work, we contribute to this line of work by performing a comprehensive set of text classification experiments on a corpus resulting from an actual systematic review in the area of Internet-Based Randomised Controlled Trials. These experiments involved applying multiple machine learning algorithms combined with several feature selection techniques to different parts of the articles (i.e., titles, abstract, or both). Results are generally positive in terms of overall precision and recall measurements, reaching values of up to 84%. It is also revealing in terms of how using only article titles provides virtually as good results as when adding article abstracts. Based on the positive results, it is clear that text classification can support the screening stage of medical systematic reviews. However, selecting the most appropriate machine learning algorithms, related methods, and text sections of articles is a neglected but important requirement because of its significant impact to the end results. 2013 Elsevier Ltd. All rights reserved. AU - Garcia Adeva, J. J. AU - Pikatza Atxa, J. M. AU - Ubeda Carrillo, M. AU - Ansuategi Zengotitabengoa, E. DA - 2014 DO - 10.1016/j.eswa.2013.08.047 IS - 4 PART 1 J2 - Expert Systems with Applications KW - Abstracting Classification (of information) data mining Decision support systems Diagnosis Experiments Learning algorithms Learning systems Text processing L1 - internal-pdf://3598526307/Garcia Adeva-2014-Automatic text classificatio.pdf N1 -Compilation and indexing terms, Copyright 2016 Elsevier Inc.
PY - 2014 SN - 09574174 SP - 1498-1508 ST - Automatic text classification to support systematic reviews in medicine T2 - Expert Systems with Applications TI - Automatic text classification to support systematic reviews in medicine UR - http://dx.doi.org/10.1016/j.eswa.2013.08.047 http://ac.els-cdn.com.ezproxy.lib.vt.edu/S0957417413006684/1-s2.0-S0957417413006684-main.pdf?_tid=d04f04f4-8335-11e6-a3d1-00000aacb35d&acdnat=1474817963_338420633476451eed2bd84d10f48367 VL - 41 ID - 1464 ER - TY - JOUR AB - Systematic reviews require expert reviewers to manually screen thousands of citations in order to identify all relevant articles to the review. Active learning text classification is a supervised machine learning approach that has been shown to significantly reduce the manual annotation workload by semi-automating the citation screening process of systematic reviews. In this paper, we present a new topic detection method that induces an informative representation of studies, to improve the performance of the underlying active learner. Our proposed topic detection method uses a neural network-based vector space model to capture semantic similarities between documents. We firstly represent documents within the vector space, and cluster the documents into a predefined number of clusters. The centroids of the clusters are treated as latent topics. We then represent each document as a mixture of latent topics. For evaluation purposes, we employ the active learning strategy using both our novel topic detection method and a baseline topic model (i.e., Latent Dirichlet Allocation). Results obtained demonstrate that our method is able to achieve a high sensitivity of eligible studies and a significantly reduced manual annotation cost when compared to the baseline method. This observation is consistent across two clinical and three public health reviews. The tool introduced in this work is available from https://nactem.ac.uk/pvtopic/. 2016 The Authors. AU - Hashimoto, Kazuma AU - Kontonatsios, Georgios AU - Miwa, Makoto AU - Ananiadou, Sophia DA - 2016 DO - 10.1016/j.jbi.2016.06.001 J2 - Journal of Biomedical Informatics KW - artificial intelligence Classification (of information) Learning systems Semantics Statistics Supervised learning Text processing Vectors Vector spaces L1 - internal-pdf://3403757026/Hashimoto-2016-Topic detection using paragraph.pdf N1 -Compilation and indexing terms, Copyright 2016 Elsevier Inc.
PY - 2016 SN - 15320464 SP - 59-65 ST - Topic detection using paragraph vectors to support active learning in systematic reviews T2 - Journal of Biomedical Informatics TI - Topic detection using paragraph vectors to support active learning in systematic reviews UR - http://dx.doi.org/10.1016/j.jbi.2016.06.001 http://ac.els-cdn.com/S1532046416300442/1-s2.0-S1532046416300442-main.pdf?_tid=81e1d074-8337-11e6-ba2e-00000aacb35e&acdnat=1474818690_469151e250ab9c6100b8a719ac9ae41e VL - 62 ID - 1045 ER - TY - JOUR AB - BACKGROUND: Different approaches can be adopted for the development of search strategies of systematic reviews. The objective approach draws on already established text analysis methods for developing search filters. Our aim was to determine whether the objective approach for the development of search strategies was noninferior to the conceptual approach commonly used in Cochrane reviews (CRs). METHODS: We conducted a search for CRs published in the Cochrane Library. The studies included in the CRs were searched for in MEDLINE and represented the total set. We then tested whether references previously removed could be identified via the objective approach. We also reconstructed the original search strategies from the CRs to determine why references could not be identified by the objective approach. As we performed the validation of the search strategies without study filters, we used only sensitivity as a quality measure and did not calculate precision. RESULTS: The objective approach yielded a mean sensitivity of 96% based on 13 searches. The noninferiority test showed that this approach was noninferior to the conceptual approach used in the CRs (P < 0.002). An additional descriptive analysis showed that the original MEDLINE strategies could identify only 86% of all references; however, this lower sensitivity was largely due to one CR. CONCLUSION: To the best of our knowledge, our findings indicate for the first time that the objective approach for the development of search strategies is noninferior to the conceptual approach. AU - Hausner, Elke AU - Guddat, Charlotte AU - Hermanns, Tatjana AU - Lampert, Ulrike AU - Waffenschmidt, Siw DA - 2015/02//undefined DO - 10.1016/j.jclinepi.2014.09.016 IS - 2 J2 - J Clin Epidemiol KW - *Review Literature as Topic *Validation Studies as Topic data mining Humans Information storage and retrieval Information Storage and Retrieval/*methods/*standards/trends Medline Publishing/*standards Reproducibility of results Retrospective studies Search Engine/*standards Sensitivity and specificity L1 - internal-pdf://0310242589/Hausner-2015-Development of search strategies.pdf LA - eng PY - 2015 SN - 1878-5921 0895-4356 SP - 191-199 ST - Development of search strategies for systematic reviews: validation showed the noninferiority of the objective approach T2 - Journal of clinical epidemiology TI - Development of search strategies for systematic reviews: validation showed the noninferiority of the objective approach UR - http://www.jclinepi.com/article/S0895-4356(14)00387-4/pdf VL - 68 ID - 122 ER - TY - JOUR AB - BACKGROUND: In the development of search strategies for systematic reviews, "conceptual approaches" are generally recommended to identify appropriate search terms for those parts of the strategies for which no validated search filters exist. However, "objective approaches" based on search terms identified by text analysis are increasingly being applied. OBJECTIVES: To prospectively compare an objective with a conceptual approach for the development of search strategies. METHODS: Two different MEDLINE search strategies were developed in parallel for five systematic reviews covering a range of topics and study designs. The Institute for Quality and Efficiency in Health Care (IQWiG) applied an objective approach, and external experts applied a conceptual approach for the same research questions. For each systematic review, the citations retrieved were combined and the overall pool of citations screened to determine sensitivity and precision. RESULTS: The objective approach yielded a weighted mean sensitivity and precision of 97% and 5%. The corresponding values for the conceptual approach were 75% and 4%. CONCLUSION: Our findings indicate that the objective approach applied by IQWiG for search strategy development yields higher sensitivity than and similar precision to a conceptual approach. The main advantage of the objective approach is that it produces consistent results across searches. AU - Hausner, Elke AU - Guddat, Charlotte AU - Hermanns, Tatjana AU - Lampert, Ulrike AU - Waffenschmidt, Siw DA - 2016/05/30/ DO - 10.1016/j.jclinepi.2016.05.002 J2 - J Clin Epidemiol KW - data mining Information storage and retrieval Medline Prospective studies Reproducibility of results Sensitivity and specificity L1 - internal-pdf://3799058530/Hausner-2016-Prospective comparison of search.pdf LA - Eng PY - 2016 SN - 1878-5921 0895-4356 ST - Prospective comparison of search strategies for systematic reviews: an objective approach yielded higher sensitivity than a conceptual one T2 - Journal of clinical epidemiology TI - Prospective comparison of search strategies for systematic reviews: an objective approach yielded higher sensitivity than a conceptual one UR - http://www.jclinepi.com/article/S0895-4356(16)30134-2/pdf ID - 75 ER - TY - BLOG AB - We have just published a new manuscript about our SWIFT-Review software in the journal Systematic Reviews! SWIFT-Review (SWIFT is an acronym for “Sciome Workbench for Interactive computer-Facilitated Text-mining”) is a freely available, interactive workbench which provides numerous tools to assist with problem formulation and literature prioritization. SWIFT-Review puts the systematic review expert … AU - Howard, Brian E., Jason Phillips, Kyle Miller, Arpit Tandon, Deepak Mav, Mihir R. Shah, Holmgren, Stephanie, Pelch, Katherine E., Walker, Vickie, Rooney, Andrew A., Macleod, Malcolm, Shah, Ruchir R., Thayer, Kristina DA - 2016/05/23/T11:13:41-04:00 L1 - internal-pdf://2044897762/art%253A10.1186%252Fs13643-016-0263-z.pdf PY - 2016 ST - SWIFT-Review Manuscript Published in Systematic Reviews Journal T2 - Sciome TI - SWIFT-Review Manuscript Published in Systematic Reviews Journal UR - http://www.sciome.com/swift-review-manuscript-published-systematic-reviews-journal/ ID - 2502 ER - TY - JOUR AB - BACKGROUND: There is growing interest in using machine learning approaches to priority rank studies and reduce human burden in screening literature when conducting systematic reviews. In addition, identifying addressable questions during the problem formulation phase of systematic review can be challenging, especially for topics having a large literature base. Here, we assess the performance of the SWIFT-Review priority ranking algorithm for identifying studies relevant to a given research question. We also explore the use of AU - Howard, Brian E. AU - Phillips, Jason AU - Miller, Kyle AU - Tandon, Arpit AU - Mav, Deepak AU - Shah, Mihir R. AU - Holmgren, Stephanie AU - Pelch, Katherine E. AU - Walker, Vickie AU - Rooney, Andrew A. AU - Macleod, Malcolm AU - Shah, Ruchir R. AU - Thayer, Kristina DA - 2016 DO - 10.1186/s13643-016-0263-z J2 - Syst Rev KW - Literature prioritization Scoping reports Software SWIFT-Review Systematic review L1 - internal-pdf://1642663198/Howard-2016-SWIFT-Review_ a text-mining workbe.pdf LA - eng PY - 2016 SN - 2046-4053 2046-4053 SP - 87 ST - SWIFT-Review: a text-mining workbench for systematic review T2 - Systematic reviews TI - SWIFT-Review: a text-mining workbench for systematic review UR - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4877757/pdf/13643_2016_Article_263.pdf VL - 5 ID - 57 ER - TY - JOUR AB - BACKGROUND: Automation of the parts of systematic review process, specifically the data extraction step, may be an important strategy to reduce the time necessary to complete a systematic review. However, the state of the science of automatically extracting data elements from full texts has not been well described. This paper performs a systematic review of published and unpublished methods to automate data extraction for systematic reviews. METHODS: We systematically searched PubMed, IEEEXplore, and ACM Digital Library to identify potentially relevant articles. We included reports that met the following criteria: 1) methods or results section described what entities were or need to be extracted, and 2) at least one entity was automatically extracted with evaluation results that were presented for that entity. We also reviewed the citations from included reports. RESULTS: Out of a total of 1190 unique citations that met our search criteria, we found 26 published reports describing automatic extraction of at least one of more than 52 potential data elements used in systematic reviews. For 25 (48 %) of the data elements used in systematic reviews, there were attempts from various researchers to extract information automatically from the publication text. Out of these, 14 (27 %) data elements were completely extracted, but the highest number of data elements extracted automatically by a single study was 7. Most of the data elements were extracted with F-scores (a mean of sensitivity and positive predictive value) of over 70 %. CONCLUSIONS: We found no unified information extraction framework tailored to the systematic review process, and published reports focused on a limited (1-7) number of data elements. Biomedical natural language processing techniques have not been fully utilized to fully or even partially automate the data extraction step of systematic reviews. AU - Jonnalagadda, Siddhartha R. AU - Goyal, Pawan AU - Huffman, Mark D. DA - 2015 DO - 10.1186/s13643-015-0066-7 J2 - Syst Rev KW - *Publishing *Review Literature as Topic Data Mining/*methods Humans Information storage and retrieval Research Report L1 - internal-pdf://3806856997/Jonnalagadda-2015-Automating data extraction i.pdf LA - eng PY - 2015 SN - 2046-4053 2046-4053 SP - 78 ST - Automating data extraction in systematic reviews: a systematic review T2 - Systematic reviews TI - Automating data extraction in systematic reviews: a systematic review UR - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4514954/pdf/13643_2015_Article_66.pdf VL - 4 ID - 20 ER - TY - CONF AB - Clinical systematic reviews are based on expert, laborious search of well-annotated literature. Boolean search on bibliographic databases, such as MEDLINE, continues to be the preferred discovery method, but the size of these databases, now approaching 20 million records, makes it impossible to fully trust these searching methods. We are investigating the trade-offs between Boolean and ranked retrieval. Our findings show that although Boolean search has limitations, it is not obvious that ranking is superior, and illustrate that a single query cannot be used to resolve an information need. Our experiments show that a combination of less complicated Boolean queries and ranked retrieval outperforms either of them individually, leading to possible time savings over the current process. Copyright 2009 ACM. AU - Karimi, Sarvnaz AU - Zobel, Justin AU - Pohl, Stefan AU - Scholer, Falk C3 - 3rd ACM International Workshop on Data and Text Mining in Bioinformatics, DTMBIO'09, Co-located with the 18th ACM International Conference on Information and Knowledge Management, CIKM 2009, November 2, 2009 - November 6, 2009 DA - 2009 DO - 10.1145/1651318.1651338 KW - Bibliographic retrieval systems bioinformatics Information services Knowledge management L1 - internal-pdf://2907797398/p89-karimi.pdf N1 -Compilation and indexing terms, Copyright 2016 Elsevier Inc.
PB - Association for Computing Machinery PY - 2009 SP - 89-92 ST - The challenge of high recall in biomedical systematic search T3 - International Conference on Information and Knowledge Management, Proceedings TI - The challenge of high recall in biomedical systematic search UR - http://dx.doi.org/10.1145/1651318.1651338 http://dl.acm.org/citation.cfm?doid=1651318.1651338 ID - 565 ER - TY - JOUR AU - Kitchenham, Barbara AU - Brereton, Pearl DA - 2013 DP - Google Scholar IS - 12 L1 - http://romisatriawahono.net/lecture/rm/survey/research%20methodology/Kitchenham%20-%20Systematic%20Review%20Process%20Research%20-%202013.pdf PY - 2013 SP - 2049-2075 ST - A systematic review of systematic review process research in software engineering T2 - Information and software technology TI - A systematic review of systematic review process research in software engineering UR - http://www.sciencedirect.com/science/article/pii/S0950584913001560 VL - 55 Y2 - 2016/09/24/15:33:48 ID - 2322 ER - TY - JOUR AB - The Cochrane Collaboration was established in 1993, following the opening of the UK Cochrane Centre in 1992, at a time when searching for studies for inclusion in systematic reviews was not well-developed. Review authors largely conducted their own searches or depended on medical librarians, who often possessed limited awareness and experience of systematic reviews. Guidance on the conduct and reporting of searches was limited. When work began to identify reports of randomized controlled trials (RCTs) for inclusion in Cochrane Reviews in 1992, there were only approximately 20,000 reports indexed as RCTs in MEDLINE and none indexed as RCTs in Embase. No search filters had been developed with the aim of identifying all RCTs in MEDLINE or other major databases. This presented The Cochrane Collaboration with a considerable challenge in identifying relevant studies.Over time, the number of studies indexed as RCTs in the major databases has grown considerably and the Cochrane Central Register of Controlled Trials (CENTRAL) has become the best single source of published controlled trials, with approximately 700,000 records, including records identified by the Collaboration from Embase and MEDLINE. Search filters for various study types, including systematic reviews and the Cochrane Highly Sensitive Search Strategies for RCTs, have been developed. There have been considerable advances in the evidence base for methodological aspects of information retrieval. The Cochrane Handbook for Systematic Reviews of Interventions now provides detailed guidance on the conduct and reporting of searches. Initiatives across The Cochrane Collaboration to improve the quality inter alia of information retrieval include: the recently introduced Methodological Expectations for Cochrane Intervention Reviews (MECIR) programme, which stipulates 'mandatory' and 'highly desirable' standards for various aspects of review conduct and reporting including searching, the development of Standard Training Materials for Cochrane Reviews and work on peer review of electronic search strategies. Almost all Cochrane Review Groups and some Cochrane Centres and Fields now have a Trials Search Co-ordinator responsible for study identification and medical librarians and other information specialists are increasingly experienced in searching for studies for systematic reviews.Prospective registration of clinical trials is increasing and searching trials registers is now mandatory for Cochrane Reviews, where relevant. Portals such as the WHO International Clinical Trials Registry Platform (ICTRP) are likely to become increasingly attractive, given concerns about the number of trials which may not be registered and/or published. The importance of access to information from regulatory and reimbursement agencies is likely to increase. Cross-database searching, gateways or portals and improved access to full-text databases will impact on how searches are conducted and reported, as will services such as Google Scholar, Scopus and Web of Science. Technologies such as textual analysis, semantic analysis, text mining and data linkage will have a major impact on the search process but efficient and effective updating of reviews may remain a challenge.In twenty years' time, we envisage that the impact of universal social networking, as well as national and international legislation, will mean that all trials involving humans will be registered at inception and detailed trial results will be routinely available to all. Challenges will remain, however, to ensure the discoverability of relevant information in diverse and often complex sources and the availability of metadata to provide the most efficient access to information. We envisage an ongoing role for information professionals as experts in identifying new resources, researching efficient ways to link or mine them for relevant data and managing their content for the efficient production of systematic reviews. AU - Lefebvre, Carol AU - Glanville, Julie AU - Wieland, L. Susan AU - Coles, Bernadette AU - Weightman, Alison L. DA - 2013 DO - 10.1186/2046-4053-2-78 J2 - Syst Rev KW - *Databases, Bibliographic *Review Literature as Topic Abstracting and Indexing as Topic Evidence-Based Medicine Humans Information Storage and Retrieval/*methods/*standards/trends Randomized Controlled Trials as Topic Registries L1 - internal-pdf://0039848341/Lefebvre-2013-Methodological developments in s.pdf LA - eng PY - 2013 SN - 2046-4053 2046-4053 SP - 78 ST - Methodological developments in searching for studies for systematic reviews: past, present and future? T2 - Systematic reviews TI - Methodological developments in searching for studies for systematic reviews: past, present and future? UR - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4015986/pdf/2046-4053-2-78.pdf VL - 2 ID - 76 ER - TY - JOUR AB - There is ongoing interest in including grey literature in systematic reviews. Including grey literature can broaden the scope to more relevant studies, thereby providing a more complete view of available evidence. Searching for grey literature can be challenging despite greater access through the Internet, search engines and online bibliographic databases. There are a number of publications that list sources for finding grey literature in systematic reviews. However, there is scant information about how searches for grey literature are executed and how it is included in the review process. This level of detail is important to ensure that reviews follow explicit methodology to be systematic, transparent and reproducible. The purpose of this paper is to provide a detailed account of one systematic review team's experience in searching for grey literature and including it throughout the review. We provide a brief overview of grey literature before describing our search and review approach. We also discuss the benefits and challenges of including grey literature in our systematic review, as well as the strengths and limitations to our approach. Detailed information about incorporating grey literature in reviews is important in advancing methodology as review teams adapt and build upon the approaches described. AU - Mahood, Quenby AU - Van Eerd, Dwayne AU - Irvin, Emma DA - 2014/09//undefined DO - 10.1002/jrsm.1106 IS - 3 J2 - Res Synth Methods KW - *Databases, Bibliographic *Peer Review, Research *Periodicals as Topic *Review Literature as Topic *Search Engine Data Mining/*methods grey literature literature searching natural language processing systematic reviews Vocabulary, Controlled L1 - internal-pdf://0628175010/Mahood_et_al-2014-Research_Synthesis_Methods.pdf LA - eng PY - 2014 SN - 1759-2887 1759-2879 SP - 221-234 ST - Searching for grey literature for systematic reviews: challenges and benefits T2 - Research synthesis methods TI - Searching for grey literature for systematic reviews: challenges and benefits VL - 5 ID - 68 ER - TY - CONF AB - The software engineering research community has been adopting systematic reviews as an unbiased and fair way to assess a research topic. Despite encouraging early results, a systematic review process can be time consuming and hard to conduct. Thus, tools that help on its planning or execution are needed. This article suggests the use of Visual Text Mining (VTM) to aid systematic reviews. A feasibility study was conducted comparing the proposed approach with a manual process. We observed that VTM can contribute to Systematic Review and we propose a new strategy called VTM-Based Systematic Review. 2007 IEEE. AU - Malheiros, Viviane AU - Hohn, Erika AU - Pinho, Roberto AU - Mendonca, Manoel AU - Maldonado, Jose Carlos C3 - 1st International Symposium on Empirical Software Engineering and Measurement, ESEM 2007, September 20, 2007 - September 21, 2007 DA - 2007 DO - 10.1109/ESEM.2007.13 KW - data mining decision making Engineering research mining Planning Resource allocation software engineering Technology N1 -Compilation and indexing terms, Copyright 2016 Elsevier Inc.
PB - Inst. of Elec. and Elec. Eng. Computer Society PY - 2007 SP - 245-254 ST - A visual text mining approach for systematic reviews T3 - Proceedings - 1st International Symposium on Empirical Software Engineering and Measurement, ESEM 2007 TI - A visual text mining approach for systematic reviews ID - 1764 ER - TY - CONF AB - In medicine, the publication of clinical trials now far out- paces clinicians' ability to read them. Systematic reviews, which aim to summarize the entirety of the available evidence on a specific clinical question, have therefore become the linchpin of evidence-based decision making. A key task in systematic reviews is determining whether the results of included studies may be affected by biases, e.g., poor ran- domization or blinding. This is called risk of bias assess- ment and is now standard practice. Standardized tools are used to perform these assessments; a notable example being the Cochrane risk of bias tool, which covers seven different types of potential biases and involves researchers extracting sentences from articles to support their bias assessments. These assessments are crucial in interpreting published evidence, but due to the exponential growth of the biomedical literature base, manually assessing the risk of bias in clinical trials has grown burdensome for clinical researchers. Aiming to mitigate this workload, we explore automating risk of bias assessment. We demonstrate that systematic re- views may be used to distantly supervise text mining models, obviating the need for manually annotated clinical trial reports. Specifically, we leverage data from the Cochrane Database of Systematic Reviews (a large repository of sys- Tematic reviews), and link clinical trial reports to structured data from the same studies found in CDSR to produce a pseudo-annotated labeled corpus. We then develop a joint model which, using (the PDF of) a clinical trial report as input, predicts the risks of bias in each of the aforemen- Tioned seven areas while simultaneously extracting the text fragments supporting these assessments. Copyright 2014 ACM. AU - Marshall, Iain J. AU - Kuiper, Joel AU - Wallace, Byron C. C3 - 5th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics, ACM BCB 2014, September 20, 2014 - September 23, 2014 DA - 2014 DO - 10.1145/2649387.2649406 KW - bioinformatics data mining decision making Health care Medical applications Risk Assessment L1 - internal-pdf://2953234493/07104094.pdf N1 -Compilation and indexing terms, Copyright 2016 Elsevier Inc.
PB - Association for Computing Machinery, Inc PY - 2014 SP - 88-95 ST - Automating risk of bias assessment for clinical trials T3 - ACM BCB 2014 - 5th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics TI - Automating risk of bias assessment for clinical trials UR - http://dx.doi.org/10.1145/2649387.2649406 ID - 1024 ER - TY - CONF AB - Despite the increasing popularity of systematic literature reviews in Software Engineering, several researchers still indicate it as a costly and challenging process. Aiming at alleviating this costly process, we propose an iterative method to support the process of building the search string for a systematic review. This method uses Visual Text Mining techniques to support the researcher by suggesting new terms for the string. In order to do so, the method extracts relevant terms from studies selected by the researcher and displays them in a way that facilitate their visualization and supports building and refining the search string. In order to check the feasibility of this approach, we developed a tool that implements the proposed method. Interviews with researchers identified their difficulties in performing systematic reviews and captured their feedback with regards the use of the proposed method in a user study. The researchers indicated that this approach could be used to improve the process of building the search strings for systematic reviews. The study indicates that our approach can be used to facilitate the construction of the systematic literature review search string. Copyright 2015 ACM. AU - Mergel, Germano Duarte AU - Silveira, Milene Selbach AU - Da Silva, Tiago Silva C3 - 30th Annual ACM Symposium on Applied Computing, SAC 2015, April 13, 2015 - April 17, 2015 DA - 2015 DO - 10.1145/2695664.2695902 KW - Cost engineering data mining Information systems Iterative methods software engineering visualization L1 - internal-pdf://3006968068/p1594-mergel.pdf N1 -Compilation and indexing terms, Copyright 2016 Elsevier Inc.
PB - Association for Computing Machinery PY - 2015 SP - 1594-1601 ST - A method to support search string building in systematic literature reviews through visual text mining T3 - Proceedings of the ACM Symposium on Applied Computing TI - A method to support search string building in systematic literature reviews through visual text mining UR - http://dx.doi.org/10.1145/2695664.2695902 VL - 13-17-April-2015 ID - 1513 ER - TY - JOUR AB - Background: Risk-of-bias assessments are now a standard component of systematic reviews. At present, reviewers need to manually identify relevant parts of research articles for a set of methodological elements that affect the risk of bias, in order to make a risk-of-bias judgement for each of these elements. We investigate the use of text mining methods to automate risk-of-bias assessments in systematic reviews. We aim to identify relevant sentences within the text of included articles, to rank articles by risk of bias and to reduce the number of risk-of-bias assessments that the reviewers need to perform by hand., Methods: We use supervised machine learning to train two types of models, for each of the three risk-of-bias properties of sequence generation, allocation concealment and blinding. The first model predicts whether a sentence in a research article contains relevant information. The second model predicts a risk-of-bias value for each research article. We use logistic regression, where each independent variable is the frequency of a word in a sentence or article, respectively., Results: We found that sentences can be successfully ranked by relevance with area under the receiver operating characteristic (ROC) curve (AUC) > 0.98. Articles can be ranked by risk of bias with AUC > 0.72. We estimate that more than 33% of articles can be assessed by just one reviewer, where two reviewers are normally required., Conclusions: We show that text mining can be used to assist risk-of-bias assessments. AU - Millard, Louise A. C. AU - Flach, Peter A. AU - Higgins, Julian P. T. DA - 2016/02// DO - 10.1093/ije/dyv306 DP - PubMed Central IS - 1 J2 - Int J Epidemiol L1 - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4795562/pdf/dyv306.pdf PY - 2016 SN - 0300-5771 SP - 266-277 ST - Machine learning to assist risk-of-bias assessments in systematic reviews T2 - International Journal of Epidemiology TI - Machine learning to assist risk-of-bias assessments in systematic reviews UR - http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4795562/ VL - 45 Y2 - 2016/09/24/20:49:28 ID - 2500 ER - TY - JOUR AB - In systematic reviews, the growing number of published studies imposes a significant screening workload on reviewers. Active learning is a promising approach to reduce the workload by automating some of the screening decisions, but it has been evaluated for a limited number of disciplines. The suitability of applying active learning to complex topics in disciplines such as social science has not been studied, and the selection of useful criteria and enhancements to address the data imbalance problem in systematic reviews remains an open problem. We applied active learning with two criteria (certainty and uncertainty) and several enhancements in both clinical medicine and social science (specifically, public health) areas, and compared the results in both. The results show that the certainty criterion is useful for finding relevant documents, and weighting positive instances is promising to overcome the data imbalance problem in both data sets. Latent dirichlet allocation (LDA) is also shown to be promising when little manually-assigned information is available. Active learning is effective in complex topics, although its efficiency is limited due to the difficulties in text classification. The most promising criterion and weighting method are the same regardless of the review topic, and unsupervised techniques like LDA have a possibility to boost the performance of active learning without manual annotation. 2014 The Authors. AU - Miwa, Makoto AU - Thomas, James AU - O'Mara-Eves, Alison AU - Ananiadou, Sophia DA - 2014 DO - 10.1016/j.jbi.2014.06.005 J2 - Journal of Biomedical Informatics KW - artificial intelligence Behavioral research Classification (of information) data mining Diagnosis Social sciences Statistics Text processing N1 -Compilation and indexing terms, Copyright 2016 Elsevier Inc.
PY - 2014 SN - 15320464 SP - 242-253 ST - Reducing systematic review workload through certainty-based screening T2 - Journal of Biomedical Informatics TI - Reducing systematic review workload through certainty-based screening UR - http://dx.doi.org/10.1016/j.jbi.2014.06.005 VL - 51 ID - 1486 ER - TY - JOUR AB - BACKGROUND: Identifying relevant studies for inclusion in a systematic review (i.e. screening) is a complex, laborious and expensive task. Recently, a number of studies has shown that the use of machine learning and text mining methods to automatically identify relevant studies has the potential to drastically decrease the workload involved in the screening phase. The vast majority of these machine learning methods exploit the same underlying principle, i.e. a study is modelled as a bag-of-words (BOW). METHODS: We explore the use of topic modelling methods to derive a more informative representation of studies. We apply Latent Dirichlet allocation (LDA), an unsupervised topic modelling approach, to automatically identify topics in a collection of studies. We then represent each study as a distribution of LDA topics. Additionally, we enrich topics derived using LDA with multi-word terms identified by using an automatic term recognition (ATR) tool. For evaluation purposes, we carry out automatic identification of relevant studies using support vector machine (SVM)-based classifiers that employ both our novel topic-based representation and the BOW representation. RESULTS: Our results show that the SVM classifier is able to identify a greater number of relevant studies when using the LDA representation than the BOW representation. These observations hold for two systematic reviews of the clinical domain and three reviews of the social science domain. CONCLUSIONS: A topic-based feature representation of documents outperforms the BOW representation when applied to the task of automatic citation screening. The proposed term-enriched topics are more informative and less ambiguous to systematic reviewers. AU - Mo, Yuanhan AU - Kontonatsios, Georgios AU - Ananiadou, Sophia DA - 2015 DO - 10.1186/s13643-015-0117-0 J2 - Syst Rev KW - *Models, Statistical *Review Literature as Topic *Support Vector Machine Biomedical Research/*classification Data Mining/*methods Decision Making, Computer-Assisted Humans L1 - internal-pdf://0304089171/art%253A10.1186%252Fs13643-015-0117-0.pdf LA - eng PY - 2015 SN - 2046-4053 2046-4053 SP - 172 ST - Supporting systematic reviews using LDA-based document representations T2 - Systematic reviews TI - Supporting systematic reviews using LDA-based document representations VL - 4 ID - 49 ER - TY - CONF AB - Background. A systematic literature review is a process in which all relevant available research about a research question is identified, evaluated, and interpreted through individual studies. The workload required for this process may bias the evaluation of the studies, affecting the result. Aim. Creating a decision support architecture to assist participants of a systematic review in the selection process of the individual studies and quality assessment of these studies, possibly improving the execution time and reducing the evaluation bias. Method. Improving the primary studies selection and quality assessment processes by using text mining techniques and ontologies to construct a decision support architecture. We will also conduct experiments to evaluate the proposed architecture. Contribution. Improve the primary studies selection and quality assessment processes, reducing its workload, and lowering the evaluation bias in systematic literature reviews. 2015 for this paper by its authors. AU - Nepomuceno, Vilmar C3 - 13th International Doctoral Symposium on Empirical Software Engineering, IDoESE 2015 - As part of the Empirical Software Engineering International Week 2015, ESEIW 2015, October 21, 2015 DA - 2015 KW - data mining Decision support systems ontology Quality Control software engineering L1 - internal-pdf://4146382461/paper2.pdf N1 -Compilation and indexing terms, Copyright 2016 Elsevier Inc.
PB - CEUR-WS PY - 2015 SN - 16130073 SP - 10-14 ST - Decision support architecture for primary studies evaluation T3 - CEUR Workshop Proceedings TI - Decision support architecture for primary studies evaluation VL - 1469 ID - 993 ER - TY - JOUR AU - O'Mara-Eves, Alison AU - Thomas, James AU - McNaught, John AU - Miwa, Makoto AU - Ananiadou, Sophia DA - 2015 DO - 10.1186/s13643-015-0031-5 J2 - Syst Rev L1 - internal-pdf://2084420925/13643_2015_Article_31.pdf LA - eng PY - 2015 SN - 2046-4053 2046-4053 SP - 59 ST - Erratum to: Using text mining for study identification in systematic reviews: a systematic review of current approaches T2 - Systematic reviews TI - Erratum to: Using text mining for study identification in systematic reviews: a systematic review of current approaches VL - 4 ID - 46 ER - TY - JOUR AB - The large and growing number of published studies, and their increasing rate of publication, makes the task of identifying relevant studies in an unbiased way for inclusion in systematic reviews both complex and time consuming. Text mining has been offered as a potential solution: through automating some of the screening process, reviewer time can be saved. The evidence base around the use of text mining for screening has not yet been pulled together systematically; this systematic review fills that research gap. Focusing mainly on non-technical issues, the review aims to increase awareness of the potential of these technologies and promote further collaborative research between the computer science and systematic review communities. AU - O’Mara-Eves, Alison AU - Thomas, James AU - McNaught, John AU - Miwa, Makoto AU - Ananiadou, Sophia DA - 2015 DO - 10.1186/2046-4053-4-5 DP - BioMed Central J2 - Systematic Reviews KW - Automation Review efficiency Screening Study selection Text mining L1 - https://systematicreviewsjournal.biomedcentral.com/track/pdf/10.1186/2046-4053-4-5?site=systematicreviewsjournal.biomedcentral.com N1 - Pages 1-22 in PDF PY - 2015 SN - 2046-4053 SP - 5 ST - Using text mining for study identification in systematic reviews T2 - Systematic Reviews TI - Using text mining for study identification in systematic reviews: a systematic review of current approaches UR - http://dx.doi.org/10.1186/2046-4053-4-5 https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/2046-4053-4-5 VL - 4 Y2 - 2016/09/18/21:35:18 ID - 446 ER - TY - CONF AB - Background: Since the introduction of the systematic review process to Software Engineering in 2004, researchers have investigated a number of ways to mitigate the amount of effort and time taken to filter through large volumes of literature. Aim: This study aims to provide a critical analysis of text mining techniques used to support the citation screening stage of the systematic review process. Method: We critically re-reviewed papers included in a previous systematic review which addressed the use of text mining methods to support the screening of papers for inclusion in a review. The previous review did not provide a detailed analysis of the text mining methods used. We focus on the availability in the papers of information about the text mining methods employed, including the description and explanation of the methods, parameter settings, assessment of the appropriateness of their application given the size and dimensionality of the data used, performance on training, testing and validation data sets, and further information that may support the reproducibility of the included studies. Results: Support Vector Machines (SVM), Naive Bayes (NB) and Committee of classifiers (Ensemble) are the most used classification algorithms. In all of the studies, features were represented with Bag-of-Words (BOW) using both binary features (28%) and term frequency (66%). Five studies experimented with n-grams with n between 2 and 4, but mostly the unigram was used. ?2, information gain and tf-idf were the most commonly used feature selection techniques. Feature extraction was rarely used although LDA and topic modelling were used. Recall, precision, F and AUC were the most used metrics and cross validation was also well used. More than half of the studies used a corpus size of below 1,000 documents for their experiments while corpus size for around 80% of the studies was 3,000 or fewer documents. The major common ground we found for comparing performance assessment based on independent replication of studies was the use of the same dataset but a sound performance comparison could not be established because the studies had little else in common. In most of the studies, insufficient information was reported to enable independent replication. The studies analysed generally did not include any discussion of the statistical appropriateness of the text mining method that they applied. In the case of applications of SVM, none of the studies report the number of support vectors that they found to indicate the complexity of the prediction engine that they use, making it impossible to judge the extent to which over-fitting might account for the good performance results. Conclusions: There is yet to be concrete evidence about the effectiveness of text mining algorithms regarding their use in the automation of citation screening in systematic reviews. The studies indicate that options are still being explored, but there is a need for better reporting as well as more explicit process details and access to datasets to facilitate study replication for evidence strengthening. In general, the reader often gets the impression that text mining algorithms were applied as magic tools in the reviewed papers, relying on default settings or default optimization of available machine learning toolboxes without an in-depth understanding of the statistical validity and appropriateness of such tools for text mining purposes. 2016 ACM. AU - Olorisade, Babatunde K. AU - De Quincey, Ed AU - Andras, Peter AU - Brereton, Pearl C3 - 20th International Conference on Evaluation and Assessment in Software Engineering, EASE 2016, June 1, 2016 - June 3, 2016 DA - 2016 DO - 10.1145/2915970.2915982 KW - Algorithms artificial intelligence automation data mining feature extraction Filtration Learning systems Optimization Paper software engineering Support Vector Machines Text processing L1 - internal-pdf://1967513926/a14-olorisade.pdf N1 -Compilation and indexing terms, Copyright 2016 Elsevier Inc.
PB - Association for Computing Machinery PY - 2016 ST - A critical analysis of studies that address the use of text mining for citation screening in systematic reviews T3 - ACM International Conference Proceeding Series TI - A critical analysis of studies that address the use of text mining for citation screening in systematic reviews UR - http://dx.doi.org/10.1145/2915970.2915982 VL - 01-03-June-2016 ID - 1624 ER - TY - BOOK AB - This project's goal was to provide a preliminary sketch of the use of text-mining tools as an emerging methodology within a number of systematic review processes. We sought to provide information addressing pressing questions individuals and organizations face when considering utilizing text-mining tools. We searched the literature to identify and summarize research on the use of text-mining tools within the systematic review context. We conducted telephone interviews with Key Informants (KIs; n=8) using a semi-structured instrument and subsequent qualitative analysis to explore issues surrounding the implementation and use of text-mining tools. Lastly, we compiled a list of text-mining tools to support systematic review methods and evaluated the tools using an informal descriptive appraisal tool. The literature review identified 122 articles that met inclusion criteria, including two recent systematic reviews on the use of text-mining tools in the screening and data abstraction steps of systematic reviews. In addition to these two steps, a preliminary exploration of the literature on searching and other less-studied steps are presented. Support for the use of text-mining was strong amongst the KIs overall, though most KIs noted some performance caveats and/or areas in which further research is necessary. We evaluated 111 text-mining tools identified from the literature review and KI interviews. Text-mining tools are currently being used within several systematic review organizations for a variety of review processes (e.g., searching, screening abstracts), and the published evidence-base is growing fairly rapidly in breadth and levels of evidence. Several outstanding questions remain for future empirical research to address regarding the reliability and validity of using these emerging technologies across a variety of review processes and whether these generalize across the scope of review topics. Guidance on reporting the use of these tools would be useful. AU - Paynter, Robin AU - Banez, Lionel L. AU - Berliner, Elise AU - Erinoff, Eileen AU - Lege-Matsuura, Jennifer AU - Potter, Shannon AU - Uhl, Stacey CY - Rockville (MD) DA - 2016/04//undefined L1 - internal-pdf://1889947178/text-mining-report-160419.pdf LA - eng PB - Agency for Healthcare Research and Quality (US) PY - 2016 ST - EPC Methods: An Exploration of the Use of Text-Mining Software in Systematic Reviews TI - EPC Methods: An Exploration of the Use of Text-Mining Software in Systematic Reviews ID - 67 ER - TY - JOUR AB - BACKGROUND: Comprehensive literature searches are conducted over multiple medical databases in order to meet stringent quality standards for systematic reviews. These searches are often very laborious, with authors often manually screening thousands of articles. Information retrieval (IR) techniques have proven increasingly effective in improving the efficiency of this process. IR challenges for systematic reviews involve building classifiers using training data with very high class-imbalance, and meeting the requirement for near perfect recall on relevant studies. Traditionally, most systematic reviews have focused on questions relating to treatment. The last decade has seen a large increase in the number of systematic reviews of diagnostic test accuracy (DTA). OBJECTIVE: We aim to demonstrate that DTA reviews comprise an especially challenging subclass of systematic reviews with respect to the workload required for literature screening. We identify specific challenges for the application of IR to literature screening for DTA reviews, and identify potential directions for future research. METHODS: We hypothesize that IR for DTA reviews face three additional challenges, compared to systematic reviews of treatments. These include an increased class-imbalance, a broader definition of the target class, and relative inadequacy of available metadata (ie, medical subject headings (MeSH) terms for medical literature analysis and retrieval system online). Assuming these hypotheses to be true, we identify five manifestations when we compare literature searches of DTA versus treatment. These manifestations include: an increase in the average number of articles screened, and increase in the average number of full-text articles obtained, a decrease in the number of included studies as a percentage of full-text articles screened, a decrease in the number of included studies as a percentage of all articles screened, and a decrease in the number of full-text articles obtained as a percentage of all articles screened. As of July 12 2013, 13 published Cochrane DTA reviews were available and all were included. For each DTA review, we randomly selected 15 treatment reviews published by the corresponding Cochrane Review Group (N=195). We then statistically tested differences in these five hypotheses, for the DTA versus treatment reviews. RESULTS: Despite low statistical power caused by the small sample size for DTA reviews, strong (P<.01) or very strong (P<.001) evidence was obtained to support three of the five expected manifestations, with evidence for at least one manifestation of each hypothesis. The observed difference in effect sizes are substantial, demonstrating the practical difference in reviewer workload. CONCLUSIONS: Reviewer workload (volume of citations screened) when screening literature for systematic reviews of DTA is especially high. This corresponds to greater rates of class-imbalance when training classifiers for automating literature screening for DTA reviews. Addressing concerns such as lower quality metadata and effectively modelling the broader target class could help to alleviate such challenges, providing possible directions for future research. AU - Petersen, Henry AU - Poon, Josiah AU - Poon, Simon K. AU - Loy, Clement DA - 2014 DO - 10.2196/medinform.3037 IS - 1 J2 - JMIR Med Inform KW - classification and clustering data mining Information storage and retrieval Meta-analysis review literature L1 - internal-pdf://4059243603/fc-xsltGalley-3037-39660-37-PB.pdf LA - eng PY - 2014 SN - 2291-9694 SP - e11 ST - Increased workload for systematic review literature searches of diagnostic tests compared with treatments: challenges and opportunities T2 - JMIR medical informatics TI - Increased workload for systematic review literature searches of diagnostic tests compared with treatments: challenges and opportunities VL - 2 ID - 53 ER - TY - CONF AB - In this article, we present a novel statistical representation method for knowledge extraction from a corpus containing short texts. Then we introduce the contrast parameter which could be adjusted for targeting different conceptual levels in text mining and knowledge extraction. The method is based on second order co-occurrence vectors whose efficiency for representing meaning has been established in many applications, especially for representing word senses in different contexts and for disambiguation purposes. We evaluate our method on two tasks: classification of textual description of dreams, and classification of medical abstracts for systematic reviews. 2009 IEEE. AU - Razavi, Amir H. AU - Matwin, Stan AU - Inkpen, Diana AU - Kouznetsov, Alexandre C3 - 2009 IEEE International Conference on Data Mining Workshops, ICDMW 2009, December 6, 2009 - December 6, 2009 DA - 2009 DO - 10.1109/ICDMW.2009.49 KW - data mining knowledge representation Technical presentations N1 -Compilation and indexing terms, Copyright 2016 Elsevier Inc.
PB - IEEE Computer Society PY - 2009 SP - 471-476 ST - Parameterized contrast in second order soft co-occurrences: A novel text representation technique in text mining and knowledge extraction T3 - ICDM Workshops 2009 - IEEE International Conference on Data Mining TI - Parameterized contrast in second order soft co-occurrences: A novel text representation technique in text mining and knowledge extraction UR - http://dx.doi.org/10.1109/ICDMW.2009.49 ID - 1332 ER - TY - JOUR AB - BACKGROUND: Meta-research studies investigating methods, systems, and processes designed to improve the efficiency of systematic review workflows can contribute to building an evidence base that can help to increase value and reduce waste in research. This study demonstrates the use of an economic evaluation framework to compare the costs and effects of four variant approaches to identifying eligible studies for consideration in systematic reviews. METHODS: A cost-effectiveness analysis was conducted using a basic decision-analytic model, to compare the relative efficiency of 'safety first', 'double screening', 'single screening' and 'single screening with text mining' approaches in the title-abstract screening stage of a 'case study' systematic review about undergraduate medical education in UK general practice settings. Incremental cost-effectiveness ratios (ICERs) were calculated as the 'incremental cost per citation 'saved' from inappropriate exclusion' from the review. Resource use and effect parameters were estimated based on retrospective analysis of 'review process' meta-data curated alongside the 'case study' review, in conjunction with retrospective simulation studies to model the integrated use of text mining. Unit cost parameters were estimated based on the 'case study' review's project budget. A base case analysis was conducted, with deterministic sensitivity analyses to investigate the impact of variations in values of key parameters. RESULTS: Use of 'single screening with text mining' would have resulted in title-abstract screening workload reductions (base case analysis) of >60 % compared with other approaches. Across modelled scenarios, the 'safety first' approach was, consistently, equally effective and less costly than conventional 'double screening'. Compared with 'single screening with text mining', estimated ICERs for the two non-dominated approaches (base case analyses) ranged from pound1975 ('single screening' without a 'provisionally included' code) to pound4427 ('safety first' with a 'provisionally included' code) per citation 'saved'. Patterns of results were consistent between base case and sensitivity analyses. CONCLUSIONS: Alternatives to the conventional 'double screening' approach, integrating text mining, warrant further consideration as potentially more efficient approaches to identifying eligible studies for systematic reviews. Comparable economic evaluations conducted using other systematic review datasets are needed to determine the generalisability of these findings and to build an evidence base to inform guidance for review authors. AU - Shemilt, Ian AU - Khan, Nada AU - Park, Sophie AU - Thomas, James DA - 2016 DO - 10.1186/s13643-016-0315-4 IS - 1 J2 - Syst Rev L1 - internal-pdf://3733474012/art%253A10.1186%252Fs13643-016-0315-4.pdf LA - eng PY - 2016 SN - 2046-4053 2046-4053 SP - 140 ST - Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews T2 - Systematic reviews TI - Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews VL - 5 ID - 45 ER - TY - JOUR AB - BACKGROUND: Scoping reviews of research help determine the feasibility and the resource requirements of conducting a systematic review, and the potential to generate a description of the literature quickly is attractive. AIMS: To test the utility and applicability of an automated clustering tool to describe and group research studies to improve the efficiency of scoping reviews. METHODS: A retrospective study of two completed scoping reviews was conducted. This compared the groups and descriptive categories obtained by automatically clustering titles and abstracts with those that had originally been derived using traditional researcher-driven techniques. RESULTS: The clustering tool rapidly categorised research into themes, which were useful in some instances, but not in others. This provided a dynamic means to view each dataset. Interpretation was challenging where there were potentially multiple meanings of terms. Where relevant clusters were unambiguous, there was a high precision of relevant studies, although recall varied widely. CONCLUSIONS: Policy-relevant scoping reviews are often undertaken rapidly, and this could potentially be enhanced by automation depending on the nature of the dataset and information sought. However, it is not a replacement for researcher-developed classification. The possibilities of further applications and potential for use in other types of review are discussed. AU - Stansfield, Claire AU - Thomas, James AU - Kavanagh, Josephine DA - 2013/09//undefined DO - 10.1002/jrsm.1082 IS - 3 J2 - Res Synth Methods KW - *Natural Language Processing *Research Report *Review Literature as Topic *Vocabulary, Controlled automatic clustering automation Cluster Analysis Data Mining/*methods Documentation/*classification Information storage and retrieval machine learning methods, mapping Periodicals as Topic/classification scoping reviews text mining L1 - internal-pdf://3289099771/Stansfield_et_al-2013-Research_Synthesis_Metho.pdf LA - eng PY - 2013 SN - 1759-2887 1759-2879 SP - 230-241 ST - 'Clustering' documents automatically to support scoping reviews of research: a case study T2 - Research synthesis methods TI - 'Clustering' documents automatically to support scoping reviews of research: a case study VL - 4 ID - 257 ER - TY - JOUR AB - Systematic reviews are a widely accepted research method. However, it is increasingly difficult to conduct them to fit with policy and practice timescales, particularly in areas which do not have well indexed, comprehensive bibliographic databases. Text mining technologies offer one possible way forward in reducing the amount of time systematic reviews take to conduct. They can facilitate the identification of relevant literature, its rapid description or categorization, and its summarization. In this paper, we describe the application of four text mining technologies, namely, automatic term recognition, document clustering, classification and summarization, which support the identification of relevant studies in systematic reviews. The contributions of text mining technologies to improve reviewing efficiency are considered and their strengths and weaknesses explored. We conclude that these technologies do have the potential to assist at various stages of the review process. However, they are relatively unknown in the systematic reviewing community, and substantial evaluation and methods development are required before their possible impact can be fully assessed. Copyright (c) 2011 John Wiley & Sons, Ltd. AU - Thomas, James AU - McNaught, John AU - Ananiadou, Sophia DA - 2011/03//undefined DO - 10.1002/jrsm.27 IS - 1 J2 - Res Synth Methods KW - automatic summarization document classification document clustering research synthesis Screening searching Systematic review term recognition text mining LA - eng PY - 2011 SN - 1759-2879 1759-2879 SP - 1-14 ST - Applications of text mining within systematic reviews T2 - Research synthesis methods TI - Applications of text mining within systematic reviews VL - 2 ID - 175 ER - TY - JOUR AU - Thomas, James AU - McNaught, John AU - Ananiadou, Sophia DA - 2011 DP - Google Scholar IS - 1 L1 - http://psych.colorado.edu/~willcutt/pdfs/thomas_2011.pdf PY - 2011 SP - 1-14 ST - Applications of text mining within systematic reviews T2 - Research Synthesis Methods TI - Applications of text mining within systematic reviews UR - http://onlinelibrary.wiley.com/doi/10.1002/jrsm.27/full VL - 2 Y2 - 2016/09/24/16:00:21 ID - 2399 ER - TY - JOUR AB - The wide variety of readily available electronic media grants anyone the freedom to retrieve published references from almost any area of research around the world. Despite this privilege, keeping up with primary research evidence is almost impossible because of the increase in professional publishing across disciplines. Systematic reviews are a solution to this problem as they aim to synthesize all current information on a particular topic and present a balanced and unbiased summary of the findings. They are fast becoming an important method of research across a number of fields, yet only a small number of guidelines exist on how to define and select terms for a systematic search. This article presents a replicable method for selecting terms in a systematic search using the semantic concept recognition software called leximancer (Leximancer, University of Queensland, Brisbane, Australia). We use this software to construct a set of terms from a corpus of literature pertaining to transborder interventions for drug control and discuss the applicability of this method to systematic reviews in general. This method aims to contribute a more 'systematic' approach for selecting terms in a manner that is entirely replicable for any user. AU - Thompson, Jenna AU - Davis, Jacqueline AU - Mazerolle, Lorraine DA - 2014/06//undefined DO - 10.1002/jrsm.1096 IS - 2 J2 - Res Synth Methods KW - *Natural Language Processing *Periodicals as Topic *Review Literature as Topic *Software Data Mining/*methods machine learning Pattern Recognition, Automated Research Design Search Engine/*methods Semantics Systematic review systematic search term Vocabulary, Controlled L1 - internal-pdf://3829176424/Thompson_et_al-2014-Research_Synthesis_Methods.pdf LA - eng PY - 2014 SN - 1759-2887 1759-2879 SP - 87-97 ST - A systematic method for search term selection in systematic reviews T2 - Research synthesis methods TI - A systematic method for search term selection in systematic reviews VL - 5 ID - 80 ER - TY - CONF AB - According to Khan et al, "a review earns the adjective systematic if it is based on a clearly formulated question, identifies relevant studies, appraises their quality and summarizes the evidence by use of explicit methodology". Conducting systematic reviews tend to be resource intensive and may suffer from problems such as publication bias, time-lag bias, duplicate bias, citation bias, and outcome reporting bias. This research aims to develop a system to facilitate the creation of systematic reviews. Starting with a clinical question, the proposed system will query ClinicalTrial.gov to search published RCTs. The system will exploit advanced data analytics techniques to systematically mine clinical trials obtained from the ClinicalTrial.gov. From the theoretical perspective, the system provides context for exploring the feasibility and efficacy of using advanced analytics techniques for generating machine readable, real time medical evidence. From a practical perspective, the system is expected to produce cost efficient medical evidence. AU - Timsina, Prem AU - El-Gayar, Omar AU - Nawar, Nevine C3 - 20th Americas Conference on Information Systems, AMCIS 2014, August 7, 2014 - August 9, 2014 DA - 2014 KW - Hardware Information systems N1 -Compilation and indexing terms, Copyright 2016 Elsevier Inc.
PB - Association for Information Systems PY - 2014 SP - et-al; Georgia Southern University; Georgia State University; Georgia Tech Scheller College of Business; IBM; SAP ST - Leveraging advanced analytics to generate dynamic medical systematic reviews T3 - 20th Americas Conference on Information Systems, AMCIS 2014 TI - Leveraging advanced analytics to generate dynamic medical systematic reviews ID - 949 ER - TY - CONF AB - While systematic reviews (SRs) are positioned as an essential element of modern evidence-based medical practice, the creation and update of these reviews is resource intensive. In this research, we propose to leverage advanced analytics techniques for automatically classifying articles for inclusion and exclusion for systematic review update. Specifically, we used the soft-margin Support Vector Machine (SVM) as a classifier and examined various techniques to resolve class imbalance issues. Through an empirical study, we demonstrated that the soft-margin SVM works better than the perceptron algorithm used in current research and the performance of the classifier can be further improved by exploiting different sampling methods to resolve class imbalance issues. 2015 IEEE. AU - Timsina, Prem AU - El-Gayar, Omar F. AU - Liu, Jun C3 - 48th Annual Hawaii International Conference on System Sciences, HICSS 2015, January 5, 2015 - January 8, 2015 DA - 2015 DO - 10.1109/HICSS.2015.121 KW - data mining Support Vector Machines L1 - internal-pdf://1129566412/7367a976.pdf N1 -Compilation and indexing terms, Copyright 2016 Elsevier Inc.
PB - IEEE Computer Society PY - 2015 SN - 15301605 SP - 976-985 ST - Leveraging advanced analytics techniques for medical systematic review update T3 - Proceedings of the Annual Hawaii International Conference on System Sciences TI - Leveraging advanced analytics techniques for medical systematic review update UR - http://dx.doi.org/10.1109/HICSS.2015.121 VL - 2015-March ID - 1478 ER - TY - CONF AB - Background: a systematic review identifies, evaluates and synthesizes the available literature on a given topic using scientific and repeatable methodologies. The significant workload required and the subjectivity bias could affect results. Aim: semi-automate the selection process to reduce the amount of manual work needed and the consequent subjectivity bias. Method: extend and enrich the selection of primary studies using the existing technologies in the field of Linked Data and text mining. We define formally the selection process and we also develop a prototype that implements it. Finally, we conduct a case study that simulates the selection process of a systematic literature published in literature. Results: the process presented in this paper could reduce the work load of 20% with respect to the work load needed in the fully manually selection, with a recall of 100%. Conclusions: the extraction of knowledge from scientific studies through Linked Data and text mining techniques could be used in the selection phase of the systematic review process to reduce the work load and subjectivity bias. AU - Tomassetti, F. AU - Rizzo, G. AU - Vetro, A. AU - Ardito, L. AU - Torchiano, M. AU - Morisio, M. C3 - 15th Annual Conference on Evaluation and Assessment in Software Engineering, EASE 2011, April 11, 2011 - April 12, 2011 DA - 2011 DO - 10.1049/ic.2011.0004 KW - Rating software engineering L1 - internal-pdf://1734575197/06083159.pdf N1 -Compilation and indexing terms, Copyright 2016 Elsevier Inc.
PB - Institution of Engineering and Technology PY - 2011 SP - 31-35 ST - Linked data approach for selection process automation in systematic reviews T3 - IET Seminar Digest TI - Linked data approach for selection process automation in systematic reviews UR - http://dx.doi.org/10.1049/ic.2011.0004 VL - 2011 ID - 1233 ER - TY - JOUR AU - Tsafnat, Guy AU - Dunn, Adam AU - Glasziou, Paul AU - Coiera, Enrico DA - 2013 J2 - BMJ KW - *Automation *Review Literature as Topic Algorithms artificial intelligence data mining L1 - internal-pdf://3862120563/bmj.f139.full.pdf LA - eng PY - 2013 SN - 1756-1833 0959-535X SP - f139 ST - The automation of systematic reviews T2 - BMJ (Clinical research ed.) TI - The automation of systematic reviews VL - 346 ID - 152 ER - TY - JOUR AB - Target identification for tractography studies requires solid anatomical knowledge validated by an extensive literature review across species for each seed structure to be studied. Manual literature review to identify targets for a given seed region is tedious and potentially subjective. Therefore, complementary approaches would be useful. We propose to use text-mining models to automatically suggest potential targets from the neuroscientific literature, full-text articles and abstracts, so that they can be used for anatomical connection studies and more specifically for tractography. We applied text-mining models to three structures: two well-studied structures, since validated deep brain stimulation targets, the internal globus pallidus and the subthalamic nucleus and, the nucleus accumbens, an exploratory target for treating psychiatric disorders. We performed a systematic review of the literature to document the projections of the three selected structures and compared it with the targets proposed by text-mining models, both in rat and primate (including human). We ran probabilistic tractography on the nucleus accumbens and compared the output with the results of the text-mining models and literature review. Overall, text-mining the literature could find three times as many targets as two man-weeks of curation could. The overall efficiency of the text-mining against literature review in our study was 98% recall (at 36% precision), meaning that over all the targets for the three selected seeds, only one target has been missed by text-mining. We demonstrate that connectivity for a structure of interest can be extracted from a very large amount of publications and abstracts. We believe this tool will be useful in helping the neuroscience community to facilitate connectivity studies of particular brain regions. The text mining tools used for the study are part of the HBP Neuroinformatics Platform, publicly available at http://connectivity-brainer.rhcloud.com/. AU - Vasques, Xavier AU - Richardet, Renaud AU - Hill, Sean L. AU - Slater, David AU - Chappelier, Jean-Cedric AU - Pralong, Etienne AU - Bloch, Jocelyne AU - Draganski, Bogdan AU - Cif, Laura DA - 2015 DO - 10.3389/fnana.2015.00066 J2 - Front Neuroanat KW - globus pallidus internus information extraction natural language processing nucleus accumbens subthalamic nucleus text mining tractography L1 - internal-pdf://4084960731/fnana-09-00066.pdf LA - eng PY - 2015 SN - 1662-5129 1662-5129 SP - 66 ST - Automatic target validation based on neuroscientific literature mining for tractography T2 - Frontiers in neuroanatomy TI - Automatic target validation based on neuroscientific literature mining for tractography VL - 9 ID - 205 ER - TY - JOUR AB - Systematic reviews are being increasingly used to inform all levels of healthcare, from bedside decisions to policy-making. Since they are designed to minimize bias and subjectivity, they are a preferred option to assess the comparative effectiveness and safety of healthcare interventions. However, producing systematic reviews and keeping them up-to-date is becoming increasingly onerous for three reasons. First, the body of biomedical literature is expanding exponentially with no indication of slowing down. Second, as systematic reviews gain wide acceptance, they are also being used to address more complex questions (e.g., evaluating the comparative effectiveness of many interventions together rather than focusing only on pairs of interventions). Third, the standards for performing systematic reviews have become substantially more rigorous over time. To address these challenges, we must carefully prioritize the questions that should be addressed by systematic reviews and optimize the processes of research synthesis. In addition to reducing the workload involved in planning and conducting systematic reviews, we also need to make efforts to increase the transparency, reliability and validity of the review process; these aims can be grouped under the umbrella of 'modernization' of the systematic review process. AU - Wallace, Byron C. AU - Dahabreh, Issa J. AU - Schmid, Christopher H. AU - Lau, Joseph AU - Trikalinos, Thomas A. DA - 2013/05// DO - 10.2217/CER.13.17 IS - 3 PY - 2013 SN - 2042-6305 SP - 273-282 ST - Modernizing the systematic review process to inform comparative effectiveness: tools and methods T2 - Journal of Comparative Effectiveness Research TI - Modernizing the systematic review process to inform comparative effectiveness: tools and methods VL - 2 ID - 1882 ER - TY - JOUR AU - Wallace, Byron C. AU - Small, Kevin AU - Brodley, Carla E. AU - Lau, Joseph AU - Schmid, Christopher H. AU - Bertram, Lars AU - Lill, Christina M. AU - Cohen, Joshua T. AU - Trikalinos, Thomas A. DA - 2012 DP - Google Scholar IS - 7 L1 - internal-pdf://2856877232/gim20127a.pdf PY - 2012 SP - 663-669 ST - Toward modernizing the systematic review pipeline in genetics T2 - Genetics in medicine TI - Toward modernizing the systematic review pipeline in genetics: efficient updating via data mining UR - http://www.nature.com/gim/journal/v14/n7/abs/gim20127a.html http://www.nature.com/gim/journal/v14/n7/full/gim20127a.html VL - 14 Y2 - 2016/09/24/16:00:21 ID - 2396 ER - TY - CONF AB - Active learning (AL) is an increasingly popular strategy for mitigating the amount of labeled data required to train classifiers, thereby reducing annotator effort. We describe a real-world, deployed application of AL to the problem of biomedical citation screening for systematic reviews at the Tufts Medical Center's Evidence-based Practice Center. We propose a novel active learning strategy that exploits a priori domain knowledge provided by the expert (specifically, labeled features) and extend this model via a Linear Programming algorithm for situations where the expert can provide ranked labeled features. Our methods outperform existing AL strategies on three real-world systematic review datasets. We argue that evaluation must be specific to the scenario under consideration. To this end, we propose a new evaluation framework for finite-pool scenarios, wherein the primary aim is to label a fixed set of examples rather than to simply induce a good predictive model. We use a method from medical decision theory for eliciting the relative costs of false positives and false negatives from the domain expert, constructing a utility measure of classification performance that integrates the expert preferences. Our findings suggest that the expert can, and should, provide more information than instance labels alone. In addition to achieving strong empirical results on the citation screening problem, this work outlines many important steps for moving away from simulated active learning and toward deploying AL for real-world applications. 2010 ACM. AU - Wallace, Byron C. AU - Small, Kevin AU - Brodley, Carla E. AU - Trikalinos, Thomas A. C3 - 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD-2010, July 25, 2010 - July 28, 2010 DA - 2010 DO - 10.1145/1835804.1835829 KW - Decision theory Text processing N1 -Compilation and indexing terms, Copyright 2016 Elsevier Inc.
PB - Association for Computing Machinery PY - 2010 SP - 173-181 ST - Active learning for biomedical citation screening T3 - Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining TI - Active learning for biomedical citation screening UR - http://dx.doi.org/10.1145/1835804.1835829 ID - 697 ER - TY - JOUR AU - Wallace, Byron C. AU - Trikalinos, Thomas A. AU - Lau, Joseph AU - Brodley, Carla AU - Schmid, Christopher H. DA - 2010 DP - Google Scholar IS - 1 L1 - internal-pdf://3516616716/art%253A10.1186%252F1471-2105-11-55.pdf PY - 2010 SP - 1 ST - Semi-automated screening of biomedical citations for systematic reviews T2 - BMC bioinformatics TI - Semi-automated screening of biomedical citations for systematic reviews UR - http://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-11-55 VL - 11 Y2 - 2016/09/24/15:32:42 ID - 2308 ER - TY - JOUR AB - OBJECTIVE: Systematic reviews (SRs) are the cornerstone of evidence-based medicine. In this study, we evaluated the effectiveness of using two computer screens on the efficiency of conducting SRs. STUDY DESIGN AND SETTING: A cohort of reviewers before and after using dual monitors were compared with a control group that did not use dual monitors. The outcomes were time spent for abstract screening, full-text screening and data extraction, and inter-rater agreement. We adopted multivariate difference-in-differences linear regression models. RESULTS: A total of 60 SRs conducted by 54 reviewers were included in this analysis. We found a significant reduction of 23.81 minutes per article in data extraction in the intervention group relative to the control group (95% confidence interval: -46.03, -1.58, P = 0.04), which was a 36.85% reduction in time. There was no significant difference in time spent on abstract screening, full-text screening, or inter-rater agreement between the two groups. CONCLUSION: Using dual monitors when conducting SRs is associated with significant reduction of time spent on data extraction. No significant difference was observed on time spent on abstract screening or full-text screening. Using dual monitors is one strategy that may improve the efficiency of conducting SRs. AU - Wang, Zhen AU - Asi, Noor AU - Elraiyah, Tarig A. AU - Abu Dabrh, Abd Moain AU - Undavalli, Chaitanya AU - Glasziou, Paul AU - Montori, Victor AU - Murad, Mohammad Hassan DA - 2014/12//undefined DO - 10.1016/j.jclinepi.2014.06.011 IS - 12 J2 - J Clin Epidemiol KW - *Efficiency *Review Literature as Topic Computer Terminals/*statistics & numerical data Data Mining/*statistics & numerical data Efficiency Evidence-Based Medicine Humans Linear Models Research Design systematic reviews Technology Time Factors Validity L1 - internal-pdf://1059961393/1-s2.0-S0895435614002376-main.pdf LA - eng PY - 2014 SN - 1878-5921 0895-4356 SP - 1353-1357 ST - Dual computer monitors to increase efficiency of conducting systematic reviews T2 - Journal of clinical epidemiology TI - Dual computer monitors to increase efficiency of conducting systematic reviews VL - 67 ID - 89 ER - TY - JOUR AB - Automatic document classification can be valuable in increasing the efficiency in updating systematic reviews (SR). In order for the machine learning process to work well, it is critical to create and maintain high-quality training datasets consisting of expert SR inclusion/exclusion decisions. This task can be laborious, especially when the number of topics is large and source data format is inconsistent.To approach this problem, we build an automated system to streamline the required steps, from initial notification of update in source annotation files to loading the data warehouse, along with a web interface to monitor the status of each topic. In our current collection of 26 SR topics, we were able to standardize almost all of the relevance judgments and recovered PMIDs for over 80% of all articles. Of those PMIDs, over 99% were correct in a manual random sample study. Our system performs an essential function in creating training and evaluation data sets for SR text mining research. AU - Yang, Jianji J. AU - Cohen, Aaron M. AU - McDonagh, Marian S. DA - 2008 J2 - AMIA Annu Symp Proc KW - *Databases, Factual *Natural Language Processing *Terminology as Topic Abstracting and Indexing as Topic/*methods Algorithms artificial intelligence Pattern Recognition, Automated/*methods Periodicals as Topic/*classification PubMed/*classification United States L1 - internal-pdf://3172686009/amia-0825-s2008.pdf LA - eng PY - 2008 SN - 1942-597X 1559-4076 SP - 825-829 ST - SYRIAC: The systematic review information automated collection system a data warehouse for facilitating automated biomedical text classification T2 - AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium TI - SYRIAC: The systematic review information automated collection system a data warehouse for facilitating automated biomedical text classification ID - 268 ER -