Quality and rigor of the concept mapping methodology: A pooled study analysis

https://doi.org/10.1016/j.evalprogplan.2011.10.003Get rights and content

Abstract

The use of concept mapping in research and evaluation has expanded dramatically over the past 20 years. Researchers in academic, organizational, and community-based settings have applied concept mapping successfully without the benefit of systematic analyses across studies to identify the features of a methodologically sound study. Quantitative characteristics and estimates of quality and rigor that may guide for future studies are lacking. To address this gap, we conducted a pooled analysis of 69 concept mapping studies to describe characteristics across study phases, generate specific indicators of validity and reliability, and examine the relationship between select study characteristics and quality indicators. Individual study characteristics and estimates were pooled and quantitatively summarized, describing the distribution, variation and parameters for each. In addition, variation in the concept mapping data collection in relation to characteristics and estimates was examined. Overall, results suggest concept mapping yields strong internal representational validity and very strong sorting and rating reliability estimates. Validity and reliability were consistently high despite variation in participation and task completion percentages across data collection modes. The implications of these findings as a practical reference to assess the quality and rigor for future concept mapping studies are discussed.

Highlights

► We generate specific indicators of validity and reliability for a pooled sample of concept mapping studies. ► We describe the distribution, variation and parameters for each characteristic across concept mapping study phases. ► We examine the relationship between select study characteristics and quality indicators. ► Strong internal representational validity and very strong sorting and rating reliability estimates are present. ► Participation and task completion percentages vary across data collection modes.

Introduction

More than 20 years ago, Trochim and colleagues published a series of papers on concept mapping in a special issue of Evaluation and Program Planning (Trochim, 1989a). In this seminal work, the theoretical and practical features of concept mapping were outlined, making a case for its utility in planning, evaluation, and research. Since then, concept mapping has been applied in a number of fields and contexts, including public and community health (Rao et al., 2005, Risisky et al., 2008, Trochim et al., 2006, Trochim et al., 2004), social work (Petrucci and Quinlan, 2007, Ridings et al., 2008), health care (Trochim & Kane, 2005), human services (Pammer et al., 2001, Paulson and Worth, 2002), and biomedical research and evaluation (Kagan et al., 2009, Robinson and Trochim, 2007, Trochim et al., 2008). The publication of the book Concept Mapping for Planning and Evaluation (Kane & Trochim, 2007) provided concept mapping practitioners with a comprehensive methodological resource.

Over the course of two decades, concept mapping has demonstrated value in addressing a variety of practical and theoretical questions. As concept mapping has gained in popularity, so too has the need to define and examine the characteristics of the method's methodological quality. No published research exists that has systematically assessed the degree to which concept mapping produces valid and reliable results across an array of different studies. The absence of such information limits researchers’ ability to articulate, assess and improve the methodological quality of concept mapping studies. To address this need, we accessed a large sample of concept mapping studies to: (a) quantitatively describe study characteristics across different phases of the process; (b) quantitatively describe specific indicators of validity and reliability; and (c) examine the relationship between select study characteristics and quality indicators. As a context for this study, we provide a succinct overview of concept mapping, followed by a rationale for examining the quality of concept mapping as a mixed-method approach. Finally, we briefly outline an explanation of validity and reliability, as they pertain to concept mapping.

Concept mapping is a type of structured conceptualization method designed to organize and represent ideas from an identified group. A participatory mixed-methods approach, concept mapping integrates qualitative individual and group processes with multivariate statistical analyses to help a group of individuals describe ideas on any topic of interest and represent these ideas visually through a series of related two-dimensional maps (Kane and Trochim, 2007, Trochim, 1989a). Concept mapping is used frequently in evaluation as a practical means of addressing stakeholder participation in ways that enhance the relevance, ownership and utilization of evaluation (Cousins & Whitmore, 1998). The multi-phase concept mapping process typically requires participants to first brainstorm a large set of statements relevant to the topic of interest. Second, each participant sorts these statements into piles based on perceived similarity, and rates each statement on one or more scales. Third, multivariate analyses are conducted that include two-dimensional multidimensional scaling (MDS) of the unstructured sort data, a hierarchical cluster analysis of the MDS coordinates, and the computation of average ratings for each statement and cluster of statements. The maps that result show the individual statements in two-dimensional (x, y) space with more similar statements located nearer each other, and show how the statements are grouped into clusters that partition the space on the map. Finally, the group interprets the maps that result from the analyses through a structured interpretation process designed to help them understand the maps and label them in a substantively meaningful way. The quantitative maps reveal how a group discerns the interrelationships between and among items and assigns values to ideas and concepts, thus constructing a basis for further discussion, interpretation, and action. We refer readers to Kane and Trochim (2007) for a more detailed description of the entire concept mapping process.

Consistent with the arguments for combining qualitative and quantitative methods in a single study (Creswell and Plano Clark, 2007, Sale et al., 2002, Tashakkori and Teddlie, 1998) concept mapping blends the two in a complementary and additive manner. Rather than data remaining distinct but connected as with some mixed-method applications, concept mapping integrates data at multiple points of the process. Qualitative and quantitative methods are combined in ways that challenge the distinction between the two, and may suggest they may be more deeply intertwined (Kane & Trochim, 2007). Given the presence of several design typologies that emphasize a range of sequencing and mixing decisions (e.g. Creswell and Plano Clark, 2007, Tashakkori and Teddlie, 1998) addressing the quality of concept mapping is pertinent. However, the absence of a comprehensive set of criteria for critically appraising mixed-method studies (Tashakkori and Teddlie, 1998, Sale and Brazil, 2004) and the conceptual variation of mixed-method quality among evaluators and researchers (Caracelli & Riggin, 1994) further compound how concept mapping quality should be operationalized. Although generic criteria have been used to assess the quality of mixed-method studies, the need for more specific evaluation criteria, depending on the design and approach, is warranted (Sale & Brazil, 2004). This perspective supports the need to address the methodological quality of concept mapping in ways unique to the approach.

The traditional notions of external and internal validity are challenging to operationalize for concept mapping, and are frequently overlooked. As defined by Cook and Campbell (1979), validity is the best available approximation of truth or falsity, both externally and internally, of a given inference, proposition, or conclusion. Because validity can be operationalized differently, we posit that external representational validity and internal representational validity may be analogues for concept mapping. External representational validity is concerned with the extent to which a conceptualized model mirrors the reality it is purported to represent. Analytical strategies for concept mapping to assess the degree to which the conceptual model is recognized as the modal representation for a group have been suggested (Cacy, 1996). These techniques, however, are exploratory and not yet practical. Typically, the assessment of external representational validity is generally managed as a function of each concept mapping study by seeking verification that the brainstormed statement set represents the topic under inquiry, using multiple data collection and analysis methods, and including independent participants with diverse perspectives. Because uniform data relevant to external validity are not routinely available for individual studies to include in a pooled analysis, external representational validity is not considered in this study.

Internal representational validity however, is particularly germane to this study. Internal representational validity refers to the degree to which the conceptualized model reflects the judgments made by participants in organizing information to produce the model. In that sense, the question of whether the conceptualized model reveals the same distinctions among groupings made by the average participant is of particular importance. A case has been made that the analytic approach that anchors concept mapping results represents the best fit of the various cognitive structures of participants (Forgas, 1979). Questions have been raised, however, as to whether the final model may obscure some of the finer details; due perhaps to variations in how participants approach the structuring task (Keith, 1989). Thus, determining the overall match between the participant-structured input and the mathematically generated output is central to assessing internal representational validity.

Several data elements common to all concept mapping studies can be used to evaluate the correspondence of the represented model to the original participant structures. First, early work by Dumont (1989) and Trochim (1989b) suggests that the degree of configural similarity between input and output matrices can be measured by computing a Pearson's Product–Moment correlation coefficient. A second measure, the stress value, is a goodness-of-fit indicator between a given set of dissimilarities as input and the resultant distances in a configuration (Kruskal & Wish, 1978). Finally, the individual sorting input (i.e. the number of sorted piles) can be examined relative to the number of clusters, to understand the relationship between the groupings from each participant and the final partitioning of the content represented in the map. Collectively, these measures computed from data routinely produced for each concept mapping study can be used to estimate the internal representational validity of the conceptualized model.

For concept mapping, the consistency of participant input can be assessed using the sorting and rating data. Reliability of participant ratings on a chosen scale for each of the final statements can be assessed by computing conventional item and rater reliability estimates. The reliability of participant input of the perceived relationships between statements can be assessed by computing a set of estimates that are specific to concept mapping sort data. As suggested by Trochim (1993), the traditional theory of reliability, as typically applied in social science research, does not readily conform to sort data in the concept mapping model. Conventional means for assessing reliability focus on estimating the repeatability of test items or total scores, based on some known or assumed correct response. Sort data in concept mapping is different. Instead of estimating the reliability of items or overall scores of a measure, sorting reliability assessment is more appropriately focused on determining the extent to which the structural arrangements, both individually and collectively, reflect an assumed normatively typical arrangement. Thus, the individual and aggregated sort configurations (similarity matrices used as input by multidimensional scaling), as well as the resulting distance matrices (the between-item Euclidean distances output generated by multidimensional scaling), provide information to calculate reliability estimates. We refer to Trochim's (1993) recommended procedures for calculating a set of reliability statistics specific to concept mapping input and output to estimate consistency of sorting within and across specific studies.

Section snippets

Overall study approach

The standardized procedures for data collection, data organization, analysis, and representation in the concept mapping process yield a set of common quantitative data, which can be configured in a comparable statistical form. This uniformity allows for the same quality constructs and relationships to be examined, and thus produces results that are more objective and exact than a narrative review. Our first step in conducting this pooled study analysis was to generate quantitative

Results

A majority of the concept mapping studies in the sample were classified as public health oriented (59.4%). Others were in the fields of human services (20.3%), biomedical research (5.8%), social science research (2.9%), and business or human resources (2.9%). Twenty-eight studies were supported through Federal sources (41.5%), with several receiving support from foundations or not-for-profit organizations (20.3%), universities or colleges (17.4%), and state government sources (11.6%). The

Discussion

As with similar mixed-method applications, the concept mapping studies that are the basis of this pooled analysis were conducted to understand complex realities using data from multiple perspectives to combine and present practical information. This quantitative analysis generated useful baseline information to address questions regarding the methodological quality of concept mapping. The study approach suggested several means for determining the quality of concept mapping that are appropriate,

Acknowledgements

The authors would like to acknowledge the support of the staff at Concept Systems, Inc., Specifically we thank Brenda Pepe for information related to the concept mapping studies included in the analysis, Perry Slack for study level data extraction, and Marie Cope for review and comments on the manuscript. We reference and acknowledge the foundational thinking and work of William Trochim in the methodology's early applications and summary reliability estimates of almost 20 years ago.

Scott R. Rosas, PhD is a Senior Consultant at Concept Systems, Inc. where he specializes in the design and use of the concept mapping methodology. His work has focused on conceptualization and measurement in evaluation using concept mapping, with attention to the validity and reliability of the approach. He received his PhD in Human Development and Family Studies from the University of Delaware, with an emphasis on program evaluation. He previously served as Associate Faculty at the Bloomberg

References (40)

  • J.B. Cousins et al.

    Framing participatory evaluation

    New Directions for Evaluation

    (1998)
  • J.W. Creswell et al.

    Designing and conducting mixed methods research

    (2007)
  • M.L. Davison

    Multidimensional scaling

    (1983)
  • R.F. DeVellis

    Scale development: Theory and applications

    (1991)
  • K.L. Dugosh et al.

    Cognitive stimulation in brainstorming

    Journal of Personality and Social Psychology

    (2000)
  • J.P Forgas

    Multidimensional scaling: A discovery method in social psychology

  • K. Jackson et al.

    Concept mapping as an alternative approach for the analysis of open-ended survey responses

    Organizational Research Methods

    (2002)
  • J.M. Kagan et al.

    Developing a conceptual framework for an evaluation system for the NIAID HIV/AIDS clinical trials networks

    Health Research Policy and Systems

    (2009)
  • M. Kane et al.

    Concept mapping for planning and evaluation

    (2007)
  • M.D. Kaplowitz et al.

    A comparison of web and mail survey response rates

    Public Opinion Quarterly

    (2004)
  • Cited by (372)

    View all citing articles on Scopus

    Scott R. Rosas, PhD is a Senior Consultant at Concept Systems, Inc. where he specializes in the design and use of the concept mapping methodology. His work has focused on conceptualization and measurement in evaluation using concept mapping, with attention to the validity and reliability of the approach. He received his PhD in Human Development and Family Studies from the University of Delaware, with an emphasis on program evaluation. He previously served as Associate Faculty at the Bloomberg School of Public Health at Johns Hopkins University and is currently Adjunct Faculty in the Department of Health at SUNY-Cortland.

    Mary Kane, MSLIS is the Chief Executive and Principal Consultant at Concept Systems, Inc. Her consulting experience includes strategic and operational planning, product and program development, education and training design, and program needs assessment and evaluation. She has coauthored several articles on the application of concept mapping across several content areas, including public and community health. Ms. Kane is co-author of the definitive volume on concept mapping: Concept Mapping for Planning and Evaluation. She holds a Masters degree in Library and Information Sciences from Columbia University.

    View full text