Author: Antonio Gutierrez

Executive Function: Can Metacognitive Awareness Training Improve Performance?

by Antonio Gutierrez, Georgia Southern University

In a recent meta-analysis of 67 research studies that utilize an intervention targeted at enhancing metacognitive awareness, Jacob and Parkinson (in press) argue that metacognitive interventions aimed at improving executive function processes are not as effective at improving student achievement as once believed by scholars and practitioners alike. In essence, the evidence in support of robust effects of these types of interventions in improving achievement is inconclusive. While descriptive research studies continue to report high associations between metacognitive awareness and performance or achievement measures, Jacob and Parkinson argue that the experimental evidence supporting a strong role of metacognitive training in improving student performance is scant. I have recently pondered a similar dilemma with research on the effect of metacognitive monitoring training on students’ performance, confidence judgments but especially calibration. The literature on these topics converges on the finding that metacognitive monitoring training improves performance and confidence in performance judgments but not necessarily calibration (see e.g., Bol et al., 2005; Gutierrez & Schraw, 2015; Hacker et al., 2008).

While Jacob and Parkinson’s meta-analysis is illuminating, I wonder whether, like the calibration literature, the conclusion that executive function interventions are not as effective at improving achievement may be due to very different conceptualizations of the constructs under investigation. In the case of calibration, the mixed findings may be due to the fact that the metacognitive monitoring interventions were not likely targeting the same thing. For instance, some interventions may have been targeting a reduction in calibration errors (overconfidence and underconfidence), others may have been targeting improvement in calibration accuracy, whereas yet others may have been targeting both, whether intentionally or unintentionally. Because these interventions were targeting different aspects of calibration, it could be that the inconclusive findings were due to a confounding of these various dimensions of calibration … comparing apples to oranges, if you will. Could the lack of robust effects of executive function interventions on achievement be due to a similar phenomenon? What if these studies were not targeting the same executive function processes, in which case they would not be as directly comparable as at first glance? Jacob and Parkinson’s (in press) study may lead some to believe that there is little to be gained in investing time and effort in executive function interventions. However, before we abandon these interventions, perhaps we should consider developing executive function interventions that are more specific and finer grained such as by targeting very specific aspects of the executive function rather than a more general approach.

References
Bol, L., Hacker, D. J., O’Shea, P., & Allen, D. (2005). The influence of overt practice, achievement level, and explanatory style on calibration accuracy, and performance. The Journal of Experimental Education, 73, 269-290.

Gutierrez, A. P., & Schraw, G. (2015). Effects of strategy training and incentives on students’ performance, confidence, and calibration. The Journal of Experimental Education: Learning, Instruction, and Cognition. Advance online publication. doi: 10.1080/00220973.2014.907230

Hacker, D. J., Bol, L., & Bahbahani, K. (2008). Explaining calibration accuracy in classroom contexts: The effects of incentives, reflection, and explanatory style. Metacognition Learning, 3, 101-121.

Jacob, R., & Parkinson, J. (in press). The potential for school-based interventions that target executive function to improve academic achievement: A review. Review of Educational Research. Advance online publication. doi: 10.3102/0034654314561338


Comprehension Monitoring: The Role of Conditional Knowledge Part 2

by Antonio Gutierrez, Georgia Southern University

In my previous post, I discussed the role conditional knowledge (i.e., the why, when, and where to apply strategies given task demands) plays in learners’ ability to calibrate their performance against their actual performance. This is in light of debates about the relations between the various dimensions of metacognition. Calibration is a component of one’s ability to monitor comprehension, which is a regulatory function. Conditional knowledge, on the other hand, is part of the knowledge component of metacognition. As a researcher I often wonder whether instead of making assumptions that these various metacognitive functions are related whether perhaps we should empirically test these assumptions. In metacognitive research it is often assumed that the knowledge and regulation aspects of metacognition are related. From a theoretical perspective, this makes sense. However, for us to assume that this is the case with all samples and populations may be a stretch, especially given the diversity and individual differences among learners. In this vein, I am currently seeking ethics approval to conduct research with middle school students because this is an understudied population with respect to metacognition. In this proposed research I plan to not only investigate calibration among middle school students and the influence metacognitive strategy training has on learners’ calibration, but I plan to empirically assess the association between the eight dimensions of metacognition (Knowledge: declarative, procedural, and conditional; Regulation: planning, information management, debugging strategies; comprehension monitoring; and evaluation of learning). I will also attempt to test the predictive power of various components of metacognition on learners’ calibration. I am especially interested in empirically measuring the association between conditional knowledge and calibration as well as the predictive power of conditional knowledge on calibration. I expect that metacognitive strategy training will improve learners’ performance, confidence judgments, and also their calibration. I also suspect that those with greater conditional knowledge will have better calibration, and hence, I expect conditional knowledge to strongly predict calibration.

This particular study is one among a series of scientific investigations on the validity of theoretical claims made when researchers discuss metacognition. In my attempt to provide educators a toolkit of domain-general metacognitive strategies they can readily apply in their classrooms, this series of studies will help me provide the empirical evidence necessary to demonstrate the utility and relevance of metacognitive strategies to not only scholars but practitioners as well. These strategies have been adapted from an unpublished pilot study I conducted prior to my dissertation. This research will help me to continue to refine these strategies to better suit adolescents. Moreover, it will shed some light on the link between conditional metacognitive knowledge and calibration, which is a topic that began with earlier posts. Stay tuned for preliminary results of my first investigation.


Comprehension Monitoring: The Role of Conditional Knowledge

By Antonio Gutierrez, Georgia Southern University

In my previous post, Metacognitive Strategies: Are They Trainable?, I explored the extent to which metacognitive strategies are teachable. In my own research on how well students monitor their comprehension during learning episodes, I discovered that students reported already having a repertoire of metacognitive strategies. Yet, I have often found, in my own teaching and interaction with undergraduate and even graduate students, that having metacognitive declarative knowledge of strategies is often not sufficient to promote students’ comprehension monitoring. For instance, students may already know to draw a diagram when they are attempting to learn new concepts. However, they may not know under which circumstances it is best to apply such a strategy. When students do not know when, where and why to apply a strategy, they may in fact be needlessly expending cognitive resources for little, to no, benefit with respect to learning.

Schraw and Dennison (1994) argued that metacognition is divided in to knowledge and regulation components. Knowledge is comprised of declarative knowledge about strategies, procedural knowledge of how to apply them, and conditional knowledge about when, where, and why to apply strategies given task demands. The more that I engage students, inside and beyond my classes, the more that I become convinced that the greatest lack in metacognitive knowledge lies not in declarative or procedural knowledge, but in conditional knowledge. Students clearly have a repository of strategies and procedures to apply them. However, they seem incapable of applying those strategies effectively given the demands of the learning tasks in which they engage. So how can we enhance students’ conditional knowledge? Let’s assume that Sally is attempting to learn the concept of natural selection in her biology lesson. As Sally attempts to connect what she is learning with prior knowledge in long-term memory, she realizes she may have misconceptions regarding natural selection. She also understands that she has a variety of strategies to assist her in navigating this difficult concept. However, she does not know or understand which strategy will optimize her learning of the concept. Thus, she resorts to a trial-and-error utilization of the strategies she thinks are “best” to help her. Here we see a clear example of lack of adequate conditional knowledge. Much time and cognitive effort can be saved if we enhance students’ conditional knowledge. Calibration (the relationship between task performance and a judgment about that performance; Boekaerts & Rozendaal, 2010; Keren, 1991), a related metacognitive process, but distinct from conditional knowledge, involves the comprehension monitoring element of metacognitive regulation. As I continue my scholarship to deepen my understanding of calibration, I wonder whether conditional knowledge and calibration are more closely associated than researchers assume.

In my recent research on calibration I have often asked why the body of literature on calibration is inconclusive in its findings with respect to the effects of metacognitive strategy training on calibration. For instance, some studies have found positive effects for calibration (e.g., Gutierrez & Schraw, in press; Nietfeld & Schraw, 2002) while others have demonstrated no effect for strategy training on calibration (e.g., Bol et al., 2005; Hacker et al., 2008). This inconclusive evidence has frustrated me not only as a scholar but as a teacher as well. I suspect that these mixed findings in the literature on calibration may be due in part because researchers on calibration have neglected to address participants’ metacognitive conditional knowledge. How can we possibly hope as instructors to improve students’ comprehension monitoring when the findings on the role of metacognitive strategy instruction plays on calibration are inconclusive? So, perhaps as researchers/scholars of metacognition we are asking the wrong questions? I argue that by improving students’ metacognitive conditional knowledge, we can improve their ability to more effectively determine what they know and what they do not know about their learning (i.e., better calibrate their performance judgments to their actual performance). If students cannot effectively apply strategies given the demands of the learning episode (a conditional knowledge issue) how can we expect them to adequately monitor their comprehension (a regulation of learning issue)? Perhaps the next line of inquiry should exclusively focus on the enhancement of students’ conditional knowledge?

 

References

Boekaerts, M., & Rozendaal, J. S. (2010). Using multiple calibration measures in order to capture the complex picture of what affects students’ accuracy of feeling of confidence. Learning and Instruction, 20(4), 372-382. doi:10.1016/j.learninstruc.2009.03.002

Bol, L., Hacker, D. J., O’Shea, P., & Allen, D. (2005). The influence of overt practice, achievement level, and explanatory style on calibration accuracy, and performance. The Journal of Experimental Education, 73, 269-290.

Gutierrez, A. P., & Schraw, G. (in press). Effects of strategy training and incentives on students’ performance, confidence, and calibration. The Journal of Experimental Education: Learning, Instruction, and Cognition.

Hacker, D. J., Bol, L., & Bahbahani, K. (2008). Explaining calibration accuracy in classroom contexts: The effects of incentives, reflection, and explanatory style. Metacognition Learning, 3, 101-121.

Keren, G. (1991). Calibration and probability judgments: Conceptual and methodological issues. Acta Psychologica, 77(2), 217- 273. http://dx.doi.org/10.1016/0001-6918(91)90036-Y

Nietfeld, J. L., & Schraw, G. (2002). The effect of knowledge and strategy explanation on monitoring accuracy. Journal of Educational Research, 95, 131-142.


Metacognitive Strategies: Are They Trainable?

by Antonio Gutierrez, Southern Georgia University

Effective learners use metacognitive knowledge and strategies to self-regulate their learning (Bol & Hacker, 2012; Bjork, Dunlosky & Kornell, 2013; Ekflides, 2011; McCormick, 2003; Winne, 2004; Zeidner, Boekaerts & Pintrich, 2000; Zohar & David, 2009). Students are effective self-regulators to the extent that they can accurately determine what they know and use relevant knowledge and skills to perform a task and monitor their success. Unfortunately, many students experience difficulty learning because they lack relevant knowledge and skills, do not know which strategies to use to enhance performance, and find it difficult to sequence a variety of relevant strategies in a manner that enables them to self-regulate their learning (Bol & Hacker, 2012; Grimes, 2002).

Strategy training is a powerful educational tool that has been shown to overcome some of these challenges in academic domains such as elementary and middle school mathematics (Carr, Taasoobshirazi, Stroud & Royer, 2011; Montague, Krawec, Enders & Dietz, 2014), as well as non-academic skills such as driving and anxiety management (Soliman & Mathna, 2009). Additional benefits of strategy training are that using a flexible repertoire of strategies in a systematic manner not only produces learning gains, but also empowers students psychologically by increasing their self-efficacy (Dunlosky & Metcalfe, 2009). Further, a common assumption is that limited instructional time with younger children produces life-long benefits once strategies are automatized (McCormick, 2003; Palincsar, 1991; Hattie et al., 1996).

In addition to beginning strategy instruction as early as possible, it should be embedded within all content areas, modeled by teachers and self-regulated students, practiced until automatized, and discussed explicitly in the classroom to provide the greatest benefit to students. Pressley and Wharton-McDonald (1997) recommend that strategy instruction be included before, during, and after the main learning episode. Strategies that occur before learning include setting goals, making predictions, determining how new information relates to prior knowledge, and understanding how the new information will be used. Strategies needed during learning include identifying important information, confirming predictions, monitoring, analyzing, and interpreting. Strategies typically used after learning include reviewing, organizing, and reflecting. Good strategy users should possess some degree of competence in each of these areas to be truly self-regulated.

Additional strategies have been studied by Schraw and his colleagues (Gutierrez & Schraw, in press; Nietfeld & Schraw, 2002). They demonstrated that a repertoire of seven strategies is effective at improving undergraduate students’ learning outcomes and comprehension monitoring, a main component of the regulatory dimension of metacognition. Table 1 contains the seven strategies explicitly taught to students. Moreover, these strategies can function not only in contrived laboratory settings but also in ecologically valid settings, such as classrooms.

Table 1. Summary of Metacognitive Strategies and their Relation to Comprehension Monitoring

 

Strategy

 LearningProcesses

Hypothesized Influence on Comprehension
Review main objectives of the text and focus on main ideas and overall meaning Review and monitor Enhance calibration through clarifying misunderstandings and tying details to main ideas
Read and summarize material in your own words to make it meaningful; use elaboration and create your own examples Read and relate Enhances calibration by transforming knowledge into something personally meaningful
Reread questions and responses and reflect on what the question is asking; go through and take apart the question paying attention to relevant concepts Review, relate, and monitor Purposefully slowing information processing allows for a more accurate representation of the problem, thus decreasing errors in judgment
Use contextual cues in the items and responses, e.g., bolded, italicized, underlined, or capitalized words Relate Using contextual cues allows the mind to focus on salient aspects of the problem rather than seductive details, thereby increasing accuracy
Highlight text; underline keywords within the question to remind yourself to pay attention to them; use different colors to represent different meanings Review, relate, and monitor Highlighting and underlining can assist one to focus on main ideas and what is truly important, increasing accuracy; however, relying too much on this can be counterproductive and may potentially increase errors
Relate similar test questions together and read them all before responding to any Relate and monitor Relating information together provides a clearer understanding of the material and may highlight inconsistencies that need to be resolved; it may point to information the learner may have missed, increasing accuracy
Use diagrams, tables, pictures, graphs, etc. to help you organize information Review and relate These strategies help simplify complex topics by breaking them down to their constituent parts; this increases accuracy by decreasing errors

Adapted from Gutierrez and Schraw (in press).

However, while the studies by Shaw and colleagues have shown that teachers can effectively use these strategies to improve students’ comprehension monitoring and other learning outcomes, they have not thoroughly investigated why and how these strategies are effective. I argue that the issue is not so much that students are not aware of the metacognitive strategies, but rather that many lack the conditional metacognitive knowledge−that is, the where, when, and why to apply a given strategy taking into consideration task demands. Future research should investigate these process questions, namely when, how, and why different strategies are successful.

Bjork, R. A., Dunlosky, J., & Kornell, N. (2013).  Self-regulated learning: Beliefs, techniques and illusions. Annual Review of Psychology, 64, 417-447.

Bol, L. & Hacker, D. J. (2012). Calibration research: where do we go from here? Frontiers in Psychology, 3, 1-6.

Carr, M., Taasoobshirazi, G., Stroud, R., & Royer, J. M. (2011). Combined fluency and cognitive strategies instruction improves mathematics achievement in early elementary school. Contemporary Educational Psychology, 36, 323–333.

Dunlosky, J., & Metcalfe, J. (2009).  Metacognition. Thousand Oaks, CA: Sage Publications.

Ekflides, A. (2011). Interactions of metacognition with motivation and affect in self-regulated learning: The MASRL model. Educational Psychologist, 46, 6-25.

Grimes, P. W. (2002). The overconfident principles of economics students: An examination of metacognitive skill. Journal of Economic Education, 1, 15–30.

Gutierrez, A. P., & Schraw, G. (in press). Effects of strategy training and incentives on students’ performance, confidence, and calibration. The Journal of Experimental Education: Learning, Instruction, and Cognition.

Hattie, J., Biggs, J., & Purdie, N. (1996). Effects of learning skills interventions on student learning: A meta-analysis. Review of Educational Research, 66, 99-136. doi: 10.3102/00346543066002099

McCormick, C. B. (2003). Metacognition and learning. In W. M. Reynolds & G. E. Miller (Eds.), Handbook of psychology: Educational psychology (pp. 79-102). Hoboken, NJ: John Wiley & Sons.

Montague, M., Krawec, J., Enders, C. & Dietz, S. (2014). The effects of cognitive strategy instruction on math problem solving of middle-school students of varying ability. Journal of Educational Psychology,106,469 – 481.

Nietfeld, J. L., & Schraw, G. (2002). The effect of knowledge and strategy explanation on monitoring accuracy. Journal of Educational Research, 95, 131-142.

Palincsar, A. S. (1991). Scaffolded instruction of listening comprehension with first graders at risk for academic difficulty. In A. M. McKeough & J. L. Lupart (Eds.), Toward the practice of theory-based instruction (pp. 50–65). Mahwah, NJ: Erlbaum.

Pressley, M., & Wharton-McDonald, R.  (1997).  Skilled comprehension and its development through instruction.  School Psychology Review, 26, 448-466.

Soliman, A. M. & Mathna, E. K. (2009). Metacognitive strategy training improves driving situation awareness. Social Behavior and Personality,37, 1161-1170.

Winne, P. H. (2004). Students’ calibration of knowledge and learning processes: Implications for designing powerful software learning environments. International Journal of Educational Research, 41,466-488. doi:http://dx.doi.org/10.1016/j.ijer.2005.08.012

Zeidner, M., Boekaerts, M., & Pintrich, P. R.  (2000).  Self-regulation: Directions and challenges for future research.  In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.),  Handbook of self-regulation (pp. 13-39).  San Diego, CA: Academic Press.

Zohar, A., & David, A. (2009). Paving a clear path in a thick forest: a conceptual analysis of a metacognitive component. Metacognition & Learning, 4(3), 177-195.