Author: Chris Was

Some Developmental Trends in Metacognition

By Chris Was, PhD; Kent State University

Recently, I have conducted some experiments with K – 6 grade students related to children’s ability to predict their ability to recall simple items. Although a simple measure, this form of calibration is a measure of a child’s knowledge of the own memory abilities. This is, at its most basic level, metacognition.

The work in which my collaborators and I are currently engaged builds on the work of Amanda Lipko and colleagues (e.g., Lipko, Dunlosky, & Merriman, 2009). What was most striking about Lipko’s work was the robust overconfidence displayed by preschool children. Granted, there is a large body of literature that demonstrates young children are overconfident in both their physical abilities (e.g., Plumert, 1995) as well as their cognitive abilities (e.g., Cunningham & Weaver, 1989; Flavell, Friedrichs, & Hoyt, 1970). Much of this work indicates that with preschool children this overconfidence is quite persistent. But Lipko et al.’s (2009) work found that even following repeated practice and feedback, specifically salient feedback when children recalled their own previous performance, this overconfidence remained.

There are several hypotheses, both tested and untested, as to why this overconfidence exists and why it is robust against correction. Perhaps it is wishful thinking (a hypothesis test by Lipko et al.), perhaps it is a developmental issue, or perhaps it serves as a learning mechanism (children who give up the first time they fail may not learn to do succeed at much). In any case, I became interested in circumstances in which young children are capable of making accurate predictions of their cognitive abilities.

A review of the experimental methodology used by Lipko et al. is warranted. In their 2009 study Lipko et al. presented young children (mean age of approximately 5 years 0 months) with pictures of common items. As children were presented with pictures they were asked to name them. If correctly named the picture was placed on a board until 10 pictures were on the board. The experimenter then said to the children, “I am going to cover up the pictures,“ and asked, “how many do you think you will remember after I cover them?” The children then made a prediction of how many pictures they would remember. Finally, the children attempted to recall the pictures. In a series of experiments, children were overconfident in their ability even after repeated trials and even after correctly recalling their poor performance on previous trials.

Are there circumstances when children are more accurate? The simple answer is, “yes.” In a recent experiment (Was & Al-Harthy, 2015) we found that when children complete the Lipko task with unfamiliar items, their predictions of how many items they might remember are significantly lower than for familiar items. This familiarity overconfidence bias is likely due something similar to the fluency effect. That is, when the pictures are familiar to children, they seem easy to remember, but when the pictures are unfamiliar, children understand that they might be hard to recall later.

We are also investigating the developmental trends of the ability to predict recall. Our most interesting finding to date, is calibration (accuracy of recall predictions) is strongly related to the increase in working memory capacity. Put differently, as the number of items children are able to recall increases, so does their ability to accurately predict the number of items they will recall. Some will argue that this is not an unsuspected finding. The argument being that as working memory capacity increases, the ability to think about one’s own memory should also increase. My response is that it is not clear if metacognition is directly related to working memory or executive functions. Perhaps a mediating relationship exists. Recent investigations have suggested that performance on many measures of working memory are more dependent on strategy than they are on cognitive ability. Perhaps, metacognition is just good strategy use, or perhaps it is a cognitive ability.

The finding of the relationship between recall performance and calibration (the difference between predicted performance and actual performance) supports the hypothesis that metacognition is not a single skill that children have or not, but rather it is a complex of many skills and processes the children acquire through experiences and maturation. I suggest that developmental research in metacognition need focus on aptitude-by-treatment interactions. Questions such as, “What variety of academic activities contribute to the development of metacognition at different stages or levels of cognitive development?” will not only forward our understanding of metacognition, but perhaps also how to help young students develop metacognitive strategies and perhaps metacognitive performance.

Cunningham, J. G., & Weaver, S. L. (1989). Young children’s knowledge of their memory             span: Effects of task and experience. Journal of Experimental Child Psychology, 48,   32–44.

Flavell, J. H., Friedrichs, A. G., & Hoyt, J. D. (1970). Developmental changes in memorization    processes. Cognitive Psychology, 1,324–340.

Lipko, A. R., Dunlosky, J., & Merriman, W. E. (2009). Persistent overconfidence despite practice: The role of task experience in preschoolers’ recall predictions. Journal of Experimental Child Psychology, 103(2), 152-166.

Plumert, J. M. (1995). Relations between children’s overestimation of their physical abilities and accident proneness. Developmental Psychology, 31(5), 866-876. doi: http://dx.doi.org/10.1037/0012-1649.31.5.866

Was, C. A., & Al-Harthy, I. (2015). Developmental differences in overconfidence: When do children understand that attempting to recall predicts memory performance? The Researcher, 27(1), 1-5, Conference Proceedings of the 32nd Annual Conference of the Northern Rocky Mountain Education Research Association.


Evidence for metacognition as executive functioning

by Kristen Chorba, PhD and Christopher Was, PhD, Kent State University

Several authors have noted that metacognition and executive functioning are descriptive of a similar phenomenon (see Fernandez-Duque, et al., 2000; Flavell, 1987; Livingston, 2003; Shimamura, 2000; Souchay & Insingrini, 2004). Many similarities can be seen between these two constructs: both regulate and evaluate cognitions, both are employed in problem solving, both are required for voluntary actions (as opposed to automatic responses), and more. Fernandez-Duque, et al. (2000) suggest that, despite their similarities, these two areas have not been explored together because of a divide between metacognitive researchers and cognitive neuroscientists; the metacognitive researchers have looked exclusively at metacognition, focusing on issues related to its development in children and its implications for education. They have preferred to conduct experiments in naturalistic settings, as a way to maximize the possibility that any information gained could have practical applications. Cognitive neuroscientists, on the other hand, have explored executive functioning using neuroimaging techniques, with the goal of linking them to brain structures. In the metacognitive literature, it has been noted metacognition occurs in the frontal cortex; this hypothesis has been evaluated in patients with memory disorders, and studies have noted that patients with frontal lobe damage, including some patients with amnesia, had difficulties performing metacognitive functions, including FOK judgments (Fernandez-Duque, et al., 2000; Janowsky, Shimamura, & Squire, 1989; Shimamura & Squire, 1986; as cited in Shimamura, 2000). Additionally, source monitoring and information retrieval has also been linked with the frontal cortex; source monitoring is an important metacognitive judgment (Shimamura, 2000). As previously stated, executive functions seem to be located generally in the frontal lobes, as well as specifically in other areas of the brain, contributing to the growing body of literature indicating that executive functions are both correlated and function independently. To explore the link between executive functioning and metacognition, Souchay and Isingrini (2004) carried out an experiment in which subjects were first asked to make evaluations on their own metacognition; they were then given a series of neurological tests to assess their executive functioning. They not only found a “significant partial correlation between metamemory control and executive functioning” (p. 89) but, after performing a hierarchical regression analysis, found that “age-related decline in metamemory control may be largely the result of executive limitations associated with aging” (p. 89).

As it relates to executive functioning, Fernandez-Duque, et al. (2008) noted that “the executive system modulates lower level schemas according to the subject’s intentions . . . [and that] without executive control, information processing loses flexibility and becomes increasingly bound to the external stimulus” (p. 289). These authors use the terms executive function and metacognition as essentially interchangeable, and note that these functions enable humans to “guide actions” where preestablished schema are not present and allow the individual to make decisions, select appropriate strategies, and successfully complete a task. Additionally, the primary task of both metacognition and executive functions are top-down strategies, which inform the lower level (i.e.: in metacognition, the object level; in executive functioning, as the construct which controls the “selection, activation, and manipulation of information in working memory” [Shimamura, 2000, p. 315]). Reviewing the similarities between metacognition and executive function, it seems that they are highly correlated constructs and perhaps share certain functions.

Executive functions and metacognition, while exhibiting similar functions and characteristics have, largely, been investigated along separate lines of research. Metacognitive research has focused on application and informing the teaching and learning processes. Executive functions, on the other hand, have primarily been researched as they relate to structures and locations within the brain. Recent literature and research indicates that executive functions and metacognition may be largely the same process.

References

Baddeley, A. (2005). Human Memory: Theory and Practice, Revised Edition. United Kingdom; Bath Press.

Blavier, A., Rouy, E., Nyssen, A., & DeKeyster, V. (2005). Prospective issues for error   detection. Ergonomics, 7(10), 758-781.

Dinsmore, D., Alexander, P., & Loughlin, S. (2008). Focusing the conceptual lens on metacognition, self-regulation, and self-regulated learning. Educational psychology review, 20(4), 391-409.

Dunlosky, J., Metcalfe, J. (2008). Metacognition. Los Angeles: Sage.

Fernandez-Duque, D., Baird, J., Posner, M. (2000). Executive attention and metacognitive regulation. Consciousness and Cognition, 9, 288-307.

Flavell, J. (1987). Speculations about the nature and development of metacognition. In F. Weinert and R. H. Kluwe, (Eds.) Metacognition, Motivation, and Understanding. Hillsdale, NJ: Lawrence Erlbaum.

Friedman, N. P., Haberstick, B. C., Willcutt, E. G., Miyake, A., Young, S. E., Corley, R.   P., & Hweitt, J. K. (2007). Greater attention problems during childhood predict        poorer executive functioning in late adolescence. Psychological Science, 18(10), 893-900.

Friedman, N. P., Miyake, A., Young, S. E., DeFries, J. C., Corley, R. P., Hewitt, J. K. (2008).  Individual differences in executive functions are almost entirely genetic in origin.  Journal of Experimental Psychology, General, 137(2), 201-225.

Friedman, N. P., Miyake, Corley, R. P., Young, S. E., DeFries, J. C., & Hewitt, J. K. (2006). Not all executive functions are related to intelligence. Psychological Science, 17(2), 172-179.

Georghiades, P. (2004). From the general to the situated: Three decades of metacognition.  research report. International Journal of Science Education, 26(3), 365-383.

Higham, P. A. & Gerrard, C. (2005). Not all errors are created equal: Metacognition and   changing answers on multiple-choice tests. Canadian Journal of Experimental   Psychology, 59(1), 28-34.

Keith, N. & Frese, M. (2005) Self-regulation in error management training: Emotion control and    metacognition as mediators of performance effects. Journal of Applied Psychology,  90(4), 677-691.

Keith, N. & Frese, M. (2008). Effects of error management training: A meta-analysis. Journal of Applied Psychology, 93(1), 59-69.

Lajoie, S. (2008). Metacognition, self regulation, and self-regulated learning: A rose by any other name? Educational Psychology Review, 20(4), 469-475.

Livingston, J. A. (2003). Metacognition: An overview. Online ERIC Submission.

Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., & Howenter, A. (2000). The unity and diversity of executive functions and their contributions to complex         “frontal lobe” tasks: A latent variable analysis. Cognitive Psychology, 41, 49-100.

Nelson, T. O., & Narens, L. (1990). Metamemory: A theoretical framework and new

findings. In G. H. Bower (Ed.), The Psychology of Learning and Knowing. Cambridge, MIT Press, p. 1-26.

PP, N. (2008). Cognitions about cognitions: The theory of metacognition. Online ERIC Submission.

Shimamura, A. (2000). Toward a cognitive neuroscience of metacognition. Consciousness and Cognition, 9, 313-323.

Souchay, C., & Isingrini, M. (2004). Age related differences in metacognitive control: Role of executive functioning. Science Direct. 56(1), 89-99.

Thiede, K. W., & Dunlosky, J. (1994). Delaying students’ metacognitive monitoring improves their accuracy in predicting their recognition performance. Journal of educational psychology, 86(2), 290-302.

Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated learning. In D. J. Hacker, J., Dunlosky, & A. Graessser (Eds.), Metacognition in educational theory       and practice, (p. 277-304). Hillsdale, NJ: Lawrence Erlbaum.


A Mindfulness Perspective on Metacognition

by Chris Was, Kent State University

If you have any interest in metacognition, you have likely come across the description of metacognition as thinking about one’s thinking. A number of posts to this blog (including my own) provide evidence to support the conclusion that metacognition can be “learned” and improved. Further, improved metacognition leads to improve self-regulation and positive academic outcomes. There is also a good deal of evidence that training in mindfulness improves cognitive function and attention (e.g., Chambers, Lo, & Allen, 2008). Flook, et al (2010) found that mindfulness-training program improved executive functions in young elementary school students. Zeidan, et al (2010) found that mindfulness training improved executive function and metacognitive insight. This post will focus on the relationship between metacognition and mindfulness.

Let me preface by stating that mindfulness need not refer to esoteric religious beliefs, but it is often defined as a mental state achieved by focusing one’s awareness on the present moment and acknowledging one’s feelings, thoughts and bodily sensations. Kabat-Zinn (1990) describes mindfulness as bringing attention to moment-to-moment experience. In my own work on metacognitive knowledge monitoring, I have required students to make moment-to-moment (more accurately, item-by-item) judgments of their knowledge.. Hypotheses, such as cue-familiarity, provide reasonable explanations for how students and research participants rate feelings of knowing (FOK), judgments of learning (JOLs), judgments of knowing (JOK), etc. However, the simple fact is one must attend to these feelings and thoughts to provide a judgment. In psychological and educational literature, we refer to one using metacognition to make these judgments. Clinical psychology programs such as mindfulness based stress reduction (MBSR) and cognitive behavior therapy (CBT) refer to the patient/participant as being mindful about their emotions, thoughts, and actions. Although the majority of research and application of mindfulness has occurred in clinical settings, there is a great deal of potential in examining the relationship between mindfulness and metacognition.

It isn’t clear how metacognition and mindfulness are related. Some argue that metacognition is not mindful because a true expert in mindfulness does not need to reflect upon his or her thinking, but only to attend to what they are presently doing. I am not convinced. Much of the work on metacognitive improvement has focused on semester long training to improve students knowledge monitoring. The mindfulness research has focused on training students to focus on their moment-to-moment experiences and thoughts. Clearly, there is a relationship between metacognition and executive function, but I have yet to see evidence that training in one improves the other.

One argument made to dissociate mindfulness from metacognition is that metacognitive processes are by necessity reflective or retrospective and that truly being mindful does not require reflection. For example, for a student to practice metacognition during study, she must ask herself, “Do I understand this concept?” Then, depending on the answer the student may or may not adjust the cognitive actions in which she is engaged to learn. This cycle is simply explained by the Nelson and Narens (1990). Now let’s think about a practitioner of mindfulness meditation. While meditating he may chose to focus his attention on the breath. Noticing when he is breathing in and noticing when he is breathing out. During this practice his mind may wander (this is true of even the most practiced at meditation). When this happens, he will gently bring his attention back to the breath. This process, just like that of the studying student, requires one to observe the cognitive processes and exert control over those processes when necessary. This to fits nicely into the metacognitive model offered by Nelson and Narens.

Imagine you are reading a novel on summer vacation. The book is enjoyable, but not a challenging read. Your are enjoying the sun and the sounds on the beach as you read, but suddenly notice you have not really attended to the last couple of pages and are not sure what has transpired in the plot. You choose to reread the last couple of pages and pay more attention. Imagine now you are a student. You are reading a very dull textbook chapter with the TV on and your smart phone near by. A student with little metacognitive resources (whether it be due working memory capacity, attentional control, executive function, etc.) is likely to mind wander (Hollis & Was, 2014). Students in my classes have often told me the hardest part of studying is staying focused, even when the topic is of interest. Ben Hollis and I found that students watching a video as part of an online course were often distracted by thoughts of checking the social media outlets. Not distracted by checking, but just thought of checking them. What if students were practiced at focusing attention, noticing when their minds wander and bringing the attention back to the task at hand?

It seems to me that if metacognition is knowledge and control of one’s cognitive processes and training in mindfulness increases one’s ability to focus and control awareness in a moment-by-moment manner, then perhaps we should reconsider, and investigate the relationship between mindfulness and metacognition in education and learning.

 

References

Bishop, S. R., Lau, M., Shapiro, S., Carlson, L., Anderson, N. D., Carmody, J., Segal, Z. V.,    Abbey, S., Speca, M., Velting, D. and Devins, G. (2004), Mindfulness: A Proposed Operational Definition. Clinical Psychology: Science and Practice,     11: 230–241. doi: 10.1093/clipsy.bph077

Chambers, R., Lo, B. C. Y., & Allen, N. B. (2008). The impact of intensive mindfulness training on attentional control, cognitive style, and affect. Cognitive Therapy and Research, 32(3), 303-322.

Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. American psychologist, 34(10), 906.

Flook, L., Smalley, S. L., Kitil, M. J., Galla, B. M., Kaiser-Greenland, S., Locke, J., … &  Kasari, C. (2010). Effects of mindful awareness practices on executive functions in elementary school children. Journal of Applied School         Psychology,26(1), 70-95.

Kabat-Zinn, J. (1990). Full catastrophe living: Using the wisdom of your mind to face    stress, pain and illness. New York : Dell.

Zeidan, F., Johnson, S. K., Diamond, B. J., David, Z., & Goolkasian, P. (2010).  Mindfulness meditation improves cognition: evidence of brief mental training.Consciousness and cognition, 19(2), 597-605.


Testing Improves Knowledge Monitoring

by Chris Was, Kent State University

Randy Isaacson and I have spent a great deal of time and effort creating a curriculum for an educational psychology class to encourage metacognition in preservice teachers. Randy spent a number of years developing this curriculum before I joined him in an attempt to improve the curriculum and use the curriculum to test hypotheses regarding improvement of metacognition with training for undergraduate preservice teachers. A detail description of the curriculum can be found in the National Teaching and Learning Forum (Isaacson & Wass, 2010), but I wanted to take this opportunity to give a simple overview of how we structured our courses and some of the results produced by using this curriculum to train undergraduates to be metacognitive in their studies.

With our combined 40+ years of teaching, we our quite clear that most undergraduates do not come equipped with the self-regulation skills that one would hope students would acquire before entering the university. Even more disappointing, is students lack the metacognition required to successfully regulate their own learning behaviors. Creating an environment that not only encourages, but also requires students to be metacognitive is not a simple task. However, it can be accomplished.

Variable Weight-Variable Difficulty Tests

The most important component of the course structure is creating an environment with extensive and immediate feedback. The feedback should be designed to help the student identify specific deficiencies in his or her learning strategies and metacognition.  We developed an extensive array of learning resources which guide the student to focusing on knowing what they know, and when they know it. The first resource we developed is a test format that helps the students reflect and monitor their knowledge regarding the content and items on the test. In our courses we have students judge their accuracy and confidence in their responses for each item and having them predict their scores for each exam

Throughout the duration of the semester in which they were enrolled in the course students are administered a weekly exam (the courses meet Monday, Wednesday and Friday with the exams occurring on Friday). Each examination is based on a variable weight, variable difficulty format. Each examination contained a total of 35 questions composed of 15 Level I questions that were at the knowledge level, 15 Level II questions at the evaluation level, and 5 Level III questions at the application/synthesis level. Scoring of the exam was based on a system that increased points for correct responses in relation to the increasing difficulty of the questions and confidence in responses: Students choose 10 Level I questions and put those answers on the left side of the answer sheet. These 10 Level I questions are worth 2 points each. Ten Level II questions were worth 5 points each are placed on the left side of the answer sheet, and three Level III questions were worth 6 points each are placed on the left. Students were also required to choose the questions they were least confident about and place them on the right side of the answer sheet. These questions were only worth one point (5 of the 15 Level I and II questions, and 2 of the 5 Level III questions). The scoring equaled a possible 100 points for each exam. Correlations between total score and absolute score (number correct out of 35) typically range from r = .87 to r = .94.  Although we provide students with many other resources to encourage metacognition, we feel that the left-right test format is the most powerful influence on student knowledge monitoring through the semester.

The Results

Along with our collaborators, we have conducted a number of studies using the variable weight-variable difficulty (VW-VD) tests as a treatment. Our research questions focus on whether the test format increases knowledge monitoring accuracy, individual differences in knowledge monitoring and metacognition, and psychometric issues in measuring knowledge monitoring. Below is a brief description of some of our results followed.

Hartwig, Was, Isaacson, & Dunlosky (2011) found that a simple knowledge monitoring assessment predicted both test scores and number of items correct on the VW-VD tests.

Isaacson & Was (2010) found that after a semester of VW-VD tests, knowledge monitoring accuracy on an unrelated measure of knowledge monitoring increased.


Predictors of college retention/success.

In a recent investigation completed with Randy Isaacson and Tara Beziat, it was found that high school GPA and SAT scores did not predict retention as well as GPA in the first semester. It was also found that first semester GPA was a good predictor of retention and student progression. Now, this is not surprising. What is important, is that individual differences in students’ knowledge monitoring accuracy was correlated with student GPA. Further, knowledge monitoring accuracy increased following a semester of simple training.

This article is accessible from the following links:

http://nrmera.org/researcher.html 

http://nrmera.org/PDF/Researcher/Researcherv26n1Beziat_et%20al.pdf


Are Current Metacognition Measures Missing the Target?

by Chris Was, Kent State University

Clearly, there is some agreement as to what metacognition is, or how to define it. In layman’s terms we often hear metacognition described as “thinking about thinking.” It is often defined as knowledge of and control of one’s cognitive processes.

There is also agreement that metacognition is necessary for one to successfully learn from instruction. Models such as Nelson and Naren’s (1990) model and that presented by Tobias and Everson (2009) stress the importance of knowledge of one’s state of knowledge as a key to learning.

In laboratory settings we have a number of “measures” of metacognition. Judgments of knowing, judgments of learning, feelings of knowing, etc. are all research paradigms used to understand individuals’ ability to assess and monitor their knowledge. These measures are demonstrated to predict differences in study strategies, learning outcomes and host of other performance measures.  However, individuals in a laboratory do not have the same pressures, needs, motivations, and desires as a student preparing for an exam.

How do we measure differences in students’ ability to monitor their knowledge so that we can help those who need to improve their metacognition? Not in the lab, but in the classroom. Although much of the research I have conducted with colleagues in metacognition has included attempts to both measure and increase metacognition in the college classroom (e.g., Isaacson & Was, 2010, Was, Beziat, & Isaacson, 2014), I am not convinced that we have always successfully measured these differences.

Simple measures of metacognitive knowledge monitoring administered at the beginning of a semester long course account for significant amounts of variance in end of the semester cumulative final exams (e.g,, Hartwig, Was, Dunlosky & Isaacson, 2013). However, the amount of the variance for which metacognitive knowledge monitoring in the models accounts is typically less than 15% and often much less. If knowledge monitoring is key to learning why then is it the case that it accounts for so little variance in measures of academic performance? Are the measures of knowledge monitoring inaccurate? Do scores on a final exam depend upon the life circumstances of the student during the semester? The answer to both questions is likely yes. But even more important, it could be that students are aware that their metacognitive monitoring is inaccurate and they therefore use other criteria to predict their academic performance.

The debate over whether the unskilled are unaware continues (cf. Krueger & Dunning, 2009; Miller & Geraci, 2011). Krueger and Dunning have provided evidence that poor academic performers carry a double burden. First, they are unskilled. Put differently, they lack the knowledge or skill to perform well. Second, they are unaware. That is, they do not know they lack the knowledge or skill and therefore have a tendency to be overconfident when predicting future performance.

There is however, a good deal of evidence that low-performing students are aware that when they are asked to predict how they will perform on an examination their predictions are overconfident. When asked to predict how well they will do on a test, the lowest performing students often predict scores well above how they eventually perform, but when asked how confident they are about their predictions these low performing students often report little confidence in their predictions.

So why does a poor performing student predict that they will perform well on an exam, when they are not confident in that prediction? Interestingly, my colleagues and I have (as have others) collected data that demonstrates that many students scoring near or above the class average under-predict their scores, and are just as uncertain as to what their actual scores will be.

An area we are beginning to explore is the relationship between ego-protection mechanisms and metacognition. As I stated earlier, students in a course, be it k-12, post-secondary or even adult education, are dealing with demands of the course, their goals in the course and the instructors goals, their attributes of success and failure in the course, and a multitude of other personal issues that may influence their performance predictions. The following is an anecdotal example from a student of mine. After several exams (in one of my undergraduate courses I administer 12 exams a semester plus a final exam) which students were required to predict their test scores, I asked a student why she consistently predicted her score to be 5 – 10 points lower then the grade she would receive. “Because when I do better than I predict, I feel good about my grade,” was her response.

My argument is that to examine metacognition of our students or to try to improve the metacognition of our students in isolation, without attempting to understand the other factors (e.g., motivation) that impact students’ perceptions of their knowledge and future performance, we are not likely to be successful in our attempts.

Isaacson, R., & Was, C. A.  (2010). Believing you’re correct vs. knowing you’re    correct: A significant difference?  The Researcher, 23(1), 1-12.

Krueger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in    recognizing one’s own incompetence lead to inflated self-assessments.    Journal of Personality and Social Psychology, 77(6), 1121-1134.

Miller, T. M., & Geraci, L. (2011). Unskilled but aware: reinterpreting overconfidence    in low-performing students. Journal of Experimental Psychology: Learning    Memory, and Cognition, doi:10.1037/a0021802

Nelson, T. O., & Narens, L. (1990). Metamemory: A theoretical framework and some    new findings.  In G. H. Bower (Ed.), The psychology of learning and motivation    (Vol. 26, pp. 125–173).  New York: Academic Press.

Tobias, S., & Everson, H. (2009).  The importance of knowing what you know: A    knowledge monitoring framework for studying metacognition in education.    In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Handbook of    Metacognition in Education. (pp. 107-128). New York, NY: Routledge.
Beziat, T. R. L., Was, C. A., & Isaacson, R. M. (2014). Knowledge monitoring accuracy    and college success of underprepared students. The Researcher, 26(1), 8-13.