Month: October 2016

Selecting a quantitative measure of metacognition  

by Dr. Jessica Santangelo, Hofstra University

If you are interested in metacognition and promoting the development of metacognitive skills, you may also be interested in measuring metacognition. But how does one assess a person’s metacognitive development?

Metacognitive development can be assessed via quantitative or qualitative measures. Quantitative measures include self-report measures, often using Likert-style survey instruments, while qualitative measures use coding of responses to open-ended prompts (e.g., Stanton 2015). While quantitative measures are generally easier and faster to score, a drawback is that self-report measures are not always accurate (Schunk 2008). Qualitative data can be more rich, providing deeper and more nuanced information, but is much more labor intensive and time consuming to analyze. Ideally, one uses a combination of quantitative and qualitative data to develop as complete a picture of metacognitive development as possible.

When I set out to assess the metacognitive development of 484 (!) students, I was overwhelmed by the number of quantitative tools available. The focus of the tools varies. Some tools attempt to assess metacognition directly while others assess factors or attributes associated with metacognition (e.g., study skills, self-regulated learning). Some are not explicitly intended to assess metacognition (e.g., LASSI), but are used by some authors as an indicator of metacognitive development (e.g., Downing et al 2007, 2011). Some have been through many iterations over the years (e.g., ASI, RASI, and ASSIST) while others remain relatively unchanged (e.g., MAI, MSLQ). Some are free while others have a per student fee. Some are longer (120 items, ILS) and others are shorter (18, RASI).

How does one choose the “best” quantitative tool? Unfortunately, there is no easy answer. It depends on the specific question being addressed and the amount of time and money available to administer the tool. I compiled a (non-comprehensive) list of tools I encountered in my search along with some information about each one to assist anyone looking for a quantitative measure of metacognitive development.

For my 484-student project, I chose to use the Metacognitive Awareness Inventory (MAI; Schraw and Dennison 1994) in combination with coding responses to open-ended prompts I created. I chose the MAI because it purports to measure metacognition directly (rather than being a study or learning skills inventory), is free, and is of moderate length (52 items). Others have found correlations between MAI results and other quantitative measures of student success (e.g., GPA and end of course grades), even suggesting using the MAI as a screening tool to identify students who could benefit from metacognition training (Young and Fry 2008). These characteristics  fit with the questions I was asking: Can we rapidly (and accurately) assess metacognitive development at the beginning of an introductory course? Does including explicit instruction and implicit practice with metacognitive skills in a course increase student metacognitive development?

While coding the open-ended responses is taking months to complete, it has revealed some clear and interesting patterns. In contrast, the quantitative data from the MAI, though gathered in about 5 minutes running scantron sheets through a machine, show no patterns at all. There does not appear to be any relationship between the quantitative MAI data and the qualitative data or any other measure of student success (GPA, exam and course grades, etc.). I’m not entirely surprised – metacognitive skills are unlikely to be wholly captured by a number generated by a 52-item self-report questionnaire. However, given the results of others (e.g., Sperling et al 2004, Young and Fry 2008) I was hopeful there would be at least some relationship between the quantitative and qualitative results.

This is not to say that rapid assessments via self-report questionnaires are worthless. It is simply a caution to not rely on these quantitative tools as one’s sole measure of metacognitive development. Indeed, I have colleagues who have had more “success” with tools other than the MAI (e.g, with the MSLQ), where success is defined as the quantitative tool reflecting similar patterns or trends as other, more time-consuming qualitative measures.

As with many things in science, there is no easy answer. My hope is that this compilation of available tools makes the choice of which one to use a little easier.

For more in-depth reading on measuring metacognition, I recommend:

Mogashana, D., J. M. Case, and D. Marshall. 2012. What do student learning inventories really measure? A critical analysis of students’ responses to the approaches to learning and studying inventory. Studies in Higher Education 37:783–792.

Schraw, G., and J. Impara, eds. 2000. Issues in the Measurement of Metacognition. Buros Institute of Mental Measurements, Lincoln, NE.

References

Downing, K., F. Ning, and K. Shin. 2011. Impact of problem‐based learning on student experience and metacognitive development. Multicultural Education & Technology Journal 5:55–69.

Downing, K., R. Ho, K. Shin, L. Vrijmoed, and E. Wong. 2007. Metacognitive development and moving away. Educational Studies 33:1–13.

Schraw, G., and R. S. Dennison. 1994. Assessing metacognitive awareness. Contemporary educational psychology 19:460–475.

Schunk, D. H. 2008. Metacognition, self-regulation, and self-regulated learning: research recommendations. Educational Psychology Review 20:463–467.

Sperling, R. A., B. C. Howard, R. Staley, and N. DuBois. 2004. Metacognition and self-regulated learning constructs. Educational Research and Evaluation 10:117–139.

Stanton, J. D., X. N. Neider, I. J. Gallegos, and N. C. Clark. 2015. Differences in metacognitive regulation in introductory biology students: when prompts are not enough. CBE-Life Sciences Education 14:ar15.

Young, A., and J. Fry. 2008. Metacognitive awareness and academic achievement in college students. Journal of the Scholarship of Teaching and Learning 8:1–10.

 


Do Your Questions Invite Metacognition?

By Arthur L. Costa and Bena Kallick, Co-founders, International Institute for Habits of Mind

Our ‘inner voice’ is what we use to reflect on what we do, how and why we behave in the way we do, how we critique ourselves and how we connect the knowledge, ideas, concepts and concept frameworks developed using each of our four learning systems. It is the voice that challenges us to strive further and the voice that condemns our foolishness.

Mark Treadwell, Learning: How the Brain Learns (2014)

One of a teacher’s most important practices is designing and posing questions.   Wise teachers pose questions consciously with deliberate intentions. They know that questions engage sometimes subtle and overt responses from students.   Questions are the powerful stimuli that evoke cognitive, behavioral and emotional responses in students. They initiate a journey in the mind. Indeed questions are the backbone of instruction. They must be employed with care (Costa & Kallick, 2008).

Building a Thinking Vocabulary

Because thinking words may not be used in students’ homes or in previous classrooms, thinking vocabulary may be a “foreign language” to them. They may not know how to perform the specific thinking skills that a given term implies. It is imperative, therefore, that students develop a vocabulary with which to express their metacognitive processes.

When adults speak usiing mindful language, using specific, cognitive terminology and instructing students in ways to perform certain skills, students are more inclined to be able to both name and use those skills. For example,

Instead of saying: Use Metacognitive language by saying:
“Let’s look at these two pictures.” “As you COMPARE these two pictures…”
“What do you think will happen when . . . ?” “What do you PREDICT will happen when . . . ?”
“How can you put those into groups?” “How might you CLASSIFY . . . ?”
“Let’s work this problem.” “Let’s ANALYZE this problem.”
“What do you think would have happened if… ?” “What do you SPECULATE would have happened if… ?”
“What did you think of this story?” “What CONCLUSIONS can you draw about this story?”
“How can you explain . . . ?” “What HYPOTHESES do you have that might explain . . . ?”
“How do you know that’s true?” “What EVIDENCE do you have to sup-port . . . ?”
“How else could you use this . . . ?” “How could you APPLY this . . . ?”
“Do you think that is the best alternative? “As you EVALUATE these alternatives….”

As students hear these cognitive terms in everyday use and experience the cognitive processes that accompany these labels, they internalize the words and use them as part of their own  metacognitive vocabulary. Teachers will also want to give specific instruction and provide awareness of experiences so that students recognize and know the meaning of the terminology.

Invite metacognitive responses.

Teachers can deliberately invite students to become spectators of their own thinking by posing questions that invite a metacognitive response. Some questions invite a behavioral response, others can invite a thought-full response. Notice how behavioral questions can be transformed into questions that invite thinking:

Questions That Invite a Behavioral Response Questions That Invite Metacognitive Responses
“Why did you do that?” “What were you thinking when you did that?”
“What did the author mean when . . . ?” “What cues were you aware of?”
“What are your plans for . . . ?” “As you envision . . . what might be…..”
“When will you start . . . ?” “How will you decide when to start . . ?”
“Was that a good choice?” “What criteria did you have in mind to make that choice?”

If teachers pose questions that deliberately engage students’ cognitive processing, and let students know why the questions are being posed in this way, it is more likely that students will become aware of and engage their own metacognitive processes.

Making Internal Dialogue External

Students can become spectators of their own thinking when they are invited to monitor and make explicit the internal dialogue that accompanies their thinking.

They reveal their own thinking as they consider questions such as:

  • “What was going on in your head when……?”
  • “What were the benefits of……?”
  • “As you evaluate the effects of . . . ?”
  • “By what criteria are you judging…..?”
  • “What will you be aware of next time?”
  • “What did you hear yourself saying inside your brain when you were tempted talk but your job was to listen?”

Keep Students Thinking About Their Thinking

While such questions will initiate students’ metacognitive journey, you will also want to sustain that momentum by:

Causing Students to Monitor their Accuracy

  • “How do you know you are right?”
  • “What other ways can you prove that you are correct?

Pausing and Clarifying but not Interrupting

  • “Explain what you mean when you said you ‘just figured it out'”.
  •  “When you said you started at the beginning, how did you know                                where to begin?”

Providing Data, Not Answers (As soon as you confirm that an answer is correct, there is no need to think further about it!)

  • “I think you heard it wrong; let me repeat the question………………”
  • “You need to check your addition.”

Resisting Making Value Judgments Or Agreeing With Students’                  Answers.

  • “So, your answer is 48. Who came up with a different answer?”
  • “That’s one possibility. Who solved it another way?”

Remaining Focused On Thinking Processes

  • “Tell us what strategies you used to solve the problem”
  • “What steps did you take in your solution?”
  • “What was going on inside your head as you solved the problem?”

Encouraging Persistence

  • “Success! You completed step one. Now you’re ready to forge ahead.”
  • “C’mon, you can do it” Try it again!”

Ultimately, the intent of all this is to have students monitor and pose their own questions that promote thinking in themselves and others. Questioning, monitoring and reflecting on our experiences are requisites for becoming a continuous, lifelong learner. When we teach students to think about their thinking, we help make the world a more thought-full place.

References

Costa, A & Kallick, B. (2008) Learning and Leading with Habits of Mind: 16 Characteristics for Success. Alexandria, VA: ASCD

Treadwell, M (2014) Learning: How the brain learns. www.MarkTreadwell.com/products