Testing Improves Knowledge Monitoring

by Chris Was, Kent State University

Randy Isaacson and I have spent a great deal of time and effort creating a curriculum for an educational psychology class to encourage metacognition in preservice teachers. Randy spent a number of years developing this curriculum before I joined him in an attempt to improve the curriculum and use the curriculum to test hypotheses regarding improvement of metacognition with training for undergraduate preservice teachers. A detail description of the curriculum can be found in the National Teaching and Learning Forum (Isaacson & Wass, 2010), but I wanted to take this opportunity to give a simple overview of how we structured our courses and some of the results produced by using this curriculum to train undergraduates to be metacognitive in their studies.

With our combined 40+ years of teaching, we our quite clear that most undergraduates do not come equipped with the self-regulation skills that one would hope students would acquire before entering the university. Even more disappointing, is students lack the metacognition required to successfully regulate their own learning behaviors. Creating an environment that not only encourages, but also requires students to be metacognitive is not a simple task. However, it can be accomplished.

Variable Weight-Variable Difficulty Tests

The most important component of the course structure is creating an environment with extensive and immediate feedback. The feedback should be designed to help the student identify specific deficiencies in his or her learning strategies and metacognition. We developed an extensive array of learning resources which guide the student to focusing on knowing what they know, and when they know it. The first resource we developed is a test format that helps the students reflect and monitor their knowledge regarding the content and items on the test. In our courses we have students judge their accuracy and confidence in their responses for each item and having them predict their scores for each exam

Throughout the duration of the semester in which they were enrolled in the course students are administered a weekly exam (the courses meet Monday, Wednesday and Friday with the exams occurring on Friday). Each examination is based on a variable weight, variable difficulty format. Each examination contained a total of 35 questions composed of 15 Level I questions that were at the knowledge level, 15 Level II questions at the evaluation level, and 5 Level III questions at the application/synthesis level. Scoring of the exam was based on a system that increased points for correct responses in relation to the increasing difficulty of the questions and confidence in responses: Students choose 10 Level I questions and put those answers on the left side of the answer sheet. These 10 Level I questions are worth 2 points each. Ten Level II questions were worth 5 points each are placed on the left side of the answer sheet, and three Level III questions were worth 6 points each are placed on the left. Students were also required to choose the questions they were least confident about and place them on the right side of the answer sheet. These questions were only worth one point (5 of the 15 Level I and II questions, and 2 of the 5 Level III questions). The scoring equaled a possible 100 points for each exam. Correlations between total score and absolute score (number correct out of 35) typically range from r = .87 to r = .94. Although we provide students with many other resources to encourage metacognition, we feel that the left-right test format is the most powerful influence on student knowledge monitoring through the semester.

The Results

Along with our collaborators, we have conducted a number of studies using the variable weight-variable difficulty (VW-VD) tests as a treatment. Our research questions focus on whether the test format increases knowledge monitoring accuracy, individual differences in knowledge monitoring and metacognition, and psychometric issues in measuring knowledge monitoring. Below is a brief description of some of our results followed.

Hartwig, Was, Isaacson, & Dunlosky (2011) found that a simple knowledge monitoring assessment predicted both test scores and number of items correct on the VW-VD tests.

Isaacson & Was (2010) found that after a semester of VW-VD tests, knowledge monitoring accuracy on an unrelated measure of knowledge monitoring increased.

Incorporating Metacognitive Leadership Development in Class

by Lauren Scharff, U.S. Air Force Academy*

During the spring 2014 semester I decided to try an explicitly metacognitive approach to leadership development in my Foundations for Leadership Development course in the Behavioral Sciences department at the United States Air Force Academy.

I had taught the course twice before and had many discussions with other course instructors. Overall, my sense was that many of our students didn’t intentionally and systematically connect what they were doing and learning in the course with their own personal leadership development. This is despite a paper that focused on a personal leadership development plan, and a video project that focused on implementing positive change within their squadrons.

This course is an upper-level, required core courses in our curriculum, and my section was one of more than 30 with approximately a dozen different instructors teaching sections. At our institution, much of the course structure within core courses is standardized across instructors, but I had 20% of the points with which to do what I desired, as long as 10% somehow assessed accountability for lesson preparation.

I was aware of a foundation of research in skill development (e.g. Svinicki, 2004), so I knew that in order to most effectively develop skills, people need multiple opportunities for practice coupled with feedback. Feedback leads to awareness of strengths, shortcomings, and possible alternate strategies. This understanding of skill development became intertwined with my increasing focus on metacognitive approaches. I came to the conclusion that perhaps part of the less-than-ideal student connection to the course material and objectives occurred because our course activities that were designed to support that connection didn’t provide (require) enough opportunities for practice and continued awareness, especially beyond the classroom and course requirements.

As I prepared for the semester I drew on resources from The Learning Record, which outlines Five Dimensions of Learning that connected well with goals I had for my students’ leadership development: confidence and independence, knowledge and understanding, skills and strategies, use of prior and emerging experience, and critical reflection. The site also shares well-developed activities and assignments that supported my goal of using a metacognitive approach to promote my students’ leadership development.

Ultimately, I designed my course to be centered around journal entries, which I also completed. During each lesson we shared our understandings, questions, and reflections based on the readings, as well as examples of personal observations of leadership and our reflections on how what we were learning might be effectively applied to real situations. More specifically, the journal entries included 1) answers to guided reflection questions about each reading for each lesson, and 2) at least two personal leadership observations and analyses each week. I created a simple grading system so that I wouldn’t be overloaded with assessing journal entries every lesson. (Journal assignment)

A quick poll of my students (N=13) indicated that none of them regularly kept personal journals, and only two had ever had any sort of journal assignment for a class. Thus, this requirement for regular journal writing required a change of habit for them that also represented increased time and energy for class preparation. Although there was some adjustment, when I asked for feedback after two weeks of class, the students were unanimous in their agreement that they were more deeply reading than if I had incorporated reading quizzes for accountability and that they preferred to continue the personal and reading reflections even though they involved frequent writing. Discussion during class was deeper and more engaging than in previous semesters.

Twice during the semester, students wrote evidence-based personal development evaluations, based on a shared example from The Learning Record. Students chose examples from their journals to support their evaluations of their own leadership development. These evaluation exercises forced them to thoughtfully review their observations across the weeks of the semester and develop ongoing awareness of their leadership strengths and weaknesses as well as an understanding of alternate strategies and when/how they might be useful for their leadership efforts. (Personal Development Evaluation assignment)

I also added a question each time that had them evaluate the journal approach and course design. We made some tweaks at mid-semester. By the end of the semester, all but one student reported that the journal entries deepened their learning and personal awareness of their leadership development. While I will likely make some further tweaks in future semesters, I believe that this approach was a success, and that it could be scaled up for larger classes (see the simplified grading scheme in the Journal assignment). Below are sample comments from the final evaluation assignment (released with student permission):

“The leadership journal has had a tremendous effect on my personal development as a leader. The journal has made me aware of my strengths and weaknesses…. The journal allowed me an avenue to give time and actually think about how I am doing as a leader and peer within my environment.”

“The personal observations were definitely helpful for documenting our successes and failures, which we can look back upon and improve. This relates not only to our personal leadership development, but to how we learn about ourselves.”

“These journals have taken all of us on a journey through the semester. They undoubtedly began as something that we disliked and looked down upon each week. However, I have really grown to love and understand this application of leadership growth. They not only provide a chance for us to look back on our leadership gains and failures, they offer an opportunity for us to challenge ourselves in order to write about the things we want to see in ourselves. The journals have become much more than a simple task of writing on a week-to-week basis. They have grown into an outpouring of our character and lives as we turn the page from underclassmen to upper-class leaders and eventually to lieutenants in a few short months. I believe that these journals are also a metaphor for many leadership challenges in that they will be frequent, difficult, and time consuming, but in the end they will let us all grow. ….my reflections are not simply babble, …they actually represent significant growth and understanding of myself.”

References:

Svinicki, Marilla. 2004. Learning and Motivation in the Postsecondary Classroom. Bolton, MA: Anker Publishing Co, Inc.

The Learning Record: http://www.learningrecord.org/

————————————-

* Disclaimer: The views expressed in this document are those of the authors and do not reflect the official policy or position of the U. S. Air Force, Department of Defense, or the U. S. Govt.

Assessing Metacognitive Awareness

Constructed by Rayne Sperling and Gregory Schraw (1994), the Metacognitive Awareness Inventory (MAI) is a well established and useful assessment of metacogntion. The MAI has been used in hundreds of studies, ranging from basic to applied research. It is a 52-item inventory with two broad categories (i.e., knowledge of cognition and regulation of cognition), with several sub-categories.

Schraw, G., & Dennison, R. S. (1994). Assessing metacognitive awareness.Contemporary educational psychology, 19(4), 460-475.

The Effects of Metacognition and Concrete Encoding Strategies on Depth of Understanding in Educational Psychology

Suzanne Schellenberg, Meiko Negishi, and Paul Eggen (2011) from the University of North Florida describe a useful method to increase the metacognition of their students. They found that when educational psychology students were taught specific encoding strategies they academically outperformed a control group in learning course material.

Schellenberg, S., Negishi, M., & Eggen, P. (2011). The Effects of Metacognition and Concrete Encoding Strategies on Depth of Understanding in Educational Psychology. Teaching Educational Psychology, 7(2), 17-24.

Changing Epistemological Beliefs in Pre-service Teacher Education Students

Joanne Brownlee, Nola Purdie, and Gillian Boulton-Lewis (2010) describe an interesting method to increase student’s epistemological beliefs using reflective journal assignments. Brownlee and colleagues found that when students engaged in these reflective practices, they had significantly improved their epistemological beliefs over that of students who did not complete these activities.

Brownlee, J., Purdie, N., & Boulton-Lewis, G. (2001). Changing epistemological beliefs in pre-service teacher education students. Teaching in higher Education,6(2), 247-268.

Metacognition and Reflective Thinking

By Steven C. Fleisher, California State University Channel Islands

Imagine that we are reading an assignment. As we read, do we think: “How long will this take?” “Will this be on the test?” If so, try this instead. Presume that we are reading the article as preparation for meeting later with an important person such as our supervisor to discuss the article. How would this situation change the questions we ask ourselves? Such thinking can make us aware of what constitutes satisfactory mastery of knowing and how to achieve it.

Think back for a moment to learning a psychomotor skill, such as learning to ride a bicycle. It is normal to master that skill with normal innate balance and strength. We might think: “That’s all there is to it.” However, watching cyclists in a serious bicycle race or triathlon, reveals that reliance only on innate ability cannot produce that kind of performance. That level of expertise requires learning to pedal with cadence, to deliver equal power from both legs, use the gearing appropriately, exploit position within a group of racers and pace oneself relative to challenges. Untrained innate ability can rarely get us far in comparison to the results of informed training.

The same is true in learning. Metacognitive skills (learnable skills) enhance academic performance. People with metacognitive skill will usually outperform others who lack such skill, even others with greater innate intelligence (natural ability). Metacognitive training requires developing three strengths: 1) metacognitive knowledge, 2) metacognitive monitoring, and 3) metacognitive control.

Metacognitive knowledge refers to our understanding about how learning operates and how to improve our learning. We should have enough of this knowledge to articulate how we learn best. For example, we can know when it is best for us to write a reflection about a reading in order to enhance our learning. We should be alert to our misconceptions about how our learning works. When we learn that cramming is not always the best way to study (Believe it!), we must give that up and operate with a better proven practice.

Metacognitive monitoring refers to developed ability to monitor our progress and achievement accurately. For example, self-assessment is a kind of metacognitive monitoring. We should know when we truly understand what we are reading and assess if we are making progress toward solving a problem. When we become accurate and proficient in self-assessment, we are much better informed. We can see when we have mastered certain material well enough, and when we have not.

Metacognitive control. This competency involves having the discipline and control needed to make the best decisions in our own interests. This aspect of metacognition includes acting on changing our efforts or learning strategies, or taking action to recruit help when indicated.

Putting it together. When we engage in metacognitive reflection, we can ask ourselves, for example, “What did we just learn?” “What was problematic, and why?” “What was easy, and why?” “How can we apply what we just learned?” Further, when we gain metacognitive skill, we begin to internalize habits of learning that better establish and stabilize beneficial neural connections.

Reflective Exercises for Students:

Metacognitive knowledge. Consider three learning challenges: acquiring knowledge, acquiring a skill, or making an evidence-based decision. How might the approaches needed to succeed in each of these three separate challenges differ?
Metacognitive monitoring. After you complete your next assignment or project, rate your resultant state of mastery on the following scale of three points: 0 = I have no confidence that I made any meaningful progress toward mastery; 1 = I clearly perceived some gain of mastery, but I need to get farther; 2 = I am currently highly confident that I understand and can meet this challenge.
Next, see if your self-rating causes you to take action such as to re-study the material or to seek help from a peer or an instructor in order to achieve more competence and higher confidence. A critical test will be whether your awareness from monitoring was able to trigger your taking action. Another will come in time. It will be whether your self-assessment proved accurate.
Metacognitive control. To develop better understanding of this, recall an example from life when you made a poor decision that proved to produce a result that you did not desire or that was not in your interests. How did living this experience equip you to better deal with a similar or related life challenge?

References

Chew, S. L. (2010). Improving classroom performance by challenging student misconceptions about learning. Association for Psychological Science: Observer, Vol. 23, No. 4. http://psychologicalscience.org/observer/getArticle.cfm?id=2666

Dunlosky, J. and Metcalf, J. (2009). Metacognition. Thousand Oaks, CA: Sage.

Leamnson, R. N. (1999). Thinking about teaching and learning: Developing habits of learning with first year college and university students. Sterling, VA: Stylus.

Pintrich, P. R. (2002). The role of metacognitive knowledge in learning, teaching, and assessing. Theory Into Practice, 41(4), 219-225.

Wirth, K. (2010). The role of metacognition in teaching Geoscience. Science Education Resource Center, Macalester College. http://serc.carleton.edu/NAGTWorkshops/metacognition/activities/27560.html

Calls for Research Collaboration

If you have a planned, new, or ongoing project for which you’d like collaborators from other institutions, submit a short description and explain the type of collaboration you are requesting. For example, you might need statistical assistance, you might want someone to try your approach using a different student population, etc.

Use the comment feature to submit your post.

Request for Assistance in Project Design

If you are interested in designing a research project that investigates the impact of your incorporation of metacognitive strategies in your classroom, use the comment feature to submit a short description of your idea and questions. The Improve with Metacognition creators and consultants will try to assist you, and other site visitors might also comment on your post. Note that it is your responsibility on your end to apply for and receive IRB approval for the ethical use of human participants.

What is your favorite metacognition assignment?

Briefly explain the assignment. Help others understand how you’ve used it. For example, in what level course have you used this? What is the best thing about this assignment? What is a limitation of this assignment?

Share by using the comments feature for this post. If you have a handout to share with more detail, please email it with a short note linking it to your post, and the Improve with Metacognition creators will save it as an online resource and link it to your post. If you are comfortable doing so, you can include your name and email contact so that others can contact you for further information.

What is your favorite in-class metacognition activity?

Briefly explain the activity. Help others understand how you’ve used it. For example, have you used this in small or large classes? What is the best thing about this activity? What is a limitation?

Waves of Insight about Teaching and Learning

James Rhem, Executive Editor, The National Teaching & Learning FORUM

“When I began The National Teaching & Learning FORUM” over twenty years ago, almost everyone in faculty development had heard of Bloom’s Taxonomy. I hadn’t since I was new to this beat; so I went to the University of Wisconsin’s Memorial Library and looked for it. There on the shelf was what they were talking about, the taxonomy of cognition, but right beside it was something that interested me as much if not more, the taxonomy of affect. No one (or at least no one I knew) had heard of this taxonomy or the one on the psychomotor domain that followed the one on affect. I began quietly beating the drum for an awareness of affect back in 1995 (according to a search of my old email). That was the same year Daniel Goleman’s “Emotional Intelligence” came out; so I suspected I’d stumbled onto what would become a cresting wave. That wave has grown, but it’s still heading toward a crest.

I go into all this because I think I can already see the waves of insight and meaningful rethinking of teaching and learning that will follow. Metacognition as a concept has been around for a long time, but coming to understand it and its place in learning (and thus is effective teaching) is a new thing. It’s something I’ve been increasingly covering in the FORUM and something I plan on continuing to cover. Indeed, NTLF created a series of small books with Stylus a few years back and I would be remiss if I didn’t mention that one of the titles in that series is about metacognition (Using Reflection and Metacognition to Improve Student Learning: Across the Disciplines, Across the Academy Edited by Matthew Kaplan , Naomi Silver , Danielle LaVaque-Manty , Deborah Meizlish Foreword by James Rhem https://sty.presswarehouse.com/Books/BookDetail.aspx?productID=298776 )

I’ve linked two recent articles from the FORUM on metacognition. And I invite readers of this list who may have written (or plan to write) articles on metacognition and teaching and learning to submit material to me for wider distribution.

1. Metacognitive Skills – why bother (and how)? by Carol Hostetter and Leah Savion at Indiana University

2. Metacognition and Disciplinary Thinking by Matt Fisher at Saint Vincent College

So what’s the next wave after metacognition? The psychomotor domain. Count on it.

James Rhem, Executive Editor, The National Teaching & Learning FORUM