May 2018 - Improve with Metacognition

by Dr. Lauren Scharff, U. S. Air Force Academy*

In my earlier post this year, “Know Cubed” – How do students know if they know what they need to know?, I introduced three challenges for accurate student self-assessment. I also introduced the idea of incorporating knowledge surveys as a tool to support student self-assessment (an aspect of metacognitive learning) and promote metacognitive instruction. This post shares my first foray into the use of knowledge surveys.

What exactly are knowledge surveys? They are collections of questions that support student self-assessment of their course material understanding and related skills. Students complete the questions either at the beginning of the semester or prior to each unit of the course (pre), and then also immediately prior to exams (post-unit instruction). When answering the questions, students rate themselves on their ability to answer the question (similar to a confidence rating) rather than fully answering the question. The type of learning expectation is highlighted by including the Bloom’s level at the end of each question. Completion of knowledge surveys develops metacognitive awareness of learning and can help guide more efficient studying.

My motivation to include knowledge surveys in my course was a result of a presentation by Dr. Karl Wirth, who was invited to be the keynote speaker at the annual SoTL Forum we hold at my institution, the United States Air Force Academy. He shared compelling data and anecdotes about his incorporation of knowledge surveys into his geosciences course. His talk inspired several of us to try out knowledge surveys in our courses this spring.

So, after a semester, what do I think about knowledge surveys? How did my students respond?

In a nutshell, I am convinced that knowledge surveys enhanced student learning and promoted student metacognition about their learning. Their use provided additional opportunities to discuss the science of learning and helped focus learning efforts. But, there were also some important lessons learned that I will use to modify how I incorporate knowledge surveys in the future.

Evidence that knowledge surveys were beneficial:

My personal observations included the following, with increasing levels of each as the semester went on and students learned how to learn using the knowledge survey questions:

Students directly told me how much they liked and appreciated the knowledge survey questions. There is a lot of unfamiliar and challenging content in this upper-level course, so the knowledge survey questions served as an effective road map to help guide student learning efforts.
Students asked questions in class directly related to the knowledge survey questions (as well as other questions). Because I was clear about what I wanted them to learn, they were able to judge if they had solid understanding of those concepts and ask questions while we were discussing the topics.
Students came to office hours to ask questions, and were able to more clearly articulate what they did and did not understand prior to the exams when asking for further clarifications.
Students realized that they needed to study differently for the questions at different Bloom’s levels of learning. “Explain” questions required more than basic memorization of the terms related to those questions. I took class time to suggest and reinforce the use of more effective learning strategies and several students reported increasing success and the use of those strategies for other courses (yay!).
Overall, students became more accurate in assessing their understanding of the material prior to the exam. More specifically, when I compared the knowledge survey reports with actual exam performance, students progressively became more accurate across the semester. I think some of this increase in accuracy was due to the changes stated in points above.

Student feedback included the following:

End-of-semester feedback from students indicated that vast majority of them thought the knowledge surveys supported their learning, with half of them giving them the highest rating of “definitely supports learning, keep as is.”
Optional reflection feedback suggested development of learning skills related to the use of the knowledge surveys and perceived value for their use. The following quote was typical of many students:

At first, I was not sure how the knowledge surveys were going to help me. The first time I went through them I did not know many of the questions and I assumed they were things I was already supposed to know. However, after we went over their purpose in class my view of them changed. As I read through the readings, I focused on the portions that answered the knowledge survey questions. If I could not find an answer or felt like I did not accurately answer the question, I bolded that question and brought it up in class. Before the GR, I go back through a blank knowledge survey and try to answer each question by myself. I then use this to compare to the actual answers to see what I actually need to study. Before the first GR I did not do this. However, for the second GR I did and I did much better.

Other Observations and Lessons learned:

Although I am generally pleased with my first foray into incorporating knowledge surveys, I did learn some lessons and I will make some modifications next time.

The biggest lesson is that I need to take even more time to explain knowledge surveys, how students should use them to guide their learning, and how I use them as an instructor to tailor my teaching.

What did I do this past semester? I explained knowledge surveys on the syllabus and verbally at the beginning of the semester. I gave periodic general reminders and included a slide in each lesson’s PPT that listed the relevant knowledge survey questions. I gave points for completion of the knowledge surveys to increase the perception of their value. I also included instructions about how to use them at the start of each knowledge survey:

Despite all these efforts, feedback and performance indicated that many students really didn’t understand the purpose of knowledge surveys or take them seriously until after the first exam (and some even later than that). What will I do in the future? In addition to the above, I will make more explicit connections during the lesson and as students engage in learning activities and demonstrations. I will ask students to share how they would explain certain concepts using the results of their activities and the other data that were presented during the lesson. The latter will provide explicit examples of what would (or would not) be considered a complete answer for the “explain” questions in contrast to the “remember” questions.

The biggest student feedback suggestion for modification of the knowledge surveys pertained to the “pre” knowledge surveys given at the start of each unit. Students reported they didn’t know most of the answers and felt like completion of the pre knowledge surveys was less useful. As an instructor, those “pre” responses helped me get a pulse on their level or prior knowledge and use that to tailor my lessons. Thus, I need to better communicate my use of those “pre” results because no one likes to take time to do what they perceive is “busy work.”
I also learned that students created a shared GoogleDoc where they would insert answers to the knowledge survey questions. I am all for students helping each other learn, and I encourage them to quiz each other so they can talk out the answers rather than simply re-reading their notes. However, it became apparent when students came in for office hours that the shared “answers” to the questions were not always correct and were sometimes incomplete. This was especially true for the higher-level questions. I personally was not a member of the shared document, so I did not check their answers in that document. In the future, I will earlier and more explicitly encourage students to be aware of the type of learning being targeted and the type of responses needed for each level, and encourage them to critically evaluate the answers being entered into such a shared document.

In sum, as an avid supporter of metacognitive learning and metacognitive instruction, I believe that knowledge surveys are a great tool for supporting both student and faculty awareness of learning, the first step in metacognition. We then should use that awareness to make necessary adjustments to our efforts – the other half of a continuous cycle that leads to increased student success.

———————————————–

* Disclaimer: The views expressed in this document are those of the author and do not reflect the official policy or position of the U. S. Air Force, Department of Defense, or the U. S. Govt.

by Dr. Ed Nuhfer, California State Universities (retired)

Courage is a metacognitive quality that enables not having our actions limited or dictated by fear. Academic courage is a unique category of “moral courage.” Unlike physical acts of courage that occur in a brief time span, moral courage governs a day-to-day way of being and acting in practice, despite recognizing the forces that merit fear. Martin (2011) describes academic courage as perseverance through academic difficulty in the face of fear. The threatening environments of universities and the nature of courage that educators, particularly faculty developers, need in their professional practice are our focus here. With the notable exceptions of Palmer (2017) and Martin (2011), data-based studies of developed courage by teacher-scholars are nearly nonexistent.

“Fear” inevitably enters nearly all discussions about “courage.” While courage does not exist in the absence of fear, courage is a developed metacognitive capacity; fear is not. Acting with courage becomes possible when a person’s learning involves affective development along with intellectual and ethical development. (See the last blog in this series, Part 3.) Hannah, Sweeney, and Lester (in Pury and Lopez, 2011, pp 125-148) portray acts of courage as the products of a developed “Cognitive-Affective Processing System.” In brief, courage enlists a lot of brainpower.

Teachers have courage (image from http://tamaravrussell.com/2015/07/teachers-have-courage/)

Academic Courage versus Suicidal Tendency

Institutions of higher education produce unique threats, and several are increasing for faculty and faculty developers. The unifying theme for most fear in academia is the potential loss of career and livelihood. Because threatening faculty livelihoods for reasons other than incompetence or ethical violations is a detriment to society, tenure developed in the U. S. as a means to shield scholars from being intimidated and having their voices silenced. But tenure has not always worked perfectly. Tenure-track positions are vanishing as academic staff, with no prospects of tenure or its protections, now replace retiring tenured professors. Increased faculty vulnerability naturally accompanies increased apprehension and fear. People involved in threatening scholars’ livelihoods are politicians & regents boards, administrators, students, and colleagues.

Politicians in parts of the federal government, and a few states, now actively suppress or work to eliminate government researchers and college faculty who work on topics where investigations produce (or might produce) evidence that contradicts partisan advocacy. Political power now is wielded to weaken tenure and eliminate programs that produce knowledge that contradicts authoritarian positions and even university benefactors’ interests. Faculty can suffer reprisals for doing politically-charged scholarship. The World Wide Web and social media open today’s college faculty and students to unprecedented surveillance, so even work performed outside universities with scholarly journals or professional organizations produces risks. Sometimes, faculty can wonder whether every group on the campus has power over them. When faculty state that their careers “live and die by student ratings,” this discloses that they also fear their students. To some faculty, it seems that even students have supervisory power to take away faculty livelihoods. All of these sources of fear represent substantial threats. When a faculty member discloses fear, a developer must never think to trivialize or discount it. The reason for fear is likely real.

College faculty have responsibilities as teachers and scholars. In both, a responsibility provides an obligation to remain current about new knowledge, to add to the knowledge base, and not merely act as dispensers of knowledge. As new evidence validates the increased effectiveness of different instructional methods, teacher-scholars have obligations to maintain command of the current literature and to apply the best available knowledge to serve their students. Yet, doing so poses risks.

Introducing unfamiliar teaching structures to the classroom is not always received favorably by students or by colleagues. Striving to enact nontraditional instructional techniques carries risks of becoming unpopular by not satisfying students with teaching as they expected to be taught. Where the retention of livelihoods rests more on popularity with those who have the power to take livelihood away than on the quality of research achieved or the learning promoted, acting from fear of becoming unpopular has potential consequences.

A courageous action is not rash or suicidal; it recognizes and respects actual threats. The temptation to bulldoze ahead with “best practices” by believing that one has superior knowledge disrespects a genuine threat. Acting courageously respects the threat and works to understand it, but without giving into fear. Acting from courage requires more brainpower than does just giving up. Building capacity to enact academic courage involves a lot of work. A courageous approach may strive to find empathy for colleagues and students, seek awareness of the reasons for their dissatisfaction and will work to gain an understanding of what changes will be needed for those in opposition to begin to accept and support the most beneficial actions. A failed first attempt will serve to inform later efforts.

Face management as a Cultural Challenge to Courage

Face management seeks to advance oneself socially through associating and being seen with those who are popular and/ or influential. Face management is a way of life in many organizations. Its dark side appears when advancing self involves marginalizing those perceived as unpopular or just different. Faculty being ostracized can include those who are struggling to make changes and newcomers with new ideas, perhaps controversial to local established practices, who are striving to be accepted and valued. Stanford professor Robert I. Sutton referenced such face-managing actions as “kiss-up; slap-down,” and recognized that the behavior could render toxic the culture of an entire organization–a department, a college, and even an institution. Ostracism is something any individual should fear. Researchers at the University of British Columbia’s Sauder School of Business studied the impact of ostracism on employee health and morale and discovered that it can be even more harmful to one’s mental and physical well-being than harassment or other forms of bullying. For a faculty developer, supporting those ostracized entails risks of also becoming marginalized.

That’s a hazard in faculty development work because faculty development is a helping profession. A faculty developer’s responsibility is to support faculty who are in need of help, and they seldom are people who hold popularity or power. Anyone who aspires to be a faculty developer needs to realize that he/she will likely experience such a threat multiple times. Often, faculty in need are also being marginalized, struggling to succeed and sometimes suffering the consequences of being caught crossways in a toxic, unit-level culture of kiss-up-slap-down. Unfortunately, face-managing cultures are tolerated, even nurtured, in universities, perhaps because courage isn’t taught in college.

While the exercise of academic courage in faculty development requires knowledge and skills, both are external qualities. Acting with academic courage is different. It’s an internal capacity that infuses knowledge and skills with empathy and affect. Internal development is a ceaseless metacognitive reflection, and it is a LOT of work.

References

Martin, A. J. (2011) Courage in the Classroom: Exploring a New Framework Predicting Academic Performance and Engagement. School Psychology Quarterly 26 2 145-160.

Palmer, P. (2017) The Courage to Teach: Exploring the Inner Landscape of a Teacher’s Life. 20th Ed. San Francisco, CA: Wiley.

Pury, C. L. S., and S. Lopez, eds. (2011). The Psychology of Courage. Washington DC: American Psychological Association.

Improve with Metacognition

Month: May 2018

Supporting Student Self-Assessment with Knowledge Surveys

Developing Affective Abilities through Metacognition Part 4: Exercising Academic Courage in Faculty Development