Month: February 2021

Learning. Design. Analytics. Post 4: Metacognitive Design To Support Metacognitive Learning Within Virtual Learning Environments

By Yianna Vovides, PhD and Marie Selvanadin, MS, MBA, Georgetown University

We see the process of design and development of products as very much a metacognitive awareness process given that designers and developers are engaged in cycles of reflection, planning, monitoring, and evaluation with every prototype produced. This is especially the case when embarking on the development of a virtual learning environment (VLE) because of the fact that the environment itself needs to enable and support engagement toward varied types of learning and meet the needs of both instructors and students. In this blog post we discuss the design and development process of a VLE, a web-based case analysis app developed by the Center for New Designs in Learning and Scholarship at Georgetown University. We highlight how the metacognitive design process of planning, monitoring, and evaluation supports a reflective practice that enables metacognitive learning.

Context: Ethical Decision Making for Global Managers (GUX)

The case analysis app was designed to augment case-based teaching and learning techniques by providing multiple ways for students to reflect on their thinking in relation to how they make decisions. In collaboration with a faculty member and program director, we approached the design of the app by contextualizing it within the Ethical Decision Making for Global Managers professional certificate offered via edx. The main goal that the app was designed to support learners in achieving is the following: “how to analyze real-world ethical dilemmas using multiple frameworks, considering many possible choices, and selecting a “best choice” option.” The courses in the certificate program focused on rules- and results-based decision-making processes.

Teaching decision-making, let alone ethical decision-making, is not easily done and doing so online, especially within an open self-paced learning environment, is even more challenging (Sternberg, n. d.).

Sternberg emphasizes that:

“[l]earning how to reason ethically is a dialectical, back-and-forth process. Simply delivering content through lectures and readings are at best supplementary forms of instruction. The primary form of instruction needs to be interactive because students need to present ideas, get feedback on those ideas, and then try out re-formed ideas that themselves will be subject to further modification.”

With this in mind, we knew that we needed to design the app in a way that enabled individuals to engage in a back-and-forth process that could support ethical reasoning. We realized early on that in order to make the interaction within the case analysis app meaningful to individual learners, the app itself needed to provide guidance at both the cognitive and metacognitive levels. Therefore, we needed to consider how the app itself could “speak” to the learner. In other words, what interaction could we design as part of the app to provide feedback to the learners along the way so help them reflect about their decision-making process and how they were using the rules-based and results based ethical reasoning approaches.

Our collaboration with the faculty and program director in relation to this project took place over two years, so we had time to understand and internalize the ethical decision-making framework that was being used in the courses which focused on rules-oriented and results-oriented approaches. For enabling both cognitive and metacognitive interactions within the app, we used the reflective sensemaking model (Vovides and Inman, 2016) to guide our design decisions. Sensemaking as a pedagogical approach to teach ethics has been gaining attention (Brandt and Popejoy, 2020).

The following section breaks down the design of the app in relation to the reflective sensemaking process. It includes screenshots taken from the web-based app itself using a case study being used in the Ethical Decision-Making for Global Managers professional certificate available on edx.

Design: Reflective Sensemaking

  1. The reflective sensemaking process begins by asking a learner to explore a real-world case. The example we show in the screenshots is related to Policing Terrorism.

screen shot of article on policing terrorism with highlights

The learner is encouraged to read the case multiple times. A learner has the option to highlight and/or annotate parts of the case and to determine whether they want their annotations to remain private (visible only to them) or made public (visible to others interacting with the same case).

screen shot of initial decision instructions and question

  1. The learner is then asked a yes/no question about the key issue described in the case study In the Policing Terrorism case study, they are asked whether private high tech companies should develop and enforce their own standards to police terrorism on the internet.

Once the learner selects either yes or no, they are asked to write down the reasoning for their decision.

  1. screen shot of example highlights and annotations along with their categoriesAfter this initial decision, the learner is presented with their own highlights and annotations from their reading of the case and asked to identify how important each highlight and annotation was in contributing to their decision.

 

4. screen shot of instructions for the the decision-making frameworkThen, the learner is shown a summary of where their highlights/annotations fall within the Rules and Results decision-making framework. They are asked to consider the following questions:

    • Which framework are you most aligned with? Perhaps you are closer to the middle.
    • What values and assumptions did you bring to your fact selection and decision?

In addition, the app provides learners the option to see their highlights and annotations in context and explore how the instructor engaged with the same case. This aims to reduce the feeling of anonymity and isolation. We also see it as a way to enable further exploration of the case itself.

screen shot of article on policing terrorism with highlights 2The learner is then asked to consider whether a rules- or results-based framework would lead to a “best choice” decision in the case they read. They are asked to use a slider to represent the degree to which they would prioritize rules- or results-based factors and then to explain their reasoning.

screen shot of ethics framework spectrum instructionBy allowing students to take the time to reflect on their learning and decision making, we believe that they are given the opportunity to think about their own thinking (which in simplistic terms is what metacognition is all about) and giving them an opportunity to reflect on their learning.

  1. screen shot of instructions for the decision step in the processAs a final step in the reflective sensemaking process, learners are asked the same yes/no question and given another opportunity to either keep the same decision or change it. In either case, they are asked to explain their reasoning. The focus in this particular example is related to ethical reasoning; however, the approach could be used for other types of reasoning.

Model: Where Learning Design and Analytics Align

screen shot of the dashboard for the case analysis platform at Georgetown University's Ethical Decision-making for Global Managers Professional Certificate ProgramThis case analysis app presents one model for how learning design and analytics can come together to create a unique experience where reflection of the learning process is prioritized. Reflection is critical in developing metacognitive awareness (Schraw, 1998). We designed the app so that learners can go through multiple cases as they move through the Ethical Decision Making for Global Managers professional certificate program. Once learners complete one case then they continue to have access to it for review purposes in the app’s dashboard (shown here). Encouraging review of one’s completed cases enables learners to become more aware of how they reasoned through that case.

To align the learning design and analytics we began with a conceptual data model which we are still refining (see Sensemaking Process with Identified Variables diagram). We took the reflective sensemaking process and have started mapping the variables that could serve as proxies to better understand how learners are making sense of the case they are engaging with. We considered the number of different actions that a learner takes when reading the case itself. We also take into account whether the learner goes back and reviews the case, and more. This type of mapping of the data to the learning process will enable us to make the learner journey visible to the learner because we will be able to create a visualization of the reflective sensemaking process as they engage with the cases.

diagram of the 5 steps in the decision making process: exploration, identification, processing, judgement, integration

Therefore, our next step is to create the visualization of the sensemaking process for each learner who goes through a case that will be available as part of the learner dashboard in the case analysis app. Given that a learner could go through multiple cases, the visualization would also represent the learner’s sensemaking process across cases and over time. We envision that the dashboard itself could then become an active learning space that would support further metacognitive awareness as it would enable learners to interrogate how they reasoned through multiple cases over time.

Our Reflection so Far in relation to using metacognitive design to support metacognitive learning

Schematic of a spiral illustrating loops of before, during, after
Figure 2. Spiral – Before, during, after Model

To design a learner dashboard as an active learning space requires an adaptive learning design process so that we can account for changes from one iteration to the next as we gain a better understanding of how learners are engaging with the app. Given that adaptive learning design is a process that strategically modifies designs based on emerging learner needs (Bower, 2016), we incorporated from the start of this project reflective pauses to become even more aware of our own iterative approach. We planned what we would do before we developed a prototype, during our prototyping process, and after when conducting formative evaluations. Yes, it has taken two years so far!

The app launched in 2020 and we are currently in the process of collecting learner data from the platform and surveys to help us solidify some of the functionality that would be part of the active learning dashboard. We are also investigating options to introduce social learning opportunities as part of the active learning dashboard to reduce the sense of isolation and anonymity. Want to learn more, contact us!

References

Bower, M. (2016). A Framework for Adaptive Learning Design in a Web-Conferencing Environment. Journal of Interactive Media in Education, (1).

Brandt, L., & Popejoy, L. (2020). Use of sensemaking as a pedagogical approach to teach clinical ethics: an integrative review. International Journal of Ethics Education, 1-15.

Schraw, G. (1998). Promoting general metacognitive awareness. Instructional science, 26(1-2), 113-125.

Sternberg, R. J. (n.d.). Developing ethical reasoning and/or ethical decision making | IDEA. IDEA. Retrieved January 10, 2021, from https://www.ideaedu.org/idea-notes-on-learning/developing-ethical-reasoning-and-or-ethical-decision-making/

Vovides, Y., & Inman, S. (2016). Elusive learning—using learning analytics to support reflective sensemaking of ill-structured ethical problems: A learner-managed dashboard solution. Future Internet, 8(2), 26.


Learning. Design. Analytics. Post 3: Scripting Asynchronous Lectures: A Metacognitive Process

By Eleri Syverson, MA, Joe King, Yiran Sun, MA, Yianna Vovides, PhD, Georgetown University

In this post we’ll look at the process of scripting in the context of metacognitive awareness by walking through the processes and considerations we recommend for scripting. When we talk about scripting in this case, we are referring to the writing process of reflecting on a subject and drafting — as a word-for-word script, or at minimum a very detailed outline — a lecture to be delivered to a class of students. Most of our own experience with scripting is in developing asynchronous lectures for online courses (or similar online format) that are either entirely new or are being redesigned into an online format from an existing face-to-face course, and this is our focus in this post. However, many of the principles and practices we consider can be useful in designing face-to-face presentations and interactions too. We first address the conundrum of whether to script, and then we offer a script of our own in relation to how we guide faculty through the process of scripting.

To Script or Not to Script? Script, script, and script some more….

When we first meet with an instructor who is working to design an online course, one of the things we talk about is whether they have any plans to develop new content, such as a video, audio or text lecture, for the course. For the most part, the answer is yes. We recommend the instructor script out their lecture materials before we begin to record or build them for an online course. This might sound like the last thing an instructor wants to do for several reasons:

  • Perhaps they have been delivering lectures on this topic, maybe even for this very course, for many years. They may know these lectures very well, even without notes.
  • Perhaps they rely on spontaneity to keep the experience fresh and therefore fear they would lose that if they scripted.
  • Perhaps the time and effort commitment required to produce scripts seems daunting.

While these are valid concerns, a well-designed script will not take away from their inherent expertise on the subject or remove those elements of personality that make their lectures engaging. In fact, we emphasize that the value of scripting is less in the final product, but in the process itself which can help them plan, monitor, and evaluate their approach to each lecture anew.

During the design and development process for online courses we approach scripting, especially when working from existing materials or lectures, as a metacognitive act that relies on planning, monitoring, and evaluation about one’s own areas of expertise. By approaching scripting as a process that supports instructors in deepening their metacognitive awareness of how they teach and how their students learn, we encourage increased efficacy, coherence, and accessibility of lectures. By the end of the design and development process, instructors have accounted for the thinking of their students as well as their own and in fact have developed a lasting metacognitive practice.

A Script of our Own

Schematic of a spiral illustrating loops of before, during, after
Figure 2. Spiral – Before, during, after Model

In the first blog post of this series Yianna Vovides described a simple model we share with instructors to recognize the iterative process of design (see figure 1). When we think of the Before phase, we think of pre-production efforts, for During, we think of production efforts and for After, we think of post-production. Scripting is part of our pre-production efforts.

This section highlights the script we use in our process to guide instructors and we explain the thinking behind the process, our reasoning.

Script

Reasoning

If you are not sure whether or not to script your materials, you might start with a test recording to see how the lecture goes without a script.

 

 

In our process, we will often give faculty a rough cut of this initial test recording to give the instructor a chance to review this initial product and go over any concerns with them.

A common observation from instructors after this initial test recording is that lecturing to a camera or microphone can be a very different experience than lecturing to a classroom full of students.

In the classroom, students nod, ask questions, and make facial expressions that signal as to their engagement and understanding. Even the lack of such signals in the classroom would, itself, be a significant sign about the lecture, your students, or something else in the environment that is impacting your teaching.

This presents a unique opportunity to reflect on one’s own thinking while teaching, and to reflect on the experience of the learner.   When lecturing in class, which moments typically brought about questions (or even were confusing to you as you learned it the first time)? Which examples do you find particularly impactful? What is essential to understanding the material? Scripting allows instructors to address these questions and plan the presentation of materials in the most helpful and more meaningful way, because it allows instructors’ time to reflect on both past and future deliveries of a lecture.

 

Through this writing process of reflection, planning, and drafting lectures as text, instructors can carefully consider meaning they intend to leave with their students, and work to ensure that their learning goals for a particular lecture are met. Writing itself is applied metacognition. “Every act of writing is an act of meaning production. Reading, re-reading, reflecting, and reviewing — processes traditionally associated with writing — serve as monitoring strategies to ensure that the production of meaning is in conformance with the author’s goals for writing…” (Hacker, Keener, Kircher 157). In scripting a lecture, instructors are not merely replicating a previously delivered lecture, but improving and adapting it for a new format. Keep in mind the following tips (and their reasonings) for writing for that new format.

Script

Reasoning

Keep your script brief and connected to learning objectives.

Remember that attention spans behave differently in the online environment than they might in face-to-face conversations. An ideal length for videos in online learning environments is 6-10 minutes, which comes out to about 2-4 pages of written text, single spaces. Audio only lectures can typically be longer, around 40 minutes to an hour. Consider the length of a typical Youtube video vs the length of a typical podcast.

Identify key terms or questions in advance. Include graphics in your script to illustrate key points. Please include the references! Once you send us the draft, we will then discuss with you how we pull all the pieces together.

By following these recommendations, post-production will be easier. It can also help you make sure that you are, in fact, addressing all of your learning objectives and covering all of the necessary topics in the video.

This kind of metacognitive planning and evaluation of your student’s thinking increases the efficacy and coherency of your lectures.

In the same vein, this planning allows you to choose the most appropriate format for lectures.

You might script a lecture you have been giving face-to-face in the classroom for many years, only to realize that in the online format, the lecture may be more suited to a podcast format than a video. Different formats present other advantages to students as well — for example, an audio-only podcast lecture would allow students to listen on a walk around their neighborhood, as opposed to at home with the distractions of noisy family or roommates. Presenting content in a variety of formats, while letting the content direct the format chosen, provides students with additional flexibility. And choosing a format after planning the content, rather than as a first step, allows you this flexibility to present the content that is most suitable for the format itself.

Scripting and preplanning lecture materials not only forces you to consider the needs of your students and how they might interpret lecture material, it inherently addresses some of those needs by making materials more accessible

Transcripts not only benefit students with hearing impairments, but also students who may be listening to materials in their non-native language, students who are experiencing poor internet connectivity and cannot stream the video, students with learning disabilities who may need more time or alternate formats to process the lecture, or students who simply prefer to read or take notes on the transcript while listening along. Producing a transcript of a lecture can itself be time consuming or, if you outsource the effort, costly task. By creating a word-for word script and drafting it with your audience in mind, providing your script to students as a transcript eliminates the need to produce a separate transcript to address the accessibility concerns noted above.

It is important that you include your sources in your script to be incorporated into the video.

While a lecture video isn’t a peer-reviewed paper, it is more formal and more lasting than a face-to-face lecture or conversation, and it takes some additional effort to communicate corrections of any errors. Including sources also gives you an opportunity to review the sources and correct any errors, and gives interested students the opportunity to engage more deeply with the subject by exploring these sources.

 

Scripting lectures is a multi-layered metacognitive process. It requires the writer to think about their own thinking in reflecting on their own understanding of the material and past deliveries lectures. It also requires them to think about the experience of their audience, both in who they are and the context and channels which will affect the composition of their experience. Successful lecture scripting means making sure the material feels like it is for these students in this course.

  • Who are the students?
  • What they have already learned, what you hope they will learn, and where they are now?
  • Where might they be listening to these lectures?
  • How can the writer best facilitate their understanding?

These questions should be considered not just in regards to a specific group of current students, but future students or others who may view the lecture in a different context. A lecture might be relocated to a different part of a course, or sometimes a different course entirely. The social and historical context in which a lecture was recorded can change in just weeks. While the process of scripting might seem daunting, it should be thought of as a reallocation of efforts rather than an additional one: the metacognitive processes of reflection on self and audience, as well as the product of the script itself, contribute to a more robust and longer-lasting lecture that better addresses the needs of the audience.

Reference

Hacker, D. J., Kircher, J. C., & Keener, M. C. (2010). Writing is Applied Metacognition. In 1132034699 853417179 D. J. Hacker, 1132034700 853417179 J. Dunlosky, & 1132034701 853417179 A. C. Graesser (Authors), Handbook of metacognition in education (pp. 154-176). New York: Routledge.

 


Learning. Design. Analytics. Post 2: Utilizing Instructional Design Methodologies and Learning Analytics to Encourage Metacognition

By Zhuqing Ding, MA

What’s the interaction like in higher education between faculty and instructional designers? While faculty often have full autonomy in their course design and teaching methodologies (Martin, 2009), instructional designers play the role of a change agent. When instructional designers propose improvements in instructional strategies and recommend significant changes to existing courses, faculty may resist adopting their recommendations. With this in mind, the online course design team at the Center for New Designs in Learning and Scholarship (CNDLS), Georgetown University, had implemented an adaptive approach to the course design process since 2018. During the pandemic, we were able to pivot and expand this approach to offer support to faculty across the university and not only to the ones who had been part of online programs.

So, what makes our process adaptive? The traditional ADDIE (Analyze, Design, Develop, Implement and Evaluate) instructional design model, focuses on linear processes in content development. Compared to the traditional instructional design approach, an adaptive approach has distinct iterative phases where we use learning analytics as evidence to initiate faculty members’ metacognition, thereby inspiring changes to future iterations of the course.

Here is how our course design team implemented the adaptive approach for transforming an existing classroom-based law course to fully online. Before moving online, this tax law course was offered once a week through one 2.5-hour lecture, accompanied by reading assignments and graded only by one final exam. During the first phase of our adaptive process our team tackled the following questions:

  • How can we transform the passive learning experiences in the classroom (receiving information) into active learning (interacting with the course materials)?
  • How can we strike a balance between good practices for online course design and the traditional methods familiar to/preferred by law school professors?
  • What are some of the strategies we will use to encourage the faculty members to evaluate their own teaching strategies to become more mindful and intentional about their own teaching?

While lecturing has been proven to be the most used instructional method, especially in higher education, it is more suitable for a face-to-face traditional classroom than an online environment (McKeachie, 1990). In an online course, passively receiving a large amount of information by watching a series of 2.5-hour lectures can be challenging for students. Following an adaptive process allowed us to guide faculty toward an awareness that designing an online course is more than recording lectures and that students need to interact with the lectures in order to be able to recall and retrieve what they are learning. The following image shows the metacognitive activities that we incorporated in the adaptive design approach. During the first phase of the course, our course design team introduced activities that support reflective practice to enable faculty members to become more self-aware about their own teaching and learning. Then, in the second phase we introduce activities that encourage a deepening of the reflective practice to spark creativity.

Flow diagram showing Phase 1: Course Translation and Phase 2: Course Adaptation

 

Interactive and short lectures

In order to collect data that can help us understand the students’ learning experience, we proposed to the faculty to create interactive and short lectures to replace the 2.5-hour long lectures in each module. To demonstrate to faculty the types of interactive elements that can be inserted into lectures, we introduced a storyboarding method. In the script, the faculty broke their long lectures into subtopics. Our instructional designers highlighted the keywords and areas that could be illustrated by graphics or animations. Then, the faculty confirmed the highlights and the graphics, and added/deleted as needed.

The resultant, short, subtopic videos were presented as playlists in the course. Within each video, the professor is shown lecturing on the right side of the screen, and on the left, relevant animated keywords, charts, and graphics appear. Certain parts of the charts were highlighted as the professor talked through certain elements within the charts. Such interactive elements within the lectures are designed to help students make a connection to the professor while also focusing on important keywords and short summaries of the lecture topics.

The analytics report provided by the video hosting platform that became available after the course launched helped with the faculty’s meta-thinking. Based on the total views, total minutes of content delivered, unique viewers, and the percentage of completion data for each video, the course team was able to understand important aspects of the students’ learning experience: the videos that had the highest views, the videos that students were not able to finish watching, and the videos that students watched again and again. Such evidence helped the faculty identify knowledge areas that students were not able to understand right away and recognize times when students’ participation dropped.

In the second iteration/phase of this course, the faculty took several actions to not only improve the course design, but also improved his teaching presence in this online course. First, during the low-participation time observed from the analytics report of the first iteration, reminders were sent to students to encourage them to keep up the pace. Additionally, more office hours including one-on-one and group office hours were scheduled, allowing students to clear up questions with the professor if they got stuck. The faculty was able to address common questions during the recorded office hour sessions, and made these sessions available to students. Overall, the student-faculty interaction was improved since, due to metacognition, the faculty became more aware of the importance of building interactive touch points to keep online students on track. The metacognition has initiated his awareness of the importance of the teaching presence in the online courses.

Weekly Activities

The law school has a long-standing tradition of using final exams as the only assessment in a given course. In face-to-face classes, interactions such as small talk among peers before and after the class and question-and-answer sessions after each lecture help students confirm whether they are on track. For online students, such checkpoints are missing and, therefore, it is necessary to periodically build them in so students can make sure they are following along.

In the first iteration of the course, we introduced weekly, ungraded quizzes, allowing students to practice, experiment, and reflect on their learning. Since this particular course is related to tax law, the quiz questions — most requiring calculation in Excel — were extracted from previous exams. Correct answers and short explanations were provided for each question at the end of the quiz. Students were allowed to take the quizzes multiple times. Such low-stakes activities provided the space for students to explore and discover the answers during their learning.

After the first iteration of the course, the professor was able to review the quiz analytics report provided by the quiz tool in Canvas, which allowed for metacognition on the activity design. There was a correlation between students’ performance on the weekly quizzes and the final exam. Students who didn’t participate in the practice quizzes at all achieved lower scores than students who did. Students who completed practice quizzes were also more active in the online office hours and found more opportunities to engage with the professor throughout the semester. Based on this finding, the professor realized the importance of motivating students to practice on a weekly basis in order to assess their understanding and ask questions before they fall too far behind.

By the second iteration of the course, a few actions were taken to improve students’ engagement in weekly quizzes. The professor improved the design of the quiz questions by adding downloadable Excel spreadsheets with formulas to the provided explanations of the quiz answers, allowing students to tinker with formulas and reflect on their own calculations. He also offered additional office hours following the quizzes in each module to make sure students had the opportunity to ask questions.

The professor moved from reluctance to including weekly quizzes at all, since they didn’t exist in the face-to-face class, to encouraging active learning processes by improving the quiz question design and proactively providing space for students to reach out to him with questions. Quiz analytics served as evidence that drove the faculty member to metacognition and improved the way he teaches online.

Summary

Metacognition is implicitly part of faculty development programs across disciplines. However, while working with law faculty, the adaptive approaches our course design team followed led to new reflections about their teaching practice. It was challenging to find the balance between traditional ways of teaching in the law school and an interactive online course that would allow students to succeed in a virtual environment. The adaptive approach allows the instructional designers and faculty to reach more agreements with iterative efforts. We used the analytics provided by the media-hosting tool and the quiz tools as evidence to encourage meta-thinking about the faculty’s teaching practice. This led to minor changes to the course with significant impacts. With such evidence, faculty are more open to adapting their long-standing teaching practices and embracing new ways of designing online courses. Sometimes these metacognitive approaches to teaching online also inspire faculty to rethink their teaching practices in traditional classrooms, such as providing more measurable learning goals or diversifying assessment methods.

References

McKeachie, W. J. (1990) Research on College Teaching: The Historical Background, Journal of Educational Psychology, 82, 2, 189-200.

Martin, R. E. (2009). The revenue-to-cost spiral in higher education. Raleigh, NC: John William Pope Center for Higher Education Policy. Retrieved from http://files.eric.ed.gov/fulltext/ED535460.pdf