Mind the Gap: The Illusion of Skill Acquisition in Computational Thinking

In this episode I unpack Bao & Hosseini’s (2023) publication titled “Mind the gap: The illusion of skill acquisition in computational thinking,” which compares learning, perceptions of learning, and confidence among adult learners participating in interactive, video-based, and text-based learning.

quote while interactive platforms
improve the perceived learning compared
to other methods educational videos also
provide a sense of confidence among
Learners Although our findings do not
show any impact on actual learning the
gap between confidence in applying
algorithmic techniques and perceived
knowledge suggests that educational
videos when used in isolation may be
detrimental to learning by providing an
illusion of confidence end quote that's
from page 783 of the article titled mind
the gap colon the illusion of skill
acquisition and computational thinking
which was written by yeding Bao and Hari
hosseini apologies have mispronounced
any names here's the abstract for this
paper quote with the Advent of online
educational platforms and the advances
in pedagogical techniques self-directed
learning has emerged as one of the most
popular modes of learning distance
education elevated by the kova 19
pandemic involves methods of instruction
through a variety of remote activities
which only rely on educational videos
for mastering in the absence of direct
student engagement the asynchronous
nature of remote activities May
deteriorate the quality of education for
Learners students often have an illusion
of skill acquisition after watching
videos which results in overestimation
of abilities and skills we focus on the
efficacy of skill acquisition through
interactive Technologies and assess
their impact on computational thinking
in comparison with delivery through
other traditional media EG videos and
text in particular we investigate the
relationship between actual learning
perception of learning and learnness's
confidence in adult Learners our results
reveal intriguing observations about the
role of interactivity and visualization
and their implications on the
pedagogical Design for self-directed
Learning modules end quote Vada
summarize this article into a single
sentence I'd say that this article
Compares learning perceptions of
learning and confidence among adult
Learners participating in interactive
video based and text-based learning and
as always in the show notes you can find
the link to this article as well as over
interviews with guests or multiple
guests or these like solo episodes where
I unpack implications of scholarship for
computer science Educators in addition
if you go to my website jaredelary.com
there are hundreds if not thousands of
free computer science education
resources as well as a bunch of gaming
and drumming stuff because I create
content for a living now even though
this article is talking about adult
Learners I'm going to talk about what
this looks like in K-12 classes as well
as just computer science education
classes in particular now if you don't
know who I am I have a background
working with all grades kindergarten
through doctoral student in a variety of
different contexts like music education
and computer science education classes
micv is linked on the website if you're
interested in that kind of nerdiness now
in the introduction of this article the
authors talk about how there can be an
illusion of skill acquisition for people
who learn something by watching a video
so in the introduction they talk about
how there are some self-directed
learning platforms like think of Khan
Academy which has like a bunch of math
stuff as well some Computer Science
Education and web development courses
and then they unpack what they mean by
computational thinking which as I've
talked about in other episodes is a very
messy term that means many different
things to many different people and here
are the questions that kind of guide
this particular study this is from page
skill acquisition in teaching
computational thinking through a variety
of learning Technologies is there a gap
between the perception of learning
actual learning comprehension and
learners's confidence end quote those
are some really important questions to
consider I've done some Publications and
even some podcast episodes that kind of
unpack self-efficacy of like teachers
and Learners which isn't to say that
those are two separate things but
basically to say that between adults and
kids but just because people gain
confidence in being able to do something
doesn't mean they're actually better at
doing that thing now as an example of
that because I have like a drum kit
right behind me I could show you a bunch
of videos that would teach you how to
develop limb Independence to make it so
that your two hands do two different
things separate from your two feet and
even your mouth you could have five
weight Independence while playing on the
drums and I could show you many
different videos on how to develop that
and it would make sense and you go oh
that makes sense however actually
putting into practice is a completely
different thing you have to develop the
myelination and the neural connections
across your brain to actually be able to
make it so that your limbs can basically
interact in an interdependent way so
there's a difference between
understanding and skill or even
confidence and understanding in an
actual application of that understanding
same thing applies for computer science
so if we're talking about programming
here I might be able to conceptually
understand an idea or an algorithm or
whatever but actually applying that and
creating that on my own and putting it
into use within a program that's a
completely different skill set which one
do you value or focus on in the classes
that you work with so this paper is
going to kind of explore that a little
bit now if you're interested in section
is on page 779 kind of gives an overview
of learning comprehension perception of
learning and confidence and interest in
learning it gives like a summary of the
paper itself so if you want a very quick
read of that go ahead and take a look at
that but this podcast will also kind of
unpack each of those areas and then in
section 1.2 they talk about some related
work so you can find some studies that
are related to this in some way alright
so section two is on design and
development so they kind of unpack well
what was the interactive platform how
did this compare with the educational
videos and how did this compare with
based instruction so all three were kind
of given to different groups if you want
to learn more about that check out
section 2.1 and 2.2 and if you want to
learn more about the experimental design
I really appreciate the authors that
actually gave a deep dive into here's
exactly what our demographic survey
looked like or here's what our
assignment and workflow will look like
Etc so that is all discussed on page 780
in the experimental design as well as in
going to nerd out on that in this
episode but it is worth reading if
you're interested in that so you can
hopefully replicate this kind of study
or design to see if you get similar or
different findings with your different
populations all right so let's get nerdy
and talk about the results so this is
starting on page 782 so 4.1 is on the
perception of learning so in this
section they kind of talk about the
perception of learning as well as their
confidence with learning here's a quote
from page 782 quote while the Learners
of the Interactive Group reported
significantly higher score for perceived
comprehension in contrast to video and
text-based groups the participants in
the video groups were as confident about
their knowledge as those in the
Interactive Group this finding suggests
that educational videos may lead
Learners to develop overconfidence about
the subject matter in contrast
interactivity provides an opportunity
for Learners to get their hands dirty by
manipulating the input and directly
interacting with the algorithms which in
turn aligns learners's perceived
comprehension with confidence end quote
now this is really important to consider
so it's great that videos might be able
to help like improve confidence with
learning or understanding something but
the interactivity actually made it so
that they were able to apply their
understanding in some way and kind of
like maybe get some feedback in terms of
what worked and what did not work
depending on what kind of feedback you
get from the platform again it's one
thing to be able to understand something
and it's a completely different thing to
be able to actually apply that
understanding consistently now what does
this mean for educators so for myself
what I really tried to focus on was
maybe introducing a concept at the
beginning of a class in the first like
five minutes out of a 40 minute period
at most and then the rest of the time
was actually students being able to
apply that on understanding in their own
projects or explore it in different ways
so if everybody happened to be working
on the exact same thing which was a very
rare circumstance in in the classes that
I work with because everybody was able
to pick a number of different projects
with a number of different programming
languages and platforms Etc and apply it
in a meaningful way but in the few
instances where students were all
working on the exact same thing I would
spend some time introducing an idea and
then that would take up a very small
percentage of the time the rest of the
time would be spent applying that and me
working with students one-on-one kind of
coaching them or having their peers kind
of Coach each other now if this had been
flipped this would have made me look
like 35 minutes of like video based
lecture or me at the front of the room
kind of talking about things in five
minutes of application while that time
spent learning from an expert whether
it's from a video from a teacher or
whatever may have like developed some
sense of confidence of understanding
that subject area the actual application
of that understanding is a very small
percentage of time so I've talked about
in other podcasts if we think of like
music or language or any kind of skill
or understanding you need to be able to
actually practice it to improve at it
and apply it in different contexts to
constantly look at a concept or practice
from many different angles and many
different perspectives to better
understand how to use it and modify in
different scenarios so for example if
you're learning something like a for
Loop if you only go and create one for
Loop and don't understand how it works
in other programs and other contexts
other scenarios Etc you're going to have
a very narrow understanding and not
understand how to use it in many
different scenarios or when to use it in
comparison to like a while loop or
something else how much time you
actually spend exploring those different
scenarios as opposed to just learning
the concept itself will have a direct
impact on how good you are at that it's
the same thing with learning like a
language or an instrument or like a
sport or whatever you can't just watch
videos of people teaching you how to do
it you actually need to do it so a
recommendation that I might give is that
while students might have an increased
sense of confidence worth learning
something from a lecture or a video or
whatever you need to actually really
think through how much time we're
actually spending applying that
understanding because that will have an
impact on learning which Speaking of
four point 2 is on the actual learning
comprehension now what's really
interesting about this is that the
authors found no statistically
significant difference between learning
comprehension among the Interactive
Group and the video group so again even
though they had an increase in
confidence they did not have an increase
in actual learning comprehension but
what about with the interest in learning
well 4.3 kind of talks about that very
quickly now when the authors compared
the text-based group with the video
group there was an increase in interest
among the video group however there was
not a statistically significant
difference between the video group and
the Interactive Group if you have an
option of being able to hand a textbook
to a student or an interactive platform
online or a video on how to do something
in this particular study it appears that
the textbook would be the least
interesting to some adult Learners and
maybe the most interesting would be the
video or the interactive platform I
would argue that kind of depends on what
the content is that is being taught and
who you are teaching it to or who is
learning that content but in general I
could totally see why video and
interactive media would be more
interesting or more engaging than just
reading through text now under 4 0.4
they talk about usability so this is a
quote from page 782 that kind of talks
about well how do these three groups
compare quote First the interactive
treatment required participants to
proactively engage in the learning
process by manipulating the input thus
more effort is required in this mode
compared to the video group second the
educational videos contained audio in
addition to the visual explanation we
hypothesized that audio can have an
important impact on the overall learning
experience as expected those in the
text-based group were less satisfied and
found the instructions complicated and
sometimes confusing end quote now if you
look into like studies on learning
theories and like cognitive load Theory
Etc you'll find that like mixed media
like something like a video that has
like visuals and audio going on
simultaneously might be more interactive
and more interesting than something
that's just text based but it really
kind of depends on how it's presented
like there are ways to create videos
that maximize cognitive load and make it
so that it's very confusing just like
with text but as I've mentioned in many
other episodes the more ways that you
can kind of like take in
multi-perspective approach or look at
something from many different angles the
better so with the text it's a little
bit harder to do that you could do that
with different stories different
vignettes kind of explaining different
applications of things or you could
actually show that through a media like
a video but again different contexts
different scenarios different
applications kind of depends on what
students are learning and whether or not
that would be more valuable than just
having a conversation or reading
something about it now in section five
they talk about the positive
correlations that they found in this so
an example of a positive correlation
would be the correlation between actual
learning comprehension with the
perception of learning so if students
had a higher perception of learning they
would likely have a higher learning
comprehension and vice versa they also
found a positive correlation between
usability and learning so the more
usable something is then the better it's
going to be for people to learn that
thing and then section 6 talks about
limitations and Lessons Learned and then
section 7 talks about some of the
concluding marks if you're interested in
reading that I recommend taking a look
at those sections which again I linked
to this paper in the show notes now at
the end of these unpacking scholarship
episodes I like to share some lingering
questions and thoughts just to kind of
share here are some things that I was
thinking of that maybe you'd be
interested in exploring as well so one
of the questions that I have is how do
you know when a student understands
something so an example that I've given
in other podcast episodes is when I used
to teach in a K-8 District that I was
working in it's kind of like the mentor
for the coding teachers in there one of
the conversations that we had one day
was looking at different student
projects and so this was using Khan
Academy's interactive coding platform
which made it so that students were
using JavaScript to be able to create
art and animation one of the projects
had students creating pizza so it's like
a visual depiction of pizza like on a
plate or on a dinner table or whatever
students could kind of be creative with
this and so they're creating with
different shapes to represent the
different toppings on the pizza
different slices or maybe a full pie of
pizza whatever now one of the examples
that we looked at it had this really
detailed there's like really neat
depiction of what pizza might look like
with code in terms of like how it looked
it was like a standout compared to every
single one of the other projects like if
this project was a 10 all the other
projects were like a seven however when
we actually looked at the code it was
very clear just looking at the code that
the student didn't understand Loops as
well as didn't understand some of the
shape commands Etc so in terms of the
overall product it was phenomenal but in
terms of like the demonstration of
understanding it actually demonstrated
they didn't understand what they were
doing and they kind of happened to have
some happy accidents that made a really
interesting looking project that didn't
actually demonstrate that they knew what
they were doing so it's important for
researchers and for educators to really
think through how you're actually going
to check for understanding are you just
going to look at the end result are you
going to look at the process that they
took to get that end result are you
gonna look at the final code are you
going to actually ask them even if their
code looks great and you look at it
you're like there's no flaws here like
this is very efficient and effective use
of code it looks great but do they
understand it if you sit down with a
student and you say can you explain to
me what this function does can they do
that but to argue with myself does being
able to demonstrate something tacitly
have more or less value than be able to
explain how to do that very thing now
this is something that I talked about in
episode 89 which is titled applications
of affinity space characteristics in
computer science education and I've
related it to musicians so do you want
to be able to have it so that a musician
can play a solo or be able to explain
how to play the solo which one are you
valuing or assessing for same thing with
coding do you want to see that students
can be able to actually create the thing
or finish an app or a program or
whatever or be able to explain how to do
that where along the Continuum between
the application of understanding and
explanation of understanding do you want
them to lie are they going to be
somewhere right in the middle where they
can do both or is it going to be leaning
towards one or the other depending on
how you answer this is going to kind of
depend on what kind of assessment you
end up using which I talked about in
many other assessment related episodes
that I'll link to in the show now
another question that I have is how
important is it to various stakeholders
that students understand Cs and then how
does that compare with motivation for
students to want to continue learning
something oh I asked this because
there's like different community members
or even like corporations or like
administrators or whatever they all have
kind of different like weights given
towards the value of computer science
for students to be able to learn but
what's not often discussed is the
motivation to actually continue to learn
something so are we just trying to get a
future population of Voters to a base
level understanding of what computer
science is or are we trying to motivate
people to actually want to continue to
learn computer science after the colors
is done so is the focus on your six to
nine to 12 month course and just making
sure that students learn the content or
the standards or whatever within that
particular period of time or is it to
develop lifelong Learners or for an
extended period of time as I've talked
about in other episodes like on serious
Leisure are we going to encourage
students to mod as I did did when
proposing to my wife through Minecraft
and engage in other leisurely Pursuits
of computer science and whatnot or are
we just focusing on developing citizens
who understand computer science or
developing future programmers Etc
depending on what you focus on whether
it's the understanding application of
understanding or motivation to continue
to learn it it's going to kind of like
determine what kind of an approach you
use in your classroom so something
important to consider when designing
classes what is your focus on is your
focus on understanding is your focus on
application of understanding or is it
focus on developing lifelong learning or
is a combination of those in different
ways so I had a conversation with
somebody earlier this week that person
was talking about how their kid was
having to label the parts of a
microscope in class and they're like
cool that's great I'm glad you're able
to do that but what did you see when you
looked through the microscope and the
kid's like well we actually didn't even
get to touch the microscope we were just
labeling the parts of it that to me is
just focusing on understanding like a
set of Standards not actually
investigating exploring and applying
those understandings to figure out well
what does like a cell structure actually
look like when I look through this how
do I actually zoom in and zoom out with
the different parts of the microscope
when would I even use a microscope
instead we're focusing so much on just
like rote labeling and understanding
rather than applying the understanding
and exploring or engaging in an inquiry
or whatever now that approach would have
in my opinion a profoundly negative
impact on learning it's going to make it
so students look at science and go this
is boring it's like what my math and
like history classes were like we were
learning facts we were learning how to
solve problems but we didn't know when
we'd use that we didn't know why these
facts were important it was just
memorize this date for the test and then
forget it after that as CS Educators we
really need to make sure that like we
learn from the mistakes that you may
have experienced as a student and go
well that didn't work for me or my peers
how can I take that failure for myself
and for my peers and actually make it so
the students that I work with don't have
to suffer through that in the classes
that are designed Etc but another
question that I posed and this was like
in episode 66 which was titled the role
of deliberate practice in the
acquisition of expert performance is how
do you develop expertise over time so I
was previously in a k6 school and a K-8
school but I've also taught high school
so at minimum I would see students for a
few years even when I taught
undergraduate and graduate courses like
I'd see them over multiple years
multiple courses Etc it was very rarely
what I only see students for like a year
or a semester when thinking about those
kinds of classes like in the K-8 classes
it was nine years that I could
potentially see those students trying to
think of how to make it so that I could
develop expertise from kindergarten all
the way to nine years later when they
got into eighth grade what did I want
them to be able to know and be able to
understand and be able to explore or
create or Express themselves Etc across
that nine-year period of Time how you
develop expertise over that time is very
different than focusing on a single
class where I'm only going to see these
students for a single quarter semester
year or whatever if you're only focusing
on that and you're not thinking of that
long-term development of expertise then
we're going to in my opinion shortchange
the students rather than focusing on
lifelong learning of computer science we
would instead focus on just learning
facts figures standards Etc without
actually thinking about how to apply it
over the course of a lifetime because
computer science is just constantly
changing and evolving I mean think of
like the developments in AI within the
last year alone this is going to
continue and likely accelerate over time
if all we do is focus on the concepts
and practices and skills and
understandings that are important today
we are going to fail the students who
are going to be out of the schools 10
years from now instead I think in my
opinion we should focus on developing
lifelong expertise and teaching students
how to develop that expertise when an
adult is no longer present to help guide
them the ability to be able to learn how
to learn is more important than whatever
facts and figures you can learn in Cross
Your K-12 tenure same thing with like
undergrad and graduate classes the way
that I designed every class that I ever
worked with kindergarten all the way
through graduate student was constantly
trying to focus on not just what they
needed to know in that moment but how to
also learn that down the road when I was
no longer with them in those classes so
a challenge for you listener is that
when you are designing your classes
think of not only what you're going to
do within that quarter semester or year
or however long you're working with your
students but think about how you're
going to encourage them to continue to
develop their learning and refine their
understanding at the conclusion of
whatever class you are designing if we
can do that then perhaps we're going to
have some lifelong Learners who are
interested in computer science whether
or not they even decide to pursue a
career in it I hope this episode was an
interesting study for you to look at I
know it was for me to consider the
different comparisons between like
text-based interactive and video based
lessons I personally think we need more
of this in computer science education
research and just education research in
general if you enjoyed this episode
please consider sharing it with somebody
else or leaving a review on whatever app
you're listening to this on there's over
phenomenal guests on this podcast as
well as many other episodes that explore
different topics and you can find a link
to that in whatever app you're listening
to this on or just by going to
jaredoleery.com stay tuned next week for
another episode till then I hope you're
all staying safe and are having a
wonderful week

Article

Bao, Y. & Hosseini, H. (2023). Mind the Gap: The Illusion of Skill Acquisition in Computational Thinking. Proceedings of the 2023 ACM SIGCSE Technical Symposium on Computer Science Education - SIGCSE ’23, 778-784.

Abstract

“With the advent of online educational platforms and the advances in pedagogical technologies, self-directed learning has emerged as one of the most popular modes of learning. Distance education— elevated by the COVID-19 pandemic—involves methods of instruction through a variety of remote activities which often rely on educational videos for mastery. In the absence of direct student engagement, the asynchronous nature of remote activities may deteriorate the quality of education for learners. Students often have an illusion of skill acquisition after watching videos, which results in overestimation of abilities and skills. We focus on the efficacy of skill acquisition through interactive technologies and assess their impact on computational thinking in comparison with delivery through other traditional media (e.g. videos and texts). In particular, we investigate the relationship between actual learning, perception of learning, and learners’ confidence in adult learners. Our results reveal intriguing observations about the role of interactivity and visualization and their implications on the pedagogical design for self-directed learning modules.”

Author Keywords

Skill acquisition, interactive learning, computational thinking

My One Sentence Summary

This article compares learning, perceptions of learning, and confidence among adult learners participating in interactive, video-based, and text-based learning.

Some Of My Lingering Questions/Thoughts

How do you know when a student understands something?
How important is it to various stakeholders that students understand CS?
- How does that compare with motivation for students to want to continue learning something?
How do you develop expertise over time?

Resources/Links Relevant to This Episode

Other podcast episodes that were mentioned or are relevant to this episode
- Applications of Affinity Space Characteristics in [Computer Science] Education
  - In this episode I unpack my (2020) publication titled “Applications of affinity space characteristics in music education,” which has twelve characteristics of informal learning spaces that I will discuss in relation to computer science education.
- Considering Leisure in Education with Roger Mantie
  - In this interview with Roger Mantie, we discuss the importance of leisure for self preservation, problematize the single focus of education for workforce readiness, discuss the importance of focusing on happiness and wellbeing, explore discourse in education and around leisure, and much more.
- The Role of Deliberate Practice in the Acquisition of Expert Performance
  - In this episode I unpack Ericsson, Krampe, and Tesch-Römer’s (1993) publication titled “The role of deliberate practice in the acquisition of expert performance,” which debunks the notion of innate abilities within a domain and describes the role of deliberate practice in achieving expert performance.
- More episodes related to assessment
- More episodes related to computational thinking
- More episodes related to expertise
- More episodes related to leisure
- More episodes related to skill acquisition
- All other episodes
Find other CS educators and resources by using the #CSK8 hashtag on Twitter