Assessment Considerations: A Simple Heuristic

In this episode I read and unpack my (2019) publication titled “Assessment Considerations: A Simple Heuristic,” which is intended to serve as a heuristic for creating or selecting an assessment.

Welcome back to another episode of the
csk8 podcast my name is Jared O'Leary
each week of this podcast is either an
interview with a guest or multiple guest
or a solo episode where I unpack some
scholarship in relation to Computer
Science Education in this week's episode
I'm unpacking an article that I wrote
called assessment considerations Colin a
simple heuristic and if you hear that
pitter patter in the background that is
my dog Minnie who has decided to join
this episode by running around so if you
hear some squeaks or some pitter patter
that's my dog all right here's the intro
for this paper quote discussions on
assessment and education often describe
summative assessments as a product
oriented assessment of learning
formative assessments as a
process-oriented assessment for Learning
and ipset of Assessments as a
self-reflective assessment as learning
each of these types of assessment can
serve different educational purposes so
which assessment would work best for you
and the students with whom you work this
article presents a simple heuristic for
creating or selecting an assessment end
quote alright so this episode is going
to build off of the discussion last week
on assessment types so I talked about
summative formative and implicitive
assessment and so so now I'm going to
provide a tool or a framework to kind of
think through okay well how do I create
or select some kind of an assessment but
these are just some things to think
through and if you want to actually read
this as opposed to listen to me talk
about it I'll include a direct link to
this in the show notes which you can
find at jaredollary.com or by clicking
the link in the app that you're
listening to the song just in the
description while you're there you'll
notice that this podcast is powered by
boot up professional development which
is the non-profit that I currently work
for in 2022 but in 2023 I will be
working at my website jaredelary.com so
stay tuned for a lot of new free
resources for gamers drummers and
computer science Educators alright so
here's from the article of six W's of
assessment so the first section is why
quote when creating or selecting an
assessment start by asking why you are
considering using an assessment your
intended purpose for an assessment
should serve as a guide for the
remaining considerations as it can help
determine if the assessment is intended
to inform how you teach or facilitate
provide feedback to students through
automated peer teacher or
self-reflective means or simply to a
sign a grade for example if you're
looking for an assessment that can
produce a final grade for a class a
summative assessment in the form of a
project or test might serve such
purposes however if you are looking for
an assessment that can inform how you
guide or facilitate each student during
a project or process formative and
ipsitive approaches might better serve
your needs end quote Yeah so if you're
working on some kind of like a coding
project or CS project or whatever and
you're like okay where do I want to
start with an assessment before you
figure out which one you're going to do
you really need to think through like
well why is it that you are going to do
this kind of assessment what's the point
of it like at the end of each one of the
classes that I use to facilitate there
are K8 coding classes and then like a
middle school maker space I would have
one half of the room go to the other
half of the room and they would check
out what they worked on their peers and
then they would ask some questions and
the reason why I wanted to do that was
one I wanted them to be able to see what
other people are doing to help get
inspired to figure out oh that's
something that I might do two I wanted
students to get feedback from their
peers through through in-process work to
be able to ask questions with them like
hey what do you think I should add here
or what do you like about my project and
then three I wanted students to be able
to communicate to their peers what
they're working on so they're practicing
talking about their processes and what
they're learning those were the goals
and so the assessment that I came up
with was having that peer-to-peer
assessment self-reflection Etc but then
I also had a different type of
assessment where I was like okay I want
students to be able to come in and kind
of have some kind of direction for where
they're going and to think longer term
bigger picture about what is it they
actually want to be able to do each week
because they had three weeks in a row
where they were in my class and I want
them kind of pace out to like oh this
week I want to be able to do this next
week I want to be able to do that I
think that's a very important skill to
have so thinking through this I wanted
students to be able to start planning
out their creative processes because
that's something that I found useful and
so I thought of okay well what exactly
do I want students to be able to do to
help with that so that leads to the next
section of this paper which is what
quote after outlining a clear purpose of
why consider what will be assessed when
an assessment involves the creation of a
product or expression it should focus on
the Core Concepts practices and
understandings evident within the
creation rather than the creation itself
in other words assessment criteria
should focus on the Core Concepts
practices and understandings associated
with District standards or Community
interests and needs rather than
subjective or arbitrary criteria for an
example an assessment of the Common Core
Ela standard ccs.ela
literacy.w.4.3 quote right narratives to
develop real or imagined experiences or
events using effective technique
descriptive details and clear event
sequences end quote should assess the
development of real or imagined places
through descriptive details and
sequences of events rather than
arbitrary criteria such as the number of
characters in a narrative although not
everything can or should be assessed
narrowing the focus to the Core Concepts
practices and understandings can provide
Clarity of purpose while simultaneously
opening up multiple Pathways for
demonstrations of understanding and cool
all right so let's let's tie this into
computer science so in the former
District that I was working in I was
running our like CS group and we would
talk about okay well how do we assess
projects as students we're submitting so
when we use icon Academy students would
submit in like artwork or like
animations using JavaScript and when we
would pull up a project we would have
not only what they created but also the
code right underneath it so that way we
could look at that and a lot of the
times teachers will look at the end
result and go oh this is really cool
artwork or animation this looks great
and so their immediate response was oh
this student deserves an a because they
made a wonderful picture but then we can
actually look at the code you see oh
they don't understand Loops they just
like repeatedly wrote out the exact same
sequence like five times in a row as
opposed to just using a loop that
repeated five times so one of the things
that we tried to focus on in this was to
not just look at the end result but also
to look at the code sometimes there are
happy accidents where it looked great
but the code was just like not good or
didn't really demonstrate understanding
and something they're supposed to learn
and so what we we often do is like just
look at the code first analyze that and
then look at what the end result was so
the artwork or whatever so after you've
thought through why you're doing doing
something think through what exactly it
is that you're looking at if you're just
counting stuff like I mentioned last
week like with number of costumes that a
Sprite has what's the purpose of that
does that demonstrate a specific CS
understanding or concept or skill that
you think is relevant if so then great
use that in the rubric if it doesn't why
is it in there maybe it can be removed
all right so the next thing to consider
is when quote knowing what is being
assessed and the purpose behind the
assessment can help determine when to
implement an assessment for example when
engaging in a multi-day project we might
assess understandings prior to the start
of the project shortly after beginning
the project toward the middle or end of
the project or well after the conclusion
of the project in addition to
considering when the assessment will
occur also consider when the assessment
will provide feedback for example an
assessment that occurs in the middle of
a project might not be useful if the
feedback is made available after the
conclusion of the project however such
delayed timing between an assessment and
feedback might be useful when the
feedback is used as data that informs a
self-reflection on how a student's
understanding change over time end quote
so I don't know about you but when I was
in like K-12 and higher education
classes I would frequently get really
helpful feedback after I received a
grade and there's really no opportunity
to apply that feedback in any kind of
meaningful way and for me that was
frustrating because it was like I wish I
knew this while I was writing it so that
way I can go back and like add in the
things that the teacher or Professor
were asking for recommending that's why
when I was teaching undergrad and
graduate classes I I had it so that you
either got like a hundred percent if
everything was great and then I'd give
you some feedback in the form usually of
questions or like a 70 whereas like you
turn it in on time but like ah it's not
there's some things to work through or
things to continue to improve and then
if you decided to go back and continue
to work on that you could resubmit it at
any point and potentially then get all
the way up to 100 and then a 50 was says
well there are things to work on but you
turned it in late and then a zero
percent was you didn't turn in anything
so there are only four grades you could
get and one of the reasons why is
because I wanted the opportunity for
that ability to improve so when I gave
feedback I wanted students to actually
read the feedback it was usually in the
form of questions or things to consider
and I wanted to give them that
opportunity to apply that feedback in
some kind of a meaningful way rather
than just get the grade and then move on
so again if you got a 70 cool take a
look at the feedback resubmit you could
get 100 most students really loved this
approach and then some students didn't
like it they were used to just like
getting a 91.4 because of the quality of
their work and then not having to like
redo stuff and so they just kind of
accepted their fate but those were all
types of feedback that I've described or
after the completion of something what I
really focused on like during class was
assessments like formative assessments
during the process constantly walking
around the room constantly trying to
figure out where students are at how
many people to help them in that moment
Etc now this kind of assessment was in
person and then the previous assessment
was usually asynchronous like via email
or LMS or whatever which kind of gets
into the next section which is where
quote depending on when you plan on
implementing an assessment you might
consider where the assessment will take
place if an assessment occurs outside of
class this can allow for more in-class
time to focus on Explorations and
creative applications of understandings
however when using out of class
assessments consider whether an
assessment requires access to physical
or digital resources that are difficult
to acquire outside of class for example
chemistry equipment pianos platform
specific software Etc and whether
students should be encouraged to consult
peers or external resources when
completing an assessment end quote so
the where side of things especially if
you're in higher education or high
school or there's like tendency to focus
on like scantrons or tests and whatnot
if you're doing something like that that
might take an entire class period and
let's say you do that every unit maybe
every month imagine if you could get
like a whole week or two worth of like
class time back by not doing the
assessment in class but if you did it
outside of class asynchronously if
assuming students had access to
resources and whatnot that might make it
so students can be more engaged in the
class but to argue with myself it can
also provide some pressures in terms of
like more things for them to do outside
of school which for some can be very
difficult or demanding if you decide to
do summative Assessments in particular
outside of school where it's like a take
home consider making an open book open
resource open peer discussion that
models what we do in real life I mean
how many of you have like gone on stack
Overflow and been like how do I do this
thing and then you just copy and paste
some code and change the parameters and
function names Etc I've done that so
many times I mean it's a meme within the
programming community so to pretend like
students can't do that or aren't going
to be able to do that just doesn't
really reflect what happens in real life
in my opinion but another thing to
consider not only like the why the what
the win and the where is well who is
going to do the assessment which leads
to the next section of this paper quote
although many discussions on assessments
position and assessment as an act done
by a teacher to a student a
multi-perspectival approach to
assessments might diversify feedback for
example while teachers can and should
assess students automated software can
provide in the moment feedback on
individual progress or understandings
and students can assess themselves their
peers or even their teachers encouraging
both external and internal feedback
distributes the assessment processes
across people and software which can
make it much easier to assess a large
number of people end quote Yeah so even
though I like generally don't like
automated graders and things like that
they can be very helpful if used as a
tool for reflection which I've talked
about in previous podcasts do a little
go oh this thing rated my I don't know
level of modularity as a four out of
five or something like that okay well
how can I improve my modularity or
whatever or just being able to have
peers work with each other can be really
helpful not only the person being
assessed but the person doing the
assessment can learn from this process
and then having it so that students
assess you the educator or facilitator
in the room can also be really helpful
so each week I had students like fill
out what their goals were for the week
like I was mentioning earlier and then
kind of like think through that but I
also included a question in there that
said how can I Jared help you with
whatever your goal is for the week and
good but sometimes a student would be
like I could really use some resources
on like how to do player controls and so
then I'd go and find a student or a
resource that would be able to help them
with that but I also got some feedback
online Kara's teaching some students
wanted more structure like hey I could
really use some more assistance with
setting my own goals oh cool yeah I can
help you out with that now I mentioned
multiple assessment types already
without necessarily labeling them and
then talked about some last week but
that leads into this next section titled
which quote when determining which
approach to use for an assessment
consider each of the following types of
assessment in relation to each of the
aforementioned W's of assessment
summative assessment of learning
summative assessment is quote an
assessment at the end of a unit scheme
term or whatever which is designed to
summarize student attainment into a mark
grade or level end quote summative
assessment is sometimes referred to as
an assessment of learning because it is
an assessment of intended learning
outcomes demonstrated through
performance examples of summative
assessment include exams projects
playing tests portfolios Etc formative
assessment for learning performant
assessment is quote an assessment which
happens as work is being undertaken and
is purposed with improving the work done
by the student often undertaken in
dialogic form in quote formative
assessment is sometimes referred to as
assessment for learning because it
provides feedback which informs future
learning examples of formative
assessments include one-on-one
facilitating and questioning daily
journals entrants or exit tickets think
pair share storyboarding Etc ipsative
assessment as learning ipsative
assessment is quote an assessment the
student makes against their own prior
performance so that they are measuring
their personal progression against their
own previous work and quote ipsive
assessment is sometimes referred to as
assessment as learning because it is an
assessment where students are active
participants in an assessment intended
to produce learning an example of an
ipset of assessment that can occur as a
daily self-reflection or after the
completion of a project is a reflection
of what students learn while working on
a project in comparison with their prior
understandings EG when working on
previous projects as well as setting a
goal for continued learning an ipsitive
assessment differs from student
completed summative Assessments in that
the focus of an ipsitive assessment is
on reflecting on changes in
understanding over time and setting
goals for future learning rather than
evaluating a completed product or
expression end quote alright so I
mentioned each of these three assessment
types the summative formative ibps that
is last week so I'll include a link to
that in the show notes if you haven't
happened to listen to it and I won't
elaborate it on here but this gets into
the very last section of this short
article which is on combining assessment
types quote each of the three assessment
types mentioned above can serve
different purposes that may address your
why of an assessment however these
assessment types are not mutually
exclusive as each assessment type can
work together to provide recurring
feedback for different purposes for
example in the K8 project-based classes
I previously facilitated I heavily
emphasize formative assessments to focus
on in the moment feedback for projects
that took weeks or years to complete
however we also engaged in summative and
ipsitive assessment as a completion of
projects for setting weekly goals for
providing peer-to-peer feedback at the
end of each class and for providing
feedback on how I might better assist
the kids with whom I work throughout
each of these assessment types I
followed up with questions to further
assess understandings as a correct
answer apparent skill or completed
product does not necessarily demonstrate
understanding and abilities that are
consistently replicable or transferable
end quote that last point is super
important like I mentioned with the
coding teachers that I mentored in the
previous District that I was working in
even when a student like created a
really cool product and their code was
great you'd think of like what are some
questions we could ask students to
really assess whether or not they
actually understand why it worked the
way that it did sometimes it was a happy
accident sometimes a Peter just wrote in
the lines of code that worked and
they're like I don't know what's going
on but hey it's functioning now thank
you so having this kind of combination
of summative formative and ipsative is
extremely important you can assess
throughout the beginning the middle the
end and well after the conclusion of
some kind of a project or product or
whatever so I guess the main purpose of
this is to Think Through the different
questions why what when where who which
and how you are going to combine these
different assessment types in the
different classes that you're working on
even though I really preferred formative
I made sure that we still had ipsative
and summative forms of assessment it's
important to have this combo of them you
might lean towards one over the others
but it shouldn't be I only do summative
or I only do formative they all have
their purposes in places but at the end
of these unpacking scholarship episodes
I like to share some lingering questions
or thoughts I kind of embedded some
elaborations throughout and then because
I wrote it I don't really have as many
questions as I normally would one of the
questions that I have for you is what
questions or Frameworks do you use when
considering creating or selecting an
assessment another question that I have
is what's missing from this discussion
yeah it was helpful to have the six W's
of assessment that's like an easy
marketing way to kind of like frame
things but what might have been useful
for the heuristic that didn't have a w
in it if you got some ideas feel free to
let me know on social media you can find
all my social medias at jaredleary.com
as well as some links to other
assessment resources and like 160 some
odd episodes that are Solo episodes like
this as well as some interviews with
some amazing people talking about
Computer Science Education research and
practice thanks so much for listening to
this episode stay tuned next week for
another one until then I hope you're all
staying safe and are having a wonderful
week

Article

O’Leary, J. (2019. Assessment Considerations: A Simple Heuristic. www.JaredOLeary.com

Intro

“Discussions on assessment in education often describe summative assessments as a product-oriented assessment of learning, formative assessments as a process-oriented assessment for learning, and ipsative assessments as a self-reflective assessment as learning (Manitoba Education, 2006; Scott, 2012). Each of these types of assessment can serve different educational purposes, so which assessment would work best for you and the students with whom you work? This article presents a simple heuristic for creating or selecting an assessment.”

My One Sentence Summary

This article serves as a heuristic for creating or selecting an assessment.

Some Of My Lingering Questions/Thoughts

What questions or frameworks do you use when considering creating or selecting an assessment?
What’s missing from this discussion?

Resources/Links Relevant to This Episode

Other podcast episodes that were mentioned or are relevant to this episode
- Discussing Computer Science in K-12 with Shuchi Grover
  - In this interview with Shuchi Grover, we discuss the importance of having a variety of assessments in a CS class, why we need more research on computational thinking, why educators and scholars should read literature outside of the field, Shuchi’s new book titled “Computer Science in K-12: An A-To-Z Handbook on Teaching Programming,” and much more.
- Effects of Automated Feedback in Scratch Programming Tutorials
  - In this episode I unpack Obermüller, Greifenstein, and Fraser’s (2023) publication titled “Effects of automated feedback in Scratch programming tutorials,” which investigates the impact of two different types of hint generating approaches among two different classes.
- How Do You Encourage Critical Thinking and Dialogue?
  - In this episode I discuss an approach I’ve used for encouraging critical thinking and dialogue through individualized feedback and group discussion.
- How to Get Started with Computer Science Education
  - In this episode I provide a framework for how districts and educators can get started with computer science education for free.
- Individualized Learning Without Grades with Sofía De Jesús
  - In this interview with Sofía De Jesús, we discuss Sofía’s book (Applied Computational Thinking with Python: Design algorithmic solutions for complex and challenging real-world problems), the importance of bringing your full self into the classroom, designing for equity and inclusion, working with individuals one-on-one rather than teaching to group averages, problematizing grades in education, collaborating with educators, and much more.
- Rethinking the Roles of Assessment in [Computer Science] Education
  - In this episode I unpack Scott’s (2012) publication titled “Rethinking the roles of assessment in music education,” which summarizes three roles of assessment (assessment of learning, assessment for learning, and assessment as learning) that I discuss in relation to computer science education.
- The Centrality of Curriculum and the Function of Standards: The Curriculum is a Mind-altering Device
  - In this episode I unpack Eisner’s (2002) publication titled “The centrality of curriculum and the function of standards: The curriculum is a mind-altering device,” which problematizes curricula and standards by discussing how both can deprofessionalize the field of education.
- Vulnerability, Reflection, and CS Education with Amy Ko
  - In this interview with Amy Ko, we discuss the importance of mentorship in education, learning what not to do with teaching, the positive results of being vulnerable, understanding and exploring the limitations and consequences of CS, problematizing grades in education, practicing teaching through mental simulations, the importance of engaging in the CS community, and much more.
- More episodes related to assessment
- All other episodes
Find other CS educators and resources by using the #CSK8 hashtag on Twitter