Accessible CS Education through Evidence-based Programming Languages with Andreas Stefik

In this interview with Andreas Stefik, we discuss the importance of using evidence-based programming languages, problems with the lack of replication in CS education scholarship and academia in general, the importance of designing for accessibility and disabilities, lessons learned designing Quorum (an accessible programming language and platform), and much more.

  • Everyone I just want to give a very

    quick disclaimer before we started the

    episode that this episode was recorded

    in January of 2020 now the reason why I

    say that is because at one point in the

    episode I mentioned doing weekly

    Japanese lessons at my house and that

    was accurate months ago but it is no

    longer accurate doo doo Cove in 19 so

    just FYI we are practicing social

    distancing and all that anyways I hope

    you really enjoy this episode and let's

    get it started

    welcome back to another episode of the

    CSK8 podcast my name is Jared O'Leary in

    this week's episode I'm interviewing

    andreas tefik in our discussion we

    talked about the importance of using

    evidence-based programming languages in

    particular we talked about it in

    relation to designing for accessibility

    for kids with disabilities we also

    talked about some of the problems with

    the lack of replication in CS education

    scholarship as well as some of the

    problems in academia in general in

    addition we discuss an evidence-based

    programming language that stefa helped

    create that's called quorum as well as

    many other topics relevant to CS

    educators there are a good amount of

    show notes that link to many of the

    papers that we've discussed as well as

    some other music related concepts that

    we discussed at the beginning and

    towards the end of the interview as well

    as many resources that can help with

    designing for accessibility in your cs

    classroom you can find those show notes

    by clicking the link in the app that

    you're listening to this on or by

    visiting jared O'Leary comm and we will

    now begin this entertaining and

    informative interview with an

    introduction by steph ik my name is

    andreas tefik almost everyone calls me

    Steph ik I'm a computer scientist that

    works at the University of Nevada Las

    Vegas I do a lot of work related to two

    major things one is investigating how

    people invent things in software sort of

    investigating programming languages but

    I also do a considerable amount of work

    related to how people with disabilities

    do computer science I had meant a lot of

    stuff in that direction can you tell me

    the story of how you got into CS

    education and research when I was an

    undergraduate I was doing a degree in

    music I've played in bands for years and

    stuff I was never really very good at it

    but I enjoyed it you know it was kind of

    fun like hobby type of thing and I was

    reading in textbooks at school on this

    composer named Ben Johnston he did a lot

    of music that's called microtonal it's

    kind of like keys between the keys type

    of stuff frequency bands not on the

    piano that could be used as notes that

    always sounds wacky when you say it

    actually there's really good reasons for

    it that's because some notes on the

    piano would actually be more in tune and

    others are actually less in tune and you

    can figure that out using some pretty

    simple math and then okay so as an

    undergrad I moved to the East Coast to

    work with this guy and when I was there

    he was using a lot of music notations to

    represent this kind of music inevitably

    kind of complicated that were really

    hard to understand and there was a

    string quartet called the Keppler

    quartet that wanted to play the music

    but they couldn't figure out what it

    would sound like based on the notation

    so the problem was the notation was

    really hard to understand they wanted to

    play it but they really couldn't do it

    so this guy Ben Johnston he sort of

    asked me when I left the training is he

    if I could figure out an alternative

    that would make it easier for groups to

    play as a notation so I started taking

    classes in computer science thinking

    maybe I could write a computer program

    or something or do some kind of math or

    some funny thing to make it easier and

    it took a couple years and by the time I

    figured it out I had a computer science

    degree eventually though a number of

    albums were produced on this and what my

    graduate school I had already done all

    this stuff related to notation to try to

    make it easier to understand in addition

    I've been doing music for a since

    forever so I was trying to think of

    projects related to who benefits from

    sound based technologies in computer

    science so I started talking to schools

    for the blind and just talking to the

    kids hey kid have you ever written a

    program before do you want to try and

    then kind of watching them and

    inevitably watching them struggle

    because doing programming when you're

    blind using off-the-shelf technology can

    be a pretty serious challenge and

    everything kind of snowballed from there

    in terms of my interest into computer

    science education now it was the

    notation that he was writing was it

    aleatory so like chance based or was it

    no this is a very specific note that i

    want you to play but it's in between

    what's typically the white and black

    keys it's the second one so okay you

    think you like John Cage's style

    notations yeah it's more like graphical

    where it's kind of like up to the

    artists interpretation like how do I

    play this thing

    yeah actually funny if I were correctly

    I think John Cage lived in Ben's attic I

    think that's true

    in any case no wasn't it wasn't

    Elliott's work but he was influenced by

    that kind of stuff so there are some

    things that kind of sound chanting his

    music but it's very specified so for

    example if you had a particular note

    combination you actually specified

    mathematically the relationship between

    the two notes so like d2a is exactly the

    frequency of D multiplied by 3/2 right

    and that's not a king on the keyboard

    right but it is notation that would end

    up with a certain set of symbols that

    represent that frequency relationship

    not the normal keyboard and it has all

    these weird names like sin tonic commas

    and stuff like that and it gets kind of

    wacky but what's interesting about is it

    actually explains when you're thinking

    about choirs and stuff why fighters tend

    to go flat and sharp over time because

    the frequency bands on piano don't

    actually match what would be in tune so

    it kind of like is somewhat explanatory

    sort of asking yeah and if this is too

    abstract for people who don't have a

    background in music I'll include some

    resources in the show notes that kind of

    explain the different types of tunings

    and whatnot that you can find could you

    elaborate on what you mentioned with

    watching a blind student trying code

    what did that look like in terms of your

    AHA's and going oh this is not at all

    how somebody who is able to see it uses

    this program I had been primed with the

    challenges related and notations I kind

    of mentioned now it's totally different

    kind of notation mm-hmm but nonetheless

    that was clear in my head so I called

    the Washington state school for the

    blind and was like hey I want to do

    something together and then eventually

    after there's some grants and some stuff

    like that I went we started observing

    kids and so one of the very first things

    we did we did some experiments where we

    had kids that age try different

    programming languages and literally just

    try and like write a loop right it is

    just simple little programs and because

    they come out through screen readers

    there's weird things that happen right

    at least weird if you're not a screen

    reader user which I usually explain

    because most people aren't and so I'll

    give an example you write a loop in C++

    you can see it visually if you know the

    code what sounds like is or left paren

    int I equals 0 semicolon I less than 10

    semicolon I plus plus right paren

    imagine hearing that is a

    fourteen-year-old kid three times faster

    than what I just said and not knowing

    what a single symbol meant right also

    that's tricky and then if you have an

    error in your code you're gonna have a

    compiler error back now in in an IDE

    development environment you might have

    like a little visual display on the

    bottom of the screen and your eyes

    you're naturally skip certain things so

    like microsoft visual studio at the time

    you would have like all sorts of weird

    error codes colons funny slashes and

    stuff like that and when i visually look

    at it with my eyes i skip stuff right

    i'm not gonna read underscore underscore

    c CDEC l colon colon four or five eight

    five you know that stuff is just useless

    right but if you're listening with a

    screen reader guess what every character

    gets read so you got a compiler error

    that compiler might last a minute in

    audio and the actual thing you did wrong

    like oh you missed a semicolon is like

    halfway through so it's real easy to

    miss so we're sitting there watching

    this and I was like man there's no way

    that someone like maliciously designed

    it this way I bet just no one cared and

    even thought about for a moment whether

    this would be the design for somebody

    that's using it screaming oh that smells

    like an opportunity and we kind of

    jumped on it so how's your research in

    accessible computing kind of informed

    your own practices as an educator and a

    scholar one of the first things that we

    thought was when we're watching blind

    students well babe this is probably just

    a gladness issue right I skip over that

    stuff I don't need it or maybe for left

    her and all that kind of stuff it

    doesn't really matter but it turns out

    that we were suspicious that that was

    probably wrong why is it the programmers

    designers use the word for you know and

    it's not just like C++ there's kids

    languages to pencil code uses the word

    for that's a block language it doesn't

    really matter like lots of language

    designers made decisions about how they

    represent their code I wonder you know

    what is the data well you know when C++

    was made like what kind of evidence did

    they use to make their decisions were

    like you know pythons much newer

    what is the evidence on various

    decisions that they made in Python

    what's the evidence on the whitespace

    about the colons all the other funny

    things and so I like kind of naively

    thought well let's just go look it up

    and gather all the data

    designers been gathering and try to

    figure out like what was the right

    answer so that maybe we can make it a

    little easier for these kids what I

    wasn't expecting well is that there

    wasn't any data and that which designers

    have in fact emphatically been against

    gathering data and I'm not kidding about

    that for a long time and this is

    actually this intuition that like this

    sort of like it's not part of their

    community gathering evidence is really

    weird sounding so people that like

    aren't used to this idea that scientific

    communities also have their flaws and

    imperfections this has actually been

    formalized by a number of scholars one

    of them is there's a guy named Adi Hani

    kiona who is this Finnish scholar and he

    actually analyzed effectively every

    paper from like 1950 till 2012 this is

    like peer-reviewed top conferences made

    decisions about National Science

    Foundation support all sorts of stuff

    and it turns out between that entire 60

    year time period there was 22

    experiments and that is nuts absolutely

    nuts like that just should not be true

    when people are designing programming

    languages there's two parts there's the

    technical you got to make sure their

    works and then people have to use the

    damn thing that's a fact we all do it

    and developers get paid an enormous

    amount of money nationwide or worldwide

    to use these things hundreds of billions

    of dollars a year gets paid to devs so

    tiny little impacts from productivity

    impact billions of dollars national it's

    a big deal and is that including even

    just a quasi-experimental or yeah so

    there's whenever you talk about

    gathering data how you gather it depends

    a lot on the context under which you're

    gathering so for example the word to use

    quasi-experimental often means that you

    lack certain methodological controls in

    an experiment but sometimes you have no

    choice you actually have to do that for

    example I can't give half of the country

    cancer right that doesn't make sense I

    don't think it passed IRB either yeah

    like just plausible right just on a

    practical level in education this is a

    big deal too because even though it's

    much less extreme I can't take kids and

    randomly assign them to schools that are

    at various levels of poverty I can't we

    assign them to new parents or stuff like

    that so there's always an element of in

    the field you have to make certain

    compromise

    when you're conducting studies and

    there's no choice about that even if you

    really wanted to it would be impossible

    to do otherwise on the other hand is

    full experiments and there's surveys and

    stuff like that too full experiments

    where you do have custom controls those

    have value to you just use them for

    different kinds of things when you're

    looking for evidence so we kind of do a

    combination things and that's really

    common

    yeah and in education for people who

    don't have a background in research it's

    particularly difficult to do full

    experimental studies one it tends to

    make things decontextualize like you

    might be going to a lab setting rather

    than a social setting where kids are

    learning but the quasi side of things

    you have a lot of factors and variables

    that you can't control for because

    you're in a live setting where kids are

    learning not just in the environment

    that you are testing and yeah so I think

    that the model that I like the most and

    you may you may not like this particular

    model I don't know there's a very famous

    scholar named Austin Bradford Hill you

    heard this guy mm-hm

    you know that the phrase that smoking

    causes cancer you know this yeah I mean

    it's common commonly known not everybody

    buys it I guess but the evidence is very

    clear right

    that's awesome Bradford Hill he was

    basically the first caller to sort of

    show pretty definitive evidence that

    smoking not is correlated with cancer

    that might be related to cancer but no

    no it causes cancer right and the way

    that that worked what's called Medical

    randomized controlled trial now when

    people say that word randomized

    controlled trial they think it's a

    custom experiment that just has very

    careful controls and all sorts of stuff

    like that but actually since Bradford

    Hill experiments even in other

    disciplines where you have these

    constraints just like you do in computer

    science education they use a combination

    of what can we control and what can't we

    and then how do we correlate with other

    factors so I'll give you an example in

    some of the original studies they would

    do things like okay I know that I can't

    give people the cancer and I know I

    can't control their smoking habits so I

    can't control for that but I can't

    control for certain other kinds of

    things and so they basically designed

    these studies where they had control

    groups and they did it for certain

    things that they cared about that they

    can manipulate and then there was other

    things that they basically just tracked

    anything related to it so that they

    could correlate it in and rule out

    alternative explanations so when

    conduct studies although nothing's

    perfect you have to just kind of do the

    best you can we always try to follow the

    Bradford Hill model where it's like

    control what we can control and then

    correlate in any other factor that we

    can switch you least clues about

    causality yeah and then I'm assuming

    doing follow-up studies to kind of like

    Rivera fie in different scenarios and

    with different participants and whatnot

    he'll upstairs replications want other

    groups to do it in a different way you

    want people to bury the experiments

    science is complicated and you want all

    that stuff one of the guests that I

    spoke with had mentioned that that was

    one of the biggest things that she

    wished there was more of was people of

    following up and doing studies and with

    different sets of people and whatnot

    just to kind of find out okay well we

    thought this thing but now when we

    applied it in this different context we

    ended up with a different result and so

    let's explore that and why in general

    not a lot of replication going on and in

    the field of education yeah and there's

    a couple reasons why that is actually

    and one of them is that especially in

    computer science education it sounds

    really weird to say but we actually have

    no evidence standard now what does that

    mean well actually it turns out there's

    a formal thing called an evidence

    standard it's not like a term I'm making

    up it's a like a standard term evidence

    standard and what it means is that when

    you publish papers there's certain

    protocols you follow for how you report

    experiments now think about this if I'm

    in the medical journals they actually

    follow and called consort Co NS ort

    consort is a medical standard that you

    have to follow if you publish at certain

    journals and it basically means that you

    have a replication package you have

    certain things you report you report

    demographics you report how the study

    was done that kind of stuff and the

    reason that matters is because anybody

    that wants to replicate it really needs

    that information able to do it in

    education there is a evidence standard

    called a what works Clearinghouse it's

    much more complicated and that's a whole

    other topic of discussion but it's not

    really the same and what do you talk

    about computer science journals the

    evidence center that uses nothing people

    just kind of publish whatever they want

    so if you want to replicate other

    people's work sometimes it's almost

    impossible because you really don't have

    the information you need let alone like

    a replication package and stuff like

    that so it's really hot because we are

    Journal

    having kind of gotten on the bandwagon

    yet for evidence standards and therefore

    were really high

    yeah and I've done some reviews for the

    Department of Ed and then applied for

    some grants that required what works

    Clearinghouse and with a problem that we

    were running into was there weren't any

    studies related to CS education that we

    could cite so what we ended up doing was

    saying well here's some professional

    development stuff that's in alignment

    with what we're doing but it's in a

    different content area so it's kind of

    like this but not really I a hundred

    percent understand exactly what you're

    saying that's completely true because we

    don't follow the standards then it is

    what it is and it's tricky to computer

    science education is also an

    international community it's not just

    Americans but the what works

    Clearinghouse only covers Americans so

    it's sort of funny like we almost in a

    way want like an international evidence

    standard that covers everybody cuz like

    are the kids in Finland the same as

    United States

    maybe but like maybe not and in fact

    there's some evidence to suggest that

    that's not true there's a woman named

    Felina Herman's that does really

    fantastic computer science education

    work I think she's in the Netherlands

    and like they speak a different language

    and the programming languages are in

    English what are the challenges there

    well they're not zero right they're

    complicated you know but yeah that's not

    gonna make it into the what works

    Clearinghouse even if it's all work

    right yeah hopefully in a couple of

    years that'll change for now it's just

    frustrating frustrating it's so much

    easier to read studies that are in an

    evidence standard like you look up study

    on PubMed that follows concert you know

    where he is like oh you want to look at

    demographics look it's in the section

    with that exact name every single time

    there's no debate it's just simple yeah

    for my dissertation I tried to make it

    as replicable as possible so it was a

    corpus assisted discourse analysis so I

    looked at like 11 million words of data

    and try to find patterns of discourse

    and then what does this mean for

    education so like in my methods chapter

    in chapter 3 it was just all about okay

    if somebody else wanted to be able to

    redo this study what do I need to

    include in there so there's very

    conscious of that so hopefully people

    could redo it if they wanted to oh

    that's lovely ten years ago we had

    replication packets but they were all

    kind of like we didn't really know what

    they should be and stuff nowadays

    replication packets are download this on

    get

    and humans right right right

    and so it's a lot at least theoretically

    a lot easier to sort of rub lick the

    studies now does anybody actually but

    it's possible and that's the first step

    yeah then there's also the issue of well

    what counts in terms of publication and

    for the institutions that you're at like

    are you gonna get as many citations if

    you're the one doing the follow-up study

    or if you're doing the initial study

    that people are then going to try and

    replicate that's a good point I mean

    like we obsess about these ridiculous

    things like h-index and citation right I

    mean they're not totally dumb like we do

    care about it to an extent of like you

    know you want your work to be used by

    other people but it misses two

    fundamental things like these kind of

    weird metrics that were obsessed with an

    academia number one is all the metrics

    related to citation counts and paper

    writing and stuff they usually emphasize

    that you should be writing right like

    your h-index how do you get your h-index

    up you write more papers and then you

    write papers in popular areas that's it

    right that's the the most likely way to

    do it so pick a big community and then

    write lots of papers to that community

    to increase your age index

    however we forget that as computer

    scientists we also can invent things and

    that has value so for example we

    invented this programming language

    called quorum and it's used last year

    but I think like eighty one thousand

    people right now that's a metric but at

    the same time like if I spend time

    writing that application I'm actually

    not spending time writing which might

    look bad from other academics

    perspective but honestly I don't care

    because I think that has more tangible

    impact on society than just obsessively

    working on writing so it feels like we

    need as a culture of academics you need

    a better balance as to like how we think

    about what we choose to do in our own

    jobs so well and it rewards and kind of

    punishes some approaches over others so

    for instance survey really easy to send

    that stuff out and to get data back but

    doing a longitudinal ethnographic study

    on a particular culture or set of people

    or whatever that's going to take much

    more time and it might be multiple years

    until you can actually start publishing

    on the actual findings from those kind

    of studies one hundred percent the first

    big study that we did

    programming language design was in 2013

    and it took my student and I for years

    to put that study together now like

    that's not what was me it took a long

    time that's because we wanted to know

    the answer right that mean that like I

    could have been doing smaller studies or

    split them up into like little mini

    papers and like just so stupid it's just

    a waste of time it's easier for the

    research community if you have one

    resource that's well vetted over a long

    period of time and lots of little mini

    pubs that get tossed out all the time

    sort of I know I just feel like we waste

    because we're so obsessed with these

    metrics it also wastes our time because

    we end up reading one the next increment

    on the same paper which is like you know

    a three percent improvement bomb is

    algorithm or something that of like

    waiting a couple years in getting a big

    foundational advance so I'm curious to

    what's something that you believed about

    accessibility in particular when you

    first began but you no longer believe

    now that you've done some research in

    that area when I first started working

    in accessibility I was working only with

    blind people now that's not true anymore

    we work with deaf people by people

    people with mobility impairments

    learning disabilities all sorts of stuff

    and I loved it I loved every minute of

    it but when I first started out working

    only with white people I didn't have

    context for all these other groups yet

    and so I sort of naively I think assumed

    that when you add support for one group

    for accessibility that that mostly

    impacts that group but I know with

    certainty that that is not true I'll

    give you a few examples if you have

    people that are blind or visually

    impaired they use the keyboard very

    heavily because you can't see the screen

    you wouldn't use a mouse that makes no

    sense right I mean some people that are

    blind can also see that's complicated

    but never mind that I even have a

    student that is blind legally blind that

    can drive but never mind depends on the

    state and since they're using the

    keyboard heavily because they're very

    used to it if the keyboard shortcuts are

    really nasty it will get very annoying

    very fast so if you're blind it's

    important to be able to remap keys to

    sort of whatever is kind of convenient

    to make sure they follow standard so

    conflict with screenreader keys and so

    like that however if you have a mobility

    impairment you must be able to remap the

    keys like you just have to otherwise you

    really can't use the product in addition

    you might think that if you have closed

    captioning for a video

    that that only impacts the death but

    actually that's not true at all and it

    impacts parents if you have closed

    captioning and you want to turn the

    sound off because your little kids are

    asleep you might want to read or if

    you're in a busy hotel or something at

    an airport or whatever it is in addition

    people that are blind often download

    closed captions so they can read the

    transcript it's our screen reader at

    higher speed because waiting for the

    video and something be very slow and

    there might be visual things on screen

    that like don't get shown except baby

    and the captions or somewhere else

    interesting yeah so a lot of the

    technologies that you use that are

    related to disabilities often get used

    not just by that group but by other

    disabilities groups that end up having

    the same reliance but also like groups

    that you wouldn't think of like a parent

    that just has a screaming kid or like

    you know someone in a wheelchair that

    has a curb cut that they go up well

    that's gonna use by a woman with a

    stroller too that has a child so I think

    that like even though I'd heard these

    phrases like curb cuts before when I was

    doing the accessibility research I think

    that that message didn't really hit home

    until later when I was sitting watching

    different kinds of groups like watching

    a kid with dyslexia use a screen reader

    and you know things like that so what is

    something that you wish more CS

    educators kind of understood about

    accessible computing there's really two

    things that I often think about that

    went with accessible computing for

    computer science educators number one I

    wish more people realized that it's

    required by law like so for example if

    you're developing a tool you're making

    blocks or something like that right you

    are putting yourself at legal liability

    if you don't add accessibility support

    in K through 12 that's never fun to say

    that's like the not nice thing about

    accessibility but actually there's laws

    protecting people with disabilities

    because they've been horribly

    discriminated against historically like

    for example if someone said I'm gonna

    have a child with darker skin excluded

    from this computer science curriculum

    all of us would flip out right that's

    just unacceptable

    or if you said I don't think women

    should be programmers we would say screw

    you that's just completely unacceptable

    that's just not not not not right but if

    almost every single

    language out there is inaccessible to

    the blind we sort of throw our hands up

    and say well it is what it is so number

    one the not nice thing about

    accessibility but I wish people knew

    more it's required by law it is not

    optional in K through 12 every child has

    the legal right to do education as

    States increasingly make computer

    science mandatory people are gonna have

    to cope with that but the second thing

    that I think people that is much nicer

    about accessibility is that like people

    with disabilities often have

    observations about how they invent

    things that are super fascinating you

    know like watch I would not have

    realized that you can make progress on a

    problem as complex as the programming

    language Wars like how you know these

    languages vary and whether they're easy

    or hard has been a matter of debate for

    a long time until I started watching

    blind children I just want to notice so

    I didn't think about it I'm so used to

    you know these looping structures and

    classes almost kind of stuff I wouldn't

    thought about it so oftentimes when you

    work with people with disabilities you

    get the big advantage that you get to

    observe things that many of your

    colleagues in computer science education

    would not have seen right you get to see

    them like oh trying to program with

    audio and seeing all the shortcuts they

    press or you get to see someone with

    deafness trying to interpret some really

    bad captions in a funny way or looking

    at the language differences people that

    are deaf have in comparison to people

    that weren't using ASL for most of my

    life or watching someone with a mobility

    impairment trying to get around some

    particular feature that they're using

    I mean it's fascinating and has a lot of

    opportunities for making things easier

    for everybody and like I feel lucky that

    I get to kind of sit and observe these

    people all the time that they kind of

    allow me to do that because it gives me

    a lot of fun insights that I get to come

    back from playing it yeah I think that

    really speaks to having diverse teams

    and having diverse users whether it be

    in beta testing or product users or

    whatever I was at WWDC and was at 2016

    and paraphrasing here one of the things

    that they had announced at the

    conference was that their approach was

    focusing on designing for the margins so

    that way you can include everybody

    whereas if you design for the

    average then you're excluding the

    margins and whatnot and that really kind

    of hit home for me or it's just like oh

    yeah that makes a whole lot of sense and

    they showcase it by having like blind

    DJ's go up and perform and like things

    like that just like little things that

    are like okay this is really cool and

    it's nice to see that one of the largest

    companies in the world is really pushing

    for making things more accessible for

    people I think Apple has put in a lot

    more work than a lot of groups and I

    think it's gotten better over the years

    nothing's perfect at any company and

    even even with really good intentions

    and stuff like that sometimes the

    challenges can be hard to make something

    accessible it's it's not like you flip a

    switch it can be difficult right so

    speaking of that what might you

    recommend for CS educators who are kind

    of designing either like curricula or

    educational experiences what would you

    have them consider to become more

    accessible with the experiences or

    curricula that they use yeah so there's

    a few things number one is for the

    curriculum itself when it comes to like

    you know the text that you write and

    stuff like that always at least do the

    basics basically understand that web

    browsers do a pretty good job for

    accessibility they're really not that

    bad but you have to not break it so like

    I'll give you some examples of ways that

    you could break it usually not

    maliciously that are coming dorable that

    I think of funny so my daughter right

    now is at the time of this recording is

    seven years old and she goes to a local

    local school and at their local school

    they have a calendar that like just kind

    of says their events and stuff like that

    I'm like it's clear they don't have an

    expert putting their website together

    which is normal it's a grade school

    right right okay and this is so adorable

    they actually have like it's like a I

    don't know if it's like a spreadsheet 11

    they like take a picture of it and then

    they put up the picture that's it and

    like as a computer scientist I look at

    that I'm like oh dear II that's just not

    that malicious they're just like trying

    to put the schedule for everybody no you

    don't do stuff like that like don't like

    take pictures of text and just put it up

    with an aunt or something like that's

    just a really bad idea so use text

    normally and that kind of thing and the

    second thing is when you do have

    pictures there's nothing wrong with that

    but there's little tags you have to put

    in or call alts that give you

    descriptions

    and those kind of things are the really

    like the bare-bones basics of making

    sure that stuff is accessible they don't

    make your site accessible they don't

    make your tools accessible or nothing

    like that but it's like if you don't do

    at least those things things are

    probably broken for people I want people

    to do more is to start making sure that

    when you're writing grants and when

    you're using tools make sure everything

    in your tool chain is accessible and

    just like if something in your chain

    isn't accessible drop it get rid of it

    use another technology that is

    accessible because it is the law and

    also because it's the right thing to do

    you know and there are challenges there

    like I think that we can make block

    language is accessible or time I think

    that will be solved I think that that's

    not like a huge technical burden I think

    that's okay and I also think that there

    are technical challenges that people

    have as they're doing stuff you know

    like if you're making a curriculum

    related to Android

    well Android naturally has some nasty

    accessibility stuff in it because the

    Android accessibility architecture is

    really not that it has issues right and

    then vice versa like accessibility for

    every platform it turns out is very

    different so apples accessibility

    architecture is 100% incompatible with

    the Windows one which is 100%

    incompatible with the Android one which

    is 100% incompatible with the web so

    cool great job so to a certain extent I

    feel empathy for many curriculum

    designers because they're like well I

    really want to use platform X I don't

    know about accessibility on that

    platform and it's complicated and I need

    help so the one other thing I would

    mention if you are someone that does

    curriculum and you don't have the

    expertise to do it reach out to those of

    us that do because we can totally help

    you we have grants to help you and we

    have like thoughts about it we can go

    through it with you and talk to you

    about it and go through some of that

    stuff it can be complicated sometimes

    depending on the platform but that's

    reality of the nature of accessibility

    today so speaking of platforms you had

    mentioned quorum already I'm wondering

    if you could describe what quorum is for

    people who aren't familiar with it yeah

    so quorum is a programming language so I

    mentioned this issue of like evidence

    with languages so my lab and I thought

    well if you were to try to make a

    programming product based on evidence

    what would that even mean and what would

    that be

    so quorum has become a programming

    language where we meticulously and

    rigorously and sometimes even

    obsessively go through every symbol

    every character every combination of

    semantics and whatever with human

    studies to try to make it easier to

    understand there's there's two pieces of

    it one of which is the language itself

    just like the raw text and all the

    libraries and all that kind of stuff and

    we evaluate that in different kinds of

    ways to making things here but another

    is very recently we've been working

    toward visualization of the technology

    oh this is really hard to make

    accessible so it's taken us a couple

    years to do it

    for example are you familiar with

    unity3d bourbonnais mm-hmm so for

    anybody listening that hasn't unit III

    is like a programming environment that

    involves 2d and 3d graphics well how do

    you make that accessible

    well we've solved that and we have a

    version that's totally blind accessible

    and other disabilities that allows you

    to do 2d and 3d gaming and all sorts of

    cool stuff inside and porom that will

    come out in July but in any case if you

    want things like blocks or

    visualizations on top of programming

    languages to be accessible you need to

    do that so when we eventually go in and

    add that kind of support to quorum we

    have to do the accessibility side first

    which so it takes a little while but for

    the language really what it is it's a

    programming language where a group of

    scholars not just me but people outside

    of my research group haven't been

    obsessively running studies on anything

    you can think of to try to make the

    language easier to understand if you

    have loops in a normal language that

    would be this for left paren int I

    equals 0 semicolon type stuff in quorum

    it would be repeat ten times and why is

    it those specific words is because we

    have survey data on the word choices we

    have randomized control trials comparing

    it to other programming languages and we

    often compare those across groups

    probably the most famous one is our

    placebo study I don't know if you know

    that one No so in 2013 we had this idea

    for a study we're trying to think of

    like how would you compare programming

    languages and what would be the line

    like you know for example in the medical

    sciences they often use placebos which

    are things like sugar pills and then we

    compare against placebo to kind of give

    like a comparison across a set of

    medicines like if a placebo works in a

    certain percentage of the time or works

    is a rough word but you get mind

    then you're comparing across but for

    programming language what's a placebo

    right so what we did was we took Orem we

    ripped out all the symbols and then I

    had my student mostly laughing rolling

    electronic dice and randomly choosing

    the symbols that go in the programming

    language so like they're awful they're

    absolutely horrific Liebeck does not

    repeat it's like left slash so you got

    like pound colon colon left brace for no

    reason and then like a axon and students

    just have to figure out that means

    repeat 10 times right so very hard to

    use and you might think cool we got a

    baseline but that seems like

    ridiculously unfair until we ran the

    study it turns out Perl and Java don't

    beat it really and when I say don't beat

    don't be in statistics usually means two

    things one is it it means it doesn't

    find a statistically significant

    difference between that in the group but

    also crucially it means that even if

    there is a difference and you can find

    it at larger sample sizes it means that

    it's inevitably pretty small the

    question that is why on earth would that

    be true and the hypothesis that we

    generally have is that the word choices

    suck yeah the word for doesn't mean

    anything to a novice and a lot of these

    little braces they have weird effects

    that like we wouldn't have expected like

    for example if you have the left brace

    instead of a programming language like

    usually it's match braces turns out and

    offices get the left brace wrong more

    often than the right one why I don't

    know but thank you and so often times in

    programming languages when we design

    quorum we use that statistical data and

    then we make directed decisions about

    which things we leave in and out so for

    example an if statement in quorum

    doesn't have matching parentheses

    because it's one of the most common

    errors and this has been shown in a ton

    of studies since but our data shudders

    as well in 2013

    Neil Brown's data in UK shows it Brent

    Becker's data shows it pulled Denny's

    data shows it I mean it's just the

    general consensus from a variety of

    programming language so that's true so

    we leave it out if the left race is more

    likely to be screwed up into something

    like a function than the right the left

    race goes away and it turns out that

    works pretty well or like if you have an

    equal sign like in programming languages

    often it's double equal to imply

    equality equivalent it

    yeah and it turns out that if you have a

    single equal it increases novices

    abilities to use that in the context of

    statement by 800% well yeah I know

    that's what I said when we saw the data

    too and in fact in the studies we then

    going to use assignment and equality

    test right next to each other to see if

    we can screw up a novice and it doesn't

    happen like novices can figure it out

    based on context the point is like you

    can use all that very strategically to

    define programming languages where

    you've really thought it through you've

    gathered the data you've thought it

    through done experiments you've

    replicated those experiments your survey

    data on the words and symbols and then

    you put that all in so that people can

    kind of use it a little bit more

    effective and then every release we do

    on quorum we work with outside scholars

    that are doing different studies

    sometimes to disprove this and sometimes

    they're right in addition we keep doing

    internal studies as well to make slow

    incremental changes so that hopefully in

    like 10 20 50 100 years programming

    languages are much more condensed in

    terms of how much garbage is it because

    right now there's a lot of garbage right

    I'm curious in the Placido study so I

    proposed to my wife by modifying the

    game minecraft which used Java so I've

    used that but I haven't used Perl I'm

    wondering if you found significant

    differences between lower level versus

    higher level languages compared to the

    placebo right so we wouldn't up that we

    didn't test that in that study but the

    reason the most perform badly is

    actually because they have a lot of

    syntax in common okay the reason where

    they tend to screw up is actually the

    same across where they're the same and

    in fact in languages that did slightly

    better than placebo like Python did

    slightly better it actually where Python

    does badly is the same places where it

    matches the syntax for Python and Perl

    and the reason is they use the same

    words and symbols right it just has a

    colon at the end instead of a brace and

    a match and stuff like that so the

    reason why Python does slightly better

    is actually accidental they still have a

    lot of the cruft

    from you know the 60s and 70s that

    didn't get rid of but they have a couple

    decisions that made a small difference

    and in fact in those original studies

    we're better than chance but it did

    pretty badly too in a lot of ways and we

    made a ton of changes since 2013 that

    are based on that evidence from that

    study and many others that's great

    yeah in my coding classes the kids had

    the opportunity to pick from several

    different languages and one of the

    things that I noticed is it was easier

    for kids to go from a syntax heavy

    language to a syntax light language like

    JavaScript to Ruby but going the other

    way around it was much more difficult

    because they were used to this like

    clear easy to understand click to right

    didn't have to worry about semicolons

    and then when you go to JavaScript it

    was like wait I have to add in all this

    other crud that I'm not used to so they

    were not able to do as well with that

    yeah I don't have evidence on transfer

    of learning in both directions but that

    might be true that there is different

    effects going either way that wouldn't

    surprise me it's interesting you

    mentioned Ruby too because a lot of what

    we found in those original studies as to

    how to make poram better we actually

    stole from Ruby because they had some

    parts of the language where they did

    really well like leaving off the

    parentheses and stuff and that did

    really well in those studies on the

    other hand interestingly because the

    whole field was non evidence base for

    such a long time most languages make

    certain mistakes that they probably

    don't know they're making Ruby's a good

    example ruby has dynamic typing so it's

    JavaScript and it turns out dynamic

    typing which is this idea that I don't

    tell you what the type is I don't force

    you to write it well it turns out that

    in languages like Ruby because they have

    dynamic typing that actually

    significantly lowers your productivity

    for people I think it's past years three

    of experience level in college at the

    beginning students that do have a

    negative impact and it's like really

    small but it goes away quickly and we

    even know why it happens the reason is

    because if you videotape programmers

    basically what you observe is that when

    it's dynamically typed if you want to

    call a function you got to know what to

    call it with the properties go with the

    past so what you observe is developers

    jump to the function try to figure out

    what to pass to it and then they jump

    back and guess what that takes time and

    therefore productivity lost and to

    hopefully clarify for people who aren't

    familiar with it so like one example is

    Khan Academy's JavaScript they use var

    is a substitute for any kind of type of

    data but instead of using something like

    var you might say int for integer or dub

    for double or say a string or whatever

    to clarify in advance I'm going to pass

    text or I'm going to pass an integer

    that has decimal points or it doesn't

    have decimal points etc yeah that's

    exactly right

    turns out that var type approach

    actually lowers prickles productivity a

    substantial amount that is absolutely

    fascinating I think actually now that

    effect of dynamic for second I think

    it's actually the most well studied and

    replicated in probably home computer

    science off the top of my head I want to

    say somewhere between ten and twenty

    randomized controlled trials on that

    topic now and they pretty consistently

    show the same effect under different

    context without code completion in and

    out of an IDE under this condition that

    condition is kind of funny like it used

    to be we used to debate that in computer

    science a lot I feel like that's a

    solved problem now I mean there's more

    to learn and there's probably contexts

    under which you know so I have to put

    mice and scientists head on for a minute

    and say there are caveats and there are

    but at the same time like if you want to

    say the other direction now you got to

    provide the data and show us the

    evidence so we can see what conditions

    you're testing I'm wondering going back

    to quorum can you describe so I was on

    the the webinar that you did a couple

    weeks ago so I'm wondering if you could

    describe in particular the positionality

    of the camera that was particularly

    fascinating to me how you had to think

    of when looking at an object how close

    or far away female you are from that

    object with the distances and then even

    possibly discuss how you sonna phi an

    environment or a landscape by like kind

    of panning through it and playing some

    of the different sounds to get a picture

    of exactly what is displayed on the

    screen

    yeah okay so just to provide anybody

    that's reading some context if they

    didn't hear that webinar and we'll have

    a link to that at some point I'll put

    that in the show notes whatever it goes

    up in any case so it's right context of

    people listening what we've invented is

    we've been trying to figure out a way

    that we can make it so that kids that

    have disabilities but even including

    blindness can create games that are

    highly visual in nature and there's a

    couple reasons why we want to do that I

    always have to say this first just

    because that sounds like why would you

    do that of course a blind child's not

    going to make a 3d game that doesn't

    make any sense but actually there's got

    the reasons and one is again in high

    school if a kid is in a class with 3d

    programming it's illegal to exclude them

    if it's required that they take the

    class you can't do that so that's number

    one but number two is well like I just

    got this hunch that if we can make that

    accessible we're gonna learn all sorts

    of weird neat interesting things but so

    far what

    built is we have an environment and

    we've tricked it to be able to be usable

    using technologies like screen readers

    which talk to you and give you sounds

    related to what's in an environment now

    how that works on a technical level is

    ridiculously complicated and unless you

    want me to I won't go into it but the

    short and skinny is that trying to make

    animation loops all this kind of stuff

    talk to screen readers is challenging it

    took us years to figure out how to do it

    in such a way that it would work

    seamlessly so what we have now is that

    we have like cameras on the screen that

    that's are in a 3d environment there's

    all sorts of 3d components like boxes or

    spheres or things like that so the

    question that I think you're getting at

    jared is like how does that work in

    terms of like what the user interface is

    so like you know if I have a camera and

    I'm looking at a 3d landscape well how

    do i how do i describe that and i think

    what we've learned so far in limited

    user testing and by trying it ourselves

    with screen readers because we haven't

    finished our beta tests yet with actual

    blind users or other disabilities is

    that so far as far as we can tell the

    most crucial thing that you want when

    you're talking about making 3d things

    accessible is actually not the gestalt

    like trying to describe the whole scene

    at once it's being able to know your

    context which is classically called -

    where am i problem in disabilities where

    it's sort of like where do I exist in

    this environment but in addition being

    able to interact with the things that

    you're working so I'll give you an

    example if I'm like building a 3d model

    of an ogre and I'm blind and I can't see

    it I might want to know whether the ogre

    which is just one model in a giant

    landscape has the focus and I might want

    to know that as a whole but once I'm on

    the model I might want to know things

    like how many parts does this ogre have

    at a high level it might be the arms and

    the legs and the head so I want to be

    able to query that and start to break it

    down and then I want to be able to

    navigate those things so that I can get

    information about what those are and

    then I can start to drill down

    mathematically or in some other way so

    now that's not always easy like if you

    had a castle and a head let's just say

    ten million triangles you need to

    basically figure out how to combine that

    down into higher-level items so that the

    and then allow drill downs so that you

    can manipulate individual pieces how

    exactly that ends up when the tool is

    done but at least the first version of

    that part is done by the summer will

    kind of be a user testing thing as we

    kind of try it with people but my

    suspicion is that we're gonna have lots

    of like drill down here and then go

    there query the camera or query where

    you are around you or what's around me

    right now or how many things are in

    front of me stuff like that is where I

    suspect the directional go as we kind of

    test it with people and they say Oh God

    why'd you do with that which is what I

    expect to happen the first time and you

    mentioned animation I'm curious how you

    would account for describing animation

    when there are multiple parts going on

    simultaneously

    is it like different pitches or textures

    and the voices or that's a great

    question okay so the first thing to know

    about animation with screenreader

    technologies is that making them work is

    like an insane challenge for reasons

    that have nothing to do with the user

    interface and I'll describe that very

    briefly just to gonna give people that

    sort of understanding notably if I have

    in an environment like Windows Windows

    uses something called the Windows

    message pump to get information back and

    forth to a computer program right and

    all this is is like you presses the Mike

    Mouse clinics and so but the Windows

    message pump when you're working inside

    of an animation library is tied to the

    frame rate in certain ways depending on

    how you pump it out so that for a second

    I get a key press it goes on a frame

    tada

    normally for animation good that's

    exactly how it should work but the

    problem is screen reading technologies

    don't work that way what they do is they

    get a key press and then they go through

    a dance with the application to try to

    make sure that it's querying for

    information a particular time so why

    Bing if I press like down on the

    keyboard and I'm in a text editor that

    might actually send back a few hundred

    on I mean hundred requests to the

    application the gather like where's the

    cursor what is it doing or there any

    annotations here or there spelling

    errors when I'm at or whatever it's

    doing right and so what we learned very

    quickly when trying to integrate an

    animation

    is that we have to do these like nasty

    equations behind the scenes to sort of

    force pumping in bizarre ways that don't

    limit the framerate too much although

    just a smidge and that's an unavoidable

    but also allow it to be accessible so

    that you can get information back and

    forth the screener without delays

    because if you had one request per

    screaming of perkele and you have a 30

    frames per second then one downer on the

    keyboard all of a sudden you got 30

    seconds until you've got your little

    animation so once we saw that we then

    realized that if you had multiple things

    animating at once you're not gonna be

    able to understand it all at the same

    time so what we then do is we have a

    system where you can provide information

    to the screen reader at strategic times

    and this is done in a couple ways

    one of which is that there are certain

    events that get fired as animations

    trigger that provide information to our

    screen reader architecture that then

    selectively chooses whether or not sends

    information to the system so that's

    number one a good example of that would

    be if you press a button you want it to

    click and you want the screen reader to

    know that and like that is technically

    an animation with lots of frames doing

    all sorts of stuff but the crucial

    information you need is the click not

    the fact that it's like you know

    flashing or whatever it's doing and same

    with 3d animation too like if I'm

    rotating something I'm not gonna provide

    screen reader information on every step

    of that rotation but I might give it

    only when it's done or maybe along the

    way or stuff like that and those have to

    be kind of tuned with user testing to

    make sure we kind of get it right and it

    feels good

    normally when we use it it's pretty easy

    to tell kind of when those should

    trigger but then the last thing is

    screen readers while they have good

    mechanisms for connecting into things

    like text editors and stuff they just

    don't connect it all up to 3d engines so

    there's some things that we have to

    present to the user that don't follow a

    standard Windows event infrastructure so

    for those we have this little thing that

    we call and notify and a notify is an

    event that doesn't match to an event but

    forces the screen reader to talk in

    appropriate ways at appropriate times an

    example of that might be when I'm using

    a camera one of the things we found

    pretty quickly is that if we try to

    using a screen reader blind

    I want to query it for all sorts of

    information like where am I

    what are my coordinates what

    what's in front of me what's in my

    frustum what's in my if i zoom out

    what's far you know stuff like that

    yeah so the way that we do that is we

    use a series of accessibility notifies

    they're not events really they give the

    screen reader context about a certain

    wearying condition and that tells us

    creator oh I need to find some

    information and then it goes from finds

    it and you're good to go so it's really

    tricky like it's not one thing right it

    involves a series of technologies but

    it's possible you just have to be kind

    of judicious and you always have to test

    it with people because it's really easy

    to get it wrong or make it just not feel

    good how do you recommend CS educators

    use quorum in the classroom somebody

    might be listening to this and go okay

    awesome I have an option for children

    who are blind or hard of sight but what

    about all the other kids would they be

    able to use it as well yeah so there's a

    couple things going on in the quarren

    project first is last year I think that

    the number of users on our website for

    forum was in the ballpark of 80,000

    something like that which is a lot more

    than students with blindness right like

    it's not just blind kids that use this

    stuff it's a lot of schools in general

    use it I don't always know the ways they

    use it they don't mail us what they're

    doing in the classroom and stuff but I

    would say the most common thing that I

    see people doing online on analytics and

    stuff is using it for computer science

    principles because we have a mapping of

    our curriculum to code orgs which they

    kindly give us permission to do except

    that all the tools and techniques for

    accessible I will say that in the future

    what I suspect people will use for is

    will probably add a block visualization

    mode on so if you're in a typical school

    and you're wanting to do whatever you

    feel like doing and you're used to that

    kind of thing

    you can just do whatever you want

    normally but it's also accessible so

    over time my suspicion is that quorum

    will follow all the same visual

    standards that some of those sort of fun

    kids tools do but at the same time will

    remain accessible so you can use it just

    for whatever the last thing I'll mention

    is teachers in the classroom that use

    quorum use it for a lot of different

    stuff that I wouldn't expect so like I

    know there's schools in Maryland that

    use it for Lego robot competitions

    there's a lot of schools that use it for

    game programming like two-year 3d games

    under all sorts

    there's people that use our digital

    signals processing libraries to write

    like little audio things there's people

    that just make audio games that like

    plant 3d sounds all over the environment

    and then run around with them and stuff

    there's things like that there's a group

    of people in mostly Wisconsin that use

    corn to connect to robotic telescopes

    and then they send back images from

    these telescopes and do analysis on them

    something like that so lots of stuff

    actually corns really powerful for doing

    all sorts of like even non-tv style of

    things

    I just don't know all the details about

    how everybody uses it but those are some

    examples of what people do right now so

    while you're talking about quorum and

    like navigating a 3d environment I'm

    curious have you looked into

    accessibility issues in relation to AR

    and VR so we've thought about it but we

    haven't done anything with it yet it

    turns out here's a way in our

    environment to connect arv our libraries

    up to our existing stack in fact it

    turns out to be relatively easy to do

    however we haven't like bought the

    devices to test with it and I don't know

    how the accessibility works on those

    devices like for example does an oculus

    have screen reader support

    I'm guessing though but we haven't

    looked into it so I think there's

    actually a lot of opportunities there or

    lots of people even if you can't see man

    the things you can do if you can turn

    your head and observe it's really

    powerful but I haven't done much I

    haven't done anything with that I think

    there's a real opportunity in orem or

    had yeah and I'm slowly learning

    Japanese and so like each weekend

    somebody comes over and she's a native

    Japanese speaker and we kind of like she

    helps me out with learning how to do

    that and one of the things she showed me

    I didn't realize in Google Translate you

    could actually pull up your camera with

    that and it'll translate an image like

    you hold up a piece of paper or

    something in Japanese and it'll on your

    screen make it into English

    oh I'm wondering if there are

    applications of that in relation to

    screen readers with an AR so if you

    potentially had on some kind of a

    headset that allowed you to basically

    audibly translate the visual environment

    as you're going through it I hadn't

    really thought about that but that makes

    complete sense I can imagine that being

    highly possible that sounds like you

    actually I think imagine that'd be

    really

    like maybe have a pair of glasses or

    something that has a little camera on

    them and as you're walking around as a

    blind person you can see and it says

    water closet if you're in Europe or

    right now bathroom if you're in Japan or

    something like that even though it might

    be in a different language that doesn't

    matter cuz it automatically translates

    so I'm wondering if you could wave your

    magic wand and instantly make CS in CS

    education more accessible what kind of

    changes would you make in terms of

    accessibility there's really two that

    immediately come to mind one is

    accessibility infrastructures really

    shouldn't be different across

    programming languages and across

    platforms and that particular issue

    sounds like it's minor because we just

    deal with this fact but the truth of the

    matter is the programming language Wars

    where every company makes its own

    language with its own platforms and own

    api's has caused serious damage to our

    discipline and it's like triple e so for

    accessibility because like you're going

    to build accessibility support in the

    Windows sorry it works

    nowhere else because the code is

    compatible and also with some

    applications like all this just accept

    well if I'm making a user interface in

    Java it's different than Swift that

    shouldn't be true either

    the languages should be more

    consolidated than that there should be

    like an API that's across languages but

    never mind that

    but for accessibility it's even worse

    because the accessibility

    infrastructures usually are kind of

    broken on each platform so like you go

    out to use Microsoft's UI 8 providers

    and it has just a million quirks because

    it doesn't work in certain ways under

    certain contacts like if you use

    multi-threading in Jaws it breaks but

    not with NVDA but if you go onto other

    platforms like Android or Windows

    I mean Apple you know is completely

    different like literally you have to

    throw your code away and start over and

    that's a huge problem because those

    accessibility infrastructures are

    massively different and they're broken

    in their own ways too so it's like you

    need almost like a team of 20 to really

    get accessibility support everywhere and

    most teams just aren't that big and

    that's just the accessibility part

    before you even touch the problem you're

    trying to work on so I wish that the

    companies would get it together and make

    a unified accessibility infrastructure

    for all platforms and all languages so

    that it could be more you

    find everywhere and you ride it once and

    everybody can back to it but that's

    tight dreamy I mean like the companies

    aren't doing that in any way right now

    and in addition the companies do then

    have a vested interest because like they

    may if Swift makes apples stuff and they

    make a custom language and they get

    schools to use it then you boxed into

    their platform right and so then

    developers are more likely to build

    stuff for their things that gets more

    apps and so there's a commercial

    interest personally I suspect long term

    the government will have to step in to

    sort of start to fix some of these

    issues but how to do that without being

    heavy handed and to make it actually

    reasonable and equitable is a tricky

    thing it's not it's not simple at all

    yeah I'm a Windows Android kind of user

    and taught Swift in my classes because

    we had a Mac lab and I love the language

    but I can't use it now because I no

    longer have a Mac you know it's

    unfortunate and that's just the platform

    but like it's a legitimate issue and if

    you have you know windows and you're

    doing accessibility well you can't do

    that immaculate I mean maybe there's a

    way to get around some of it we have to

    figure that I'm curious in your CS

    classes that you teach do you use quorum

    only as the language and platform or if

    use others how do you consider the

    various affordances and constraints when

    you're picking a language or a platform

    for your class oh that's a good question

    so I always actually feel uncomfortable

    pushing my own work on to my students

    yeah I wouldn't say that it's unethical

    like legally there's no mandate about it

    but it just feels like you know I want

    my students to feel able to explore I

    might be able to do kind of more

    interested in and that's okay

    but in my classes I usually teach like

    upper division stuff so they're

    project-based students kind of get to

    choose stuff so I almost always have a

    student team that's at least willing to

    try it and then you know not my mandate

    but my choice and so one some of the

    more popular ones are like building like

    games that are accessible and stuff like

    that but even then it's not always forum

    so I I had a few years ago I had a team

    try to make world of warcraft accessible

    am i that is really hard they had like I

    don't know 50 accounts banned by

    Blizzard I'm trying to automate things

    to make it accessible they were trying

    to cheat right right right

    blizzard didn't I see it that way

    because they in the bone all

    time that light will make like a

    role-playing game trying to make it

    accessible and they usually use quorum

    there because there's libraries built in

    to make it accessible so it's a lot here

    then trying to do it in unity where it

    would be a nightmare trying to make it

    accessible my own personal balance which

    other professors might have their own

    own way is I never mandate it for

    students I let them choose whatever they

    want but oftentimes since students know

    that I do this kind of stuff I do work

    with some of them on quorum stuff and

    that can sometimes lead some pretty fun

    projects yeah I know it's a little weird

    kind of asking people to use something

    that you've created like having taught

    classes or I'm like hey I wrote this

    publication on this thing but here are

    some other publications sort of related

    to this topic again it's not like it's

    unethical it's fine like there's nothing

    wrong with it it's a legit product that

    you can use and stuff right but it also

    feels like you know am I just too close

    to it am i pushing something you know

    and maybe that's maybe that's wrong

    maybe it's fine and I shouldn't even

    think about it but I always second-guess

    myself

    maybe it's an imposter syndrome I don't

    know maybe maybe I should but I like to

    let students choose yeah I agree with

    that sentiment having had some

    professors that would push their books

    on you and then you never end up using

    it in your class yeah exactly right

    stuff like that it just feels like it's

    not necessarily like malicious it's just

    you know thankfully those were

    professors outside of my major they're

    just like some of the general Studies

    classes so I'm curious working in higher

    education and research in particular is

    a pretty demanding job in terms of like

    some of the pressures in terms of output

    for publications presentations other

    works and whatnot so how do you kind of

    take care of yourself or try and stave

    off some of the burnout that comes with

    jobs like these oh that's a good

    question I don't need ever asking with

    that before you're right it is

    challenging is that kadai Mia you know

    there's always these phrases like

    publish or perish and they are true you

    know if you don't publish you don't get

    to keep your job at least until you get

    tenure and then there's weirdly there's

    a lot of protection but you know for me

    I would actually say a better balance

    after I got 10 years and before and

    that's probably pretty typical but you

    know in my own case I am careful about

    my health number one I exercise

    regularly i watch my calories and trying

    to keep myself physically fit so that

    that's important to me and I I know

    have direct evidence that that helps but

    I feel like it helps me mm-hmm that's

    one thing but then number two I also

    feel like academics should especially

    once they have tenure when you're still

    trying to keep your job you kind of

    don't have as much choice you know

    because you're trying to prove yourself

    but once you do I also feel like

    academics should take a step back and

    realize that a lot of the academic

    metrics that universities use are stupid

    like for example if you wrote an

    academic paper and you said I think we

    should hire and fire academics based

    purely on the number of papers that they

    publish you would get a rejection letter

    because that's stupid but in tenure

    promotion guidelines

    that's happen what we actually do in

    practice right there's a lot of

    universities they don't even read the

    papers but their academics put in they

    just say well you published six in this

    time good job you get to keep your job

    that just makes absolutely no sense it's

    totally ludicrous and so I think part of

    my own sanity is I've come to the

    conclusion that I'm some of it I'm just

    going to ignore and I will publish

    enough that the admins will keep off my

    back and stuff like that and I like to

    publish anyway so I guess there's that

    advantage but like I'm also that going

    to spend a lot of time doing what I love

    and to me that's programming and so I

    spend a lot of my days like thinking of

    how am I gonna hack together this thing

    to make 3d work for a blind kid my

    colleagues are supported but if they

    didn't like it I would just I'm fine

    with that

    then give me a bad evaluation

    fortunately I'm lucky and that we've

    gotten awards and you know all sorts of

    stuff and you know people are sort of

    okay but at the same time like you know

    some of it too is you have to kind of be

    willing to speak out a little and remind

    your colleagues that like some of these

    academic metrics are really not

    evidence-based they're just kind of like

    you know we chose paper counts because

    the admins can count them easily in

    academia it's funny they always talk

    about teaching research and service

    right and it's funny with research we

    have these absurd metrics for teaching

    we often use teaching evaluations and

    mine are very high but there's lots of

    issues with them having gender bias that

    I mean they're just not really good and

    that's service like we don't take any

    metrics and so therefore somehow people

    just ignore service and like it's just

    not taken seriously I'm like is

    really weird that we evaluate academia

    this way and it should probably change

    if we want to keep our own sanity as one

    academic I only have so much control in

    such things anyway so for me it's like

    health stuff and then kind of ignoring

    some of them that makes sense

    and what do you wish there is more

    research on that could inform your own

    practices well I mean this is my own

    bias talking but I really want like

    impact of programs I mean look this

    affects more than 300 billion dollars a

    year in wages in the US alone it's a

    huge deal for students it's a

    monstrosity

    and then in k-12 computer science think

    about this right now we've got these

    things like computer science principle

    right which i think is generally

    actually a pretty well thought through

    standard however because there's no

    language attached to it and computer

    scientists cannot seem to agree that for

    is a bad word choice having 34 years

    we're in this weird situation where you

    might go to one school you're learning a

    notation about programming the school

    write over use this one different and

    the problem is that's absurd I'm like in

    math they don't do that if you have an

    integral we use the integral symbol I

    mean there are alternative notations if

    you want to use the plus symbol in math

    everybody agrees but the freaking plus

    symbol means science we don't not even

    the Plus do people agree on what that

    means in C++ it means something

    different in Java and that's stupid we

    need to get rid of that so like you know

    in my magic wand waving way but I would

    want I want a lot more human factors

    research on that because I think if we

    can sort a lot of that stuff out and

    rule out a lot of these bad decisions

    that language designers may have made we

    can start to construct products that are

    based on sound evidence and then get

    them adopted slowly over time

    get rid of some of these languages like

    you know like maybe like Perl that

    aren't very well constructed in terms of

    their human factors impact slowly let

    them die and let languages that are

    using evidence that are slowly making

    improvements over time

    let those slowly get adopted there's one

    reason why I'm quorum we focus on

    adoption as well that's why when we

    first started we had four kids and then

    now it's like more than 80,000 I don't

    want it to grow too fast and I don't

    want it to be

    only option out there because arm still

    has a lot of areas where we need more

    data to say the least especially right

    now we're redoing our compiler error

    messages because there's good strong dad

    we need to but in any case I think that

    would probably make a big difference

    because this language horse problem has

    been systemic huge problems for our

    discipline and we don't really talk

    about it that much just kind of except

    that it's this big nasty mess yeah and

    in the k-12 in particular in the younger

    grades it is based more on how shiny it

    looks and the marketing that's behind it

    rather than the evidence that supports

    the learning hopefully that's involved

    with whatever platform is being used or

    language yes right like I mean you know

    to a certain extent we all want students

    to be engaged and so like I'm all for

    like making little animations and doing

    this kind of stuff that's all good I get

    it I mean like I don't want to just have

    kids use a console that's pretty damn

    boring but on the other hand too like

    keynote the data does matter and even if

    like that stuff is shiny you know the

    fact that there's a bazillion different

    products and the kids are all learning

    different things you think that I don't

    know in my magical dream world which you

    know maybe in like 10 lifetimes from now

    what I would really like to see is that

    there's one standardised syntax for

    languages that is like a bare minimum

    and then any language that out there and

    this super evidence-based there's hard

    core data people have tried to refute

    the evidence countless times a lot a lot

    and then any language that exists they

    can innovate on top of that standard but

    the that standard for basic things like

    looping like everybody uses repeat ten

    times because it just makes damn common

    sense right and then the standard

    libraries are all built on the common

    part not the experimental part thereby

    making it so we can get to closer

    standards over time over like a 50-year

    century so that maybe we can get rid of

    some of these problems before the next

    three or four generations come by cuz

    like the promise is they don't do

    anything kids three generations from now

    are gonna have the same damn problem for

    the same reason it's just not right it's

    gonna screw up generations of kids if

    you don't fix it so I don't know maybe

    I'm being pedantic but that's five you

    know I like it I mean in general like

    multi-perspective all approaches with

    most things in life but we need to have

    those perspectives built off of some

    kind of valid research that supports the

    claims that are typically marketed

    with it yeah exactly well and in other

    disciplines too to my brain it feels

    like there's sort of a core and then

    there's an experimental part where we

    tweak like for example generally in

    schools teachers aren't gonna change the

    meaning of the plus symbol right because

    like that's just part of our standard we

    just thought we accept that that symbol

    means one thing or at least you know at

    least didn't eight the twelve setting if

    you want to make some innovative

    mathematics you can use your own crazy

    symbols but if that becomes standard

    where everybody's learning about it then

    the textbook is trying to get some kind

    of normal notation for it we try to kind

    of boil it down so everybody's doing we

    need that computer science to like if

    we're talking about looping we should

    try to let use the same symbols and

    terminology so that any kid no matter

    what school they're at and start to like

    you know communicate with each other it

    just feels like it'll help society you

    know like I mean think about this all

    right any evidence of this but I wonder

    if you were to take the number of hours

    work from building the standard

    libraries that do the same thing in

    every programming language that exists

    lists a race games all this good stuff

    how much money have we blown down the

    drain by rebuilding the same ceremony I

    mean it's gotta be one of the most

    massive duplications of effort in human

    history to do that and like we don't

    talk about that in computer science

    we're just like tripod foam that's great

    whatever language you want to want to

    pick in the irony of that in relation to

    CS in particular is not loss of me in

    the fact that it's like oh why don't I

    just find somebody else's code that will

    do this automate this thing that I make

    my life a whole lot easier

    exactly I've been there - absolutely

    it's just the nature of the beast cuz we

    still live in the society I mean we

    can't change it immediate looking at

    your google scholar profile there's a

    lot of publications on there and in your

    CV I'm curious if there's in particular

    that you really wish more people read or

    kind of understood the study on that

    placebo study that's probably the one

    that's the most one done I don't know if

    I have one in particular where I'm like

    this is the one thou shalt know I feel

    like the the broader theme of the work

    that we've been doing is to try to get

    at these broader like language design

    type issues and I wish that people had

    not necessarily one of the publications

    but what I wish

    you know these text books for like

    principles of programming

    and we talk about like they often

    they're like they make a crap ton of

    claims like this thing is better than

    nothing and stuff like that I wish those

    would be rewritten and like thrown in

    the trash and then we would make a new

    generation of those text books where

    instead of just talking about like this

    is what scoping is and then also here's

    the impact on people that are working

    when they use these things here's the

    impact on students like for example when

    students leave a class on principles of

    programming shouldn't they actually know

    that differences in programming

    languages in fact people very strongly I

    mean like that isn't that a fundamental

    principle of programming languages all

    we really do is just say here's a

    smattering of languages that exist and

    like you can try them all and then

    that's it somehow they're not better or

    worse they're just different

    right which is crap because it's not

    true at all or like this other common

    phrase you hear in classes like that is

    you just have to choose the right tool

    for the job

    right and like that's just such a

    ridiculous nonsensical statement yes if

    you're on Apple you only have so many

    choices for what languages you use but

    that doesn't make it the right tool it

    means the only one that you can use

    because of either license agreements or

    something else it's not quite the same

    so I mean to me which these principles

    of programming books would be rewritten

    completely and that inside of them would

    contain the smattering of evidence

    across a lot of scholars on what we know

    now well so so that when students leave

    school they actually have some

    background in like what's going on in

    the the myriad of research groups that

    are finding these particular effects in

    the last 10 years or so yeah so even

    though the last few years have been

    working full-time in CS education like

    I've mentioned outside of this interview

    that all my degrees are music education

    so I'm technically coming from from an

    outside perspective and in music

    education and other areas within

    education there's a lot more critical

    discussion on the discourse that is

    going on in those fields and I think

    that is something that's kind of like

    lacking in CS education in particular is

    we need to have more conversations about

    some of the things that you're talking

    about that no these are not equally as

    good they're better or worse in certain

    ways and others and we need to kind of

    talk about those things I think so too I

    don't know what the actual debates are

    music education can I ask what are they

    yeah so the biggest debate right now is

    the debate about whether large ensembles

    anachronistic so yes they provide some

    awesome opportunities for people to

    create music but what opportunities are

    there for people to continue that music

    making once they finish their k12 tenure

    oh man isn't that the truth that's

    interesting so that is like one of the

    biggest things and there's a huge divide

    among the field where people are like no

    I'm a hundred percent for a large

    ensembles and other people are like no

    we need to just burn that down and just

    start with like popular music making

    today's like deejaying and rapping or

    whatever and do only that and then

    there's like people who are like well

    they both have their places and so

    that's a large one that's pretty

    interesting especially given that you

    and your experience might be different

    but when I was doing my music singular

    degree in just a lot a lot of what you

    do learn it does feel really old right

    there's a thing called counterpoint and

    it has to do with like when notes are

    sorted together and how they're kind of

    linked up and stuff like that what that

    ends up sounding but when you if you

    take a class an actual counterpoint the

    actual theories are from the scanning

    Fuchs if I recall if Fuchs Ian

    counterpoint the cotton is from like the

    haven't been updated you know 400 500

    years right 400 I forget exactly and so

    it is funny that the music education did

    feel like it needed more evidence to

    sort of update itself as well that's a

    whole other lifetime of scholarly work I

    suppose I had run into a lot of conflict

    through my degree in that the last

    course I ended up taking was a Russian

    music theory course for my PhD and it

    had absolutely nothing to do with

    anything I was interested in scholarship

    wise it had no sway on anything I was

    doing but I had to do it because it was

    a music degree technically in a music

    school and the electronic studio courses

    that I took would not count as music

    theory so I took a class that was about

    creating music software through max/msp

    which is a graphical programming

    language and that would not count as

    music theory and then even though it was

    a music theory course housed on the

    music theory department so like they

    even have that divide where oh if it's

    anything electronic sorry that doesn't

    count as real music theory doing air

    quotes there it you need to do this

    other thing that is

    completely irrelevant to whatever it is

    you're doing that is crazy because you

    know so I used to program and I don't

    even know if it's still around anywhere

    a programming language called super

    collider yeah yeah yeah so it was it's

    like kind of a text-based version of

    Maximus P I think max is a lot easier to

    use frankly but nonetheless different

    letters what we were kind of messing

    with and it is very theoretical right

    like it's you need to know some math you

    need to know how all these pieces link

    together there's these weird Jen's and

    make up I mean like that they wouldn't

    count that as serious just crazy yeah I

    felt the same way that's why I kept

    pushing off that course because it was

    like maybe if I petition it they'll

    eventually count these as an actual

    music theory course but they wouldn't so

    that was the last thing I had to do that

    is so funny it does just feel like that

    it's like that's legitimately

    theoretical in nature but clearly not I

    guess do you have any questions for

    myself or to the field at large nothing

    immediately comes to mind I will say

    though for people they can that I should

    try to make yourself accessible it's

    good for your students there's a lot of

    people that have disabilities challenges

    or whatever and it does people good it's

    a good way to not only feel good but

    like it does good it's a good thing it's

    for the betterment of society so if you

    can give it a shot where might people go

    to connect with you and the

    organizations you work with they can

    email me at my normal University email

    which is online if you're interested in

    corner stuff there's a quorum mailing

    list and that's linked on the quorum

    website which is at quorum language comm

    if you're interested in accessibility

    the name of our grants that we do we

    call it access yes for all and that's

    Google Bull for sure but I can get you I

    can toss you a link before for some of

    these things so people can get up on the

    site or whatever access you use for all

    has a number of various failing lists

    for different kinds of interests that

    people have their relatively low traffic

    on purpose so we don't bombard people if

    you want to get even more involved we

    have regular meetings for developers

    related to accessibility for people that

    part of want more or just want to learn

    more like for example I think you

    actually went to our most recent webinar

    which was on 3d technologies that are

    yeah and I think I became aware of that

    webinar because of being on the

    newsletter in the mailing list rather oh

    and there's others too that we have on

    that too I think we had Stephanie Moody

    does also work in Texas on accessibility

    and I think we had someone from Google

    that was doing some cool accessibility

    stuff lots of people like that and with

    that that concludes this week's episode

    of the cska podcast I hope you enjoyed

    listening to this episode as much as I

    enjoyed interviewing Steph ik just as a

    friendly reminder I'd really appreciate

    it if you would share this with somebody

    else or simply write a review on

    whatever platform you're listening to

    this on up so that people find it

    speaking of finding stuff friendly

    reminder you can find me show notes in

    the description that you listen to this

    on or by visiting jared O'Leary com stay

    tuned next week for another unpacking

    scholarship episode and two weeks from

    now you will get another interview until

    then I hope you all have a wonderful

    week

Guest Bio

Andreas Stefik is an associate professor of computer science at the University of Nevada, Las Vegas. For the last decade, he has been creating technologies that make it easier for people, including those with disabilities, to write computer software. He helped establish the first national educational infrastructure for blind or visually impaired students to learn computer science and invented the first evidence-based programming language, Quorum. The design of Quorum is created from data derived through methodologies similar to those used in the medical community. He has been a principal investigator on 5 NSF-funded grants, many of which related to accessible graphics and computer science education. Finally, he was honored with the 2016 White House Champions of Change award and the Expanding CS Opportunities award from Code.org and the Computer Science Teachers Association.

Resources/Links Relevant to This Episode

More Content

Featured
Unpacking Scholarship
The intersection of videogames, music, and computer science education
Unpacking Scholarship
Unpacking Scholarship
Professional Development
Project-based Learning with Scratch
Professional Development
Professional Development
Professional Development
Facilitating Multiple Programming Languages in One Space
Professional Development
Professional Development
Interview
Recommendations for Preventing Burnout in Education (Part 4)
Interview
Interview
Unpacking Scholarship
Depression, Suicide, and Computer Science Education (Replay)
Unpacking Scholarship
Unpacking Scholarship
Professional Development
The Apprenticeship of Observation and Computer Science Education
Professional Development
Professional Development
Unpacking Scholarship
The Making of [an Educator] Athlete
Unpacking Scholarship
Unpacking Scholarship
Other
Thanks for Listening and an Update on the Future of This Podcast
Other
Other
Professional Development
How Do You Encourage Critical Thinking and Dialogue?
Professional Development
Professional Development
Unpacking Scholarship
Scratch for Sports: Athletic Drills as a Platform for Experiencing, Understanding, and Developing AI-Driven Apps
Unpacking Scholarship
Unpacking Scholarship
Featured
Scholarly Article
Wind River Elementary Computer Science Collaborative: Connecting Computer Science and Indigenous Identities and Knowledges on the Wind River Reservation
Scholarly Article
Scholarly Article
Practitioner Article
Preventing Burnout
Practitioner Article
Practitioner Article
Scholarly Article
Computer Science && Popular Music Education
Scholarly Article
Scholarly Article
Scholarly Article
Fostering Intersectional Identities through Rhizomatic Learning
Scholarly Article
Scholarly Article
Practitioner Article
Trans Voices Speak: Suggestions from Trans Educators about Working with Trans Students
Practitioner Article
Practitioner Article
Scholarly Article
Measuring the Effect of Continuous Professional Development on Elementary Teachers’ Self-efficacy to Teach Coding and Computational Thinking
Scholarly Article
Scholarly Article
Book Review
Music, Technology, and Education: Critical Perspectives, edited by Andrew King and Evangelos Himonides [Book Review]
Book Review
Book Review
Chapter
Applications of Affinity Space Characteristics in Music Education
Chapter
Chapter
Chapter
Hip Hot Cross Buns
Chapter
Chapter
Scholarly Article
Intersections of Popular Musicianship and Computer Science Practices
Scholarly Article
Scholarly Article
InterviewJared O'LearyAccessibility, Quorum, Evidence-based programming, (dis)abilityComment