Eric Mazur sits down with Dr. Jarek Janio, ESL assistant professor and faculty coordinator at Santa Ana College and founder of the Student Learning Outcomes (SLO) Symposium, to unpack the case for treating learning as observable competency rather than inferring it from proxies like grades, attendance, or engagement. Eric and Jarek explore the difference between performance-based and product-based assessment, how ungraded and multiple-measures approaches can reduce bias in evaluation, and what it might mean to build a classroom — and a student-teacher relationship — around a shared, visible record of ability.

‍

Eric: Thank you for joining us today for this episode on Redefining Achievement Through Observable Learning in the Social Learning Amplified podcast series. I'm your host, Eric Mazur, and our guest today is Doctor Jarek Janio. Jarek is an ESL assistant professor and faculty coordinator at Santa Ana College, where he leads critical initiatives in workforce education and assessment. He's a prominent voice in the national conversation on pedagogy, as the founder of the Student Learning Outcomes or SLO Symposium. His work challenges our traditional reliance on indirect measures like grades and engagement in favor of defining learning as an observable skill and competency. He joined us today to discuss his forthcoming book from Routledge on assessment of student learning outcomes and observable competencies. I was delighted to contribute a vignette to this book, specifically on how peer instruction helps make student thinking visible. Jarek, thank you so much for joining me today.

Jarek: My pleasure. Thank you very much for having me, Jared.

Eric: To dive right in, the core of your work is, I'll call it, the observable behavior imperative. You argue that learning only matters when it shows up in what students actually do. Can you explain why you feel it is so dangerous for educators to rely on proxies like attendance or grade point average, rather than direct evidence of competence?

Jarek: When I started studying student learning outcomes as a phenomenon that's required by accreditors, and that's what stirred up my thinking, my work on student learning outcomes or SLOs. As far as I remember, what drew my attention to the concept is the fact that we really do not have an operational definition of what learning means, and over the years I spoke with a number of faculty. The student learning is not really something that we clearly identified, and therefore it's really very difficult for us to attest to actually evaluate even student learning. What I mean by that is, perhaps I could use grading proxies that you just mentioned. As an example, if a student leaves a course with a grade of B+ respectable grade, right? Under any circumstances. Here's the question, do students read better or write better? Do they know theory better, or do they apply it in practice better? The grade doesn't really tell you this. As we know, there are a lot, a lot, a lot of things that come into the issuance of a grade. There is the question of punctuality of the student, timely submission of the required work, which really do not have much to do with learning.

If a student is late, remember 5 p.m. on Thursday, you are supposed to turn in your paper and you're late. 15 minutes. Sorry, but that's 10% of your grade. What does this have to do with the content of the course, or with any specific capacity that student has developed as a result of instruction? So that was my driving question. What is it that students can do as a result of a given course of a given program, or even attending a specific institution? This question is not readily answered. That's where I started doing the research. And after having listened to a number of cognitive theories and learning sciences, I arrived at the understanding or realization, perhaps, that learning must be something that we can all attest to, that we can actually observe. It can be anything that's happening inside our minds. We can't just look at the student's face and, you know, observe this twinkle in their eyes that, oh, yes, they got it. I mean, I'm such a good professor. They actually understand what I'm talking about. See, I can't tell by looking at anyone's face if they understand anything, unless I ask them to reciprocate. Unless I ask them to do something in return. And in the process of them doing it, I can evaluate and I can attest to the fact that they can do something as a result of our interaction.

Eric: Very interesting. And I actually see a lot of parallels also with backward design, which says we should start from the outcomes and work towards the front rather than content, and then think as assessment of an afterthought. And, and maybe what people in backward design call their list of action verbs are sort of related by your observational evidence. I'd love to come to that. But first, let's go back for a moment to, you know, these proxies, grades and so on which are persisting. Right now, at my own institution, we have a discussion, grades should be lowered. And I mean, it's like going back to the early 20th century. So you've mentioned that these practices persist because of administrative convenience or, you know, historical inertia. And we have, in fact, been using these symbols, you know, like grades for over a century. I mean, I completely buy into what you propose, but it's the biggest hurdle to change the system itself. Or is it our own psychological comfort? And when I say our, I don't mean you and me. I mean of the system as a whole.

Jarek: I'm very happy you phrase the question that way. Let me just perhaps start with the issue of the system, for example. You are absolutely right. Many times the discussions go into this rabbit hole, if I can call it off system. Our psychological orientation culture of the institution faculty by those are sort of very well meaning well sounding notions that we have to struggle with. And we completely understand it, that there is an institutional culture, that there is institutional accountability, things of that nature is just that. Again, those are not operationalized definitions. What's happening in our institutions, they are not broken down to any type of one specific or a series of observable phenomena that must be taking place in order for institutions to behave one way or the other. So if I were to address that, the most important phenomena that really impacts how institutions of higher learning behave is how they are funded. That's their job, is to make sure that their institutions are funded. So to that end, we do have the attendance, we have persistence. We have graduation rates. We have diploma rates, the diploma attainment rates. We even have employment surveys. Those things, as we report them through our reporting channel and through our institution effectiveness, the activities get funded at some point. 

What that means, however, is that the aspect of learning as we understand it now, which is again a lack of the definition of learning, results in the fact that we pay attention to how well institutions generate funding, rather than what it is that students learn in the process of taking our courses and attending the institution. And I believe that we can have those conversations about the culture of an institution and administrative purview and faculty purview and all those things that, again, seem so, so lofty and so important for us to tackle. But unless and until funding is addressed, there's just nothing that we are going to do. There aren't any administrators in their right mind who are going to be spending time talking about learning when funding is out the window in order to generate a formula for funding that would be reimbursable, right? That's going to generate a portion, meaning we need to have a clear definition of what learning is, because that's something that I, as an impartial observer, can attest to.
And I suppose that all of us as faculty, as teachers, as people who educate others, who take responsibility for making sure that our students learn well, it just makes perfect sense for us that we know what it is that our students getting out of our classrooms, what our students are getting out of our classrooms as a result of our instruction. And for that, there are a number of steps that need to be taking place, starting with instruction, then designing assignments, assessment, then generating the data. That's going to tell us what it is that students are learning and how we need to mitigate the environment around that learning, so that it results in specific skills and competencies that, again, can be observed rather than just looking at the student saying, oh, they are very much engaged, they are very motivated. Oh, they have grit or motivation. Obviously it's all there. Well, is it obvious? Again, just by looking at their faces we cannot tell. So with that, the cognitive science that attempts to inform the discussion about what's happening in our brains may very well be applicable. It's just that it's not really quite practical when I think about it in the classroom.

Eric: Regarding grading and grades and easy metrics, I mean, we tend to see such a focus on easy, easily collected quantitative metrics. I've been arguing these past, I think probably 12 or 13 years. Easy metric, probably a useless metric. And talking about assessment in your book, you categorize assessment in sort of two main buckets: performances that occur in real time and products that are durable artifacts. Why is it so important for an instructor to distinguish between these two when, you know, designing a course?

Jarek: So first of all, if I ask a question, does this conversation between us would probably could be used as some type of an assessment of what it is that I know about student learning outcomes by observing my behavior, how I articulate my thoughts about it. However, a book that you mentioned would be a product of that behavior. So there we go. That's why those are the two types of assessment that we have available for us in the classroom. We can either expect a live event or even if it's recording, let's just say it's going to be observable behavior or a product of that behavior. And the product could be an artifact that we can handle with our hands, I suppose, if I were to define it. But the reason this is important is because those are the two phenomena that can be observed that I could use for evaluation purposes, so that I can look at how students behave when they are talking about the concept, studies, and what it is that's expected of them as a result of instruction. I can look at them at the recording of their behavior or them on the stage speaking to everyone. The part next to it would be this artifact. What it is that they are turning in to again attest to the fact that they have acquired skill and competency. I don't want my students to be focused on course completion. I want them to focus on learning. And I think that's what all of us collectively have in mind.

Eric: It's kind of very right. And in education learning is not front and center. And at the same time, you know, as I listen to you here, I'm thinking this is like a huge mountain to move because it's not within the control of a single instructor. I mean, you have to comply with whatever the assessment policy is of the institution of the accreditation agency. When I started doing this, my goal was really to take the internal, invisible process of understanding and turn it into a visible social performance. In your framework, would you say that the discourse in a structured class, you know, the questions and explanations students and traded with order is a primary performance that we've been ignoring for too long.

Jarek: Ultimately, if students will have to demonstrate their skills and competencies anywhere they go and classroom is probably the best place where they can actually practice demonstrations of those skills and competencies. If I could just go back for just a second to the underlying philosophy or underlying guideline, it really needs to be based on what it is that students demonstrate in this period of the moment. So they are asked to react to a certain prompt in a certain way. And that's what we can attest.

As a matter of fact, the skill demonstration, the competency demonstration is really everything and anything that we have. It is always disposal. So it's permeating all our communications. So again, defining learning is really not going to be that hard. What pushes back at it I think is the issue of, again, cognitive science that I feel emotionally or intellectually rewarded by a certain type of interaction. Therefore, my assumption is that the other party, namely my students, are being rewarded by that interaction just as much. How do we know we simply don't? So once we figure out what that definition of learning is, and again, that is based on what it is that students can demonstrate so that we can observe it, we will just certainly will just not be happy when it comes to the performance versus artifact.

Again, the issue here is that whatever it is that I produce in terms of my behavior is up to a public scrutiny. We don't even know what students are getting out of this. All this work that went into the design, the development, the production of the text, and then all the work that faculty are doing on the weekends, my weekends are gone. I start on Friday night and I'm not done until Sunday, and I'm just doing, you know, all this oldest grading. And at the end of the day, we do need to have a discussion. To what end? How is that learning going to be informing the discussion of what's been even covered in the class? With that, students need to have an opportunity to observe how others interpret reality. Every single response to a prompt is different. If the only way for me to compare my point of view is between me and my professor, well, then that's just not enough. Because once I leave the classroom, there is the whole world out there, not just me and myself versus my favorite person in an organization. It doesn't work that way. So having this not articulated enough, none of it is defined, none of it is specified. And worse yet, nobody really knows what needs to happen as a result. We know that standardized tests are not something that's going to do it for us, not even in math.

Eric: You have a fascinating section in your book on reframing soft skills. You already mentioned one of one of them earlier, like grit or mindset and teamwork as observable competencies. I would actually prefer calling them durable skills. I mean, so many people have a negative perception of the soft skills, right? They're actually the things that are left after we forget all the content in our courses. But to come to my question, how do we move these durable skills from intangible traits to behaviors that we can actually document and judge fairly.

Jarek: We simply specify what they are. So the notion of soft skill again lends itself to a number of interpretations. Ask different people about what soft skills are, and you're going to get different types of answers. And that's probably the problem that they are not operationalized. They are not specific enough for us to make a difference in the classroom.

Jarek: It seems like they appear when it fits the conversation, when it fits the context, but they disappear once we try to identify them and specify what they are. Again, to me, everything and anything that we do is a matter of skill, whether that's a soft skill or hard skill. Well, you be the judge and tell me where the border is. Everything that we do is observable behavior. I have to speak English well enough to be understood. I have to know what I'm talking about, to be understood. I have to be engaging enough to be understood. So those are all those things that can be attested to, can be evaluated, can be observed.

Eric: I want to come back to fairness in grading too, because that resonates with me deeply. I've moved to ungraded this year, and I'm really excited by it. I've realized that the traditional grading approach is often penalizing students for their sort of internal states or backgrounds rather than actual growth. Does your system of, I think you call it multiple measures offer a way to grade fairly without the bias of impressions.

Jarek: That's right. If you were to again focus on what it is that students do or used to stand, you focus on their behavior, then that's what you're going to get in return. You're going to get the behavior. And behavior is shaped as a result of the environment in which humans behave. So when it comes to grading, that's as far away from it as you can possibly imagine.

Jarek: It's not only very subjective, but it's also very, very, very murky to begin with. Again, different faculty members, different professors, different teachers have different understanding of what it means to meet the course requirements to the students, show appreciation or they show understanding, or we know that they know it. It's just not a serious way to evaluate anything when you think about it.

Eric: So two final quick questions when a faculty member finishes your book, what is the one. How shall I call it a moment or shift in perspective that you hope stays with them for the rest of their career, or maybe even changes their trajectory?

Jarek: From my perspective, my book would be a part of the environment in which the faculty member orients himself or herself themselves. You know, the idea is that there are certain notions we need to convey as faculty. And I think that taking responsibility for assessment of our student learning would be a permanent focal point of that, of those discussions. So that's what I would like to certainly make sure that it happens. And I think I articulate the area in the book on numerous occasions from different perspectives. Same thing with our students. Our students will demonstrate their learning from all kinds of perspectives. That's where multiple measures come in. So many disciplines in one activity, we can certainly do it if you put our minds to it.

‍Eric: Well, I surely hope that will happen. Your mantra is “teaching arranges conditions, assessment records behavior.” If we truly left grading, would the student teacher relationship become less about authority and more about a shared record of ability?

Jarek: Yes, and that would be the beginning of the discussion. It's not that teacher-student relationships are not important. But if you were to transfer that to other disciplines, imagine what type of a conversation. Do you have with your doctor, with your lawyer, with your plumber, with your whoever? It's not even until we get to the academia where all of a sudden we believe that this, this, this developing, this relationship with students is so important. Let's say that it is important, but the bottom line is that it needs to result in learning. We can't just develop a relationship for the sake of having a relationship, and feel good about having a relationship with someone. Whatever this relationship entails, students need to learn, just like our lawyers need to help us in our, you know, lawyer things. And doctors need to help us with our doctor things. And architects have to have a job in our lives and veterinarians do. Why not teachers?

‍Eric: Absolutely. It's been a real pleasure. And I want to thank you for pushing us to, how shall I say, to make the invisible visible. I would like to conclude by thanking our audience for listening and inviting everyone to return for our next episode. On behalf of all our listeners, Jarek, thank you again for joining us today.‍

Jarek: Pleasure. Thank you very much for having me.

‍Eric: You can find Jarek’s forthcoming book on assessment of student learning outcomes through Routledge and other major bookstores. To find our Social Learning Amplified podcast and more, go to www.perusal.com/amplified all one word and please subscribe to find out about upcoming episodes. I hope to welcome you back on a future episode.