Assessing educational innovation: learning the hard way

Leave a comment

Screen Shot 2014-12-19 at 7.17.31 AM

My master data table

One of the first tasks I was given when I started in my current position in July 2012 was to be the lead faculty for a non-majors introductory biology course. That course is one of the workhorses of the university as it is part of the GE curriculum, and it is basically the one and only science class most non-science majors will take. Run in several sections in multiple campuses and online, it was a formidable task to tackle. When I took over, the course had been in place with the same textbook for many years, and especially in the online version, plagiarism was rampant.

Over the next months I explored options for a new textbook, mainly looking for something fresh and attractive, with plenty of ready-made supplementary material, low(er) cost, and options to customize. I wanted something that was “ready-made” enough for a new adjunct to tackle, but flexible enough for an experienced instructor to make changes. An instructional designer helped me to develop a nice sequence of activities and assessments that would go hand in hand guiding the students. Weekly quick surveys were added to pick up any early student issues. I asked instructor feedback, and in August 2013 we switched. I expected to see a positive change right away.


My university’s accelerated schedule means that we run the course monthly in multiple sections. So there was no real time off to test drive the system. The first months were full of glitches and student  frustration. Some instructors kept their old exams with the new textbook, resulting in vociferous protests for lack of matching between material read and material evaluated. Things calmed down over the next months and currently the course runs quite smoothly.

A few months ago I decided to use the large amount of data generated to compare before and after. I had student end-of-course survey and GPA data easily accessible, as well as a number of assessment data with the new course and tons of (anonymous) student comments.

What I learned:

  1. Student survey data don’t mean anything. This is not new of course, but it did hit me with full force going over the numbers of 40 something courses. Response rate was usually around 50% in the best case. The few cases that it was higher it was usually due to issues with the instructor.
  2. With the above I mean not only that there was no difference in student perception, but that the data were not really robust. If the same instructor who has been teaching the same class forever gets really different evaluations in back to back courses, chances are there are confounders. One can be different student population. Other may be just sampling bias (who answers the surveys?) A colleague with biostatistics experience is lending me a helping hand as we speak.
  3. The hardest lesson of course, was not keeping some of the previous assessment questions in place to compare. However, I do not really know it would have been feasible. The written assignments were so plagiarized that they could be found online. The exams were straight multiple choice questions. In any case, for a flipped classroom project I am participating now I had the precaution of designing a few strategic critical thinking questions placed in the “unflipped” class serving as control.
  4. Not all is lost of course. As I learn more about how to analyze education experiments, I have been given some ideas, such as rank the students and compare their grade in the biology class and then the grade in a subsequent lab class. Maybe the approach does not help high achieving students (who will do well no matter what), but there may be some difference in the low achieving group.

Of course even negative results are results, but it would be nice to see “something” improving. The only factor that moved significantly in the positive direction was the students’ opinion about the textbook. Will see how the next round if data crunching goes…

Dear readers, do you have any insight/advice about measuring learning effectiveness? Please share in the comments…any help is much appreciated.

Riding with Miss Coco

Leave a comment

Miss Coco and I

Miss Coco and I

When I was a teen, I spent a couple of vacations horseback riding. It was in Cuba, and there was not much system to it. My brothers and I would rent horses in the morning (it was a big country park called Escaleras de Jaruco) and spend the whole day on horseback. I loved it.

Fast forward a few decades to now. In the time in-between, I may have ridden less than ten times, mainly on designated trails. So when I decided a few months ago to take advantage of a free lesson deal, I did it with trepidation. Learning English riding at my age feels borderline ridiculous. Good thing that at my age I do not really care about ridiculous anymore.

Sara the trainer is young and bubbly, and oozes positivity. During the past months (I take one class per week), she has taken me through the very basics and some bareback riding to practice leading with movement and voice only. Every time I miss a class (or a whole month as recently) she patiently walks me through the process again from square one.

All in all, I love it. I love the connection with the horse, Miss Coco. I enjoy Sara’s approach of trusting the horse and communicating with them. And of course I relish the exercise.

So what this has to do with my usual topic on education?

This morning, as I was painfully two-pointing and getting frustrated by how much I have lost for being off one month, I thought about what a silly client I must be for Sara. She trains people to become competitive riders, and here she is walking me around and tugging at my toes and knees to have them in the right position. We both know that I will never become good at this. But she is encouraging and positive. She gives me feedback and shows little tricks. As a result, I enjoy horseback riding, and want more challenges. I enjoy it even if my muscles burn and my back protests after every class.

In summary, Sara is a great teacher.

It came to me how easily we focus in class on our stellar students, the ones who do well, who want to become scientists and doctors, the ones who perform closer to our level. And how often we lose sight of those who know they will not be A students, but still may enjoy the ride, may still learn a lot and feel a deep sense of accomplishment.

From this year’s courses, I have five students who are coauthors of posters that will be presented at the AAAS conference  student competition next February. They are the typical non-traditional students: all work, and one is an active military.

I am immensely proud of them. They are extremely accomplished students who did very well in my classes.

But after this morning’s exercise, I wonder about the many others. And I promise to myself to look harder after each student. Because there is much happiness and accomplishment that can result even at a “lower” level of knowledge. And as educators, it is our goal to reach the “bliss point” of learning for each of our students.

CUREing Ocean Plastics

STEM education exploring ocean plastic pollution

about flexible, distance and online learning (FDOL)

FDOL, an open course using COOL FISh

Main Admin Site for the WPVIP multisite

This multisite hosts public sites for and WordPress VIP


An Online Summer Book Club of Science


Teaching and learning reflections around science education

Disrupted Physician

The Physician Wellness Movement and Illegitimate Authority: The Need for Revolt and Reconstruction

The Blog of Author Tim Ferriss

Tim Ferriss's 4-Hour Workweek and Lifestyle Design Blog. Tim is an author of 5 #1 NYT/WSJ bestsellers, investor (FB, Uber, Twitter, 50+ more), and host of The Tim Ferriss Show podcast (400M+ downloads)

Here is Havana

A blog written by the gringa next door


A blog full of humorous and poignant observations.

Jung's Biology Blog

Teaching biology; bioinformatics; PSMs; academia, openteaching, openlearning


Reflexiones sobre asuntos variados, desde criminologia hasta artes ocultas.

Humanitarian Cafe

Think Outside the Box

Small Pond Science

Research, teaching, and mentorship in the sciences

Small Things Considered

Teaching and learning reflections around science education

1 Year and a 100 Books

No two people read the same book