12 Lessons That Changed The Way We Do Research
This article was originally published on the Leanlab Blog
Research can be done remotely (but relationships are still paramount)
Last year, we added education research to the list of things that could actually be done well remotely. Conducting research while schools were completely remote seemed unfathomable before that. But, like everyone else, we didn’t have a choice.
The fact that we were researching edtech tools during that time definitely worked in our favor. Teachers were looking for any edtech tool that could improve learning and make their lives easier, and we helped them find the right tools and assess if they actually worked.
What ultimately allowed us to continue conducting research was our relationships with schools. The schools that we had already established relationships with and were familiar with the expectations of a research study allowed us to communicate better. The pre-existing relationships gave teachers and staff more confidence to approach Leanlab and the research team when things weren’t going well, and to request more time or amendments to the study. The trust and honesty allowed iterations to the study to occur in a way that worked for the participants and still met the goals of the research plan.
There’s no one-size-fits-all research
When we first started doing research on edtech tools, our approach was to design one study that incorporated a full spectrum of research goals. For each study, we looked at everything from usability questions to correlational results. We quickly found that the lack of specificity wasn’t helpful for anyone involved.
That’s why in the last year we developed a suite of research offerings that better align with an edtech company’s stage of development. We have product validation studies that are used to validate a prototype, all the way up to Correlational studies that seek to uncover relationships between usage and outcomes.
We’ve found that there’s no, universal, best research practice and there’s no research study that can answer all questions. The research we do works best when we can identify developmentally-appropriate research questions with a company and align a research project offering that is best-suited to answer those questions.
Research is a journey
There may be some incongruence between how we portray research linearly and the messiness of research in the real world, because the types of research that we offer build upon each other. However, we don’t expect every company to start their partnership with us by validating a prototype, then easily progress towards measuring correlational outcomes.
If a company is at a later development stage, it sometimes makes more sense for them to start with correlational research. Other times it might be more valuable for a company to replicate usability or feasibility studies. There’s no one path to doing research and, more importantly, there’s no end point where research stops being beneficial.
Many companies aren’t ready to do correlational research
There’s one question that every edtech founder wants to know: does my product have a positive impact on student outcomes? This is a great question and one that edtech companies should ask themselves every day. But in reality, it’s a very hard question to answer.
Before that question can be answered, dozens more have to be asked, such as:
Do teachers and students even see value in my tool?
Can any teacher from any school pick up my edtech tool and start using it with fidelity and with minimal training?
Is it a hassle for teachers to integrate my edtech tool into their current Learning Management System?
Is my edtech tool engaging enough for teachers to continue using it throughout the semester and beyond?
Companies often take these questions for granted and want to first answer questions about impact. However, these are the questions that will make most attempts to understand impact come back null because the products don’t have sustained and quality engagement. Companies will get infinitely more value from research by asking the right questions, and most of the time these questions are about understanding the perception of the tool, its usability, and how teachers actually implement it in the classroom.
Usability research is underrated
The ability for an edtech tool to be easily understood and highly user-friendly is make-or-break for edtech companies. No one, especially teachers, has the time to deal with a clunky user interface, no matter the perceived value. If the usability of an edtech tool isn’t there, a teacher will put down that tool and never pick it back up.
It’s important to remember that users aren’t comparing edtech tools to other instructional materials like textbooks. Instead, they’re expecting the user experience to be on par with every other app that teachers and students use, from Apple iOS to Google Maps to TikTok.
We have found that, across school systems, usability is one of the major roadblocks that causes underutilization of edtech to that we find across school systems. Even established companies can benefit from usability and feasibility studies in order to increase quality engagement.
We can’t do research without teacher buy-in
There is a path for us to conduct research inside of schools by solely organizing research projects with the school district’s central office and only talking to teachers once the research is underway. Many third-party vendors use this approach when trying to get their product into schools, but with research, we’ve found this way is harder and more difficult than first talking to teachers and understanding their challenges and needs.
That’s why we do as much as we can to develop relationships with teachers. We go inside the schools and talk with them about what’s going on. Because when researching an edtech tool aligns with priorities that all stakeholders believe in, the commitment to research is stronger, and the outcomes are thus more valid and reliable.
A gift card goes a long way
Have you ever received an email from a company that asks you to fill out a survey because they value your feedback? In a capitalist economy, ‘value’ is measured in monetary terms. So if your feedback is so valuable, why aren’t you getting paid for it? Even small incentives can help you get more survey responses and show that you actually value the opinions of teachers and parents.
But we’ve also learned some hard lessons about surveys and gift cards. Without captchas and other authentication methods, bots and individuals can game the system and create a lot of noise in your data. Because of this, we don’t often advertise our surveys on mass communication channels and, instead, started working directly with our school networks to find survey respondents.
Teacher expertise has value
Most of our studies require more commitment from teachers than completing a survey, though. We provide larger stipends for teachers who do research with us because teachers have years of expertise about what works for the classroom. They’re providing real monetary value for the companies they work with through our research studies. We want to honor that by actually paying teachers and, at the same time, making sure teachers feel heard and feel valued in the process.
Communications norms are different for schools and edtech companies
Since we work at the nexus between school and edtech companies, we are especially attuned to the flow of communication between these two groups. To briefly summarize the characteristics of each group, educators can be hard to get a hold of, and edtech companies value prompt, high-frequency communication. Or as the anthropologist Edward T. Hall might put it: edtech companies are high-context communicators, while schools are somewhat lower-context communicators.
Ultimately, it’s our job to serve the needs of educators, so it’s important that we cater to the communication preferences of schools in this scenario. To do this, we have to act as a bit of a buffer between the schools and companies. By now, we’ve adapted our timelines to appreciate the hectic and less flexible of schools, and we’re able to set those expectations with companies early on in the process.
Companies need actionable results
Our research reports are really long; many times upward of 50 pages. We think that level of detail is necessary so that companies and the research participants are able to get a rich and full picture of the data. But if we gave companies a fifty-page report and wished them luck, nothing would be accomplished.
That’s why we developed our Action Plan this past year—a simple 2-4 page document that distills the feedback from the teachers into something akin to a product roadmap. We believe it’s our obligation in this process to make sure that teacher and student voices are heard and are acted on. And aside from including ample direct quotes from teachers and students in our full report, the Action Plan assures that there’s no ambiguity about the changes teachers and students want for the product.
Codesign Research is Accessible
One of the most exciting things we’ve seen in the past year is companies taking some of the research methods and techniques that Leanlab used in their research studies and applying it themselves. Of course it helps greatly to have a third-party design research instruments and conduct independent, unbiased research. But there’s plenty of value in a company embracing codesign principles to create a survey and collect systematic and valid feedback from end users to help iterate a product.
Research is humbling
The type of codesign research that we do requires openness from all parties. As researchers, we’re familiar with some level of pushback from companies after they receive feedback or impact data that they perceive as unfavorable. We always try to take the time to make room for this type of feedback from the companies and adjust where necessary. It’s also necessary that we’re open to accommodating the unique contexts and needs of participants.
On the part of the schools, it’s important that they’re open to trying new things, and even more crucially, it’s important that they’re open to sometimes failing at it. It’s those failures that often tell us the most about why a product is or isn’t successful.
The role of edtech companies is probably the most humbling, though. They need to open themselves up to hear a lot of very hard truths about their product. They also need to be open to iterating their product because teachers have an inherent expertise about those products that sometimes surpasses the company’s, even though they are the product’s developers.
