Inside Education Perfect with CO-Founder Shane Smith

Paul Spain:
Greetings and welcome along to the New Zealand Tech Podcast. I’m your host, Paul Spain. In today’s episode, we’re diving into the story behind Education Perfect, a local edtech platform that is already helping change New Zealand’s trajectory by improving learning outcomes for our school students. The platform is an adaptive online learning system, and it already impacts the future of 1.8 million students every year in high schools across New Zealand and Australia. In 2021, KKR acquired a majority stake in Education Perfect in a deal that valued the Dunedin-based firm at over $450 million NZD. Joining me is Shane Smith, co-founder of Education Perfect.

Paul Spain:
He shares about the founding days of Education Perfect right through to today, including how it grew from a humble —vocabulary tool into a comprehensive online learning environment and the pivotal moments that shaped its evolution. Shane takes us behind the scenes, sharing tales of classroom experiments, the challenges of scaling up, and even navigating a serious cyber attack, something that we really hear about. We’ll also delve into the future of education, chatting about the role of AI in the classroom, its impact on learning and how technology is changing the way students and teachers engage. Before we begin, a big thanks to our show partners, to One NZ, 2degrees, Spark, Workday, Fortinet, and Gorilla Technology. Let’s jump in.

Paul Spain:
Shane Smith, great to have you on the podcast.

Shane Smith:
How are you today? Thanks so much, Paul. I’m fantastic. And thanks so much for having me along. Congratulations for such an amazing podcast. I’ve really enjoyed listening to some of your episodes recently.

Paul Spain:
Oh, thank you so much. What’s the short version of what Education Perfect is and, and, you know, why it’s, it’s taken up so much of your life and, you know, why you’ve ended up with, you know, million— literally millions of students and teachers using this technology?

Shane Smith:
EP, Education Perfect, is an adaptive online learning platform. It helps teachers to identify strengths and weaknesses of their students. It helps to remediate the weaknesses that students have. So it’s both assessment and learning deeply tied together. And there’s, there’s real strength in that synergy there. And it’s used by high school students, um, across English, maths, science, humanities. And we actually started off as LanguagePerfect. Initially it was a vocab acquisition tool.

Shane Smith:
Teaching French, German, Japanese, or all the foreign languages that are taught in New Zealand and has broadened out to a full language learning environment that, that lang— language students use across Australia and New Zealand.

Paul Spain:
You know, keen to hear a little bit about, you know, what your backstory is, where you grew up, where you went to school, those sorts of things. No, absolutely.

Shane Smith:
My mum was a high school teacher, but then when she had us, uh, decided that she wanted to raise us full-time, which was sort of complemented how busy my dad was. And so education was a, was a key part of our family. We were very fortunate to have someone who was very passionate about it, supporting us all the way through, through school as well. I was lucky enough to be born the brother to my, my brother Craig, who’s 2 years younger than me. Yeah. Yeah. Okay. Oh, fantastic.

Paul Spain:
When you kind of look back at those, those younger years, you know, do you see any, any connections to what you ended up, you know, doing? Are there any particular things that sort of stand out?

Shane Smith:
I, I, I think there are a few things that, that do kind of flow through. My dad, when we were very young, we must have been about 7 or 8 at the time, uh, got hold of one of the first computers available in New Zealand. And I remember as a child, all going all the way through my childhood, being captivated by what’s possible on, on the computer. And along the way, he realized, well, there are two ways you can interact with this machine. You can either kind of consume stuff on it, or you can create things. And one day— well, we were really interested in video games, played them as much as we were allowed to, which wasn’t very much. And one day my dad brought home a big, thick book, which was a games programming book. It was developing in, I think, C++, which is a pretty rough language for a, I think, 10 or 11-year-old to learn.

Shane Smith:
But he said, like, you know, if you’re going to be spending this time on the computer, here’s how I want you to do it. And from that, from that day forward, We spent— I spent quite a bit of time actually self-teaching, learning to program. My brother and I both were really interested in doing things on the computer. He picked up Adobe’s Flash when it came out and was really good at making little, little games, which he did in his spare time. One of the things that he really enjoyed doing was working out ways to embed them on the screensaver so that if he was in class and a teacher walked past, He could press one button and all of a sudden the screen went from game mode to work mode immediately. He had great fun doing that. And, and so, I mean, definitely the computing part of our, of our background went all the way through. It was a real interest.

Shane Smith:
It was never the core thing that we did. I don’t think we ever studied it formally in school or at university, but it was always an interest there. With that passion for education, I guess we, we both not only spent a lot of time learning and really enjoyed that learning, but also did quite a bit of reflection on the process of the learning. And I think that there was a lot that we learned from there that we’re able to bring into LanguagePerfect and EducationPerfect eventually around, I guess, our own experiences as students as well. Yeah, fantastic.

Paul Spain:
What, what, what was that first computer that you were, you were using? Can you remember what you used in those?

Shane Smith:
I think it was a very, very old Pentium processor, but it was, it was, it was, I think it was like 486 or something. It was, it was, it was really old.

Paul Spain:
Yeah, yeah, yeah. Oh, fascinating. And what was the journey from, from there through, through, through school and, you know, into studies?

Shane Smith:
What did that, what did that look like? I was fortunate enough to get an academic scholarship to St. Kent’s College out in Pakuranga, and that really did open a huge number of doors for me. It was the first school in New Zealand to have a one-to-one laptop program. It had some really visionary teachers. I remember in particular Walter Cheng being, being one of the teachers there driving their ICT program forward, and not only driving it forward from, from the perspective of providing resources, but also really providing opportunities for the students in the school to engage with these, with these machines. And so for example, he held completely voluntary classes for students. We did the Cisco CCNA networking course in our spare time, which was firstly an example of like quite, quite quite well put together e-learning, but also an example of, hey, you can actually go out and do things as full adults in the world way before the rest of the world thinks you’re ready for this. That’s brilliant.

Shane Smith:
So we, so we, we had a, an, an amazing range of opportunities at St Kent’s. Uh, Craig got really into language learning at, at, at, at St Kent’s, and actually the very first version of LanguagePerfect was built in one of the classrooms. There was a, a take-home assignment where Doug Anderson, who was his teacher at the time, said, “Hey guys, you’ve got these laptops. I’d like you to go away and build something to help you learn languages with the tools that you have.” And people used all sorts of skills that they had to come back with something. And Craig had a look at vocabulary learning in particular and said, “Hey, this is really hard. It’s really hard for the teacher to teach. It’s really hard for me as a student to—” to learn. I mean, traditionally we were talking about flashcards that you flipped over to see whether you remembered the words or not.

Shane Smith:
And, and he thought that there was a real opportunity there to do something on the computer that gamified that and brought in a whole lot of, uh, best practice pedagogy to make something really cool. So he is a— I think he would have been a Year 11 or Year 12 student at that stage, put together the first highly gamified version of this thing, which he really enjoyed. And then the nugget of that sat there for a few years until he got to university and then sort of grew with that. But definitely those opportunities in the setting. And I think the other thing that we were really fortunate about is that being some of the first people to have these one-to-one devices, it was really clear that there was potential here, but it was not yet fully realized. And so when we went through high school, everyone was still trying to work out, well, What do we actually do with these machines to, to really maximize the learning experience for students? And there was a process of trial and error that happens with any new technology. We’re seeing it happen at the moment with AI, almost sort of replaying exactly, actually. And, and we were sort of— we had that firsthand experience of, of people trying to work this out and knowing that there wasn’t yet a great solution to this.

Shane Smith:
And from that, seeing that there’s a real opportunity to build something that’s really tailored for this new reality and can unlock some really cool new learning experiences. And so we were doing all sorts of things. And then alongside all of this, we were sort of fiddling with the edtech side of things as well. And then in 2007, we were— we entered in what’s called the auda— what was called the Audacious Business Competition down, down in Dunedin. And this was a business comp where you pitched an idea, you built a business case, you pitched to the judges, and then there was a $20,000 prize for, for the winner. That’s worth making a bit of an effort. That’s worth making a bit of an effort, particularly as students with no money. Yeah, yeah, that’s brilliant.

Shane Smith:
And so I had no— I had no business experience, so Craig was the one building the business case. But I actually flew down to Dunedin around this time, and we sat down and, and we sort of built this business case, um, together, and we submitted it. So we ended up winning the Audacious Business Prize and having $20,000 strings-free to, to kick this off. So we actually then both took a year out of our university courses, moved back up to Auckland into our parents’ garage— typical startup story, uh— and, and for that year worked on this thing full-time. And we got to near the end of that first year and hadn’t actually made that much progress. And then we had to go back and give a talk to the next group of the Audacious Prize people saying the things that we’d done, what we’d done with this money, how, how successful we’d been. And all of a sudden there’s this pressure of this event coming up.

Paul Spain:
So what did that pressure create?

Shane Smith:
So that, that pressure created Craig to reach out to the New Zealand Association of Language Teachers and pitch what became the— our— the first version of it was, was the Vocab Olympics. It’s, it’s now become the Language World Champs. We worked out that there was copyright issues there after the fact, but it was the Vocab Olympics in that very first year, and it was going to be a free competition for anybody in New Zealand, uh, who was interested in languages to, uh, to take part, to use our software One of the things that we’d done with LanguagePerfect is that we’d— we’d worked out that motivation was a key issue. Learning vocab is not inherently that rewarding. It’s hard work. And so we gamified on top of that. And this was hugely popular. And so that really, really helped us to kick off.

Shane Smith:
And that was, I guess, the start of our growth engine within language teachers at the time. Fantastic. That’s great.

Paul Spain:
And so this was a— what did you work it out as, an in— an annual in-person kind of event? How did it—

Shane Smith:
Yeah. So, so I mean, when we were starting off, one of the very early decisions we needed to make, which just shows you how far back we go, we are 17 years now, was do we do this as a CD which we print and sell to people, or do we do, you know, this new weird thing of actually putting it on a website and then making logins and getting people to pay for for access to a website, which was— it’s, you know, it’s software as a service, but it, it was so new then that it was really quite a step. And we decided, hey, we don’t want to print CDs. And so, so everything was online, uh, the whole, uh, Vocabolympics was done by students online. But then for the, for the award ceremony each year, we’d bring the winners together, we’d bring in— we’d fly in their teachers, we’d do a, like, a full, you know, full-on award ceremony. An MC and a band, and we’d really make a big deal about it. Brilliant. And, and the kids deserved it.

Shane Smith:
I mean, you know, we had gamified this to the point that the kids who decided that they were interested in these competitions really got into it. In fact, we had to set maximum time limits in each day that the students could do LanguagePerfect because we kept getting complaints from the parents that their kids just weren’t sleeping. They were so— they were so engaged. They so wanted to win. They so wanted to be at the top of the leaderboard that they were doing all-nighters. They were— they were just going full out for this.

Paul Spain:
Must have been unheard of for, you know, from the— from the teacher’s perspective, right? That’s a huge change.

Shane Smith:
That’s right. I mean, the— the idea that bringing in a piece of software can generate so much excitement, I think for language learning in particular, was brilliant. Brand new and, and, you know, really was a revolution. We— I think the other thing that, like, the excitement is really important. And then I think the other side of it is that we were— we built something that worked in terms of learning. Like, I, I think it’s enti— it’s very possible with educational products to build something where you buil— build the excitement in, but it doesn’t really do anything. But with, particularly with LanguagePerfect, We’d built in a series of algorithms to really drive— drive that vocab home into students’ brains. And that then created a fantastic platform for those kids to then go and learn the higher-level skills of putting sentences together.

Shane Smith:
It set them up really well for the rest of their language learning. And so teachers could see that. Teachers could see the results. They could see the excitement. It got to the point where a number of years in a row, there were comment— commentaries from the NCEA markers to say that the quality of vocabulary knowledge across the country had markedly improved. So, so, you know, we generated excitement, but we delivered impact. And those two things together was, was, was really a winning formula there.

Paul Spain:
Yeah, obviously that, that sort of, you know, early interest and experimentation with you know, C++ as youngsters and, you know, all of that must have laid some amazing foundations. But of course, you know, technology, you know, has never stopped evolving. That’s true. It moves quite quickly. So, so you’ve made that, you know, decision that, hey, we, we don’t want to release this on, on, you know, CDs, which was one form of software distribution, but to, but to do it online. So you know, what did that look like in terms of the technologies and so on that, that you used when you, when you, when you got started? Yeah.

Shane Smith:
So Craig, going through high school, had picked up Adobe Flash as being a platform which he was able to iterate and innovate really quickly on without requiring a deep base of coding skills. And he built the very first version of LanguagePerfect in Flash. And, and so that became our starting point. And actually, we published that little Flash applet into a website that was languageperfect.com. That was where we started. Initially it was just a frontend only. We— I think we stored progress to local storage and then we realized, well, okay, if people are logging in, we need, we need a backend to this. And so I’d had the experience in C++, so I picked up C#, which was kind of the, the, the closest there that I could then, uh, build into web apps.

Shane Smith:
So, so we built a C# backend on that. We had a SQL Server database. That served us really well over time. Then as the, the Vocab Olympics started kicking up, all of a sudden you go from being low throughput to very high throughput. And while it’s not an examination, the kids were so excited and so into this world champs, you did not want the site to go down. And so we had these big load spikes coming through. Uh, we ended up putting a Redis cache. In place, which was— I love Redis, it’s a fantastic piece of technology, and a lot of hyper-optimization and pre-caching of data as we went through there.

Shane Smith:
Redis also then drove all of our scoreboards, the scoreboards, so we weren’t hitting our database as often. But along that route to get there, I mean, this probably happened over a number of years, and each year we had these language perfect world champs that would come around Each year we’d have the, the largest audience that we’d ever had before. Each year they would flood our servers on the start of the World Champs, sort of no matter what we did. And I mean, during this time I was still splitting my time between medicine and NLP. So I remember one year, for example, I was training down in Hamilton, Waikato Hospital. Language Perfect World Champs starts and an hour in the whole thing goes down. And so I’ve got to— frantically drive up to Auckland to, to go to our server farm and turn the machine off and on again. And, and in, you know, future years there was one, there was one year where someone decided for whatever reason that they didn’t like us and, and sent an enormous DDoS attack against us.

Shane Smith:
And all of a sudden we’d never dealt with, I guess, network security in this form before. Yes.

Paul Spain:
We were still on a— Must have been a real wake-up call.

Shane Smith:
It was a real wake-up call. We were still, I think, on a single server at that stage. And I think Craig picked up the phone and talked to— it was either Rod Drew or someone at Xero and said, look, this is happening to us. We don’t know what to do here. Can you give us any suggestions? And he put us in touch with, uh, one of the network firewall companies. The network firewall people said, well, we can put something in place, but it’s gonna cost you $63,000 upfront. And there was this moment of, well, what do we do here? I mean, $63,000 is a huge amount of money for the company. Then, but Craig said, “Do it, just get it in there.” And we, we got the firewall in place, um, the site came up, you know, the World Champs was a huge success and the business keeps growing.

Shane Smith:
And there’s just all these points, I guess, where things could have failed. And, and then through just grit and determination and willingness to try different things, each year we had new technical challenges. We went from, you know, just being a, a standalone front-end Flash application to suddenly, you know, needing subscriptions, which you need, which means you need a database, a way to manage authentication. We had no idea how to do any of that. And each stage along the way, we had to go and had to learn, pick up books from the bookstore and the library, ask anybody who we could get our hands on for advice. And it, by growing organically, it also meant that both from a technical side and a business side, I was focused on the technical side. My brother was really great on the business side. And we were both growing our skillset as we went.

Paul Spain:
So you saw that, that opportunity, this was really pivotal, right? In your future of going from being software that was gonna help youngsters learn languages to software that was gonna help, you know, right across, I guess, you know, virtually every, every potential subject. How did that, you know, how did that look? How did you have that kind of, you know, come to that conclusion? Because that’s a really big sort of, you know, flip from, from this really tight focus to then, you know, zooming out a little bit.

Shane Smith:
Absolutely.

Paul Spain:
And, you know, potentially being able to apply that, that impact and create that sort of level of engagement. But, you know, much, much more, much more broadly. Yeah.

Shane Smith:
So, I mean, I think there were two things that, that, that drove that decision. Back where, where, when we were in university, studied business down at Otago and went, went through an accounting class. And, uh, there was one year where the students were really struggling with the course material. And, and the lecturers came to the class and said, look, for this exam, we’re gonna simplify things a little bit. We are going to tell you what the questions are gonna be upfront, but all of the numbers are gonna be randomized. So you need to know how to engage with all of these formulas and, and all of these types of questions with any sets of numbers. And if, if you can do that, then we know that you’ve learned the course material. And so Craig and I sat down at that stage and, uh, we actually built a little website.

Shane Smith:
This would’ve also been around, uh, 2007, 2008-ish, which was called Underground Accounting. And it took all those questions and, uh, randomized all the numbers and, and the students could have as, as much practice as they wanted on those. Oh. And so we saw that there was something in here around digital assessment and preparation for digital assessments that could add a lot of value, that, that kind of practice there and practice with questions that are, you know, far more complicated than vocab words are. Uh, so that was the one piece. We had that sort of sitting in our back pocket for the, for the following 5 years, knowing that we’d done that and there was something there that, that was quite interesting. And then we hit a point in, in 2012 where we looked out at the world around us and we realized that language learning is a really quite narrow niche. And there were starting to be online competitors who were providing free solutions.

Shane Smith:
They, they didn’t do many of the things that we do, and to this day they don’t do many of the things that we, that we do. But we saw, hey, well, there’s, there’s a massive risk here. That we’re so concentrated on this one space that we could end up sort of crushed out. But also we sort of looked to the future and we thought, well, what are the winning companies going to be like 10 years from now? And we realized, well, those winning companies are probably going to do lots of things and then they’re going to be able to integrate them really effectively and get synergies from that integration. And so we realized, well, actually we need to take some of these ideas that we’ve been holding in our back pocket We need to branch out. We need to take a big risk and we need to take all the things that were hugely successful for language learning and bring them to science and to English and to maths.

Paul Spain:
So this was, this was effectively the moment where you both put on your futurist hats and you looked out to building the half a billion dollar, you know, and beyond business that that Education Perfect is today?

Shane Smith:
I mean, I, I must say this is, this is where my brother Craig has had amazing vision. I think this is something that, that is reflective of the vision that, that he had. He always knew that we had the potential to build something that had significant impact on the world, that had significant scale. I remember him looking around when he was, you know, in his last few years at school, you know, starting university and saying, well, hold on, When you look at a Microsoft or you look at an Apple, those are people just like us who started those. There’s no difference really. And we can do something like that. Even where we were, I mean, at that stage, our, our software was being used by, I think, the majority of languages departments around New Zealand and, and Australia. And knowing that every day we would come into the office and be doing things that impacted hundreds of thousands kids— of kids’ lives.

Shane Smith:
And today, you know, we, we, we touch 1.8 million students every year providing them education services alongside their teachers.

Paul Spain:
Through these journeys, there’s always ups, there’s always downs. Things don’t always go according to plan. You shared with me that there were, you know, that it wasn’t all sort of rosy, particularly that sort of in that, that window there where, you know, you grew the team very quickly. Maybe you can walk us through sort of what the, you know, what happened and, and you know, what you take as the the lessons from, you know, from that period?

Shane Smith:
Yeah, this was about the time that we were shifting from LanguagePerfect to EducationPerfect. So we were, we were branching out and suddenly building a brand new product for a brand new set of users. To do that, we expanded our team and, and brought in— I mean, in particular, our EducationPerfect product is not just the software, but it’s, it’s a base of curriculum-aligned material as well. And so we grew a team around being able to build out that material there. And we went into the first year with an idea of how we could make an awesome education experience for science and for English and for maths. And we sort of boldly forged ahead, built out a whole lot of content for this. But after about 12 months or so, we were getting a whole bunch of feedback from teachers and students saying, “Look, this is interesting.” but it’s just not hitting the spot. It’s— it just doesn’t quite meet us where we need to be at the moment.

Shane Smith:
And we realized in the process that some of our assumptions around what made learning really work for languages had been quite specific to vocabulary. And, you know, with vocab, each, each word is its own little data point as such. Whereas when you’re learning concepts in science or maths equations in maths, Actually, the learning process behaves differently. The amount of repetition needs to be different. You need context around all these things. And so we, we hit a point about 12 or 18 months in, we had set up this, this team to build out educational content. We hired university students. Everyone came in with, with absolute best of intentions.

Shane Smith:
In the process of doing it and, and growing that team as, as fast as we did, that team ended up with their own subculture. And, and that subculture just ended up being completely at odds with, with the culture of the rest of the business. And we found it incredibly difficult to try and merge these things together or to try and make them interface nicely. And we found it very difficult to drive the change and the urgency that we needed in that part of the team with, with, with the current setup that we had. So Craig and I both moved back to Dunedin so we were there in person. And over, over the course of about 12 months then, we did do quite a significant change-up. What we were doing wasn’t working. Mm.

Shane Smith:
But there was, you know, there was a strong call to say, look, we tried this other subject thing, it didn’t work. Why don’t we go back to the profitable business that we had that was working so well? But Craig and I knew that if we looked forward to that business in 10 years from now, that a language learning business by itself was never gonna have the scale or the synergies to have the impact that we really wanted or to be the size business that, that we wanted to build. And so we had to make a whole bunch of changes. So on the product side, we had to eat a whole bunch of humble pie. We had to step back, go and spend a lot of time just talking with students, talking with teachers in particular, and rethinking what it was that— what the value was that we were trying to deliver to them and how we could do that. It was around that time also that we started hiring teachers onto our staff, which I think has been an incredibly positive shift for us. Uh, around 2014, we, uh, got involved with NZQA, that’s the qualifications authority, who wanted to run the first high-stakes online examination in New Zealand. And we showed them the assessment platform that we had at the time and where we thought it could go.

Shane Smith:
I think when we, when we started off, we thought that they’d, you know, get kids to like write an essay online and then submit that, which seemed pretty easy. They came back to us and said, well, no, actually we want to do a maths exam online. And actually we want like this to be as good as writing on paper kind of thing, which was a much, much harder, harder task. Yes. We spent probably 9 months just hyper-focused on these high-stakes exams. So this, you know, these exams were going to count towards kids’ final grades. They were proving, you know, they were proving what they knew. And so we needed to make an interface that had as low a friction as possible.

Shane Smith:
And it just— it needed to work, which was a completely different experience to what we’d previously had with the learning. And then preventing cheating, getting— making maths work in a way that, that kids were able to engage with was a huge task. In the process of doing all of that, for the first time, there were a bunch of teachers who could see, oh, these guys are actually doing something meaningful here. And we were able to hire our first teachers to come and help us implement that, that EM-CAT. So the EM-CAT was the Electronic Mathematics Common Assessment Task, which was an internal assessment run by schools but done like an exam. Once we had a few people on staff, then the floodgates broke open and all of a sudden people looked and said, well, hold on, this is actually a pretty attractive career path here. You know, they’re building something really meaningful. It’s, it’s different to the work that I’ve previously been doing as a teacher.

Shane Smith:
And that was a massive unlock for us at the stage. So, so, so building that, that, that gravitas, building that, that sense of credibility was so, so key there. I know I’ve meandered a little bit, but here, but— No, that’s fascinating.

Paul Spain:
Now, you know, when, when you look back, I imagine, you know, points like that, which would have been, you know, challenging to deal with and, you know, stressful, expensive, and, you know, you had to make those executive decisions. But, you know, presumably that, that actually helped you, you know, get more strategic on things like, you know, you know, securing data and, you know, protection from, you know, cybersecurity risks, whether it be those, those denial of service attacks or be it, you know, any other sorts of, you know, cyber and hacking type, type risks. So, you know, how do you, how do you look back on those, those sorts of pressures?

Shane Smith:
Yeah, I mean, the interesting thing is that when you get a bunch of bored students and you put them in front of a web page, there’s going to be a subset of those students who are going to try anything and everything. And particularly if they can work out ways that they can cheat and get to the top of the leaderboard, that’s going to be a, that’s going to be a big incentive there. And so we’ve had anything and everything thrown at us over the years, and each time we see something new, we’re able to learn from it, put in place mitigations, and, and then sort of grow from that. So in some ways, having those repeated events each year gave a point in time where everything was going to be stress tested to, to the max, and then the rest of the year was plain sailing after that. The other point, obviously, that we had to put a huge amount of time and thought into this was When we did this, uh, the work with NZQA on the EMCAT, we knew that there was going to be an enormous incentive for students to cheat in that as well. This was high-stakes exams, and so we put in lots and lots of effort, uh, made sure that we had everything audited and, and all set to go, um, from that perspective, with one— which once again just put us in a really strong place. And over the years, that piece around data security and really respecting our teacher and student data, the location of that data, how we manage it, has been something that we’ve grown and grown for. And it actually put us in a fantastic position when some of this new AI stuff came around that we were already a trusted partner with schools.

Shane Smith:
We already had systems and processes in place. And we were able to go from that base to say, OK, what’s going to be OK here? What’s going to— what’s not OK? And, and, and then build from that. There was a, there was a, a, a significant journey, day zero of this being a brand new technology, getting it signed off by the various Australian states’ education departments. And there is a group in Australia, ESA, who does a certification, ST4S, which is a digital privacy and security, we, which we worked with very closely around that as well. Yeah, but it’s definitely something that builds and grows over time. And in some ways, you’re actually much better to be tested and stressed repeatedly rather than having plain sailing up until one day all of a sudden something goes horribly wrong.

Paul Spain:
So maybe you can explain, you know, MCAT as that sort of started out and what that— what that evolved into. So what was the initial goal? How, you know, how, how, how far and wide was this going to go? And, and how did that, that evolve? Because this is Yeah, obviously it’s pretty, pretty important stuff, right?

Shane Smith:
That’s right, exactly.

Paul Spain:
To get it right. But, but yeah, who— what was the— what was the audience and the scale?

Shane Smith:
So I think the genesis of this was Craig was invited to a meeting along a bunch of other people with NZQA where they were talking about their future plans around looking into digital assessment. And I think they had a roadmap of 6 or 7 years before they were going to try and do their first digital assessment at as, as their starting point there. And Kurae said, well, hold on, we can do that in 12 months if you want. And from there, there was a conversation that was started to say, well, what could we do in 12 months? How could we test things? And actually, the ministry put their best foot forward, I think, and said, well, okay, well, if we can actually try something here, you know, we’ve got to trust a set of partners who are already in New Zealand schools, who are already delivering this to a significant proportion of the schools around New Zealand already. And if we can try something, we can learn from it, and then that’s able to then shape our ongoing trajectory around where we want to go with online assessment. And so the, the eMcat was born out of that, and we delivered, I think from memory, 3 online assessments. One was, I think, history. There might have been a biology one, and then, and then, and then the most challenging one was a maths one, which was an NCEA Level 1 algebra paper.

Shane Smith:
And we had originally thought that we’d be doing text-based input, but when maths was raised, I, I think, you know, the ministry made a really sensible, interesting choice to say, well, let’s, let’s stress test, um, something really challenging here and see what the limits of this are. And so we went from— I think we had about 9 months from, from signing up to this to delivering it. And during that time, we built from the ground up math input system that both accepted handwriting as well as almost like an IME setup where, where students can get a supported way to enter math, uh, that worked really well for them. In the backend, an auto-matching system to, to do— to assist markers with their marking. We built in a high-stakes assessment system which was able to go into a lockdown full-screen mode, streaming monitoring for, for the proctors and the exam setup. And then we delivered that, I think it would have been September that year, as part of a high-stakes internal assessment by schools, but run as an assessment. And that was an incredibly intense year.

Paul Spain:
I can imagine that’s, that’s a really, really important capability to deliver, but it also could go really wrong.

Shane Smith:
That’s right.

Paul Spain:
If people are gaming the system and, you know, if it just doesn’t, doesn’t work or lead to sort of, you know, fair marking and so on as well. That’s right.

Shane Smith:
And in fact, we had a challenge internally to the team. I said, right, I’m not going to shave until we deliver this successfully. And so there was a 9-month period, right? It grew scruffier and scruffier and scruffier as we went. There were many, many nights where, yeah, as you said, huge amount of pressure to deliver it and get it right and for it to work. Not only pressure on us looking bad, but like, we didn’t want to stuff up some student’s attempt at demonstrating what they know in algebra. We wanted it to work for them. We wanted it to be a great experience. We wanted it to be as frictionless as possible.

Shane Smith:
We really cared about, about giving them the best possible shot to demonstrate their knowledge there. And in the process of doing that, obviously, we built out this fantastic assessment capability, which has served us incredibly well ever since. Yeah. March 2024, ChatGPT comes out with GPT-4. And I think I was sitting in New Zealand at the time in the South Island, and I, I saw this thing and my brain just exploded. Like, I’m sure many of our brains exploded and, Immediately I could see that this new technology was gonna have an enormous impact in the education space if managed well. At the time, I spent a number of days putting together a whole lot of different ways that we could engage with this technology. And, and as part of that, I put together a, a set of reports and prototypes that I, that I then took to our board and said, hey, I think this is gonna be really important for our business going forward.

Shane Smith:
And over about a 6-month period, I’d come into each of the board meetings and say, what are we doing in this space? And, and we had a fantastic management team in place, still do have a fantastic management team in place, but they were all fully committed and, and focused on the, the direction that the business was going in that stage. And no one had any capacity. They were doing their jobs and doing their jobs well. There wasn’t any space to then really explore something as experimental as this. So in October I said, well, would you like me to come back and just kind of kickstart the process here and get things moving? By that stage, I’d already built out pretty fully functional prototype of how we could integrate this into our product.

Paul Spain:
How, how did you do that? Because you weren’t, you know, you weren’t part of the sort of software dev team and, and so on at that point. So how did you— tell us how you built out that, that prototype because you were telling me about this. Yeah, previously.

Shane Smith:
So the interesting thing here is that I’d actually been out of programming entirely for, I think, close to 5 years. And so I was coming in pretty rusty. But what I started doing is I realized that I could open up Google Chrome and jump into the Chrome Dev Console and actually inject JavaScript into the page of our website. So I still had to log in, and so I jump into the site, I’d log into the page that I wanted to play around with, and then I just start playing around with the page through the, through the JavaScript console. And over a period of time, I built up a script that I could inject, which had a whole lot of this additional behavior. I then went and built a backend for it. I actually leaned quite heavily on GPT-4 at the time to help me with all the coding because I was rusty and I was working in frameworks and the like that I hadn’t worked in before. But lots of vanilla JavaScript hacks.

Shane Smith:
But it was to the point where literally you could give someone a link, they grab the text of that link, put it in their, their JavaScript, in their Chrome console, they would have a whole lot of functionality in addition that they could interact with in a fully interactive sense. And it was, it was a really cool way to interact with the app. Very low friction. Obviously not super easy to test and iterate, but, but you could, you know, you could do all sorts of things just tinkering around in there. Yeah. Yeah.

Paul Spain:
Fantastic. So what, what’s your, you know, view now on, you know, on the role of sort of generative AI, you know, you must have some interesting insights from, you know, how it can help inside, you know, business for the team, its role within the education perfect software for students and teachers. And, and, and maybe you’ve got some, some thoughts, you know, more broadly on, on the impact of AI on education, you know. And, and I think, you know, there have been some you know, probably some, some people are pretty scared around where AI fits into education. Firstly, that, that opportunity for the Education Perfect product and, and, you know, how you felt that OpenAI and, and the others in this generative AI space, you know, would be able to deliver something that would, would help Education Perfect, you know, evolve as a product.

Shane Smith:
Absolutely, yeah. So when I saw this technology, the thing that really struck me— previously, with, with previous sets of technology, we’ve been able to give feedback to students about closed answers, so multi-choice or a few words at a time. But these large language models work with text natively, and, and that means that they can open up a space for for feedback and coaching and direction of students that was simply not possible before. And so the first area that we looked into, which I’m probably the most passionate about, is this— is this feedback space. And we’ve always had open-ended questions in our lessons. We include a model answer once the student submits, and then they sort of self-reflect, and then the teacher has a look at their responses and, and gives them some feedback. But I’m sure that as you experience through schools, teachers don’t have time to look at every single answer and every, you know, page in a, in a, in a workbook. And so there was a significant proportion of these responses from students that were not getting any feedback.

Shane Smith:
And, and this is no, no slight to teachers. They’re doing an incredible job here, but they just worked off their feet and they, they simply can’t— don’t have the capacity to look at every single one of these responses in, in a learning context. Assessment’s slightly different. I, I, I think assess— teachers should be looking still at, at, at student responses and, and giving feedback then. But in a learning context, there’s a real opportunity for real-time feedback for students as they, as they respond, um, to these open-ended responses. And what we found on top of that, we built out a system which gave feedback to students on, on these long responses. And we found that in addition to providing the feedback, if we provided it quick enough, then students could actually iterate on their response and actually try again and improve and actually have what we call the learning loop. So the way that the learning loop works in the, in the platform is that when students submit their first response, we, we give feedback and we rate it against a, a rubric out of 5 stars.

Shane Smith:
And if students do really well, then they can progress straight away. Um, it’s up to them what they do with their feedback. But for the student who is bored or, you know, unmotivated for what— for whatever reason coming into this lesson and doesn’t really engage with that material, we can now identify that in real time, give the student pointers for where they need to go. And then get them to do it again. Equally, the student who comes in and puts in a response that is their best effort but is way off track, we can identify that, we can give them positive affirmation for their effort, and we can identify the areas of their response where they’ve gone wrong and help them to move in a positive direction and then say, “Let’s give this another go.” And so the ability then for AI to be there in the moment when students are responding repeatedly, as many times as the student wants, is really exciting and, and really transformative. And in fact, we ran a, a comparative study where we put, uh, 20,000 students using, using the system. Half of them used it before we had the learning loop in place, and half of them used it after the learning loop. And I think we had something like a 47% improvement in the quality of student responses.

Shane Smith:
With that learning loop being in place. So measurable, meaningful impact on student understanding, on student retention as they go through these lessons. So I think that, that feedback space is one really, really exciting place. On top of that though, I think a real challenge is that AI models out of the box can be really helpful to the point where it can be counterproductive. And we’re really careful that our feedback system gives direction, but never goes as far as actually answering the, the, the, the question for the student themselves. I think one of the biggest challenges for students these days is that it’s so easy to reach for assistance that is immediate and at their fingertips and very often free. And in doing so, denies them the opportunity to struggle a little bit. There is, there is a zone of productive struggle which is really important to learning.

Shane Smith:
And so one of the things that we’ve been very conscious about in our own programs is ensuring that we maintain that zone of productive struggle.

Paul Spain:
Zone of productive struggle, I like that. It sounds— it sounds like this is actually a, a real key for the, you know, for those who are interested in, in, in great education outcomes that you do have to go through that, right?

Shane Smith:
Absolutely. I think the strict technical term is the zone of proximal development. But the, but struggle, I think, is a really actually useful term here. Yeah. Because it indicates that it’s hard and that it’s meant to be hard and that if you don’t go through that struggle, then you remember less and you learn less in the process.

Paul Spain:
This is the nature of life, isn’t it? That’s true. And business and everything. Yeah. Now I’m sure that, that over time, there’ll be more opportunities and so on in terms of, you know, how you, how you’ll evolve this. But maybe we can just take a little sort of focus off, you know, more broadly on, on the Education Perfect. Did you see other roles for, for AI to, you know, to help and assist the business outside of the Education Perfect product?

Shane Smith:
Yeah, I mean, there is, you know, the next phase that we are currently looking into is that our teachers spend an incredible amount of time building resources for their students. That’s an area that— a space where there can be co-design with, with the teacher and AI together can be, can be really productive. And we’re building out capabilities there at the moment. On top of that, there are lots of opportunities around coaching and direction, which I think at some stage we’ll, we’ll, we’ll have a good look at. But then— and that’s sort of the product itself. But then looking broadly at the business, I think all businesses today have this, have this opportunity. We are sort of stepping into a new era, I think, where there are so many internal business processes which currently require human input. Many of them are repetitive, many of them aren’t particularly interesting as, as, as tasks.

Shane Smith:
And I think as we go forward, we’re going to see more and more of those be enabled with AI. To then free up our staff to, to be able to work faster, to be focusing on more interesting, engaging parts of their role as well. And we’re certainly looking at that sort of business efficiency side of things and ensuring that our team have the best tools possible to deliver fantastic outcomes and material for, for teachers and students.

Paul Spain:
The other perspective I’m interested in your thoughts on AI for— and, you know, this is something that’s you know, really, you know, there’s been a fair bit of debate on over the past, you know, past few years since, since ChatGPT landed is, you know, where, where should generative AI fit in to education? And I remember some of the first, you know, discussions and calls that I was receiving from media around, hey, you know, should, should this technology be banned in schools, banned in universities, and, and, and so on. That, that has evolved. But, you know, now we’ve got a period where some water’s gone under the bridge. You know, we’ve been able to develop, you know, the, the realities of what, what’s possible and the thinking. What is your, you know, general kind of viewpoint and, and the advice that, that, that you give on, you know, how generative AI should AI fit in broadly in education? And, you know, do you, do you have any of these concerns sort of still humming around in your mind, you know, that were being raised, you know, over this period?

Shane Smith:
I think, I think that probably the best analogy to how I view AI is how we view calculators in the classroom. Sure. So just because we have calculators today doesn’t, doesn’t mean that learning arithmetic tables is less valuable. And in fact, we need to know our times tables even if we’re going to use the calculator because it helps us to sanity check things. And so I think the way that maths teachers use calculators in targeted— in a targeted fashion where sometimes the calculator is made available and sometimes it is deliberately removed to be able to assess students’ skills without that, that crutch is actually a fantastic analogy for how I see us using AI going forward. I think we’re going to have parts of our lessons where we want to give students as much leverage as we can to see what they’re capable of doing, in which case AI then is going to be fantastic. They’re going to be other parts of the lessons where perhaps we want, want to focus on interpersonal communication or parts of the lessons where actually we want to ensure that students do understand a baseline— the baseline set of skills, a baseline set of knowledge without reaching for the crutch of AI. And we’re going to exclude it from those parts of the lesson.

Shane Smith:
I think it’s far too powerful and far too useful to, to blanket say, hey, we— this should not be a part of education. I certainly know that each time I personally go to learn something these days, I make heavy use of AI to help me understand new concepts. And help me to understand things faster and more effectively. But I just make sure that alongside doing that, I also take the time to understand it myself rather than just relying on the crutch. So as an example of there, I will often write things out full form myself, maybe get some AI feedback on it. But the process of the writing is, is part of the learning. And, and if I’d take— if I’d given that to the AI first, then I would have missed the opportunity to, to, to, to learn myself. Yeah, that’s good.

Paul Spain:
Yeah, I mean, I agree. I think personally in terms of learning, leveraging AI helps the learning, you know, helps embed it and helps you understand more, more complex things. There’s, there’s, yeah, there’s, there’s so many, you know, positive aspects there. But, you know, it’s easy to understand educators being concerned around— Oh, absolutely. Exams that are done electronically and, you know, and, and some of these things that, yeah, you, you want to ensure that the student is, is getting the deep learning as well as what they can get with, with the technology. So, you know, as you say, the, the times tables learning as well as how to use the calculator.

Shane Smith:
Absolutely. Yeah. And, and to be fair to teachers, you know, the experience that they had initially with AI was all of the negatives. Yeah. With students, you know, using it to fill out assignments and the like without any of the positives initially. And particularly initially, it was also a wild west out there in terms of data and privacy laws. I mean, I know that we’ve been very, very intentional around where we, where we host our models, what we do with student data. We don’t use student data to train any of the models.

Shane Smith:
We’re, we’re very careful with the partners we choose for those models. And, and that’s not a given in the industry. And it’s definitely not a given if the students just click onto a random site on the internet that happens to have a chatbot in it. And so, you know, I think schools are waking up to the fact that, you know, working with trusted partners is a, is a fantastic way to safely engage with this technology. We’re certainly not saying that we’re the only people who do this safely. There, there are lots of people out there who do. But definitely working with partners who know what they’re doing, who have that depth of understanding around where AI can safely fit into a positive learning experience and ensuring that, that student data is gonna be safe and secure as well is a must for schools. And I, I, I think actually schools are pretty on the ball with that these days as well.

Shane Smith:
Yeah. Yeah.

Paul Spain:
Great. Well, it’s been absolutely fascinating, uh, talking with you. So thank you very much, Shane Smith. Thanks, Paul.

Shane Smith:
This has been a fantastic chat. It’s been really, really fun. Thanks for having me on the show. Great. Thank you.

Paul Spain:
Cheers.

Paul Spain:
Well, I trust you really enjoyed hearing from Shane Smith about Education Perfect. And of course, keep an eye out because we have another episode coming about Education Perfect that will be across on the New Zealand Business Podcast. And that’s going to dive deeper into the business aspects of the journey. And share further learnings from Education Perfect’s story. And don’t forget, it’s just a few days before entries close for this year’s New Zealand High Tech Awards, 5 PM on the 2nd of March. Thanks again to our amazing show partners for their support and everything that they do for the New Zealand tech and innovation ecosystems. That’s Fortinet, Workday, One NZ, 2degrees, Spark, and Gorilla Technology. Well, thanks again for listening in, and we will catch you on the next episode next week.

Paul Spain:
And of course, jump across to New Zealand Business Podcast, uh, if you want to hear some of the business stories behind many of our top tech successes.