This is Part 2 of a retrospective on how we used app data to evaluate Even V1; Part 1 is here!
Once we’d gotten better at acquiring Even members, the next thing was to see if we could retain them.
We started off by measuring how many members stayed who fit into the category “Active.” To be active simply meant that a member had an active account — they’d moved on to using the app after vetting, and hadn’t since deactivated.
We soon realized that the “active” category wasn’t doing very much to help us understand retention. Some active members were sticking around for months; others were deactivating their accounts after a few weeks. It wasn’t very clear what was allowing us to retain some and not the others. “We’d see people come in, and then we’d see them quit,” Evan remembers. “It was a quick interaction; it was like, ‘I don’t know what to make of this.’”
A hint came from the Advisors team. They’d noticed it felt like we were skipping lots of paydays. (Skipping a payday just meant that, on a member’s payday, instead of making either a withdrawal or deposit in order to even their pay, we weren’t doing anything. There were many reasons for skipping a payday, ranging from the member requesting that we skip it, to us being forced to skip it by the fact we didn’t have the necessary account information from the member.) Upon investigation, we found we were actually skipping a full third of member paydays.
Given that handling paydays was all the app did back then, Evan points out this meant a lot of the members we were counting as “active” hadn’t truly made it all the way through our funnel. It was kind of like they were stalled a step before the real final step.
To try to separate out members who we thought were really experiencing the app, we created an new category called Engaged. To qualify as engaged, a member had to have had at least 2 paydays evened by Even.
Looking at retention numbers of members who were engaged vs. simply active, a pattern finally started to emerge. “The engaged number actually had really great retention,” says Evan. “We were like, ‘Okay, this works. If you can last through two paydays, you’re tremendously likely to stay on for another six months.’”
As for people who never became engaged: “Our business output is paydays being evened; if you’re not participating in the payday process, then there’s no question that you’re going to, at some point — especially when you start getting charged $3 a week — say, ‘This is not for me.’”
While this new engaged number helped us get a better understanding of what drove member retention, it absolutely did not mean we were off the hook. In the next post, I’ll finish by discussing the other half of what we learned from the data about retention, and how it forced us to take a hard look at whether we were providing enough Even members with enough utility.
Thanks again to Mr. Evan for helping me remember our data story.