Can AI Detect and Identify Changes in Our Well-Being? Meet Karin Andrea Stephan of Earkick
/LDV Capital invests in people building businesses powered by visual technologies. We thrive on collaborating with deep, technical teams leveraging computer vision, machine learning, and artificial intelligence to analyze visual data. We are the only venture capital firm with this thesis. We regularly host Vision events – check out when the next one is scheduled.
Our Women Leading Visual Tech series is here to showcase the leading women whose work in visual tech is reshaping business and society.
Karin Andrea Stephan is a serial entrepreneur, community leader, and startup coach.
Karin Andrea has deep domain expertise in tracking mental health. Her recent master thesis, titled “Therapist in the Ear - How hearables can extend the efficacy of treatment into a patient’s daily life”, was completed in 2020 at SRH University in Riedlingen, Germany.
In 2021, she co-founded Earkick to build the world’s first data-driven mental health tracker via multimodal sentiment analysis of physiological biomarkers. The Earkick iOS app provides automated measurements and actionable suggestions based on real-time data.
LDV’s Abigail Hunter-Syed chatted with Karin about the potential of an AI-powered tool that is able to track and treat mental health disorders in real-time. (Note: After five years with LDV Capital, Abby decided to leave LDV to take a corporate role with fewer responsibilities that will allow her to have more time to focus on her young kids during these crazy times.)
The following is the shortened text version of the interview and the unedited video can be found below.
Abby: How would you describe what you do to a 2-year-old?
Karin: I’m measuring mental health. Imagine we make drawings of how a person is feeling – think clouds, sunshine, or thunderstorms and we draw all these little emojis a person carries inside – ☀️🌦🌥🌩🌪 sad, happy, tired, or grumpy ones and how they change. The more detailed the drawing is, the more a person can understand what's going on inside herself, and the better she can do something about the emojis that are not “okay”.
Earkick app features: Intuitive anxiety level slide bar & auto-complete symptoms repository
Abby: Understanding what's going on inside of you is a hard journey. I've got a 4-year-old and one of the things we do every morning is I lay out all of “the emojis” in front of her, and I ask her, “how are you feeling today?” so she can select and start to identify how she's feeling. What has made you so passionate about mental health or trying to understand and measure it better?
Karin: I've always been interested in what goes on inside.
My co-founder and I repeatedly witnessed close relatives, friends and colleagues slip into mental health disorders without anyone noticing. Mental health disorders hide in plain sight. It's frustrating. People's lives are tragically impacted and you keep wondering who's next and why we didn't see this coming. It can be any of us and we dive into that question knowing that there are 8 billion people on the continuum of mental health.
We realized that there's a huge gap because there's no objective measurement. You can't manage, let alone improve what you can't measure. We say it's like wanting to keep a healthy weight and knowing thousands of diets, but having no scale to access the progress and to know where you're at. We, and I especially, want it to be that scale, that accountability piece for mental health so that we can improve things at scale.
Abby: We're so used to tracking things like our BMI, our weights, our resting heart rate, and we do those every day with wearables, but there's nothing out there from a mental health perspective. Especially today, as we're dealing with everything in the pandemic and so many of our interactions are limited to a Zoom screen with others. We know that there's severe burnout. There are tons of anxiety and stress today that we're not seeing in others or able to pick up on because we're not even in the same room as them, let alone how it was even pre-pandemic.
Karin: Yes. It's such a sad thing to have all this data being collected somewhere and not putting it together like in the drawing I described. There's huge potential in getting this going, and that's why we decided to found Earkick.
Abby: What put you onto this idea that hearables were the way forward to start measuring mental health? I know that Earkick is built around your master thesis that was on this, but where did that start?
Karin: It started because we are living in exciting times. Technology is advancing in leaps and we have all this knowledge about what can be done with physiological signs. We have a lot of knowledge about a person's journey when they have a mental health issue, but what's missing is an intervention or support that is not episodic, that is ongoing, continuous.
Here comes the technical part – hearables are part of this. The soon-to-be-seen trend of augmented reality, be it glasses or... Hearables are basically nothing else than these cute little earbuds, but with more functions, way smarter.
© Torben Stephan
A person with a mental health disorder benefits from early detection of a developing issue from a full understanding of physiological signs because this can prevent unnecessary suffering on a big scale. This applies throughout the whole journey. It's before they develop something, while they try to seek help, during treatment, but also after treatment. It's this ongoing knowing where you're at. People with phones and wearables are already collecting data.
Hearables would take it a step further because in the ear you have amazing opportunities. You can measure not only heart rate, temperature, and movement, you collect voice, even brainwaves, but you can also feed people with the perfect intervention. You can be there for someone at the moment and give them actionable suggestions that are tailored to their history, what they're doing in therapy, their environment, and where they're at. This makes hearables a perfect companion. It's like having this trusted little companion sitting in your ear that gives you a second opinion, that has your back on top of everything else that you can have on a mental health journey.
Abby: To read people's vital signs and do the sentiment analysis of their voices and of their faces, you are using state-of-the-art machine learning and taking a multimodal approach to track mood and mental health.
Earlier in our Women Leading Visual Tech interview series, we spoke with Rana el Kaliouby, who is co-founder of Affectiva, and she says that today, AI is like a toddler in terms of sophistication and understanding of human emotions. Can you tell me a little bit more about the AI that's powering the tools to listen to people's voices and understand not just what they're saying, but how they're feeling?
Karin: I love the toddler analogy. Never in the life of humans, we have had a steeper learning curve than in the toddler age.
I'm convinced that we will have tremendous progress in the AI, emotion AI, voice, and visual biomarkers in multimodal approaches in the coming years, so it's exciting to be working in that field.
Earkick app features: Voice memo instead of typing
You can think of spoken communication and facial expression as the windows into your mind and your soul. Voice and facial expression are not only highly ubiquitous biomarkers, but they also carry rich information about how we are doing. Our mother, for example, would look at us as a child, at our faces, and listen to our voices to detect and identify changes in our well-being, for our own good, and AI does the same.
It is especially helpful in detecting and monitoring incremental changes. It's the mass of rich data, putting it together like an engine in a smart way that you get the full picture, like the drawing about where somebody's at and where they're heading towards.
People suffering from depression speak slower and they pause more than a healthy individual. People with anxiety show tiny changes in their vocal tone. AI can detect that. Once captured and processed, the data will help to define a personal baseline and monitor trends.
Abby: It sounds like you've got the vocal tone and sentiment analysis. What other things are you tracking today?
Karin: The environment and things that people do also affect wellbeing. They could be doing a lot of physical exercises and have mindfulness sessions.
The current version of the app is focused on anxiety and stress. It tracks heart rate, especially when you're doing a breathing exercise. It tracks the voice, facial expressions, typing patterns, mood, anxiety levels, and panic attacks symptoms.
Earkick app features: Set personalized goals
It tracks various symptoms – insomnia, increased heart rate, restlessness and many more. The app also tracks daily goals or healthy habits that you try to introduce to your lifestyle.
Weather and location tracking has been added to show correlations between them and a person's mental health state. Location is also used to let a trusted person know where you are when you press the panic button.
HRV (heart rate variability), and movement will follow soon.
Some of it still needs manual input at the moment, but with every member that joins and every iteration that we make, we get closer to that full automation that we envision.
Abby: One of the big questions that I'm sure that you get all the time is: what are you doing with data? Outside of advising your AI, where does that data go? How is it protected and stored today?
Karin: Computing at edge is more privacy-friendly: only zeros and ones leave the phone. There are no hearables (yet). You can connect Apple Watch, but everything is done on the phone. No login is required.
Earkick app tracks mood swings, anxiety levels and other symptoms like irritability, insomnia, or increased heart rate on a day-to-day basis
We're Swiss. We like privacy. It's important that people feel safe. If they don't say the truth to their therapist or a family member that's their thing, but they should in the app. They should be like looking in the mirror in the morning and saying, “Hey, how are you? And this is what I feel. Those are my thoughts. That's what I'm going to do. That's what I did”. They should keep it like a rich journal that is honest and that will inform our engine and our technology to the best.
Abby: From a long-term vision of Earkick, what do the next five years look like? What are you hoping to be able to accomplish with Earkick?
Karin: A lot. In 5 to 10 years, our technology will enable individuals and organizations to shorten the time between the onset of a health disorder and a suitable individualized intervention. It will empower them to manage their mental health and well-being, as they manage their physical health, from anywhere and at any time.
They will enjoy continuous support, rather than episodic and reactive, unsustainable care. I believe, services and products will only be reimbursed and paid if they show efficacy and desired outcomes, and if they're data-driven, rather than what we're doing right now is relying on trial and error approaches and counting the hours that we spend in therapy. In short, Earkick will help humanity save time, money, and suffering, and will help to prevent future mental health epidemics like the one we're going through now.
Abby: Mental health today is not only subjective based on the therapist you have and what you tell them, but it's done only incrementally. Earkick is going to revolutionize the way in which we approach mental health to make it objective and to make it real-time applicable to real people in your daily life.
Karin: It will make it accountable too. Accountability empowers quality. Yes, you need to measure and put data, and that makes it accountable for a lot of players. It also opens the door to improvement because then you see what works and what doesn't work on an individual level but also on a general level. It's not looking back in hindsight knowledge or gut feeling, it's something that is proven and fed on real-time, real people's data.
Abby: We know that there's been a tremendous amount of burnout in the past two years. What was it? Over 21% of people report an increase in burnout and physical symptoms of stress-related simply to work. Why is it that you're focused on work? Is it to take on that aspect of it? Or why do you think that this is a great place to solve this by going to businesses first instead of direct to consumers?
Karin: All the avenues that we will take with measurement are helpful, but business – their complete productivity – depends on that. And it's not just the health of individual members of the workforce, it's the whole workforce that can be completely imbalanced.
There is only one more thing needed, like the pandemic or another challenge, and the whole thing can go south.
We recorded this interview in late January and a month later, we all heard about Russia’s invasion of Ukraine. As a result, Earkick tracked increased levels of anxiety in people around the world.
As a way to help Ukrainians, Earkick’s team released their app in the Ukrainian language. It’s now available in the AppStore. Read more and discover a new feature – the ambience mixer that was developed by Earkick’s Ukraine-based team member.
We believe that individuals and organizations would profit from having ongoing measurement and support. When the waves are not as high, so you don't go down the rabbit hole as a company or as an individual, and you don't have to recover all the time, you can keep ups and downs much more balanced because you have intervention in real-time. You have real-time help, knowledge and insight, and there are many things that can come out of this. Other technologies and business models need to have that scale and accountability.
Abby: As I think about your line in terms of recovery, does anybody ever recover from mental health issues, or do you think the reason why Earkick is so profound is that it's going to work with you over the course of your entire life, as you go through the ebbs and flows of your health journey?
Karin: People do recover from mental health issues, but not from all. In some severe cases, people have to live as well as they can with given resources.
What matters is that people realize early on how many possibilities they have to live a fulfilled and balanced life (even with a mental illness). It has a lot to do with psychoeducation (with knowing about oneself and about one's condition). It matters to have the right tools in the right moment, in the right surroundings, and the right amount of knowledge. You do not have to go all the way down the rabbit hole to learn about yourself.
There are things in life that you can’t prevent. You can have terrible incidents and triggers, but having the right approach that works for you, whether it's in a company or it's an individual, will make all the difference. There is all this knowledge out there, there is all this content, all these amazing therapies and people, it needs to be put on a scale and be individualized. It needs to be precision intervention rather than putting everything in two buckets and giving everyone the same line of treatment.
© Jasmin Frei
Abby: You were managing director of Impact Hub for a long time. I can imagine in your role as a startup advisor in the founder community, burnout, stress, anxiety and imbalance have always been the norm, and so I'm sure that that's like one of the places where you've seen such a need for this arise out of is how do you maintain a work-life balance in startup land and as a founder?
Karin: It's challenging, especially because you are driven, you want to do this, and then you don't notice how you overwork. You don't notice how pressure from all sides starts to pile up. It boils down to those incremental little things that add up and suddenly, you're in the rabbit hole.
My co-founder and I are both runners, and we see building a company as a marathon and not as a sprint. We leave space for recovery. We leave space for self-care and for meaningful connections that have nothing to do with work. We leave space for curiosity and silliness to make sure that we're not on that one route, where you're banging and breaking things because again, it's a long journey. And of course, we track our physical and mental health. We use Earkick.
Abby: If you had to give one lesson learned to other founders from your entrepreneurial journey thus far, what would that be?
Karin: For me, picking the right co-founder and the right people who back you, in terms of money or resources or smartness was the most important lesson learned. I was incredibly lucky to have found my perfect co-founder – Dr. Herbert Bay – and also to have found the perfect lead investor Evan and the team at LDV.
It's the basis. These are the fundamental things that you need to have in place before you start this crazy journey, because there are so many surprises, so much going on that you can't control. At least you can control that if you do your due diligence if you let yourself take a moment and build that basis well. Earkick is my second company. This is big learning from my first company. I would advise everyone to not rush with that part. You can rush with other things, but not with the people that are going to embark on that journey to the end of the world.
Abby: If it wasn't Earkick, what would've been your alternative career choice?
Karin: I'm in the right place now. Mental health is such a real and global challenge and I believe I can have such a tremendous impact together with everyone else in that mission that I wouldn't want to be in any other place right now.
Next month, we are thrilled to publish our interview with an electrician for the heart – a physician-scientist who focuses on the development of novel treatments for heart failure, vascular dysfunction, and arrhythmias such as atrial fibrillation. Stay tuned!
