Open Hours: An Exercise in Building and Learning Quickly
Open Hours is a new project by Rohan Pavuluri and myself. We believe phone calls are one of the most powerful and meaningful ways to stay in touch with the people we care about.
But phone calls don't happen as often as we'd like because we don't know when people are free to talk. Social norms prevent us from calling the people we want to talk to out-of-the-blue. Our mission is to make them more common, fun, and spontaneous.
So we built an app to make that possible. Open Hours enables you to let your friends know when you're available for a phone call in real-time. Flip a switch and your friends get a notification that you're free to chat. We made it for people to use when they're bored, lonely, or feel like talking to a friend.
Timeline
November 16 [Private Beta]
We launched our first private TestFlight build to a few of our closest friends — the type of people we call regularly and hypothesized Open Hours would be most useful for.
This version was raw. We had to walk our friends through signing up for a new account because they couldn't enter their phone numbers without finding a bug. We would teach our friends to force-quit the app when it would crash every time they tapped the button to sync their device's contacts. Yet, our core functionality — a notification to let your friends know when you were available to chat — worked perfectly and that was compelling enough to start learning from.
Rohan and I started using the app to talk nearly every day, constantly prioritizing the most critical elements we'd observed in our respective social networks.
November 24 [Public Beta]
We reached our 6th private TestFlight iteration and deemed it stable enough for our first (small) public release. We'd iterated on the interface to mark yourself as "available" and resolved many of the pesky bugs around signing up and syncing contacts.
We used Instagram Stories to start seeding Open Hours within our broader social networks. Despite hundreds of views, tens of excited comments, and even more website clicks, we'd only managed to attract 22 new users to the app.
This release also uncovered a few fundamental flaws. We'd realized our core network effect-creating features — custom links to invite friends to Open Hours — didn't work. And while the app wouldn't crash, most users wouldn't wait for the 30 seconds it would take to sync their contacts before getting bored and leaving the app.
December 11 [Public Launch]
Rohan received an invitation to join Clubhouse and invited me onto the platform as well. We thought they'd nailed their onboarding and used it as inspiration as we refined every part of Open Hours.
We helped our users build out their networks as a core aspect of the onboarding flow. Speed was also no longer a concern for our users as we found ways to sync contacts within 6 seconds and within the background as they completed other onboarding tasks. We made it easy to add new users as friends when they joined the app and added notifications to remind our users of Open Hours when we thought they had downtime.
We were jazzed. The app felt good. We resolved a lot of the usability issues we had found in the weeks before. And so we launched on HackerNews, ProductHunt, and LinkedIn.
Crickets.
December 13 [Pivot]
We built, tested, and iterated on our first experiment in a few very dense weeks. Rohan and I still needed to communicate with each other nearly every day and used Open Hours as an alternative to scheduled meetings. Our networks, on the other hand, would rarely return to the app.
By now, we had a large backlog of ideas we had to improve the core functionality of the app. But as we spoke to our friends that started using the app, we realized our approach to Open Hours didn't make phone calls more common, fun, and spontaneous. We weren't solving the problem. Here's what we learned:
Learnings
Network density is a pre-requisite for enabling spontaneous phone calls. Open Hours' notifications increase the chance of successful phone calls, but each user needs to have friends on the network for it to matter. Our users need to invite their own friends, which didn't happen.
And despite having a large network of friends, there would be a less than 20% chance that our users would receive a phone call. Our users didn't have friends who are always free or willing to talk. Mentees didn't have questions often enough.Creating habitual behavior requires a single-player mode. For Open Hours to be successful, we needed it to be top of mind whenever our users felt bored, lonely, or like talking to a friend. And we needed them to have a way to be instantly gratified, but couldn't rely on their friends calling them.
Many consumer products — Dialup, Honk, Roadtrip, Houseparty, Instagram, etc. — face this challenge and have found clever mechanisms for working around it.Missed phone calls and text messages to coordinate conversations are "good enough." We crafted Open Hours to be most useful for close friends — the ones that wouldn't mind being notified anytime their friends wanted to talk. The app wasn't providing the 10x value it needed to be a better alternative.
Separate use cases require separate thought. We named and crafted the Open Hours app to be use-case agnostic. We wanted to explore social, academic, and business use-cases from the start. Building without a singular target customer in mind, however, makes learning difficult.
What's Next?
Our first experiment in making phone calls more common, fun, and spontaneous taught us a lot. We're going back to the drawing board, this time focusing on facilitating one-on-one phone calls within existing communities.
Online forums like Slack, Facebook Groups, and Reddit keep communities connected, but don't lend themselves to deep one-on-one conversations. Members are required to keep returning to these forums to participate and even then, only a subset of the community actively contributes while the rest consumes. They can still make you feel lonely.
And while these online forums are great for some communities, we're excited to explore how one-to-one matching and audio can better meet the needs of other communities in a remote environment or supplement the needs of existing communities.
We're looking into:
Workplaces. How can we help remote employees feel more connected with their companies? Can we break down organizational silos and improve information exchange through structured conversations?
Social communities. What conversations can increase the value of a community to its members? How can we help deepen relationships beyond what's possible in a Slack or FB group?
Elderly companionship programs. How can we unite volunteers with members of elderly communities who might be lonely?
Check us out at www.getopenhours.com. Subscribe to my mailing list to stay in touch with future updates.
Special thanks to Hoyd Breton who’s been a beacon of support through this journey.
pee(k)
at the end of my time at USC, i was in a class where i was asked to come up with an idea for a startup and pitch it to prominent angel investors in LA. to date, this is the most interesting idea i've ever come up with. and yet, i've done nearly nothing with it. i'm hoping to use this piece to outline where i am and what i'm hoping to do next. let's start:
the problem
i have a 76 year old grandmother who lives alone. she copes with it well, but sometimes she'll forget simple things like going to a doctor's appointments, eating the right healthy foods, or telling others when she's showing signs of symptoms of being sick. this makes it hard on my mom and uncle, who have to check up on her every once in a while to make sure she is healthy and getting the proper care she needs. and it's especially difficult because my mom lives in Indiana and my grandmother thousands of miles away in India.
my mom wants more frequent, up to date, and accurate health information about my grandmother than she's currently getting through infrequent phone calls and WhatsApp messages.
the idea
that's how i came up with pee(k): an internet-of-things (IOT) module that attaches to a toilet and analyzes one's urine for symptoms of disease and general signs of health. this information is then made be available in a web and mobile application that you or those who care for you can use to track your health. pee(k) will even send recommendations, so you loved ones can improve their health over time.
the target audience
pee(k)'s target users are senior citizens, but the target customers — those paying for the device — are caregivers to the elderly. these are unpaid individuals involved in assisting the elderly with activities of daily living or medical tasks. we'd specifically be looking at caregivers who are the children of the elderly.
this initial market is particularly interesting in that:
a typical caregiver spends $12,700/year to provide care for an adult over the age of 50 if they live over an hour away from the care recipient. there's a lot of money and time being spent here and everyone is aching for a better way to do things. [source]
a UTI is the second most frequent infection in long-term care facilities. limited mobility among its residents makes it more difficult to gather urine samples to run frequent diagnostic tests. [source]
a potential business model
the total addressable caregiving market is 470 billion dollars in the US. there's tremendous spending and investment here, which will continue to rise as the baby boomer generation needs more and more help. [source]
a business model approach i've been excited by is one that whoop has pioneered for specialized fitness devices: give the device for free and charge a monthly subscription for access to the data.
it is currently unclear exactly how much consumers would be willing to pay for this information (read: i need to do more surveys), but i believe pee(k) can charge a premium for the depth of data it would provide. let's assume that is $30/month (equivalent to whoop's price). further assuming the unit economics of the device is $100 and a consumer uses it for 2 years (akin to the time a user would keep their apple watch or fitbit), i would expect an LTV of $720.
the space
i started working on this 3 years ago, but paused as my time at USC ended. there's been a lot of innovation since.
i view the space along two axis: target audience and depth of data output. unaliwear (raised $7.1 million) and carepredict (raised $19.7 million) stand out as companies that create fitbit-like devices for the elderly — meant to track their movement and not anything more. the others are far more interesting:
scanadu — it was a venture-backed private company which raised over $56,000,000. in 2014, it developed a urine test kit that enabled a consumer to analyze their urine with a dipstick and a phone. the test received FDA approval, but the company did a number of shady things and it went out of business. [source] [scandal]
toto — in 2005, toto, the world's largest toilet manufacturer, launched their intelligence toilet. it was a toilet version of pee(k). for $3,500, anyone could have a toilet in their home that would analyze sugar levels, check blood pressure, body weight (from the toilet seat), and even hormonal balances. it never took off, although i believe this was from poor timing and high price, not a lack of innovation. the wearable trend and smartphones weren't a thing yet — it would have been far more successful now. but for reasons mentioned earlier, a new toilet introduces more friction than is worth — i wonder how this would've performed as an attachable module. [source]
panasonic — in 2018, they also built an entire toilet to do all these fancy urinalysis things. the toilet also randomly included an armrest to measure a person's body fat. and the flush included a fingerprint sensor to identify users. lots of cool concept ideas. on a random note, urine itself can be used as a digital fingerprint — no fingerprint sensor necessary. it's unclear what has happened to this toilet since its initial press rounds. [source]
toi labs — this is the most similar company to pee(k) i could find. they've also focused on the older customer niche. their product, trueloo, takes a picture as you use your toilet to "analyze urine and stool for abnormalities in color, consistency, volume, and frequency." neat. [source]
where i am today
i've started to validate the broad idea and nearly everyone i've spoken to has been fascinated and eager to learn more. but that's it.
to date, i've done more speaking than taking action. there's soundness in the idea, but does this solution actually work? there's a lot that remains in assessing the viability of pee(k).
my challenges
urinalysis is a common test to perform and the easiest way to do so is with a dipstick test. the modern-day version of that requires a consumer to record a video of a small chemically-coated paper strip changing color as it reacts to their urine. this would be easy to use, but having to reload cartridges of paper strips voids the device's ease of use. various universities have been developing lab-on-a-chip devices that can analyze urine for very specific conditions. while cheap to produce, it is unclear if multiple units can be placed together that each analyze for specific conditions, which can altogether beam their data for processing on a mobile device.
any sort of medical device like this — especially if it offers health recommendations, needs FDA approval. by building upon existing FDA-approved lab-on-a-chip's, the cost of this may be reduced to $500,000. that's a lot of frickin money. in the short-term, pending lawyer consultations, there may be a path where this device doesn't provide the results of what it observes — instead, just indicating to its users "you might want to drink more water." the nuances are tricky.
next steps
i'd like to have conversations with urologists, gastroenterologists, and the folks working on existing lab-on-a-chip devices. i've identified a few at Columbia University here in New York to start. by the end of May, i hope to have a much thorougher understanding of the procedures and technology at play here.
a faq
q1: why would you make this for senior citizens? i can imagine so many more use cases!
i have no doubt in my mind there are far more use cases for such a device, but this entire concept needs more validation and seemed like the best place to start. the wearable trend, in particular, has popularized a notion of health and fitness metric tracking. pee(k) offers a key advantage in that users don't need to remember to charge them or put them on. this would be a game-changer for nearly everyone who cares about their health. and for parents, pee(k) provides a non-intrusive to track their childrens' health. for environmental scientists, pee(k) can offer insights into the type of waste flowing into our sewage systems. i'm certain there's a longer list of use cases i haven't even begun to think of.
q2: a module? why not a toilet? why not a toilet seat?
when is the last time you purchased a new toilet? unless you recently remodeled your bathroom or purchased a new home, chances are your toilet has been around for a while. and even though replacing a toilet isn't expensive, no part of the process is glamorous or easy. as a result, pee(k) must be an attachment to a toilet — it must be an extension that anyone can install.
q3: why even toilets? and why urine?
no matter who you are, one of the things we have in common is our need for a toilet — beyonce and i both urinate; Queen Elizabeth and i both poop.
and yet, toilets have not evolved in centuries. in the United States, toilet-seat bidets are the hot new things as toilet paper vanishes off store shelves. the origins of bidets date back to the 1600s. the flush toilet was the hot new invention before then, assumed to have been invented in 1592. and since then, not much has happened.
the most recent widespread change in toilet technology may have been forced by the 1992 U.S. Energy Policy Act which required toilets use only 1.6 gallons of water per flush — a far cry from innovation. as a result, toilets became more efficient. the shape, the materials, the technology — nothing really changed.
what we put in toilets hasn't changed either — it's urine and fecal matter (and other products, but we'll gloss over those). we flush away some of the most valuable information of our health and don't think twice about it.
urine alone can be used to start diagnosing UTIs, health problems, kidney problems, diabetes, liver damage, blood disorders, and infections. for many of these abnormalities, a urinalysis test is not a diagnosis, but is a vital tool before more specific follow-up testing.
its less sexy cousin, feces, are strong indicators of digestive health. observing changes in their size, shape, and texture can reveal signs of infection, digestive issues, and cancer.
but fecal matter is chemically complicated and somewhat disgusting. with pee(k), starting with urine felt like the right idea.
other content on the topic
does this pee(k) (yes, i'm hilarious) your interest? if so, drop me a note.
on Universe
NOTE: I am a product manager at Squarespace working on mobile apps. The products I work on exist in a similar domain. There’s bias here and lots of it.
Wix, Wordpress, Weebly, GoDaddy, and Squarespace dominate the website builder industry. These titans have existed since the early 2000's and have led massive transformations in web design and truly democratized website creation. In doing so, they have amassed a massive subscriber base. Technology has massively shifted since then, and while each titan continues to iterate on their mobile offerings — they've fallen behind on the mobile curve and what it means to offer a true "mobile-first" approach to website building.
Universe is a mobile app that enables user to create websites using their mobile phones. And it's been pretty damn successful at that. They've raised $17.3 million since their founding in 2014, supporting over 400,000 websites.
a Senor Tower report for Universe
This is an analysis of the Universe app, broken down by overarching key areas I had thoughts on. It exists solely as a personal learning exercise and does not remotely cover everything. Feel free to reach out at saraswatayu@gmail.com or on Instagram @saraswatayu if anything — good or bad — catches your eye here.
1/ ONBOARDING
WINS
Getting started with Universe is SO easy. A variety of options that cater to a broad range of users ensures there's something for everyone. With nearly each step, a directed flow makes picking a template easy. And the template previews load quickly, making it clearly exactly what you're signing up for.
Universe makes accepting payments particularly easy. The store set up process takes less than 30 seconds. There's been no easier way for a micro-business to get started selling online. Props. More on e-commerce further down in this blog.
Further down the funnel, Universe introduces the grid-based editor through a series of directed prompts. They're short and easy to act upon. And holy smokes — the education here really is key. The editor isn't the easiest to get a grasp on but boy does this onboarding process make it a whole lot easier to understand.
MEHS
I'm obsessed with Instagram — it's one of my favorite products to use. As such, Connect with Instagram is an enticing option, but Universe doesn't enable me to explore what that might be at all. Even as a paywalled "PRO" feature, I'd expect more information than Universe's "Instagram Magic Builder". I would expect this to be one of the easiest ways to get started: bake your existing Instagram brand into a website that represents you or your project. It's a shame Universe doesn't make this free and add upgrade paths further down the funnel.
2/ GUIDES
the "Build Your Mailing List" guide in Universe
WINS
Universe continues to build out a collection of some of the most amazing guides I've seen in a mobile product. Building a website if oft-complicated, forcing most website guides to become large text blobs. Distilling this down to short, animated tidbits is incredibly effective at conveying what is possible and how.
MEHS
Guides make it easy to set up a website on Universe, but it isn't clear how to make these websites successful. The success of a website builder is dependent on ensuring the websites its users build are successful. I'd love to see future guides that lean on tools Universe provides to help users make their websites more successful. Adding support for additional tools that fall into this domain (i.e. SEO management and other marketing tools) also may be increasingly important as Universe expands into selling.
Some other lingering curiosities:
The omnipresent "Get Started" action makes it easy to directly jump to the editing experience, but the experience it redirects to starts out jarring. I'd be curious why this part of the experience doesn't continue the guided journey that exists through other onboarding processes in the app.
Maintaining recorded content can be incredibly expensive as new features are introduced and core UX patterns evolve. References to past interaction paradigms are prevalent, but easy to ignore. I'd be curious to see how this evolves as Universe scales rapidly and taps into different customer bases.
3/ EXPLORE
WINS
I enjoy being able to browse a catalog of Universe websites. And to search through them with a topic in mind is fun. And that's where it ends for me. There's tremendous potential here, it just hasn't be capitalized on.
MEHS
There doesn't seem to be any curation in the way Universe depicts the websites on its platform. Combining an element of storytelling, or narrative videos, or some other information about the websites presented will make the experience feel more relatable than just an endless list of websites.
It'd be interesting to vet Universe's appetite for enabling their users to duplicate websites they found interesting. And because content theft will likely be a concern, at least enabling the duplication of styles or layout would make this experience more actionable & purposeful.
Lastly, explore is also used to advertise their premium subscription. This is their least compelling value proposition and makes the product feel cheap. It may have been a growth driver while the platform was small, but I don't see a future in which this bodes well for the company.
4/ E-COMMERCE
WINS
Universe optimizes for speed of getting started, and it feels most prevalent when setting up an online store. With Universe, payments work automatically and Universe collects the money for you until you're ready to move to a more widely-known payment processor. Contrast this with most prevalent website building titans — Wix, Squarespace, and Weebly — that force users to set up a payment provider to start accepting payments. Wix and Weebly have first-party offerings in the form of Wix Payments and Square, respectively, but even those require a separate configuration process. This is a BIG deal.
They also haven't invested in building out an inventory management system, rather ensuring they're leverage third-party APIs to deliver a store set up experience that's still brilliant. Special kudos to offering Square Cash — you don't normally see that, but seemingly everyone has it and is an easy way to get started if you're just starting out.
MEHS
10% cut on payments. Might be acceptable for people just experimenting with online selling. Luckily, it's free for the $1000. For people that are able to validate their ideas successfully, I can't imagine this fee will encourage them to stick around for longer.
Universe relies on the Shopify platform to power its Commerce block. Square also caters heavily to the "just getting started" audience Universe does — I'd expect that to be a better partnership. Seamlessly integrate with Square's core business — physical selling — would be extremely powerful.
5/ MOBILE WEBSITES
Featured Universe websites: Taka Enoki, Adam Watters, Klub Kid Records, Frida
WINS
Universe makes mobile-optimized websites and it shows. The approaches they take to subpages is interesting — paging content horizontally is easy to navigate on mobile.
A responsive grid layout, customizable with up to 5 boxes per row, makes it easy to build a website. Drag, pinch, and panning gestures are easy to learn and the instant updates make building a breeze.
MEHS
Universe deems itself "the easiest way to make a website". And while it's super easy to get started, the websites I've been able to produce are far from what I'd consider a "good website".
Let's break that down:
The options to navigate between pages on Universe isn't easy to customize. Nearly all examples of a navigation drawer I've been butcher any attempt at being pleasing to the eyes.
Customization of website content is too detailed. There's a lot of visual clutter when trying to build a website, especially when starting with a template. I wish the templates had an "advanced" mode, but defaulted to a more restrictive setting — it'd make getting started easier. Onboarding content creation for the templated pages may also help. For instance, a guided experience to name the product, add descriptions, and set the price would make setting up a store page WAY easier.
And lastly, an important nit:
Universe websites are mobile-only. There's no tablet or desktop view and no ability to tweak any experiences to be better on a larger screen. This reflects in other design choices such as horizontal scrolling, which in Universe's case only works with swipes. Swiping doesn't work at all with a mouse. Even if optimizing non-mobile experiences is out of question, content shouldn't break.
6/ “PRO”
WINS
All-in-one. Universe pricing is simple and upfront. Over the last few years, they've transformed from offering modular products (i.e. domains + websites separately) to a single annual subscription that bundles a custom domain, an email address, an "advanced" website, and analytics. With "PRO" displayed throughout the Universe experience, the messaging is clear and easy to act upon.
MEHS
Annual or lifetime subscriptions only. The flipside of a subscription that includes a custom domain is that there's significant upfront cost for Universe. And as a result, it makes sense that the only viable business offering be at least one year in length. But... a starting price of $120 is just a lot.
Two recommendations:
"PRO" trial period. The majority of Universe's functionality is inaccessible to free users. I would imagine opening up a short trial period to test out all the blocks and analytics tools may help a lot with lower-intent users. Downside risk of offering a trial seems low — I certainly can't imagine a significant number of users would take advantage.
Offer a separate monthly subscription. There's certainly a world in which the LTV of a user is high enough with a monthly subscription to be more profitable. Focusing on longer-term retention by creating automated social tools (i.e. offer configurations to automagically update the page when there are new Instagram posts), or connecting email lists directly Shopify inventory notification, etc may help offset the risk here.
7/ SPEED
WINS
It feels really frickin fast. There's barely any loading anywhere. Even videos load quickly. And when there's even a hint of loading, I appreciate the attention to detail paid to the loading states. Props to the team on this one.
MEHS
Large websites on Universe aren’t the fastest things in the world. That said, I’ve only come across two websites that felt slow, Klub Kid Records among them.
Overall, I’m jelly. Incredible work Universe.
CONCLUSION
Universe is an incredibly polished tool for making mobile websites. There's tremendous opportunity that lies ahead of it, especially with it being so far ahead of the pack in mobile-first content. Its biggest opportunity is also its newest: payments. With a few tweaks and experiential polishes, it'll be a dominant player for people who want to test small ideas in no time.
the splitty chronicles
note: splitty is available on the App Store. check it out! since it’s also COVID season. splitty won't entirely be relevant until the world opens back up for business. but i thought i'd write some up thoughts anyways.
i've long had a fascination with food. i used to LOVE cooking. over the last few months, that's transitioned into going to restaurants with pals, checking out the bars, and dining at some random spots all over the world.
a meal in Cambodia with some pals
as a not-yet baller, recently-graduated college student, each dining experience introduced one complexity that perpetually made things a bit awkward: paying the bill. for a while, we'd split the bill equally. but introduce dietary restrictions and substantially different meal prices, drinks, or appetizers, and we had to iterate. we'd have one person in the group pay, and then do a bunch of back-of-the-napkin math to figure out who owed what. the person paying would be left to do the grunt of the work and just hope they'd get their money back.
after a while of dealing with this nonsense, i discovered Plates by Splitwise and it was frickin game changing. no more complicated math — instead, a cute lil app that made splitting the bill after a dining experience EASY. with a lot of inputting numbers and dragging items around each person would know exactly how much to pay back and dining was fun once again.
using Plates by Splitwise
so why make splitty?
the weird thing with Plates was that most of the work in splitting the bill was just having to input all of the items into the app. why couldn't it just take a picture of the receipt and make things easy? i'd eat out at least twice a week and this really started to get to me.
i couldn't stop thinking about fixing that. so i decided to. it turns out, that was also the hardest aspect of the experience to improve. more on that later.
my vision was simple. hold your camera up to a receipt and voila — things just work! i wanted it to work like scanning a barcode works. i didn't want to take a picture of the receipt. i wanted it to be smart enough to know if a receipt was present and take the picture by itself. and afterwards, i wanted splitty to know everything that was on the receipt.
that was the part i enjoyed most about Plates. the moment you added an item, you would assign it to a plate by a super elegant drag. and when you shared an item, you would drag it to a communal plate and tap on other plates to assign it to the group. as i used it more, i realized i was sharing a whole lotta food. and therefore, i wanted splitty to just split everything among everyone, but also make it really easy to assign items to smaller groups of people. oh and also, while we're at it, i wanted splitty to make it easy to charge people on Venmo right there and then so it's easier to get your money back!
to me, this was the minimum viable product. it's what i had wanted, and i'd asked around and my friends seem to love it as well. a million more ideas had come to mind — why not pull in photos of your meal from the restaurant's Yelp profile? why not have splitty automatically know who you were eating with so you could charge them automatically? why not make a Venmo, but specifically for splitty, so the experience could be so much more seamless? or integrate it with expensing platforms to target businesses? or even just make a version of Splitwise i could be more down with? those ideas would have to wait though. they were untested, and first and foremost, i just wanted to ensure splitty was a better experience for me and for anyone who used it.
building even this tiny version of splitty was the most complex challenge i've taken on over the last few years. let's jump back to that question i would continuously ask myself: why couldn't it just take a picture of the receipt and make things easy? as i worked more on splitty, i realized the answer was a helluva lot more complicated than i had anticipated:
let's start with receipts
i've always found receipts to be straightforward to read. there's some information about the business on top, a list of items and their prices below, a section on the amount you owed, as well as payment information towards the bottom. some receipts even had a few promotions, all below the important stuff.
an average receipt
and while that's mostly consistent, did you know there's a multi-hundred page document that governs what you see on a receipt in the US? on top of that, each point-of-sale (POS) system has a different interpretation of that. the end result: receipts are kinda wack in the US. while they all contain similar information, there are a lot of permutations in the types of receipts that can be printed.
these permutations are similar enough to where they're almost a non-factor to you and me, but they make receipts a whole lot more complicated for a computer to read. and computers reading receipts is something that's happening now more than ever — people turn to invoice tools, budgeting software, and other organizational tools to keep track of their finances. and because of that, there's bound to be progress here. the EU, decades ahead of us in receipt policy, has an entire series of legislation in place for ensuring there's a digital copy of a receipt that's easy to interpret. as Square, Clover, and other more digitally-advanced POS systems are used in more establishments, an increasing number of receipts are going digital, making them easier to interpret.
the future of receipts
longer story short: receipts are complicated. there are a lot of people working hard to understand this domain. but it's still early, and that's part of the reason why you don't really see any tools that can understand receipts in the US very well. i started this project a bit of a dummy, naively thinking it was easy. little did i realize.
how splitty tackles receipts
note: the world of machine-learning is relatively new to me, so i apologize if this all sounds basic.
splitty on the hunt
when you launch splitty, it instantly goes into search mode. it obviously just hunting for a receipt? not quite... it turns out training a model to search for a receipt is a whole lot of work, so it's starts by searching for rectangles. it does a bit of magic to figure out which the most probable rectangle are. as you hold still, splitty takes a high resolution photo of the whole screen and relies on this rectangle information to come up with a cropped image of just your receipt. often, the receipt will be on a flat surface (like a table) and you'll be taking the image from an angle, so splitty accommodates for that as well by skewing the image until it's a rectangle instead of a quadrilateral. thanks to a lot of this being built-in-ish functionality with the latest versions of iOS, starting here was relatively easy.
okay, so now we have an image of the receipt. to breakdown the items on it, we need to understand what it says. for that, splitty relies on a lot of the optical-character recognition work that Apple has already done. they're pretty frickin good at this (but not perfect) and splitty just latches on because there's no chance i was going to make something better. by the way, this is the same type of stuff that enables you to search some PDFs, or Google Translate's live camera translation to just work. as splitty does this, it identifies the full text of the receipt. mind you, this really is just a massive list of words and information about where they are positioned on the image.
splitty knows what's on the receipt, but it certainly doesn't know how everything goes together. does "$15.00" go with the carrot cake or the quesadilla? and because i couldn't come up with a better way, i decided that geometry must be the right solution. it knows where the text is positioned, so it tries its damndest to identify which boxes are on the same line. that way, splitty's interpretation of the receipt will be closer to "Quesadilla $15.00" instead of everything being spread apart. this doesn't always work as some receipts will put this information on multiple lines. screw them. this is good enough.
how splitty understands a receipt. yellow boxes are groups of words it things belong together. red lines indicate groups of boxes that are on the same line.
cool. now we have a transcribed receipt! all we have to do is figure out what the items are. easy, ya? not quite.
there's a lot of gibberish on a receipt and it's hard to make up a set of rules that makes it easy to identify what something is. for instance, this is just a small example of the variation of text that a receipt may have on the item line:
"5 Quesadilla $15.00 T"
"Quesadilla 15.00 5"
"Quesadilla 15 X"
"15.0 Quesadilla"
it's just all over the place! i started by doing what seemingly everyone on the internet recommended — subtotal, tax, tip, and total all were easy to identify because there was a limited number of words that could preceded the amount that mattered:
"Amount Due: $15.00"
"Sub total: $10.00"
and anything else that had a price in it — it's likely to be an item! this approach was straightforward but boi did it get complicated and fragile quick. it turns out, there are a lot of ways to say "total" on a receipt. and because the OCR didn't always get each character correct (t0tal or tootal were common), things just never seemed to work.
this was all that mattered to me. it was the most important part of splitty and it had to work, at least most of the time. and so after countless other trials, i ended up trying to leverage more unfamiliar tools the tech world was fascinated by. i made a cute little machine learning model trained on a few thousand receipts i scraped on the interwebs (this took AGES) to help splitty out. it runs each line of the receipt through this model and the model spits out what it thinks the line is most likely to be: item, subtotal, tax, tip, total, or other. business information (name, location, phone number, email address) is also extracted and captured. the best part: it worked ~98% of the time. a whole lot more than i was getting classifying things with a bunch of rules myself.
the best part of all this: it all happens in a split second. splitty finds a bunch of rectangles in your image and identifies the one most likely to be a receipt. it takes a photo and makes it look pretty. it reads through it and does 6th grade geometry to figure out what goes where. and finally, it works real hard to be like "this line's an item. this line's the total." and that's mostly how splitty works!
what's next for splitty
it's a weird time for splitty. all this COVID talk is certainly not helping me test it more thoroughly and get feedback from the friends that have played around with it. delivery and grocery store receipts receipts only do so much. because of that, i'm going to pause work on splitty to tackle other projects (such as this kicking off this blog) for the next few weeks. and as soon as the world re-opens for business, i'll push to get more feedback around making the experience a whole lot better for splitty's users.
i already have a bunch of short-term improvements in the works i'd love to add to splitty. i want to make it better at the whole receipt detection thing, but the small incremental improvements require a lot of effort. i also want to make it remember where you've been so you can access receipts from the past. and lastly, i want to make the Venmo integration it has a bit better by recognizing the basic folks that you dine out with the most. but i'll pause on that for now and weather this storm with the rest of you.
and that's all. have any comments or questions about anything? or have thoughts on ways to make splitty better? i'd love to hear from you! reach out at saraswatayu@gmail.com or via @saraswatayu on Instagram!