PART : 1
My Secret Shopper Experience in Vacation Rentals
Why share this study?
You know that mix of optimism and wonder that hits your brain after the first few sips of morning coffee? That’s exactly how this project started. I was genuinely excited to have the opportunity to conduct research before diving into design. While the idea of dedicating time to research before design is often championed, it’s rarely feasible in practice due to time or budget constraints.
This time, however, I had the runway to truly immerse myself in our customers' experiences—not just through computer simulations, but by stepping into their world firsthand. This hands-on, dogfooding approach was both eye-opening and incredibly enjoyable (after all, we’re talking about vacation homes—how could it not be fun?).
Ethnographic(ish) research Data synthesis Journey mapping Cross-functional collaboration
The quick take (tldr version)
Company
Evolve is a property management company specializing in short term rentals, responsible for all the digital aspects of management such as dynamic pricing, listing maintenance and 24/7 guest support.
Problem
As the company continued to scale, so did the inquiries from guests and this in turn put considerable strain on our customer service teams. We had some rough data on why people were calling, but no one had ever taken the time to review the experience holistically and as such it was difficult to identify big bucket problems.
Solution
I implemented a three-pronged research approach: acting as a secret shopper for our three biggest distribution channels (VRBO, Airbnb, and Evolve Direct), analyzing internal employee surveys, and conducting customer interviews.
Result
After synthesizing the data collected from the exercises listed above it was clear that poor communication, specifically our Evolve emails, was to blame for our customers confusion when it came to preparing for their vacations.
One of the wordy, the trip preparation emails
The full length version
Background
The complexity of Evolve’s business model is the fact that most guests actually book one of our properties on a website other than our own. A common situation we as customers find ourselves in as we purchase more goods through a marketplace, especially in travel. So imagine going to dinner at a friend’s house only to find out they’ve actually hired a professional chef to cook for you instead– delicious homemade ravioli but a bit of an awkward surprise.
Process
My PM and I knew the post-booking experience was broken, but it was unclear what kind of big swings we could take to clean up the mess. There was no comprehensive research done on the end-to-end guest experience. After being granted a small budget and the company‘s credit card (wahooo), we went shopping for vacation properties. We each took turns playing the role of trip planner. Each booked a home, planned an excursion, and prepared various questions to call into our customer service team to mimic what we were hearing anecdotally from customer service.
The day of check-in was a collaboration between the Product and Customer Insights team. We piled into my Subaru Crosstrek, drove across town, and visited our booked property armed with guest-related queries. Keep in mind no one outside of our team knew we were conducting this exercise, so when we called and asked for the pet fee to be removed or an early check-in (both common questions), the person on the other line had no idea we actually worked at Evolve.
Starting in the upper left:
Attempting to request an early check in for the Airbnb rental but was struggling to find the contact number for the host
Calling guest contact about the fallen tree branch
Kyle, a true UX hero, holding a fallen tree blocking the path into our Airbnb home
Connecting to wifi in the Vrbo property
Journey map
After about a month, we had checked in and out of three separate properties. Each one was booked through a different platform (roughly 60% of bookings came through Airbnb, 30% Vrbo, 10% Evolve directly). After cataloging all our data—screenshots, handwritten notes, emails, and videos—I distilled all our reflections back into one comprehensive customer journey map. I chose Miro to easily share among departments but also for the ability to view information at various levels of detail. The final product was presented to the product org as both a resource and appetizer for design work to come.
Problem identified
Once we had our customer journey map, it was easy to see that our transactional email communication was the biggest area of improvement. Oftentimes critical information like check-in criteria or payment reminders was getting lost in our inboxes. In addition, the emails themselves were difficult to read. The content was verbose, type styles didn’t pass accessibility standards, and the email wasn’t optimized for smaller screen sizes. Having identified this area of opportunity, we proceeded to the next phase of our research to incorporate a broader range of feedback beyond our own.
Actual footage of a customer opening our welcome email
Reflection point
At the time, my PM and I decided it was important to weave in feedback from outside our own experience. While I understand the importance of removing your own personal bias out of the design equation, reflecting back today, the additional data gathering was unnecessary given that we had rough call center data and our own qualitative research. For that reason, I’m providing a brief overview of our secondary research process below.
Additional research
My second initiative involved synthesizing responses from an internal employee survey. This voluntary survey consisted of 19 questions aimed at identifying the behaviors and actions that contribute to a positive guest experience. We received over 150 responses from employees who had recently returned from vacation and stayed in short-term rental properties. Bookings could have been made through any platform, not just Evolve.
Check out my completed affinity map
The last stage of our research was to collect feedback from people outside of Evolve. Using UserZoom, I ran six unmoderated interviews with people who had recently rented a vacation property. The themes that emerged were not surprising. We found a similar sentiment to what people in our internal survey reported. Here are a few of the questions asked that helped inform our overall perspective of the guest experience.
How would you describe your style of trip planning?
Most people are medium to heavy planners (5 of 6)
What types of experiences with or features of a vacation rental are most likely to leave a positive lasting impression on you?
Easy access, good communication, clean and well stocked (3 of 6)
What types of experiences or issues with a vacation rental are likely to leave a negative lasting impression on you?
Poor communication (5 of 6)
Building a persona of each participant
Conclusion
While reviewing each set of results, it became clear that communication was the biggest recurring theme. It can easily make or break a guest’s vacation experience. The key is finding the right balance—a "Goldilocks" situation: you don’t want too many points of communication, but you also don’t want too few. Finding a healthy balance was going to be our next challenge.
Instead of revamping our email design, which was owned by the marketing department, I resolved to tackle better communication by creating a fresh experience: sending a welcome text message to guests, which, after authenticating, directed them to a new web-based app, offering all the necessary information for planning their vacation. I’ll share more on my solution design in the next project.