EXPEDIA GROUP TECHNOLOGY — ENGINEERING

Events like a Super Bowl™ commercial don’t happen often here at Expedia™, but when they do, it’s at great expense and effort. The worst thing that could happen is that we get the traffic surge we hope for and our site can’t handle it. For a company that needs its users to have confidence and trust in us, “Any publicity is good publicity” does not apply.

Our team first heard about the Expedia Super Bowl 2022 commercial sometime in Nov 2021. We didn’t panic since Vrbo™ (part of Expedia Group™) has sponsored the Citrus Bowl™ for a few years now, and we’ve come up with ways to plan, prepare and monitor.

There are two major aspects to how we handle big ad events/sponsorships:

Testing

We started running peak capacity tests (PCT) on Expedia lodging pages in Spring of 2021 in preparation for a major platform convergence project, so we already had a head start. Our original plan was to complete testing by the end of 2021 and pause for a month or two before starting back up again. January–February is historically peak season for Vrbo, so we pause PCTs during that time. With the news of the Super Bowl commercial, we changed our plans to resume testing immediately after the year began.

Compared to Vrbo, for which we’ve been running capacity tests for a few years, we were still in early stages of capacity testing for Expedia. We replay a handful of initial GET requests and a couple of POST requests for the pages with the highest traffic and most potential for increased traffic from Hotels.com.

Also, we were only running load tests, where we increase traffic over 60 minutes, then hold for 30 minutes. This is to mimic gradual traffic increases that happen as we head into “peak season”, the time of year when the site experiences the highest traffic levels.

From our experience with Vrbo Citrus Bowl sponsorship and a couple of ads run during other bowl games in 2019 and 2020, we know this sort of event often results in traffic spikes, which happen very quickly — 1–2 minutes from base to peak, with a slightly longer time to drop off, 5–15 minutes. Since auto-scaling doesn’t happen that quickly, we have other approaches to testing and preparing for spikes.

In early January, we learned that the new Vrbo ad campaign was going to debut on Super Bowl Sunday, though not as an official Super Bowl commercial.

There was some chance that ads for both brands might run in close proximity to each other, meaning shared services would get hit with more traffic than usual. We implemented what we call “tandem tests”, running both Vrbo and Expedia lodging PCTs at the same time to make sure those services could handle the increase.

With any big marketing event, we don’t get much detail to start with since the teams are constantly working out details and negotiating changes, right up to when the ads run. We started with some general, high-level information about the ads, CTAs (calls to action) and timing. We had to make decisions based on that information and adapt as more details were available.

We started off with some rough estimates of expected traffic and ran load tests based on that.

We had no idea then what time of day the ads would run, so we were basing estimates on the highest traffic typically seen on Sunday for Expedia. For Vrbo, we were able to get traffic numbers from the 2021 Super Bowl game day.

Next, we made some predictions about expected spikes. For Vrbo, we used the information from 2019 and 2020 Citrus Bowls to determine the spike. For Expedia, we based it on industry reports from other brands running Super Bowl Commercials in the past.

As Super Bowl Sunday drew closer, we were able to shift our testing to focus on the most likely scenarios. Once we got the approximate ad times and knew that the brands would not have ads running at the same time, we discontinued tandem tests.

Originally the Expedia commercial was to run during Q4, when there are typically fewer viewers, but when this got moved to Q1, we refocused on spike testing for Expedia.

Vrbo ads in the US were all to run during the pregame which usually has lower viewership than later in the day, so we dropped spike testing for Vrbo and focused on Expedia.

Another benefit of these tests is that teams get an idea of how their services might fail, and can plan ahead for what actions they would need to take when it happens “for real”.

In the last few days before the game, we worked with a few teams to fine-tune their pre-scale levels. As mentioned before, auto-scaling doesn’t happen fast enough to handle such fast bursts in traffic, so when this is expected, teams will pre-scale their systems.

Our first impulse was for all teams to pre-scale as much as possible, but quickly discovered we’d hit the upper limit of AWS instances. We spent a couple of days running tests with systems pre-scaled and with auto-scaling to determine the “sweet spot” where we weren’t pushing the instances to the point of falling over, but we could still handle a big increase in traffic.

Roll Call

Yes, we have pager duty, but when traffic spikes happen so quickly, and our teams are distributed around the globe, it makes sense to have team members online, actively monitoring during ad events. This way, no time would be lost waiting for people to get to a computer, wake up (for those in earlier time zones), catch up on what’s happened and/or for escalation in the case of someone being unable to respond to the page.

We first introduced Roll Call for the 2020 Citrus Bowl. We were able to avoid outages when teams discovered issues and early and took immediate action.

We know how disruptive this is, especially for our colleagues who had to be online during their night, so we do our best to minimize the amount of time folks need to be online, and break up the times so it’s easy to spread the burden across a few team members.

We also plan on cancelling any roll calls later in the day/evening if there are no issues found during the earlier ones.

We did see increased traffic levels, but thanks to all the preparations, there were no disruptions to any services and no degradation to user experience. In addition, we learned some valuable lessons on how to better prepare for peak season and whatever next big event is in our future.

Learn more about technology at Expedia Group