S2 #3: Trailblazing New Content Paths With A/B testing with Christina Stolfo

When it comes to digital health marketing, we desperately need trailblazers who challenge us to do things differently. That’s why I really wanted Christina Stolfo, audience development director at Northwell Health, to join me on the podcast - to show us how marketing can leverage verifiable metrics while also challenging the status quo.


Christina has directed Northwell’s content marketing down an intriguing path. Her strategies leverage data optimization and unique content to help grow their audience base.

Her storytelling career began in New York and Philadelphia where she worked as a producer for a local news stations. During this time, she learned how narratives can impact and influence people.

From there, she moved to Northwell Health where she used her storytelling expertise to create a new form of digital health content marketing. To do this, she uses A/B testing to foster audience development, messaging strategy, and digital distribution.

Christina Stolfo

What is A/B testing?

In the digital space, A/B testing compares two different variables to see how audiences respond. Christina uses this experimentation to optimize Northwell’s marketing strategy so that images, headlines, tone, and topics hone in on the right people.

Sure, it takes some financial investment, but in the long run these tests yield better strategies that give you the greatestROI. Even if sometimes A/B testing delays marketing operations, Christina has found that the extra time expense results in the right content reaching the right people, leading to better brand awareness and consumer loyalty.

Step 1: Define your hypothesis

At the beginning of A/B testing, it’s vital to establish a thesis that you want to verify. Ultimately, this hypothesis should sum up what you want to learn about your audience and the content they prefer.

For instance, Christina used this to validate images before Northwell launched their editorial site. They wanted to find out which connected better - stock photography or real, authentic images. Before the test, they assumed that authentic images would win out, but they were wrong - stock images resonated better with this particular audience.

According to Christina, this is why marketers shouldn’t assume. The stuff that goes viral doesn’t always make sense, and you never quite know your audience’s appetite until you test it. That’s why analysis like A/B testing is so important because it helps you to grow and evolve with your audience over time.

Step 2: Create the variable to test

Christina noted that it’s important to only test a couple variables at a time. For instance, if you test headlines and images at the same time, you won’t actually know which variable influenced the results.

This could cause you to mistakenly interpret conclusions, leading to less effective content marketing. That being said, limiting your variables doesn’t mean that you can’t conduct several different tests to analyze other variables. In the end, your sequence of testing may look something like this:

  • Test the images

  • Test the headlines

  • Test the tone

By conducting tests in succession, you’re then able to get black and white results for each category that can influence buyer’s decisions.

Step 3: Interpret the results

The last step is to use the outcomes to inform your marketing strategy. Facebook especially gives you extra insight into your audience. Even if you target a broad age group, for example, this site will still break down the metrics into smaller age brackets so you can see who was more responsive.

Here’s why this matters. It can help you discover who you should be targeting in the first place. Say, for instance, that 25-30 year olds show very little interest in each test. Why then are you targeting them? This can help you go to scale since you’ve eliminated a whole age group that isn’t interested.

Christina’s used A/B testing to inform other content as well - blog posts, white papers, ebooks (you get the gist). But she really loves running tests on social media because of the comments.

She said that it’s an amazing social listening tool that gives you insight into the conversations you’re starting. In the first place, are you starting a conversation? And, secondly, are you starting the right conversation?

One example she gave was when she and her team were trying to reach caregivers of aging parents. Many women are bearing the load of caregiving. What Northwell discovered (through A/B testing on social media) was that they started a conversation where women were helping each other through experiences and struggles.

For Northwell, this was eye-opening. These women even gave insight into angles of caregiving that Northwell had never considered before. As a result, they were able to create pinpointed content for these types of situations.

How has this been done really well?

Christina mentioned that before they launched their editorial site, The Well, they did a lot of testing. They wanted to take a different approach than other health systems like Mayo and Cleveland Clinic who focus on facts-based blogs.

Northwell wanted to create a first-person style account of medical procedures and health scenarios. In the end, they hoped that patients could relate to and learn from these experiences, making it easier for them to face medical procedures.

But before the site ever launched they conducted a boatload of testing. In this instance, they used Medium, a user-friendly blog platform, to test their content strategy. In addition, they conducted testing on social media platforms like Facebook.

They were totally blown away by the results. The first article, one that walked through the colonoscopy process, generated a lot of conversations. People were sharing tips and tricks that made the procedure easier, and Northwell realized they’d tapped into a very powerful source of content marketing.

Ultimately, this led to one of their main pillars called “So You’re…” Articles cover topics such as “So You’re Too Young For Reading Glasses” and “So You’re Not Feeling Sexy After Baby.” These topics have given people a chance to learn from others’ experiences and share their own.

A/B testing applied in B2B models

Now I hear you, many in health tech aren’t selling to consumers. So how can this tactic be applied in a B2B setting?

Christina said that this type of testing isn’t limited by audience. No matter what your topics are, you can always analyze your audience to learn what connects with them. Maybe the content may not be as funny or lighthearted as B2C content, but there’s always something you can learn from your audience.

What kind of time frame are we talking?

Depending on the budget, you can get insights pretty quickly. Christina mentioned that on Facebook especially she normally has results within 24 hours. If she doesn’t have stable findings by then, that’s when she knows to reevaluate and adjust her hypothesis.

Usually within a week Northwell shifts their budget to the variable that’s performing the best. Christina also noted that after two weeks they normally starts to notice creative fatigue - very few new and interesting insights.

The takeaway

The thing A/B testing has shown Christina is that you always need to be testing and learning from your audience. If you ever think you have them figured out, you’ll start to lose touch with them.

Time changes people, and time changes audiences. That’s why you need to evolve with them and let them inform your marketing strategy.

Must-read book

Christina enjoys reading Ernest Hemingway with her favorite book being The Sun Also Rises. In addition, nonfiction biographies and autobiographies are especially interesting to her. Hearing others’ struggles and experiences is one way she likes to learn from others.

If you hadn’t noticed, listening to and learning from others is a huge thing to Christina. So let’s leverage that valuable human quality in our marketing to connect with our audiences in meaningful ways and encourage brand loyalty for the long run.

Want to connect with Christina? Find her here:

Twitter: @CmStolfo

LinkedIn: Christina Stolfo

Like what you hear?

Please subscribe on iTunes, Google Play, or Stitcher. I’d love it if you would leave a review as well! And of course, tell all your friends!

And, as always, if you need help with your marketing…

Let’s talk. You can book your free consulting session with me today.