Andrew Chen

DECEMBER 26, 2018

Conservation of Intent: The hidden reason why A/B tests aren’t as effective as they look. Everyone’s had the frustrating experience of running an A/B test, seeing a big lift, closing it out, and expecting the top level metrics to move a lot. But they don’t.

Metrics 80

Metrics A/B Testing Transportation Marketing 80

Andrew Chen

AUGUST 8, 2022

These larger headline numbers always tested much better on A/B tests, whether in email marketing or banner form, and while it might feel like the reward becomes unattainable, it’s possible to create a second or third or fourth tier to go along with the big headline number. The Payback.

A/B Testing 94

A/B Testing B2B Onboarding ROI 94

UVA Darden

JANUARY 12, 2023

The star analyst is now a data scientist, a private equity associate is tasked with reinventing a company’s IT instead of engineering its finances, and the marketer is now a “growth hacker” running A/B tests.

A/B Testing 111

A/B Testing Digital Transformation Engineering Onboarding 111

Harvard Business Review

APRIL 18, 2023

Testing your approach with a randomized sample of employees can help you find out what works and what doesn’t.

A/B Testing 20

A/B Testing Inclusion Diversity 20

Harvard Business Review

FEBRUARY 5, 2014

The secret to getting value from data is testing, and if you’re looking to grow your online business, implementing well-executed, consistent A/B testing is a necessity. A/B Testing in Action. A/B testing is a sure way to get to the bottom of a decision without relying on anyone’s gut instinct.

A/B Testing 13

A/B Testing Benefits Metrics User Experience 13

Andrew Chen

AUGUST 23, 2021

People solve this with ideas like wait list hacks, A/B tested invite flows, contact importers, “people you may know” algos. Without the right users there at the beginning, a user-generated content product would inherently be too shallow on content. Low engagement would lead to more low engagement.

A/B Testing 96

A/B Testing eCommerce Media Travel 96

Harvard Business Review

JUNE 23, 2021

It can be a valuable tool to prevent customer churn — if used correctly.

A/B Testing 18

A/B Testing Marketing 18

Andrew Chen

FEBRUARY 26, 2024

I had a superpower in my domain knowledge of CAC, A/B testing, funnel optimization, lead gen, etc, and began to merge all of this thinking with consumer products. I had previously worked in online ads, with customers from WSJ, CBS, MySpace, etc, and had even gotten a patent filed on ad targeting (yes, US7747676B1).

A/B Testing 123

A/B Testing Retention Metrics Engineering 123

Vantage Circle

MARCH 13, 2019

One way of achieving this is through A/B testing. A/B testing is basically a way of testing two variants of the same variable by testing the subject's response to both the variants and statistically determining the more effective of the two. Final Touches.

Advertising 40

Advertising A/B Testing Media Implementation 40

Alison Green

OCTOBER 19, 2022

The other option would be for you and your fellow team members to push back on your PI’s preferred strategy, possibly suggesting some A/B testing so you get real data on what does and doesn’t work … but it doesn’t sound like you want to take that on, which is completely reasonable. Putting video game playing on your resume.

Media 105

Media Insurance A/B Testing Benefits 105

Andrew Chen

OCTOBER 17, 2017

Optimization : A/B testing growth levers – from email/SMS/push copy – to when/how to reach out. Use cases : Understanding use cases and how to up-sell and cross-sell the stickiest ones. Offers/promotions : Using offers and content throughout the calendar cycle to engage.

Workflows 0

Workflows Marketing Retention A/B Testing 0

Andrew Chen

JUNE 21, 2017

He’ll ask about the sample sizes on your A/B tests. As it’s been widely reported, TK cares about the details. He’ll look at your graphs and ask about week-to-week deviations. I’ve worked on projects where he wants to give final approval for the final mockups of flows.

Transportation 110

Transportation A/B Testing Scaling Energy 110

SME Strategy

MARCH 6, 2023

For instance, in Marketing, data is being used to calculate ROI on marketing campaigns, or come up with new pricing strategies based on A/B testing of campaigns which helps marketing and managers bring in more revenue, and stay ahead of the competition.

Strategic Planning 98

Strategic Planning Manager Implementation Software 98

Harvard Business Review

JUNE 28, 2017

One of the most common methods, particularly in online settings, is A/B testing. What Is A/B Testing? A/B testing, at its most basic, is a way to compare two versions of something to figure out which performs better. A/B testing, in its current form, came into existence in the 1990s.

A/B Testing 8

A/B Testing Metrics Software Sales 8

Zenefits

AUGUST 25, 2022

A/B testing. A/B testing is a great way to test marketing campaigns and ensure that your organization takes advantage of the most effective means of attracting customers or users. Let’s dig into 10 of the top trends that go along with modern growth marketing efforts.

Marketing 52

Marketing A/B Testing Media Onboarding 52

Harvard Business Review

OCTOBER 3, 2024

A/B testing by Instacart suggests that telling customers about stocking times can increase spending and decrease the proportion of items replaced or refunded.

A/B Testing 24

A/B Testing Operations 24

AIHR

JULY 31, 2023

A/B testing everything will be incredibly valuable. It can also tell you if certain questions in your application process deter candidates from filling them out or completing them so you can eliminate these questions. You may also discover that images and videos highlighting company culture could encourage applications.

Recruitment 98

Recruitment Onboarding Diversity Human Resources 98

Andrew Chen

DECEMBER 26, 2018

Conservation of Intent: The hidden reason why A/B tests aren’t as effective as they look. Everyone’s had the frustrating experience of running an A/B test, seeing a big lift, closing it out, and expecting the top level metrics to move a lot. But they don’t.

Metrics 40

Metrics A/B Testing Transportation Marketing 40

15Five

JANUARY 25, 2024

Here’s a quick example of an OKR for a marketing department: Objective : Increase conversion rates across all channels Key results : Increase the conversion rate of key landing pages from 10% to 20% Improve email conversion rate from 2%-4% Conduct A/B testing on product pages with the goal of running two statistically significant improvements Benefits: (..)

Implementation 52

Implementation Scaling Metrics A/B Testing 52

Get Lighthouse

DECEMBER 15, 2022

So they let the data do the talking and ran an A/B test. In the case of Netflix, that included admitting when you're wrong, even publicly. In a heated debate between the CEO, Reed Hastings, and another executive, they could not come to an agreement.

Manager 98

Manager A/B Testing Recruitment Engineering 98

Andrew Chen

JUNE 25, 2019

The biggest gains in optimization don’t come from brute-force A/B tests, but from trying to understand the real barriers to people using your product. Question: Is this a channel your competition hasn’t tried yet? Tactic #17: Optimize conversion. All conversion optimization should start with user research.

Retention 111

Retention User Experience Travel Marketing 111

Andrew Chen

DECEMBER 10, 2018

It has tracking codes so you can see how well it circulates, and you A/B test the whole thing to make sure it’s highly optimized to be viral and spread. Of course, we’ve improved the whole thing based on the latest tech. It integrates into Facebook Messenger and your email addressbook.

Media 100

Media Advertising Scaling Marketing 100

Vantage Circle

JANUARY 31, 2023

So they ran an A/B test, and the results were in Tom’s favor. This value is best exemplified in the story Patty McCord recalls in her book “ Powerful.” ” In the story, she recalls how CEO Reed Hastings was once involved in a heated debate with another executive Tom.

Diversity 104

Diversity Inclusion Accounts Operations 104

Andrew Chen

NOVEMBER 1, 2018

Of course, as an investor you can’t run A/B tests or analyze results directly, but you can form hypotheses, ideate, and apply the same type of thinking. 2) Then, to identify potential upside based learnings from within the company as well as across benchmarks from across industry.

Metrics 111

Metrics Scaling Marketing Accounts 111

Andrew Chen

NOVEMBER 1, 2021

There were projects to help with A/B testing (Morpheus!), There were teams working referral programs, landing pages, adtech integrations, funnel optimizations, activation sequences over email, retention and loyalty programs, as well as all the underlying infrastructure.

Scaling 82

Scaling Marketing Recruitment B2B 82

Harvard Business Review

DECEMBER 30, 2016

Traditional operations research process optimization, for example, seldom embraces A/B testing or incorporates customer recommendation engines. ” When fulfillment processes become platforms, opportunities to segment customers, A/B test bundled offers and make targeted recommendations explode.

A/B Testing 15

A/B Testing User Experience Software Compensation 15

Harvard Business Review

JANUARY 30, 2015

Reverse causation investigations have the opposite structure from A/B tests , in which we vary known causes, and observe how the variations affect an outcome. Note: Good A/B test construction can help you see most likely causes; bad A/B test construction creates its own set of problems.).

A/B Testing 13

A/B Testing Metrics Marketing Sales 13

Walk Me

AUGUST 24, 2022

Netflix is one example of such a company, using A/B tests and sharing data collection findings with all aspects of the organization to use continuous change to foster a learning culture. Organizational knowledge is used more efficiently in this way. Trans-organizational Change.

Organizational Development 52

Organizational Development Organizational Change Human Resources Change Management 52

Harvard Business Review

OCTOBER 16, 2012

A/B testing is very much part of the culture of Silicon Valley web start-ups, but hasn't yet made its way to corporate America. For the CDO, this would mean ensuring that processes are being instrumented and that all necessary data is being captured and stored. This, however, is only the first step.

A/B Testing 16

A/B Testing Sales Sales Banking 16

Harvard Business Review

MARCH 10, 2014

Cartoons can make A/B testing easier — and more fun. Draw several different cartoons and captions to test out which ones get to the heart of the customer need and convey value in the most meaningful way. This means we understand, remember and retains images (and their meaning) better than words.

A/B Testing 15

A/B Testing Strategic Planning Advertising Benefits 15

Harvard Business Review

AUGUST 21, 2012

gamification), A/B testing , and mobile deployment can be applied in the enterprise, just as they are used in the consumer space. How do you create applications that are as compelling to corporate employees as they are to those same people in their personal lives? Techniques such as self-reinforcing behavior loops (e.g.

Media 18

Media Workflows A/B Testing B2B 18

Walk Me

JULY 18, 2022

How they communicate. Segmenting target audience . Use audience demographic personas to reflect demographic to attract. Speaking to the target audience. Unique campaigns. Acquisition. Lead acquisition is an integral aspect of CRM, as it cements customers’ initial interest by allowing them to engage with the service or product.

Change Management 59

Change Management Manager Benchmarking Strategic Planning 59

Harvard Business Review

FEBRUARY 4, 2020

While A/B testing on large transaction volumes is common practice at Google, Booking.com, and Netflix, Thomke says even small firms can get a competitive advantage from experiments. He explains how to introduce, run, and learn from them, as well as how to cultivate an experimental mindset at your organization.

A/B Testing 8

A/B Testing 8

Harvard Business Review

DECEMBER 27, 2010

Test Groups and A/B Testing. One way is to let a test group try out the new page, measure the gains from the visits, and compare them against the control group. Previously, stakeholders in that desert state had to rely on educated guesswork using reports and charts. Think of the problem of redesigning a web page.

A/B Testing 15

A/B Testing Education Revenue Diversity 15

Harvard Business Review

JUNE 13, 2013

For this research, the site ran a randomized controlled trial, also called a "field experiment" or "A/B test", where they randomly showed consumers either a generic retargeted ad or a dynamic retargeted ad. Our findings illustrate the importance of implementing randomized controlled trials, also called A/B tests or field experiments.

Advertising 10

Advertising A/B Testing Travel Marketing 10

Harvard Business Review

AUGUST 5, 2014

Doing an A/B test comparing whether people like a new product design more than an old one? The person who runs that test might come back providing average ratings for each design showing that B’s average rating is significantly better than A’s. The same issue arises in business.

A/B Testing 14

A/B Testing Software Scaling Sales 14

Harvard Business Review

AUGUST 12, 2015

My guess is that I was part of a Maps traffic A/B test, because of the lack of other people commenting on this interface change and the fact that I didn’t need to download an update in order to change it. ” I was surprised I got a response at all, but even more amazed at the personalization of the response.

User Experience 14

User Experience A/B Testing Scaling Sales 14