Tag Archives: ajax

[Case Study] Ecwid sees 21% lift in paid plan upgrades in one month

Reading Time: 2 minutes

What would you do with 21% more sales this month?

I bet you’d walk into your next meeting with your boss with an extra spring in your step, right?

Well, when you implement a strategic marketing optimization program, results like this are not only possible, they are probable.

In this new case study, you’ll discover how e-commerce software supplier, Ecwid, ran one experiment for four weeks, and saw a 21% increase in paid upgrades.

Get the full Ecwid case study now!

Download a PDF version of the Ecwid case study, featuring experiment details, supplementary takeaways and insights, and a testimonial from Ecwid’s Sr. Director, Digital Marketing.



By entering your email, you’ll receive bi-weekly WiderFunnel Blog updates and other resources to help you become an optimization champion.

A little bit about Ecwid

Ecwid provides easy-to-use online store setup, management, and payment solutions. The company was founded in 2009, with the goal of enabling business-owners to add online stores to their existing websites, quickly and without hassle.

The company has a freemium business model: Users can sign up for free, and unlock more features as they upgrade to paid packages.

Ecwid’s partnership with WiderFunnel

In November 2016, Ecwid partnered with WiderFunnel with two primary goals:

  1. To increase initial signups for their free plan through marketing optimization, and
  2. To increase the rate of paid upgrades, through platform optimization

This case study focuses on a particular experiment cycle that ran on Ecwid’s step-by-step onboarding wizard.

The methodology

Last Winter, the WiderFunnel Strategy team did an initial LIFT Analysis of the onboarding wizard, and identified several potential barriers to conversion. (Both in terms of completing steps to setup a new store, and in terms of upgrading to a paid plan.)

The lead WiderFunnel Strategist for Ecwid, Dennis Pavlina, decided to create an A/B cluster test to 1) address the major barriers simultaneously, and 2) to get major lift for Ecwid, quickly.

The overarching goal was to make the onboarding process smoother. The WiderFunnel and Ecwid optimization teams hoped that enhancing the initial user experience, and exposing users to the wide range of Ecwid’s features, would result in more users upgrading to paid plans.

Dennis Pavlina

Ecwid’s two objectives ended up coming together in this test. We thought that if more new users interacted with the wizard and were shown the whole ‘Ecwid world’ with all the integrations and potential it has, they would be more open to upgrading. People needed to be able to see its potential before they would want to pay for it.

Dennis Pavlina, Optimization Strategist, WiderFunnel

The Results

This experiment ran for four weeks, at which point the variation was determined to be the winner with 98% confidence. The variation resulted in a 21.3% increase in successful paid account upgrades for Ecwid.

Read the full case study for:

  • The details on the initial barriers to conversion
  • How this test was structured
  • Which secondary metrics we tracked, and
  • The supplementary takeaways and customer insights that came from this test

The post [Case Study] Ecwid sees 21% lift in paid plan upgrades in one month appeared first on WiderFunnel Conversion Optimization.

See original article:

[Case Study] Ecwid sees 21% lift in paid plan upgrades in one month

Capturing supermarket magic and providing the ideal customer experience

Reading Time: 6 minutes

The customer-centric focus

Over the past few years, one message has been gaining momentum within the marketing world: customer experience is king.

Customer experience” (CX) refers to your customer’s perception of her relationship with your brand—both conscious and subconscious—based on every interaction she has with your brand during her customer life cycle.

Customer experience is king
How do your customers feel about your brand?

Companies are obsessing over CX, and for good reason(s):

  • It is 6-7x more expensive to attract a new customer than it is to retain an existing customer
  • 67% of consumers cite ‘bad experiences’ as reason for churn
  • 66% of consumers who switch brands do so because of poor service

Across sectors, satisfied customers spend more, exhibit deeper loyalty to companies, and create conditions that allow companies to have lower costs and higher levels of employee engagement.

As conversion optimization specialists, we test in pursuit of the perfect customer experience, from that first email subject line, to the post-purchase conversation with a customer service agent.

We test because it is the best way to listen, and create ideal experiences that will motivate consumers to choose us over our competitors in the saturated internet marketplace.

Create the perfect personalized customer experience!

Your customers are unique, and their ideal experiences are unique. Create the perfect customer experience with this 4-step guide to building the most effective personalization strategy.



By entering your email, you’ll receive bi-weekly WiderFunnel Blog updates and other resources to help you become an optimization champion.


Which leads me to the main question of this post: Which companies are currently providing the best customer experiences, and how can you apply their strategies in your business context?

Each year, the Tempkin Group releases a list of the best and worst US companies, by customer experience rating. The list is based on survey responses from 10,000 U.S. consumers, regarding their recent experiences with companies.

And over the past few years, supermarkets have topped that list: old school, brick-and-mortar, this-model-has-been-around-forever establishments.

Customer experience - brick-mortar vs. ecommerce
What are supermarkets doing so right, and how can online retailers replicate it?

In the digital world, we often focus on convenience, usability, efficiency, and accessibility…but are there elements at the core of a great customer experience that we may be missing?

A quick look at the research

First things first: Let’s look at how the Tempkin Group determines their experience ratings.

Tempkin surveys 10,000 U.S. consumers, asking them to rate their recent (past 60 days) interactions with 331 companies across 20 industries. The survey questions cover Tempkin’s three components of experience:

  1. Success: Were you, the consumer, able to accomplish what you wanted to do?
  2. Effort: How easy was it for you to interact with the company?
  3. Emotion: How did you feel about those interactions?

Respondents answer questions on a scale of 1 (worst) to 7 (best), and researchers score each company accordingly. For more details on how the research was conducted, you can download the full report, here.

In this post, I am going to focus on one supermarket that has topped the list for the past three years: Publix. Not only does Publix top the Tempkin ratings, it also often tops the supermarket rankings compiled by the American Customer Satisfaction Index.

Long story short: Publix is winning the customer experience battle.

WiderFunnel Customer Experience Ratings Tempkin 2017
2017 Customer Experience ratings from Tempkin.
WiderFunnel Customer Experience Ratings Tempkin 2016
2016 Customer Experience ratings from Tempkin.

So, what does Publix do right?

Publix growth - WiderFunnel customer experience
Publix growth trends (Source).

If you don’t know it, Publix Super Markets, Inc. is an American supermarket chain headquartered in Florida. Founded in 1930, Publix is a private corporation that is wholly owned by present and past employees; it is considered the largest employee-owned company in the world.

In an industry that has seen recent struggles, Publix has seen steady growth over the past 10 years. So, what is this particular company doing so very right?

1. World-class customer service

Publix takes great care to provide the best possible customer service.

From employee presentation (no piercings, no unnatural hair color, no facial hair), to the emphasis on “engaging the customer”, to the bread baked fresh on-site every day, the company’s goal is to create the most pleasurable shopping experience for each and every customer.

When you ask “Where is the peanut butter?” at another supermarket, an employee might say, “Aisle 4.” But at Publix, you will be led to the peanut butter by a friendly helper.

The store’s slogan: “Make every customer’s day a little bit better because they met you.”

2. The most motivated employees

Publix associates are famously “pleased-as-punch, over-the-moon, [and] ridiculously contented”.

Note the term “associates”: Because Publix is employee-owned, employees are not referred to as employees, but associates. As owners, associates share in the store’s success: If the company does well, so do they.

Our culture is such that we believe if we take care of our associates, they in turn will take care of our customers. Associate ownership is our secret sauce,” said Publix spokeswoman, Maria Brous. “Our associates understand that their success is tied to the success of our company and therefore, we must excel at providing legendary service to our customers.

3. Quality over quantity

While Publix is one of the largest food retailers in the country by revenue, they operate a relatively small number of stores: 1,110 stores across six states in the southeastern U.S. (For context, Wal-Mart operates more than 4,000 stores).

Each of Publix’s store locations must meet a set of standards. From the quality of the icing on a cake in the bakery, to the “Thanks for shopping at Publix. Come back and see us again soon!” customer farewell, customers should have a delightful experience at every Publix store.

4. An emotional shopping experience

In the Tempkin Experience Ratings, emotion was the weakest component for the 331 companies evaluated. But, Publix was among the few organizations to receive an “excellent” emotion rating. (In fact, they are ranked top 3 in this category.)

widerfunnel customer delight
Are you creating delight for the individuals who are your customers?

They are able to literally delight their customers. And, as a smart marketer, I don’t have to tell you how powerful emotion is in the buying process.

Great for Publix. What does this mean for me?

As marketers, we should be changing the mantra from ‘always be closing’ to ‘always be helping’.

– Jonathan Lister, LinkedIn

In the digital marketing world, it is easy to get lost in acronyms: UX, UI, SEO, CRO, PPC…and forget about the actual customer experience. The experience that each individual shopper has with your brand.

Beyond usability, beyond motivation tactics, beyond button colors and push notifications, are you creating delight?

To create delight, you need to understand your customer’s reality. It may be time to think about how much you spend on website traffic, maintenance, analytics, and tools vs. how much you spend to understand your customers…and flip the ratio.

It’s important to understand the complexity of how your users interact with your website. We say, ‘I want to find problems with my website by looking at the site itself, or at my web traffic’. But that doesn’t lead to results. You have to understand your user’s reality.

– André Morys, Founder & CEO, WebArts

Publix is winning with their customer-centric approach because they are fully committed to it. While the tactics may be different with a brick-and-mortar store and an e-commerce website, the goals overlap:

1. Keep your customer at the core of every touch point

From your Facebook ad, to your product landing page, to your product category page, checkout page, confirmation email, and product tracking emails, you have an opportunity to create the best experience for your customers at each step.

customer service and customer experience
Great customer service is one component of a great customer experience.

2. Make your customers feel something.

Humans don’t buy things. We buy feelings. What are you doing to make your shoppers feel? How are you highlighting the intangible benefits of your value proposition?

3. Keep your employees motivated.

Happy, satisfied employees, deliver happy, satisfying customer experiences, whether they’re creating customer-facing content for your website, or speaking to customers on the phone. For more on building a motivated, high performance marketing team, read this post!

Testing to improve your customer experience

Of course, this wouldn’t be a WiderFunnel blog post if I didn’t recommend testing your customer experience improvements.

If you have an idea for how to inject emotion into the shopping experience, test it. If you believe a particular tweak will make the shopping experience easier and your shoppers more successful, test it.

Your customers will show you what an ideal customer experience looks like with their actions, if you give them the opportunity.

Here’s an example.

During our partnership with e-commerce platform provider, Magento, we ran a test on the product page for the company’s Enterprise Edition software, meant to improve the customer experience.

The main call-to-action on this page was “Get a free demo”—a universal SaaS offering. The assumption was that potential customers would want to experience and explore the platform on their own (convenient, right?), before purchasing the platform.

Magento_CTA_Get
The original Magento Enterprise Edition homepage featuring the “Get a free demo”.

Looking at click map data, however, our Strategists noticed that visitors to this page were engaging with informational tabs lower on the page. It seemed that potential customers needed more information to successfully accomplish their goals on the page.

Unfortunately, once visitors had finished browsing tabs, they had no option other than trying the demo, whether they were ready or not.

So, our Strategists tested adding a secondary “Talk to a specialist” call-to-action. Potential customers could connect directly with a Magento sales representative, and get answers to all of their questions.

Magento_CTA
Today’s Magento Enterprise Edition homepage features a “Talk to a specialist” CTA.

This call-to-action hadn’t existed prior to this test, so the literal infinite conversion rate lift Magento saw in qualified sales calls was not surprising.

What was surprising was the phone call we received six months later: Turns out the “Talk to a specialist” leads were 8x more valuable than the “Get a free demo” leads.

After several subsequent test rounds, “Talk to a specialist” became the main call-to-action on that product page. Magento’s most valuable prospects had demonstrated that the ideal customer experience included the opportunity to get more information from a specialist.

While Publix’s success reminds us of the core components of a great customer experience, actually creating a great customer experience can be tricky.

You might be wondering:

  • What is most important to my customers: Success, Effort, or Emotion?
  • What improvements should I make first?
  • How will I know these improvements are actually working?

A test-and-learn strategy will help you answer these questions, and begin working toward a truly great customer experience.

Don’t get lost in the guesswork of tweaks, fixes, and best practices. Get obsessed with understanding your customer, instead.

How do you create the ideal customer experience?

Please share your thoughts in the comments section below!

The post Capturing supermarket magic and providing the ideal customer experience appeared first on WiderFunnel Conversion Optimization.

Visit source: 

Capturing supermarket magic and providing the ideal customer experience

How to get evergreen results from your landing page optimization

Reading Time: 7 minutes

Landing page optimization is old news.

Seriously. A quick google will show you that Unbounce, QuickSprout, Moz, Qualaroo, Hubspot, Wordstream, Optimizely, CrazyEgg, VWO (and countless others), have been writing tips and guides on how to optimize your landing pages for years.

Not to mention the several posts we have already published on the WiderFunnel blog since 2008.

And yet. This conversation is so not over.

Warning: If your landing page optimization goals are short-term, or completely focused on conversion rate lift, this post may be a waste of your time. If your goal is to continuously have the best-performing landing pages on the internet, keep reading.



Marketers are funnelling more and more money into paid advertising, especially as Google allocates more and more SERP space to ads.

In fact, as an industry, we are spending upwards of $92 billion annually on paid search advertising alone.

landing-page-optimization-SERP-space
The prime real estate on a Google search results page often goes to paid.

And it’s not just search advertising that is seeing an uptick in spend, but social media advertising too.

It makes sense that marketers are still obsessing over their landing page conversion rates: this traffic is costly and curated. These are visitors that you have sought out, that share characteristics with your target market. It is extremely important that these visitors convert!

But, there comes a time in every optimizer’s life, when they face the cruel reality of diminishing returns. You’ve tested your landing page hero image. You’ve tested your value proposition. You’ve tested your form placement. And now, you’ve hit a plateau.

So, what next? What’s beyond the tips and guides? What is beyond the optimization basics?

1) Put on your customer’s shoes.

First things first: Let’s do a quick sanity check.

When you test your hero image, or your form placement, are you testing based on tips and recommended best practices? Or, are you testing based on a specific theory you have about your page visitors?

landing-page-optimization-customer-shoes
Put on your customer’s shoes.

Tips and best practices are a fine place to start, but the insight behind why those tactics work (or don’t work) for your visitors is where you find longevity.

The best way to improve experiences for your visitors is to think from their perspective. And the best way to do that is to use frameworks, and framework thinking, to get robust insights about your customers.

– Chris Goward, Founder & CEO, WiderFunnel

Laying the foundation

It’s very difficult to think from a different perspective. This is true in marketing as much as it is in life. And it’s why conversion optimization and A/B testing have become so vital: We no longer have to guess at what our visitors want, but can test instead!

That said, a test requires a hypothesis. And a legitimate hypothesis requires a legitimate attempt to understand your visitor’s unique perspective.

To respond to this need for understanding, WiderFunnel developed the LIFT Model® in 2008: our foundational framework for identifying potential barriers to conversion on a page from the perspective of the page visitor.

Get optimization ideas with the LIFT poster!

Get the LIFT Model poster, and challenge yourself to keep your visitor’s perspective in mind at all times. Use the six conversion factors to analyze your pages, and get optimization ideas!



By entering your email, you’ll receive bi-weekly WiderFunnel Blog updates and other resources to help you become an optimization champion.


The LIFT Model attempts to capture the idea of competing forces in communication, narrowing them down to the most salient aspects of communication that marketers should consider.

I wanted to apply the principles of Relevance, Clarity, Distraction, Urgency and Anxiety to what we were delivering to the industry and not just to our clients. And the LIFT Model is a part of that: making something as simple as possible but no simpler.

– Chris Goward

When you look at your page through a lens like the LIFT Model, you are forced to question your assumptions about what your visitors want when they land on your page.

landing-page-optimization-LIFT-Model
View your landing pages through a framework lens.

You may love an interactive element, but is it distracting your visitors? You may think that your copy creates urgency, but is it really creating anxiety?

If you are an experienced optimizer, you may have already incorporated a framework like the LIFT Model into your optimization program. But, after you have analyzed the same page multiple times, how do you continue to come up with new ideas?

Here are a few tips from the WiderFunnel Strategy team:

  1. Bring in fresh eyes from another team to look at and use your page
  2. User test, to watch and record how actual users are using your page
  3. Sneak a peek at your competitors’ landing pages: Is there something they’re doing that might be worth testing on your site?
  4. Do your page analyses as a team: many heads are better than one
  5. Brainstorm totally new, outside-the-box ideas…and test one!

You should always err on the side of “This customer experience could be better.” After all, it’s a customer-centric world, and we’re just marketing in it.

2) Look past the conversion rate.

“Landing page optimization”, like “conversion rate optimization”, is a limiting term. Yes, on-page optimization is key, but mature organizations view “landing page optimization” as the optimization of the entire experience, from first to last customer touchpoint.

Landing pages are only one element of a stellar, high-converting marketing campaign. And focusing all of your attention on optimizing only one element is just foolish.

From testing your featured ads, to tracking click-through rates of Thank You emails, to tracking returns and refunds, to tracking leads through the rest of the funnel, a better-performing landing page is about much more than on-page conversion rate lift.

landing-page-optimization-big-picture
On-page optimization is just one part of the whole picture.

An example is worth 1,000 words

One of our clients is a company that provides an online consumer information service—visitors type in a question and get an Expert answer. One of the first zones (areas on their website) that we focused on was a particular landing page funnel.

Visitors come from an ad, and land on page where they can ask their question. They then enter a 4-step funnel: Step 1: Ask the question > Step 2: Add more information > Step 3: Pick an Expert > Step 4: Get an answer (aka the checkout page)

Our primary goal was to increase transactions, meaning we had to move visitors all the way through the funnel. But we were also tracking refunds and chargebacks, as well as revenue per visitor.

More than pushing a visitor to ‘convert’, we wanted to make sure those visitors went on to be happy, satisfied customers.

In this experiment, we focused on the value proposition statements. The control landing page exclaimed, “A new question is answered every 9 seconds!“. Our Strategy team had determined (through user testing) that “speed of answers” was the 8th most valuable element of the service for customers, and that “peace of mind / reassurance” was the most important.

So, they tested two variations, featuring two different value proposition statements meant to create more peace of mind for visitors:

  • “Join 6,152,585 satisfied customers who got professional answers…”
  • “Connect One on One with an Expert who will answer your question”

Both of these variations ultimately increased transactions, by 6% and 9.4% respectively. But! We also saw large decreases in refunds and chargebacks with both variations, and large increases in net revenue per visitor for both variations.

By following visitors past the actual conversion, we were able to confirm that these initial statements set an impactful tone: visitors were more satisfied with their purchases, and comfortable investing more in their expert responses.

3) Consider the big picture.

As you think of landing page optimization as the optimization of a complete digital experience, you should also think of landing page optimization as part of your overall digital optimization strategy.

When you discover an insight about visitors to your product page, feed it into a test on your landing page. When you discover an insight about visitor behavior on your landing page, feed it into a test on your website.

It’s true that your landing pages most likely cater to specific visitor segments, who may behave totally differently than your organic visitors. But, it is also true that landing page wins may be overall wins.

Plus, landing page insights can be very valuable, because they are often new visitor insights. And now, a little more advice from Chris Goward, optimization guru:

“Your best opportunities for testing your value proposition are with first impression visitors. These are usually new visitors to your high traffic landing pages or your home page […]

By split testing your alternative value propositions with new visitors, you’ll reduce your exposure to existing customers or prospects who are already in the consideration phase. New prospects have a blank canvas for you to present your message variations and see what sticks.

Then, from the learning gained on landing pages, you can validate insights with other target audience groups and with your customers to leverage the learning company-wide.

Landing page testing can do more than just improve conversion rates on landing pages. When done strategically, it can deliver powerful, high-leverage marketing insights.”



Just because your landing pages are separate from your website, does not mean that your landing page optimization should be separate from your other optimization efforts. A landing page is just another zone, and you are free to (and should) use insights from one zone when testing on another zone.

4) Go deeper, explore further.

A lot of marketers talk about landing page design: how to build the right landing page, where to position each element, what color scheme and imagery to use, etc.

But when you dig into the why behind your test results, it’s like breaking into a piñata of possibilities, or opening a box of idea confetti.

landing-page-optimization-ideas
Discovering the reason behind the result is like opening a box of idea confetti!

Why do your 16-25 year old, mobile users respond so favorably to a one-minute video testimonial from a past-purchaser? Do they respond better to this indicator of social proof than another?

Why do your visitors prefer one landing page under normal circumstances, and a different version when external factors change (like a holiday, or a crisis)? Can you leverage this insight throughout your website?

Why does one type of urgency phrasing work, while slightly different wording decreases conversions on your page? Are your visitors sensitive to overly salesy copy? Why or why not?

Not only are there hundreds of psychological principles to explore within your landing page testing, but landing page optimization is also intertwined with your personalization strategy.

For many marketers, personalized landing pages are becoming more normal. And personalization opens the door to even more potential customer insights. Assuming you already have visitor segments, you should test the personalized experiences on your landing pages.

For example, imagine you have started using your visitors’ first names in the hero banner of your landing page. Have you validated that this personalized experience is more effective than another, like moving a social proof indicator above the fold? Both can be deemed personalization, but they tap into very different motivations.

From psychological principles, to validating your personalized experiences, the possibilities for testing on your landing pages are endless.

Just keep testing, Dory-style

Your landing page(s) will never be “optimized”. That is the beauty and cruelty of optimization: we are always chasing unattainable perfection.

But your landing pages can definitely be better than they are now. Even if you have a high-converting page, even if your page is listed by Hubspot as one of the 16 best designed landing pages, even if you’ve followed all of the rules…your landing page can be better.

Because I’m not just talking about conversions, I’m talking about your entire customer experience. If you give them the opportunity, your new users will tell you what’s wrong with your page.

They’ll tell you where it is unclear and where it is distracting.

They’ll tell you what motivates them.

They’ll tell you how personal you should get.

They’ll tell you how to set expectations so that they can become satisfied customers or clients.

A well-designed landing page is just the beginning of landing page optimization.

The post How to get evergreen results from your landing page optimization appeared first on WiderFunnel Conversion Optimization.

More: 

How to get evergreen results from your landing page optimization

“The more tests, the better!” and other A/B testing myths, debunked

Reading Time: 8 minutes

Will the real A/B testing success metrics please stand up?

It’s 2017, and most marketers understand the importance of A/B testing. The strategy of applying the scientific method to marketing to prove whether an idea will have a positive impact on your bottom-line is no longer novel.

But, while the practice of A/B testing has become more and more common, too many marketers still buy into pervasive A/B testing myths. #AlternativeFacts.

This has been going on for years, but the myths continue to evolve. Other bloggers have already addressed myths like “A/B testing and conversion optimization are the same thing”, and “you should A/B test everything”.

As more A/B testing ‘experts’ pop up, A/B testing myths have become more specific. Driven by best practices and tips and tricks, these myths represent ideas about A/B testing that will derail your marketing optimization efforts if left unaddressed.

Avoid the pitfalls of ad-hoc A/B testing…

Get this guide, and learn how to build an optimization machine at your company. Discover how to use A/B testing as part of your bigger marketing optimization strategy!



By entering your email, you’ll receive bi-weekly WiderFunnel Blog updates and other resources to help you become an optimization champion.



But never fear! With the help of WiderFunnel Optimization Strategist, Dennis Pavlina, I’m going to rebut four A/B testing myths that we hear over and over again. Because there is such a thing as a successful, sustainable A/B testing program…

Into the light, we go!

Myth #1: The more tests, the better!

A lot of marketers equate A/B testing success with A/B testing velocity. And I get it. The more tests you run, the faster you run them, the more likely you are to get a win, and prove the value of A/B testing in general…right?

Not so much. Obsessing over velocity is not going to get you the wins you’re hoping for in the long run.

Mike St Laurent

The key to sustainable A/B testing output, is to find a balance between short-term (maximum testing speed), and long-term (testing for data-collection and insights).

Michael St Laurent, Senior Optimization Strategist, WiderFunnel

When you focus solely on speed, you spend less time structuring your tests, and you will miss out on insights.

With every experiment, you must ensure that it directly addresses the hypothesis. You must track all of the most relevant goals to generate maximum insights, and QA all variations to ensure bugs won’t skew your data.

Dennis Pavlina

An emphasis on velocity can create mistakes that are easily avoided when you spend more time on preparation.

Dennis Pavlina, Optimization Strategist, WiderFunnel

Another problem: If you decide to test many ideas, quickly, you are sacrificing your ability to really validate and leverage an idea. One winning A/B test may mean quick conversion rate lift, but it doesn’t mean you’ve explored the full potential of that idea.

You can often apply the insights gained from one experiment, when building out the strategy for another experiment. Plus, those insights provide additional evidence for testing a particular concept. Lining up a huge list of experiments at once without taking into account these past insights can result in your testing program being more scattershot than evidence-based.

While you can make some noise with an ‘as-many-tests-as-possible’ strategy, you won’t see the big business impact that comes from a properly structured A/B testing strategy.

Myth #2: Statistical significance is the end-all, be-all

A quick definition

Statistical significance: The probability that a certain result is not due to chance. At WiderFunnel, we use a 95% confidence level. In other words, we can say that there is a 95% chance that the observed result is because of changes in our variation (and a 5% chance it is due to…well…chance).

If a test has a confidence level of less than 95% (positive or negative), it is inconclusive and does not have our official recommendation. The insights are deemed directional and subject to change.

Ok, here’s the thing about statistical significance: It is important, but marketers often talk about it as if it is the only determinant for completing an A/B test. In actuality, you cannot view it within a silo.

For example, a recent experiment we ran reached statistical significance three hours after it went live. Because statistical significance is viewed as the end-all, be-all, a result like this can be exciting! But, in three hours, we had not gathered a representative sample size.

Claire Vignon Keser

You should not wait for a test to be significant (because it may never happen) or stop a test as soon as it is significant. Instead, you need to wait for the calculated sample size to be reached before stopping a test. Use a test duration calculator to understand better when to stop a test.

After 24 hours, the same experiment had dropped to a confidence level of 88%, meaning that there was now only an 88% likelihood that the difference in conversion rates was not due to chance – i.e. statistically significant.

Traffic behaves differently over time for all businesses, so you should always run a test for full business cycles, even if you have reached statistical significance. This way, your experiment has taken into account all of the regular fluctuations in traffic that impact your business.

For an e-commerce business, a full business cycle is typically a one-week period; for subscription-based businesses, this might be one month or longer.

Myth #2, Part II: You have to run a test until reaches statistical significance

As Claire pointed out, this may never happen. And it doesn’t mean you should walk away from an A/B test, completely.

As I said above, anything below 95% confidence is deemed subject to change. But, with testing experience, an expert understanding of your testing tool, and by observing the factors I’m about to outline, you can discover actionable insights that are directional (directionally true or false).

  • Results stability: Is the conversion rate difference stable over time, or does it fluctuate? Stability is a positive indicator.
ab testing results stability
Check your graphs! Are conversion rates crossing? Are the lines smooth and flat, or are there spikes and valleys?
  • Experiment timeline: Did I run this experiment for at least a full business cycle? Did conversion rate stability last throughout that cycle?
  • Relativity: If my testing tool uses t-test to determine significance, am I looking at the hard numbers of actual conversions in addition to conversion rate? Does the calculated lift make sense?
  • LIFT & ROI: Is there still potential for the experiment to achieve X% lift? If so, you should let it run as long as it is viable, especially when considering the ROI.
  • Impact on other elements: If elements outside the experiment are unstable (social shares, average order value, etc.) the observed conversion rate may also be unstable.

You can use these factors to make the decision that makes the most sense for your business: implement the variation based on the observed trends, abandon the variation based on observed trends, and/or create a follow-up test!

Myth #3: An A/B test is only as good as its effect on conversion rates

Well, if conversion rate is the only success metric you are tracking, this may be true. But you’re underestimating the true growth potential of A/B testing if that’s how you structure your tests!

To clarify: Your main success metric should always be linked to your biggest revenue driver.

But, that doesn’t mean you shouldn’t track other relevant metrics! At WiderFunnel, we set up as many relevant secondary goals (clicks, visits, field completions, etc.) as possible for each experiment.

Dennis Pavlina

This ensures that we aren’t just gaining insights about the impact a variation has on conversion rate, but also the impact it’s having on visitor behavior.

– Dennis Pavlina

When you observe secondary goal metrics, your A/B testing becomes exponentially more valuable because every experiment generates a wide range of secondary insights. These can be used to create follow up experiments, identify pain points, and create a better understanding of how visitors move through your site.

An example

One of our clients provides an online consumer information service — users type in a question and get an Expert answer. This client has a 4-step funnel. With every test we run, we aim to increase transactions: the final, and most important conversion.

But, we also track secondary goals, like click-through-rates, and refunds/chargebacks, so that we can observe how a variation influences visitor behavior.

In one experiment, we made a change to step one of the funnel (the landing page). Our goal was to set clearer visitor expectations at the beginning of the purchasing experience. We tested 3 variations against the original, and all 3 won resulted in increased transactions (hooray!).

The secondary goals revealed important insights about visitor behavior, though! Firstly, each variation resulted in substantial drop-offs from step 1 to step 2…fewer people were entering the funnel. But, from there, we saw gradual increases in clicks to steps 3 and 4.

Our variations seemed to be filtering out visitors without strong purchasing intent. We also saw an interesting pattern with one of our variations: It increased clicks from step 3 to step 4 by almost 12% (a huge increase), but decreased actual conversions by -1.6%. This result was evidence that the call-to-action on step 4 was extremely weak (which led to a follow-up test!)

ab testing funnel analysis
You can see how each variation fared against the Control in this funnel analysis.

We also saw large decreases in refunds and chargebacks for this client, which further supported the idea that the right visitors (i.e. the wrong visitors) were the ones who were dropping off.

This is just a taste of what every A/B test could be worth to your business. The right goal tracking can unlock piles of insights about your target visitors.

Myth #4: A/B testing takes little to no thought or planning

Believe it or not, marketers still think this way. They still view A/B testing on a small scale, in simple terms.

But A/B testing is part of a greater whole—it’s one piece of your marketing optimization program—and you must build your tests accordingly. A one-off, ad-hoc test may yield short-term results, but the power of A/B testing lies in iteration, and in planning.

ab testing infinity optimization process
A/B testing is just a part of the marketing optimization machine.

At WiderFunnel, a significant amount of research goes into developing ideas for a single A/B test. Even tests that may seem intuitive, or common-sensical, are the result of research.

ab testing planning
The WiderFunnel strategy team gathers to share and discuss A/B testing insights.

Because, with any test, you want to make sure that you are addressing areas within your digital experiences that are the most in need of improvement. And you should always have evidence to support your use of resources when you decide to test an idea. Any idea.

So, what does a revenue-driving A/B testing program actually look like?

Today, tools and technology allow you to track almost any marketing metric. Meaning, you have an endless sea of evidence that you can use to generate ideas on how to improve your digital experiences.

Which makes A/B testing more important than ever.

An A/B test shows you, objectively, whether or not one of your many ideas will actually increase conversion rates and revenue. And, it shows you when an idea doesn’t align with your user expectations and will hurt your conversion rates.

And marketers recognize the value of A/B testing. We are firmly in the era of the data-driven CMO: Marketing ideas must be proven, and backed by sound data.

But results-driving A/B testing happens when you acknowledge that it is just one piece of a much larger puzzle.

One of our favorite A/B testing success stories is that of DMV.org, a non-government content website. If you want to see what a truly successful A/B testing strategy looks like, check out this case study. Here are the high level details:

We’ve been testing with DMV.org for almost four years. In fact, we just launched our 100th test with them. For DMV.org, A/B testing is a step within their optimization program.

Continuous user research and data gathering informs hypotheses that are prioritized and created into A/B tests (that are structured using proper Design of Experiments). Each A/B test delivers business growth and/or insights, and these insights are fed back into the data gathering. It’s a cycle of continuous improvement.

And here’s the kicker: Since DMV.org began A/B testing strategically, they have doubled their revenue year over year, and have seen an over 280% conversion rate increase. Those numbers kinda speak for themselves, huh?

What do you think?

Do you agree with the myths above? What are some misconceptions around A/B testing that you would like to see debunked? Let us know in the comments!

The post “The more tests, the better!” and other A/B testing myths, debunked appeared first on WiderFunnel Conversion Optimization.

Excerpt from:

“The more tests, the better!” and other A/B testing myths, debunked

Reimagining Single-Page Applications With Progressive Enhancement


What is the difference between a web page and a web application? Though we tend to identify documents with reading and applications with interaction, most web-based applications are of the blended variety: Users can consume information and perform tasks in the same place. Regardless, the way we approach building web applications usually dispenses with some of the simple virtues of the readable web.

Reimagining Single-Page Applications

Single-page applications tend to take the form of runtimes, JavaScript executables deployed like popup shops into vacant <body> elements. They’re temporary, makeshift and not cURL-able: Their content is not really there without a script being executed. They’re also brittle and underperforming because, in service of architectural uniformity and convenience, they make all of their navigation, data handling and even the basic display of content the responsibility of one thing: client-side JavaScript.

The post Reimagining Single-Page Applications With Progressive Enhancement appeared first on Smashing Magazine.

View original post here – 

Reimagining Single-Page Applications With Progressive Enhancement

Thumbnail

Web Scraping With Node.js

Web scraping is the process of programmatically retrieving information from the Internet. As the volume of data on the web has increased, this practice has become increasingly widespread, and a number of powerful services have emerged to simplify it. Unfortunately, the majority of them are costly, limited or have other disadvantages. Instead of turning to one of these third-party resources, you can use Node.js to create a powerful web scraper that is both extremely versatile and completely free.

In this article, I’ll be covering the following:

  • two Node.js modules, Request and Cheerio, that simplify web scraping;
  • an introductory application that fetches and displays some sample data;
  • a more advanced application that finds keywords related to Google searches.

Also, a few things worth noting before we go on: A basic understanding of Node.js is recommended for this article; so, if you haven’t already, check it out1 before continuing. Also, web scraping may violate the terms of service for some websites, so just make sure you’re in the clear there before doing any heavy scraping.

Modules

To bring in the Node.js modules I mentioned earlier, we’ll be using NPM2, the Node Package Manager (if you’ve heard of Bower, it’s like that — except, you use NPM to install Bower). NPM is a package management utility that is automatically installed alongside Node.js to make the process of using modules as painless as possible. By default, NPM installs the modules in a folder named node_modules in the directory where you invoke it, so make sure to call it in your project folder.

And without further ado, here are the modules we’ll be using.

Request

While Node.js does provide simple methods of downloading data from the Internet via HTTP and HTTPS interfaces, you have to handle them separately, to say nothing of redirects and other issues that appear when you start working with web scraping. The Request module3 merges these methods, abstracts away the difficulties and presents you with a single unified interface for making requests. We’ll use this module to download web pages directly into memory. To install it, run npm install request from your terminal in the directory where your main Node.js file will be located.

Cheerio

Cheerio4 enables you to work with downloaded web data using the same syntax that jQuery employs. To quote the copy on its home page, “Cheerio is a fast, flexible and lean implementation of jQuery designed specifically for the server.” Bringing in Cheerio enables us to focus on the data we download directly, rather than on parsing it. To install it, run npm install cheerio from your terminal in the directory where your main Node.js file will be located.

Implementation

The code below is a quick little application to nab the temperature from a weather website. I popped in my area code at the end of the URL we’re downloading, but if you want to try it out, you can put yours in there (just make sure to install the two modules we’re attempting to require first; you can learn how to do that via the links given for them above).

var request = require("request"),
	cheerio = require("cheerio"),
	url = "http://www.wunderground.com/cgi-bin/findweather/getForecast?&query=" + 02888;
	
request(url, function (error, response, body) 
	if (!error) 
		var $ = cheerio.load(body),
			temperature = $("[data-variable='temperature'] .wx-value").html();
			
		console.log("It’s " + temperature + " degrees Fahrenheit.");
	 else 
		console.log("We’ve encountered an error: " + error);
	
});

So, what are we doing here? First, we’re requiring our modules so that we can access them later on. Then, we’re defining the URL we want to download in a variable.

Then, we use the Request module to download the page at the URL specified above via the request function. We pass in the URL that we want to download and a callback that will handle the results of our request. When that data is returned, that callback is invoked and passed three variables: error, response and body. If Request encounters a problem downloading the web page and can’t retrieve the data, it will pass a valid error object to the function, and the body variable will be null. Before we begin working with our data, we’ll check that there aren’t any errors; if there are, we’ll just log them so we can see what went wrong.

If all is well, we pass our data off to Cheerio. Then, we’ll be able to handle the data like we would any other web page, using standard jQuery syntax. To find the data we want, we’ll have to build a selector that grabs the element(s) we’re interested in from the page. If you navigate to the URL I’ve used for this example in your browser and start exploring the page with developer tools, you’ll notice that the big green temperature element is the one I’ve constructed a selector for. Finally, now that we’ve got ahold of our element, it’s a simple matter of grabbing that data and logging it to the console.

We can take it plenty of places from here. I encourage you to play around, and I’ve summarized the key steps for you below. They are as follows.

In Your Browser

  1. Visit the page you want to scrape in your browser, being sure to record its URL.
  2. Find the element(s) you want data from, and figure out a jQuery selector for them.

In Your Code

  1. Use request to download the page at your URL.
  2. Pass the returned data into Cheerio so you can get your jQuery-like interface.
  3. Use the selector you wrote earlier to scrape your data from the page.

Going Further: Data Mining

More advanced uses of web scraping can often be categorized as data mining5, the process of downloading a lot of web pages and generating reports based on the data extracted from them. Node.js scales well for applications of this nature.

I’ve written a small data-mining app in Node.js, less than a hundred lines, to show how we’d use the two libraries that I mentioned above in a more complicated implementation. The app finds the most popular terms associated with a specific Google search by analyzing the text of each of the pages linked to on the first page of Google results.

There are three main phases in this app:

  1. Examine the Google search.
  2. Download all of the pages and parse out all the text on each page.
  3. Analyze the text and present the most popular words.

We’ll take a quick look at the code that’s required to make each of these things happen — as you might guess, not a lot.

Downloading the Google Search

The first thing we’ll need to do is find out which pages we’re going to analyze. Because we’re going to be looking at pages pulled from a Google search, we simply find the URL for the search we want, download it and parse the results to find the URLs we need.

To download the page we use Request, like in the example above, and to parse it we’ll use Cheerio again. Here’s what the code looks like:

request(url, function (error, response, body) 
	if (error) 
		console.log(“Couldn’t get page because of error: “ + error);
		return;
	
	
	// load the body of the page into Cheerio so we can traverse the DOM
	var $ = cheerio.load(body),
		links = $(".r a");
		
	links.each(function (i, link) 
		// get the href attribute of each link
		var url = $(link).attr("href");
		
		// strip out unnecessary junk
		url = url.replace("/url?q=", "").split("&")[0];
		
		if (url.charAt(0) === "/") 
			return;
		
		
		// this link counts as a result, so increment results
		totalResults++;

In this case, the URL variable we’re passing in is a Google search for the term “data mining.”

As you can see, we first make a request to get the contents of the page. Then, we load the contents of the page into Cheerio so that we can query the DOM for the elements that hold the links to the pertinent results. Then, we loop through the links and strip out some extra URL parameters that Google inserts for its own usage — when we’re downloading the pages with the Request module, we don’t want any of those extra parameters.

Finally, once we’ve done all that, we make sure the URL doesn’t start with a / — if so, it’s an internal link to something else of Google’s, and we don’t want to try to download it, because either the URL is malformed for our purposes or, even if it isn’t malformed, it wouldn’t be relevant.

Pulling the Words From Each Page

Now that we have the URLs of our pages, we need to pull the words from each page. This step consists of doing much the same thing we did just above — only, in this case, the URL variable refers to the URL of the page that we found and processed in the loop above.

request(url, function (error, response, body) 
	// load the page into Cheerio
	var $page = cheerio.load(body),
		text = $page("body").text();

Again, we use Request and Cheerio to download the page and get access to its DOM. Here, we use that access to get just the text from the page.

Next, we’ll need to clean up the text from the page — it’ll have all sorts of garbage that we don’t want on it, like a lot of extra white space, styling, occasionally even the odd bit of JSON data. This is what we’ll need to do:

  1. Compress all white space to single spaces.
  2. Throw away any characters that aren’t letters or spaces.
  3. Convert everything to lowercase.

Once we’ve done that, we can simply split our text on the spaces, and we’re left with an array that contains all of the rendered words on the page. We can then loop through them and add them to our corpus.

The code to do all that looks like this:

// Throw away extra white space and non-alphanumeric characters.
text = text.replace(/s+/g, " ")
	     .replace(/[^a-zA-Z ]/g, "")
	     .toLowerCase();

// Split on spaces for a list of all the words on that page and 
// loop through that list.
text.split(" ").forEach(function (word) 
	// We don't want to include very short or long words because they're 
	// probably bad data.
	if (word.length  20) 
		return;
	
				
	if (corpus[word]) 
		// If this word is already in our corpus, our collection
		// of terms, increase the count for appearances of that 
		// word by one.
		corpus[word]++;
	 else 
		// Otherwise, say that we've found one of that word so far.
		corpus[word] = 1;
	
});

Analyzing Our Words

Once we’ve got all of our words in our corpus, we can loop through that and sort them by popularity. First, we’ll need to stick them in an array, though, because the corpus is an object.

// stick all words in an array
for (prop in corpus) 
	words.push(
		word: prop,
		count: corpus[prop]
	);
}
	
// sort array based on how often they occur
words.sort(function (a, b) 
	return b.count - a.count;
);

The result will be a sorted array representing exactly how often each word in it has been used on all of the websites from the first page of results of the Google search. Below is a sample set of results for the term “data mining.” (Coincidentally, I used this list to generate the word cloud at the top of this article.)

[  word: 'data', count: 981 ,
   word: 'mining', count: 531 ,
   word: 'that', count: 187 ,
   word: 'analysis', count: 120 ,
   word: 'information', count: 113 ,
   word: 'from', count: 102 ,
   word: 'this', count: 97 ,
   word: 'with', count: 92 ,
   word: 'software', count: 81 ,
   word: 'knowledge', count: 79 ,
   word: 'used', count: 78 ,
   word: 'patterns', count: 72 ,
   word: 'learning', count: 70 ,
   word: 'example', count: 70 ,
   word: 'which', count: 69 ,
   word: 'more', count: 68 ,
   word: 'discovery', count: 67 ,
   word: 'such', count: 67 ,
   word: 'techniques', count: 66 ,
   word: 'process', count: 59  ]

If you’re interested in seeing the rest of the code, check out the fully commented source6.

A good exercise going forward would be to take this application to the next level. You could optimize the text parsing, extend the search to multiple pages of Google results, even strip out common words that aren’t really key terms (like “that” and “from”). More bug handling could also be added to make the app even more robust — when you’re mining data, you want as many layers of redundancy as you can reasonably afford. The variety of content that you’ll be pulling in is such that inevitably you’ll come across an unexpected piece of text that, if unhandled, would throw an error and promptly crash your application.

In Conclusion

As always, if you find anything related to web scraping with Node.js that you think is helpful or just have questions or thoughts you want to share, be sure to let us know via the comments below. Also, follow me on Twitter7 and check out my blog8 for more on Node.js, web scraping and JavaScript in general.

(il, rb, al)

Footnotes

  1. 1 http://nodejs.org/
  2. 2 https://www.npmjs.com/
  3. 3 https://github.com/request/request
  4. 4 https://github.com/cheeriojs/cheerio
  5. 5 http://en.wikipedia.org/wiki/Data_mining
  6. 6 https://gist.github.com/elliotbonneville/1bf694b8c83f358e0404
  7. 7 https://twitter.com/bovenille
  8. 8 http://heyjavascript.com

The post Web Scraping With Node.js appeared first on Smashing Magazine.

View original article: 

Web Scraping With Node.js

Thumbnail

Creating Well-Behaved Sites With The Page Visibility API

We’re all resigned to it: launching a browser reloads every tab you previously had open, blasting a cacophonous mix of sound and video. While browsers have made it easier to control this experience with tab icons and extensions like MuteTab, for most people this behavior presents a confusing and disorienting experience. As developers and designers it’s our job to make the web welcoming, not overwhelming.

Doesn’t it make sense that sites should only be active when they are the primary focused tab? Why are we burning up batteries and processor cycles with animation that can’t be seen?

Thankfully, there is a solution: the HTML5 Page Visibility API. You can see it used particularly effectively in recent projects by Active Theory1, such as their work for Under Armor2 and A Spacecraft For All3: click to another tab and you’ll find that the multimedia presentation pauses and the music fades away. This behavior typifies what I like to call the “polite web”: sites that are considerate of users’ attention, bandwidth and abilities.

In the past developers have tried to create this behavior by adding onblur() and onfocus() window handlers. While they are considerably better than nothing, the approach is limited by the fact that it can’t tell if the window is actually hidden to the user. For example, having two browser windows side-by-side and switching between them would still register as onblur() or onfocus(), even though the content in each remains perfectly visible.

Implementing Page Visibility

While there are many possibilities for using the Page Visibility API, perhaps the most obvious use case is when a page contains video: there’s usually little point in continuing to play video content if the user can’t see it.

<video autoplay controls id="videoElement">
	<source src="rar.mp4">
	<source src="rar.webm">
</video>

<script>
var videoElement = document.getElementById("videoElement");
document.addEventListener("visibilitychange", function() 
if (document.hidden)      
videoElement.pause();  
 else 
videoElement.play();   
 
});
</script>

The one problem with this is that it’s a little abrupt: the audio on the video starts and stops as if cut off with a guillotine when the user switches tabs. The presentation could be improved significantly by fading the audio in and out as the tab is focused. If we’re using jQuery, we can employ a neat use of the animate method to do so.

<script>
var videoElement = document.getElementById("videoElement");

document.addEventListener("visibilitychange", function() 
if (document.hidden)      
$("#videoElement").animate(volume: 0, 1000, "linear", function() 
videoElement.pause();
);
} else 
videoElement.play();  
$("#videoElement").animate(volume: 1, 1000, "linear");
} 
});
</script>

The Spec

The Page Visibility API spec is surprisingly simple, consisting of just two methods: document.hidden returns true or false depending on the status of the browser window; and those same states are reflected in string form for document.visibilityState (hidden and visible, respectively, with two optional values of prerender and unloaded), with visibilitychange available as an event. document.visibilityState can be useful in determining why a document is not visible, but otherwise document.hidden covers most needs.

Precautionary Principles And Browser Support

The Page Visibility API takes a deliberately conservative approach to reporting a hidden document: your page will be reported as hidden if the user switches to another tab in the same browser window, but not if you move another window to obscure your page. The API is not foolproof, and will return false positives in some circumstances, erring on the side of caution.

Support for the Page Visibility API is excellent4: all modern browsers, with the exception of Opera Mini, fully support the API, including IE10+. Vendor prefixes are also refreshingly absent from implementation of the spec: the only browsers that currently need a –webkit prefix are Android and Blackberry. For that reason (and for the sake of simplified illustration) I have not included any prefixes in the code samples above, although they are easy enough to include and test for.

The Page Visibility API is also an excellent example of progressive enhancement. If the browser doesn’t support the API, the script will be ignored, and the user will simply be subjected to the usual uncontrolled cacophony.

Other Uses

While audio and video content are obvious candidates for using the Page Visibility API, there are many other possibilities: pausing a slider or presentation while the site remains out of focus, or changing the visual state of the page as it remains neglected.

Considerate use of the API can help make the web a better, greener, and more responsible place, and I would strongly encourage developers to consider how to integrate it into their projects.

(il, og, ml)

Footnotes

  1. 1 http://activetheory.net/
  2. 2 http://gisele.underarmour.com/
  3. 3 http://spacecraftforall.com/
  4. 4 http://caniuse.com/#feat=pagevisibility

The post Creating Well-Behaved Sites With The Page Visibility API appeared first on Smashing Magazine.

Originally posted here: 

Creating Well-Behaved Sites With The Page Visibility API