Bionic Gloves is an online store that designs and sells a range of gloves, such as golf gloves, fitness gloves, and more. Their focus is to provide customers with gloves that have fine grip, comfort, and durability.
To increase sales from their eCommerce shop, they decided to optimize their website. The task was given to Portland-based marketing & conversion optimization agency, Sq1.
To test their hypothesis that removing the ‘special offer’ and ‘gift card’ code boxes from the shopping cart page would result in more sales and less cart abandonment, they set up an A/B test in VWO.
This is how the original shopping cart page looked like:
The test was run on close to 1400 visitors for a duration of 48 days. This is how the variation page (without the code fields) looked like:
The primary goal that they were tracking was the revenue made. The variation won and increased the total revenue by 24.7%, and revenue per visitor by 17.1%.
Why the Variation Won?
In the words of David from Sq1, “Anytime you leave the door open for a user to leave the conversion funnel – even if it seems like they’d come right back – you risk losing sales. By showing the Promo Code field on the cart, users were enticed to leave the site in search of a promo code. At that point, the conversion process is interrupted and you are more likely to lose potential customers. As such, hiding it was a very logical test.”
A shopping freak myself, I wouldn’t lie that I, too, have gone looking for coupon codes a number of times in the middle of my purchasing process. This, as David pointed out, has a number of risks:
The sight of the coupon box triggers visitors to look for one on Google and other places. I did a quick Google search of “Bionic Glove”, and look what I found in the auto-complete searches:
eCommerce websites also risk losing money to affiliates and websites offering deals, coupons, etc.
Many a times, visitors end up finding a better deal on another web store.
To avert this, I have seen many websites now show all available coupon codes right on the product page and also on the cart page. Not only does this help them reduce cart abandonment, but also helps them increase their average order value as many shoppers go ahead and buy more stuff to cross the threshold at which coupons can be applied.
See how Myntra, a fashion ecommerce website based out of India, does this beautifully:
Tell me what you think about this case study in the comments section below. I am also available for intellectual discussions on CRO and A/B Testing which can fit in less than 140 characters on Twitter @taruna2309. See ya!
Over in startup land, one of the big stories of 2014 was, without a doubt, the success of Product Hunt1. It’s is a community where people post, vote on and comment on new products they’ve discovered or launched. Whether you’re looking for the next big thing to invest in or just want to find a better weather app, Product Hunt has got you covered.
Coincidentally, in addition to being a fan of the website, I also have a pretty personal connection to the company. I’ve been online friends with Product Hunt’s designer Jonno Riekwel2 for years, and I was part of founder Ryan Hoover3’s previous project, Startup Edition4.
All this to explain that Product Hunt has been on my radar for a while now. In fact, what many people don’t know is that Ryan initially considered7 basing Product Hunt on Telescope8, an open-source app that I had just started working on! Sadly, Telescope was still too immature at the time, and Ryan wisely decided to build his own solution.
But I’m here to tell you that things have changed. As we know, Product Hunt has gone on to become Silicon Valley’s latest darling. But Telescope has also kept quietly evolving, and it’s now a full-featured solution for people to build their own communities. So, today, let me show you how you too can set up your own personal Product Hunt-style website.
If you’re on Mac OS or Linux, installing Meteor is as simple as opening a Terminal window and typing:
curl https://install.meteor.com/ | sh
Though Meteor doesn’t quite support Windows yet, a preview release13 is available, and an official version should be coming very soon.
Next, you’ll need to clone Telescope’s code base from GitHub14. Just type:
Almost there! First, drill into the newly created telescope directory with this:
Then, run your app with this:
Your app might take a minute or two to initialize the first time. But once it’s done, your brand new Telescope app should be up and running! Head to http://localhost:3000 to check it out.
If at any point you need to stop your Meteor app (maybe it’s crashed or you want to get the command prompt back), you can do so by opening the Terminal tab where your app is running and hitting Control + C.
Let’s personalize the app a bit. The first thing to do is create a user account by clicking the “Sign Up” link in the top navigation menu. Because this will be the first user account you’ve ever created, it will automatically be assigned administrative rights, which will let you access Telescope’s admin area.
Once your account is created, head to the “Settings” page via the “Admin” menu. There, you can set various options, such as your website’s title, description and color scheme.
For example, try setting the header’s color to #1282A2 (or any other color!), and then click the “Submit” button at the bottom of the form.
Activating “Product Hunt Mode”
Telescope was originally based on the popular hacker hangout Hacker News, which is why the default view is just a uninterrupted list of posts. But with the success of Product Hunt, more and more people started asking for an option to use a day-by-day view, which I like to call “Product Hunt mode.”
To enable it, just go to the “Settings” page, scroll down to the “Default View” option, and set it to “Daily.” Save the settings and refresh the home page. You should now see a much nicer layout!
Let’s show the world the result of our hard work. Deploying an app is usually a painful process, but not today. It turns out that Meteor provides its own free hosting service for quickly deploying small apps and prototypes.
Open a new Terminal window, browse to your app’s directory, and then type the following:
meteor deploy my-app
Replace my-app with the name of your app.
If this is your first time deploying on Meteor’s service, you’ll be prompted to create a developer account25.
Once the deployment process is finished, your app will be live at http://my-app.meteor.com!
Note that the settings of a Telescope app are stored in its database, which is not transferred as part of the deployment process. So, you’ll need to re-enter any settings that you’ve previously set.
Customizing Your App With Packages
So far, we’ve changed the default view and tweaked a few colors. We can take things much further than that. Let’s start by adding a few extra CSS customizations.
To do so, we’ll create our own package. At this point, we could just go ahead and modify Telescope’s code, but that could lead to Git conflicts when we try to update the app down the road.
Instead, we’ll use Meteor’s package system to make sure that any customizations we implement stay independent of the main code base and will not be overwritten even when we pull in future updates.
Inside your app, locate the /packages directory, and create a new telescope-custom directory inside it (either with the mkdir command or by going to File → New Folder). From now on, we’ll put the files we create inside this directory.
Each Meteor package needs a file named package.js, which contains the package’s manifest, a simple set of instructions that tell Meteor what the package actually contains.
So, create the package.js file inside telescope-custom, and type the following:
The Package.describe block provides some basic information about the package. But the interesting part is the Package.onUse block, which tells Meteor which files to load in our package. In this case, we’ll say we want to add a custom.css file.
Remember I said Meteor could handle code on both the client and server? This is why we have to specify where we want to add our files — in this case, on the client.
All that’s left is to actually create this custom.css file!
Customizing The CSS
Let’s open the custom.css file we just created and add Product Hunt-style yellow hover states to each of our posts:
We’re almost done. We still need to tell Meteor that we want to use this new package. We can do so with the following command:
meteor add telescope-custom
Once you’ve added the package, your browser tab should automatically refresh, and you’ll see that our new yellow hover state is working!
But wait, we didn’t even tell Meteor where to import that new style sheet from. How can that even work?!
This is one of Meteor’s cool time-saving features. It’s smart enough to notice when you add a new CSS file to your code base. It will automatically concatenate it, minify it and load it in your app.
A nice touch in Product Hunt’s UI is how it includes banners between days. It’s a great way to link out to other content and break up the monotony of listed links, while maintaining a clean layout and not disrupting the experience. Let’s see how to do the same thing with Telescope.
Telescope already has templates before and after each day, but the default templates are empty. So, our mission will be to override the afterDay template with one that contains actual content.
Add a banner.html file to your package, and add a simple banner:
<div class="custom-banner"><a href="http://twitter.com/telescpe">Follow us on Twitter!</a></div>
templates["afterDay"] = "banner";
Before calling the afterDay template, Telescope will check in the templates array to see whether we have provided a replacement. And since we have, it will use the template code from banner.html instead!
We’re not quite done yet, though! Just as before, we need to add our new files to the package’s manifest. And this time, we’ll also need to add package dependencies to it.
We’ll add the templating package, which lets us use templates, as well as the telescope-base package, which exposes the global templates object that we’re calling in custom.js. So, let’s go back to package.js and modify its contents:
Things are looking good, but you’ll notice that our banner is repeated after every single day.
This might get a little repetitive. Let’s see how to show our banner after only the third day.
<div class="custom-banner"><a href="http://twitter.com/telescpe">Follow us on Twitter!</a></div>
We haven’t yet defined what isThirdDay is supposed to be. We’ll do that through a template helper.
isThirdDay: function ()
return this.index == 2;
Because our afterDay template is called from a loop, we can access the current day’s index using this.index. And because that index is zero-based, the third day will be the one whose index is 2, which in turn gives us the contents of our helper.
Here you go! Now our banner is appearing after only the third day and no other.
We’re running out of time, but this is by no means the extent of Telescope’s features. Out of the box, Telescope also makes the following possible:
Add a banner to sign people up to your email list.
Generate and send a newsletter of the best posts on your website.
As Product Hunt’s success has shown, there’s a big demand for websites that help us deal with information overload by streamlining and centralizing content. Telescope is a fast, modern platform on which to build your own community, social news app or link-sharing website. And, as you’ve just seen, extending it is very easy. I encourage you to give it a try. And if you end up building the next Product Hunt with Telescope, I hope you’ll remember who first told you about it!
Thanks to Loren Sands-Ramshaw, Phil Pickering, Nigel Anderson, Austin Stoltzfus and others for reviewing a draft of this article.
Mobile application ecosystems — let’s count Android and iOS here — are unbelievably dynamic, but they also suffer from both software and hardware fragmentation. This is especially true for Android, but fragmentation also exists in the iOS ecosystem1, as experienced with the rollout of iOS 8. As the latest version of iOS was released, many existing apps were made clumsy on updated devices.
Even the new iPhone 6 and iPhone 6 Plus have had not-so-typical issues2 for Apple devices. In addition, a significant proportion of users with older devices have very few options: essentially, buy new hardware (i.e. a new device) to get everything working well.
In the Android world, things are different. As OEMs launch new devices, software updates and customizations for their devices, application and game developers get serious headaches3 trying to keep their latest products up to snuff and fully compatible with all possible device variants. Making a certain app or game work only on high-end devices is out of the question, though. Why would a developer want to miss out on a significant chunk of potential users?
Professional automated testing software is a solution to a common problem: how to produce high-quality, robust and reliable software with the ever-growing complexity of technology and under massive competitive pressure. Automated software testing is a cost-effective solution to this problem. Not to mention, it provides three business benefits:
increased testing efficiency,
increased testing effectiveness,
faster time to market.
This article walks through a sample use case for test automation and provides a downloadable example to get you started. Also, we’ll focus on different aspects of mobile test automation and explain how this relatively new yet popular topic can help mobile app and game developers to build better, more robust products for consumers. With the advanced example later in the article, we’ll show how image recognition can be used to test mobile games; specifically, we’ll run Appium’s test automation7 framework against Supercell’s Clash of Clan8 game to illustrate how image recognition can be built into the test-automation process.
Test Automation Is Perfect For Mobile App Development
Developing mobile applications is very different from developing PC software or even embedded software. Mobile development is meant to be agile9, and a lot of great tools and practices have been developed for that agility. However, doing something manually — such as testing an app — is never agile, which is why test automation has shown tremendous growth among app and game developers, speeding up their doings and yielding robust and better results.
To achieve compatibility between users, devices and the market, including test automation as a part of the agile development process10 is typical. Fortunately, a lot of tools are available, and test automation is a perfect fit for this process. For example, let’s say your typical development sprint is two weeks. You have daily standups and a lot of scrum activities, and you own internal policies that gear development to the final product. Test automation offers a significant value-add by enabling testing to be done in parallel — for example, as nightly test sessions. By the next morning, the tests will have been finalized and the results of the latest regression will be ready for review. Fixing an issue earlier will save a lot of time and get developers to finalize the product sooner; most importantly, it cumulates to better quality11, with fewer bugs.
Test automation offers the possibility to test mobile apps instantly and effectively. Once tests have been automated, they can be executed quickly and repeatedly, again and again. In almost all cases, this is the most cost-effective method for regression testing15 of software products that have a long maintenance life. In fact, test automation of any mobile app is the best way to increase the effectiveness, efficiency and coverage of the testing process. The true benefit of automation comes not only from the repeatability of tests, but also from the ability to execute tests that probably could not even be performed manually.
Things To Consider When Automating Mobile App Testing
Let’s look at how test automation (as opposed to manual testing) can improve the development process, add value and speed up development.
Costs and Assets
Regardless of whether you go for manual or automated testing, you’ll need the following assets and resources (all of which cost money): time, people, infrastructure, tools and training. Depending on the size of the project and the application, test automation will quite obviously provide a good return on investment. For example, once test cases have been created, automated tests can be run over and over again at no additional cost, and these can be more rapidly completed than any manual tests. Automated software testing can reduce the time required to run repetitive tests from weeks to hours. This is a significant time-saving that translates directly into cost-saving.
Integrated App Development And Testing Cycle
Increasing efficiency and productivity with automation actually starts with adopting a new mindset. Software tests have to be repeated often during all development cycles to ensure the best possible quality of an application. Every time source code is modified19, software tests should be repeated. For each release, your software should be tested on all supported variants of operating systems and all configurations of hardware. Manually repeating these tests would be costly and time-consuming. For example, running comprehensive tests manually on all variants of Android and on actual devices would take a lot of time.
Tools and Technology: Test Automation Frameworks
To get the most out of your efforts and maximize testing coverage, select the most robust and most cross-platform20 method. That automated methods can be used both to validate requirements and to reduce costs through automated test-case generation is well known. However, the full automatization of large software entities also comes with a cost that many companies haven’t been ready to pay for. Historically, one of the reasons has been a concern over the lack of adequate integration with well-established development life cycles.
Test Coverage and Reusability: Open Standards Mean No Vendor Lock-In
Automated testing can increase the depth and scope of tests and significantly improve software quality. Lengthy and thorough tests — often not doable with manual testing — can be run automatically. Ideally, test cases should have full access to an application and test all aspects of it21 — memory contents, data tables, file contents and internal program states — to determine whether the product behaves as expected. Automated tests can easily execute thousands of complex test cases during every test run, providing coverage that is simply not possible with manual testing. Developers, freed from repetitive manual tests, will have more time to create new automated test cases and build more compelling features (and then more test cases).
Improve Effectiveness and Finalize Sooner
In a nutshell, professional automated testing software is a solution to a common problem: how to produce high-quality, robust and reliable software with the ever-growing complexity of technology and under massive competitive pressure. Automated testing improves business results in three ways: greater testing efficiency, greater testing effectiveness and a shorter time to market.
Different Ways to Automate Mobile Testing
In general, there are three ways to automate the testing of mobile apps.
Handwritten Test Scripts
Typically, this is the best choice when you know what you’re doing and when you have programming-capable people doing the test scripts. Plenty of options are available for test automation frameworks, tools and integration — both commercial and open-source offerings. Having your engineers write all of the test scripts will take up some time and tie up resources, but it will get you exactly what you want: well-structured, thorough scripts that test precisely the aspects of your app that you want to test.
This approach is less error-prone because nothing needs to be written in code, but it is typically more limited in functionality. Tests can be quickly recorded and then played back again and again against different OS versions and device configurations. These tests focus on user interactions and user-driven activities. Some things might fall beyond a test’s scope (such as integration with other technologies and compatibility with other software).
Automatic Test Exercisers
Automatic test exercisers provide a great way to smoke-test applications. No specific tests are needed; rather, the focus is on testing user-interface logic (such as opening menus, clicking buttons, swiping and multi-gesture actions). Automatic test exercisers yield the least exact results but provide quick feedback on each iteration of an app.
Focus Areas In Testing For Mobile Apps And Games
User Interface and Functionality
A user interface and its overall functionality will directly affect how successful your app or game30 will be. These two things, which encompass visual appeal and gameplay, are the most important things to get right — and you must ensure that device fragmentation doesn’t break any of these. Various things in the UI need to be tested:
UI layouts and elements
Games especially are typically targeted at a high number of different screen resolutions and screen types. Regression testing should be done each and every time the UI’s layout changes to ensure that the game works.
Menu structure and functions
Testing menu structures, functionality and behavior can be automated with instrumentation and the help of different test-automation frameworks.
Surprisingly, so many apps and games out there get this wrong. If a screen’s orientation changes during an interaction, for example, what happens? What is supposed to happen? Does the app or game work well in both landscape and portrait modes?
A lot of screen resolutions exist, especially on Android, and auto-scaling will usually help developers. However, test your game across these resolutions to ensure that the graphics do not stretch.
Performance needs to be consistent across all device variants31 among your users. Because of this, test on as many real devices as possible. To determine how well your app or game responds to various levels of usage, including performance and battery usage, consider creating tests that last for hours. To determine whether your game runs effectively under a heavy load for a long time, run load (or stress) tests. These performance tests will measure, for example, how responsive your game is on real devices.
Usability and User Experience (i.e. Good Entertainment)
Testing usability, navigation flow and user experience simply cannot be done on a desktop with a mouse and keyboard. So, forget emulators and use only real devices35. And to test how usable and entertaining your app is, consider these two important things:
User interaction and responsiveness
Testing performance is critical because this will make make or break the user experience. Performance lag, for example, is easy to expose with real devices.
Interruptions, battery consumption and the effect of battery chargers on overall performance and usage all have a significant impact on the user experience — and entertainment value.
Nowadays, multi-user support is common in both apps and games. Testing multi-player capabilities is important and is naturally more challenging, requiring real users to measure performance. A typical case is a game communicating with the back-end server. In this case, connectivity is essential, to synchronize the back end with devices that need to get information about the gameplay. You should test a ton of different scenarios, many of which could severely affect the game’s experience, resulting in negative feedback and the game being uninstalled by users.
Integration with social networks is another important factor. Being able to share something across an ecosystem, with friends or just with oneself, is essential in many apps. Test this thoroughly with real Android and iOS devices, with different OS versions and different device configurations, to assess functionality and ease of use.
Security and Liabilities
Nearly all developers use some open-source components in their apps. This practice is widely accepted and highly recommended because it offloads the development of code for non-core functionality. However, identifying vulnerabilities and licensing restrictions with third-party code is often neglected by developers.
Breakdown: Android Test Automation Frameworks
Robotium42 is an Android test automation framework that fully supports native and hybrid applications. Robotium makes it easy to write powerful and robust automatic black-box UI tests for Android applications. With the support of Robotium, test case developers can write function, system and user acceptance test scenarios, spanning multiple Android activities.
UIautomator5143, by Google, provides an efficient way to test UIs. It creates automated functional test cases that can be executed against apps on real Android devices and emulators. It includes a viewer, which is a GUI tool to scan and analyze the UI components of an Android app.
Espresso44, by Google, is a pretty new test automation framework that got open-sourced just last year, making it available for developers and testers to hammer out their UIs. Espresso has an API that is small, predictable, easy to learn and built on top of the Android instrumentation framework45. You can quickly write concise and reliable Android UI tests with it.
Calabash46 is a cross-platform test automation framework for Android and iOS native and hybrid applications. Calabash’s easy-to-understand syntax enables even non-technical people to create and execute automated acceptance tests for apps on both of these mobile platforms.
And then there’s Appium. OK, let’s get into this one.
Appium: Executing Tests On Real Devices On A Cloud Service
In its architecture, Appium is an HTTP server written in Node.js that creates and handles multiple WebDriver sessions. Appium starts tests on the device and listens for commands from the main Appium server. It is almost the same as the Selenium server that gets HTTP requests from Selenium client libraries.
Appium enables users to execute tests on mobile devices regardless of OS. This is possible because the Appium framework is basically a wrapper that translates Selenium’s WebDriver commands to UIAutomation50 (iOS), UIautomator5143 (Android, API level 17 or higher) or Selendroid52 (Android, API level 16 or lower) commands, depending on the device’s type.
For Android, this is how Appium compares to other test automation frameworks:
One of the best things about Appium is that, despite sounding architecturally complex, it actually isn’t — at all. For developers, it provides support for various programming languages, freedom from having to select tools, compatibility across the most important platforms (Android and iOS), freedom from having to install and configure devices to test and more.
If you are familiar with Selenium, then you’ve got Appium covered. An Appium test is pretty much the same as a Selenium test: They use the same WebDriver, and DesiredCapabilities59 is used the same way. Configuring an application to run on Appium has a lot of similarities to Selenium — for example, those DesiredCapabilities. We’ll configure a sample test later in this article.
Also, Appium includes a component called the inspector. This inspector enables a host of functionality — for example, showing all of those UI elements in the application and enabling basic recording and playback.
However, you may not need the inspector because everything can be done in the code. The example later provides all of the scripts and instructions step by step, without using it.
If you’re on Linux, install pip if it’s not already:
$ sudo apt-get install python-pip
Install the Selenium module for Python:
$ sudo pip install selenium
Verify that Selenium is installed:
$ pip freeze | grep -i selenium
Ensure that Python 2.7.x or later is installed:
$ python --version
If it’s not installed, download and run the setup from Python’s download center64. To add Python environment variables, go to “System properties” → “Advanced System Settings” → “Environment Variables” → “System Variables” → “Edit Path,” and then insert C:Python27;C:Python27Scripts at the end (assuming you have installed Python in the default location). Make sure to restart the command prompt to bring new environment variables into effect.
Check whether Python’s pip module is already installed:
appium_example_android.py (for Android API level 17 and above)
appium_example_selendroid.py (for Android API level 16 and below)
Step 1: Create Account and Upload the APK
Create an account.66 Note that this service provides a freemium option, and you don’t need a plan to complete these examples using real devices on a cloud service. If you want to access hundreds of device models, then different plans are available.
Before proceeding with running the test script, you’ll need to upload the APK or IPA to Testdroid’s cloud service via HTTP POST. Let’s try that using cURL.
Open the test script, appium_sample_ios.py, in any text editor. Set screenshotDir to the path where you want those screenshots to be saved on your machine. Set your credentials on testdroid_username and testdroid_password in DesiredCapabilities. Set the myAppFile identifier against testdroid_app in DesiredCapabilities. It should look like this:
The screenshots will be available locally on your machine in the directory that you specified in step 2. Log into your cloud service and navigate to the project’s name, as defined in the _project attribute of DesiredCapabilities, to access the following log files:
Appium’s server log,
Logcat and instruments log.
Advanced Example: Using Appium For Mobile Game Testing With Image Recognition
In this example, We’re using Supercell’s popular mobile game Clash of Clans. It’s a fantastic game and I’ll bet many of you have played it, so you should be pretty familiar with its look and gameplay. This example does basic clicks through the game’s tutorial.
## Example script that tests Clash of Clans tutorial first steps
## Works on different resolutions, both iOS and Android
from time import sleep
from AppiumCloudTest import AppiumCloudTest, log
from selenium.common.exceptions import WebDriverException
# AppiumCloudTest takes settings from environment variables
driver = self.get_driver() # Initialize Appium connection to device
sleep(10) # Wait for game to load
# Use this to get detected screen hierarchy
# print self.driver.page_source
# Dismiss the in-app purchases dialog if it shows
okayButton = None
okayButton = driver.find_element_by_id('button3')
log("There was no in-app purchases dialog")
else: # iOS
self.driver.implicitly_wait(5) # Wait only 5 seconds to find it
okayButton = driver.find_element_by_accessibility_id('Okay')
# No need to sleep here
log("There was no in-app purchases dialog")
# Cancel iOS Game Center login
cancelButton = driver.find_element_by_accessibility_id('Cancel')
log("Cancelling iOS Game Center login…")
log("The Game Center login was not displayed")
# Check that a goldmine is on screen
rect = self.find_image("queryimages/tutorial_goldmine.png",
"There should be a goldmine on screen at beginning of tutorial")
log('Gold mine found at %s %s! Continuing…' % (rect, rect))
# Dismiss the bubbles
sleep(2) # second blabla
sleep(2) # Goblin appears
# Go to shop
# NOTE: tap_image does also assert, fails test if target not recognized
# Buy cannon
# Place the cannon
# Use gem to finish right away
# Bring it on!
# To be continued…
if __name__ == '__main__':
Let’s look some of the stages in this test script. The test_tutorial contains the following steps:
It first figures out whether the test executes on Android (self.isAndroid()) or iOS. As you can see, it looks for UI content differently. On Android, it tries to find a button by using an element’s ID (element_name) and on iOS by using an accessibility ID with a description (“Okay”). The same check happens for iOS’ Game Center login.
Screenshots are taken at various steps and stored in files entered as a parameter in a function call.
A check is done to see whether “goldmine” appears on screen by comparing two PNG files using a self.find_image call. If these pictures match (i.e. “goldmine” appears on screen), then the script continues executing the game’s tutorial.
The tutorial proceeds with the following steps:
Go to shop.
Place the cannon.
The information about all three of these items is stored in those PNG files: shopbutton.png, cannon.png and place_the_cannon.png.
Finally, the tutorial finishes and the battle starts. After the battle, the application is closed.
The video below shows how the test executes at each step. For the video, I’ve used three devices: one iOS (iPhone 4S) and two Android phones (Samsung Galaxy S3 Mini and HTC One X). You can also watch the video on YouTube70.
How Is Image Recognition Used Here?
The example shown uses image recognition (i.e. template matching71) to identify which features — basically, pixels and graphic content — are shown on screen and to compare the two pictures to each other. The algorithm built to recognize images was used on real devices and on two different platforms (Android and iOS), and it used a single test script for both platforms. This sort of image comparison is even very handy for recognizing UI elements and graphics that are resized and/or rotated.
Let’s say the template image has some distinctive features, such as text that can be easily abstracted from the background content. In this case, feature-based recognition can be used. In our example, if the “Button” text has been resized or rotated (or otherwise transformed), we can quickly and easily identify this and take further action.
The following functions explain the approach of comparing images:
def tap_image(self, query_image,
width_modifier=0.5, height_modifier=0.5, retries=2):
retries_left = retries
rect = None
while retries_left > 0 and not rect:
rect = self.find_image(query_image)
retries_left = retries_left - 1
image_name = os.path.split(query_image)
self.assertIsNotNone(rect, "Image %s is on screen" % image_name)
x = rect+rect*width_modifier
y = rect+rect*height_modifier
log('%s button found at %s %s (%sx%s), tapping at %s %s' %
(image_name, rect, rect, rect, rect, x, y))
def tap(self, x, y):
log('Tapping at %s,%s' % (x,y))
driver = self.get_driver()
touch_actions = TouchActions(driver)
action = TouchAction(driver)
else: # iOS
action = TouchAction(driver)
# TODO: Temporary hack
# On iPhone 4S = res. 480 x 320 for taps but screenshots are 960 x 640
Testing is crucial to ensuring success in the highly competitive landscape of mobile apps and games72. But even poorly planned testing can take up 20 to 50% of your total development effort, in which case it would also account for the single biggest cost in your budget. To ensure that testing is extremely efficient, covering the breadth of today’s mobile ecosystems and device models, the best option is an online cloud-based service.
If you only start thinking about testing a few days before the app hits the market, it’s too late. You’ll need to test a wealth of elements, data and functionality from day one. Here are some things to consider in making testing a part of your development process:
Plan carefully: Automate generic processes as much as possible.
When you’re building a mobile app, a well thought out strategy is critical, a great user experience and design are paramount, and solid development and testing are fundamental. Many aspects of testing can be automated, and this automation will increase the depth and scope of your testing and significantly improve the app’s quality. Ideally, test cases should have full access to the application and test all aspects of it — memory contents, data tables, file contents and internal program states — to determine whether the product performs as expected.
Your app will change during development: The same goes for testing.
Choose a testing technology and provider you can grow with.
If you already have an app on the market and are looking to create a similar one, then select a technology74 and vendor that meet your needs. For example, building your test cases according to a certain method or framework will enable you to reuse those test cases for your new application — at least to some extent. So, choose a technology and vendor that is able to handle your needs as your product scales up and as your testing has to cover new geographical areas and even support new platforms (for example, going from Android to iOS).
Test automation is available 24/7.
Automation will reduce the time it takes to test new features and even the app itself by running 24/7.
Use a cloud-based platform for truly global reach.
With an online cloud-based service, you’ll get instant access to hundreds of real Android devices75. Especially with Android, having access to devices that are used in volume holds significant value. Running automated tests on these devices is easy and fast and provides all of the information you’ll need, preprocessed, summarized and in full detail.
What are your thoughts on test automation for mobile? Please let us know in the comments section!
Ever found yourself puzzled with a sudden spike or drop in test results? Did it finally take you hours of brainstorming and discussions to figure out what caused the change? Now you can avoid all that hassle by simply adding notes to your graphs.
VWO allows you to annotate your graph reports with relevant notes to make your analysis more meaningful. Think of these as sticky notes on your graph reports. You can use these annotations to add comments, mark important events and even add relevant links to your graphs.
Let’s say you are the optimization expert with a large software product company. To get more users for your product, you are running a campaign on your ‘sign-up’ page with a goal to improve conversions. Meanwhile, your marketing team decides to launch a well-targeted newsletter campaign that directs visitors to the sign-up page. The awesome campaign results in a sudden spike in conversions on your sign-up page, right on the date when the newsletter was sent.
A few months down the line, when you analyze the results of your campaign, it would be hard for you to identify what caused the sudden spike in conversions. This is where annotations come to your rescue. Annotations allow you to add context to your results by including notes for other activities/events happening during the test duration. The moment you see the comment with newsletter campaign in your graph, you know what to attribute the spike to.
The annotations are visible to all users in your account so that everyone can have similar context for interpreting results. It also comes with the added advantage of viewing who added the annotation so that any further discussions can flow seamlessly.
Check out this article in our knowledge base to get a step by step view of how to use annotations for your account.
And There’s More…
1. Now you can make changes to the iframe tag on your website. The iframe editor allows you to edit iframe URL, change style and placement, and so on. Simply load the webpage containing the iframe content into the campaign builder and you are ready to go!
2. The select parent option expands to allow you to select the largest parent from any element on the page. If you are making changes to an element on your page, use this option to go up to the largest parent of the element.
Let us know about your experience with this month’s updates in the comments section. And we would love to hear if you have anything to share with us about VWO!
Product pages are the doorways that lead to conversions and revenue for an eCommerce store. A good product page plays a pivotal role in taking a customer from just browsing to actually adding the product to the cart. Most eCommerce websites, therefore, invest a significant portion of their time and money in making their product pages more attractive and persuasive.
But then, how do you do this?
Essentially, it comes down to creating a great User Experience for your customer. An effective User Experience (UX) requires relevant and useful information designed in a manner which takes the customer seamlessly through the buying process. In this blog post, we are going to talk about 15 product page elements that add to the UX for your product pages and makes them high-converting. You will be familiar with most, if not all, of these. My intention here is to create a short checklist for every element that you can use to optimize that element. There’s also a surprise for you at the end of this post, so hold on!
1) Product Name
Product Name should be descriptive. This gives more clarity to the customer as well as boosts SEO for that page. For example, ’24 Carat Gold Ring’ is much more descriptive and SEO-friendly than ‘Gold Ring’
The name should be unique. This sounds like common sense but you’ll be surprised by how often eCommerce websites give similar names to their products which makes finding them through search really difficult. Needless to say, this also negatively affects the SEO for the products.
2) Call to Action (CTA) Button
This is the most important element on any product page. The CTA affects the decision – making of the customer through its size, color and text. Thus, marketers should pay significant attention to optimizing all these aspects.
Different colors denote different things across different cultures. When deciding on color, keep in mind two things. Firstly, whether that color triggers the emotion in your target audience that you are hoping for. And secondly, how does it contrast with the color scheme of the rest of the page. Ideally, you want the CTA to stand out so that it grabs customer attention.
When deciding on the text, keep in mind that certain words mean different things in different countries. For example, see how Amazon changes the CTA text on its US and UK websites to adhere to the local flavor.
Amazon US Page
Amazon UK Page
3) Product Price
Understandably, this is the least controllable of the page elements. As a marketer, your main focus should be the appearance of the price tag.
The price should be placed close to the Buy button. In fact, the product name, price and CTA should form a kind of visual hierarchy so that the act of buying flows seamlessly in the mind of the customer.
Use multiple images clicked from different angles to give the overall look of the product.
Use inspirational product images (product image used in a setting; for example, a customer wearing the dress that is displayed on the product page). These images add social proof as well as make the product more desirable.
5) Product Description
Keep product descriptions short and make sure to include important keywords so that the page ranks well in search engine rankings
6) Quantity Option
Quantity option takes a very small amount of screen real estate but can lead to a bigger sale. For example, if I want to buy 5 copies of a book, I shouldn’t be forced to go to the page and click 5 times on the ‘Buy’ button. Not having this element means you are potentially leaving a lot of money on the table.
It should be placed near the CTA.
7) Product Reviews & Testimonials
According to an iPerceptions study, 61% of customers read online reviews before making a purchase decision, while 63% of customers are more likely to make a purchase from a site which has user reviews.
The problem with having reviews on your product pages is that so many pages just have so few reviews. This can result in negative social proof. To combat this, follow the eBay model of buyer/seller feedback. That means, as soon as someone purchases a product, send them an email asking for feedback about the product. Many customers feel that such feedback request is part of the buying process and they are more likely to fill in the feedback form. This will ensure that your review section is filled.
Ask people to rate reviews. This ensures that the most helpful reviews rise to the top.
8) ‘Add to Wish List’
This option is particularly helpful in case of indecisive customers who are evaluating your product. Also, while browsing, some customers may stumble upon a product which they like but are not yet ready to buy. This option gives them the flexibility to ‘bookmark’ the product to which they can return later.
9) Cost Savings
Many times, if a product has been discounted, the product page shows the original price along with the discounted price. This is a smart trick used by marketers to cash in on the loss aversion tendency of people.
Show both the percentage saving as well as the actual saving made on the product. Different customers are induced by different messages.
10) Cross-selling & Up-selling Options
As a business owner, you want the user to purchase add-ons, related products and accessories of the products they buy. One way to achieve this is to provide good options for up-selling and cross-selling on the product pages.
Good suggestions for similar and related products not only improve the browsing experience but also aid in product exploration.
11) Social Media Integration
Social media buttons provides the friends/followers with social proof, which may result in increased purchases of the same item.
It helps to spread brand awareness.
It may also increase traffic on your website which will help in increase in overall sales, quite apart from the particular product which has been shared.
Keep in mind, however, that if the number of likes and share are too few in number, it may result in negative social proof.
12) Delivery & Returns Information
Nothing is more irritating to a customer than selecting a product, making a purchase and reaching the checkout section only to discover the addition of unexpectedly high delivery charges and hidden taxes.
Posting the total cost on the product page eliminates the surprise of a high shipping cost and also lets the customer factor in the total cost before adding the product to the cart.
13) Live Chat Widget
According to an Econsultancy report, Live chat has the highest satisfaction levels for any customer service channel, with 73%, compared with 61% for email and 44% for phone.
Often times people have questions that may not have been answered in the product description or in the customer reviews. If you force them to guess, or leave them wondering, they are going to leave and find the answer to their question elsewhere.
14) Product Videos
Visuals work much better than text when it comes to conveying a message. Many eCommerce sites are using videos in different ways to improve customer understanding
of their products as well as make their content more engaging and intimate. SixPackAbsExercises.com, a VWO customer, A/B tested videos on the sales page which resulted in increased conversions by 46%.
Videos also help the customers to gain a more intimate understanding of product look as well as functionality.
If you decide to have videos on your website, you may also look at having transcripts for the
videos. These not only ensure that viewers with hearing disability can access your video content, it also improves the SEO for that video.
Breadcrumb navigation helps the user to understand the product hierarchy as well as navigate to other areas of interest. They are also known to reduce bounce rates.
That’s it! One last piece of advice: keep testing. One thing that we at VWO have learned over the years is never to trust experts. The best way to improve product page performance is to keep testing out new ideas and concepts with A/B testing and keep optimizing your product pages.
Now it’s time for the surprise we told you about. We’ve designed a beautiful eCommerce product page template for you which employs many of the elements we just talked about. Scroll down to see it.
We have designed 4 more templates specially for you. Click on the button below to get a free eBook on ’5 eCommerce Product Page Templates to Boost your Sales and Conversions”.
Manna is a Hungary-based online store that sells chemical-free, organic and handmade personal-care products like soaps, body butters, essential oils, etc. Their products are available for purchase in Hungary, Germany, Austria and Serbia.
Being a Hungarian brand in Germany, they have to deal with a lot of trust issues. To overcome the same they decided to test various certifications and payment seals on the website. The hypothesis was that adding these certificates and payment icons near the “add to cart” button on the product page will increase clicks on the button as well as sales and revenue.
This is how the original product page looked like:
The team at Manna decided to a/b test 2 variations of the product page against the original. The first version had a big banner showing the payment options and 3 certificates below it. This is how it looked:
The second version had 2 rows of various certificates. This is how it looked:
The test was run on more than 3000 visitors split equally between the 3 variations.
Variation 1 won and 11.26% of the visitors, who viewed this version, clicked “add to cart” button, in comparison to 1.48% in original (and 8.73% in variation 2).
Here’s a screenshot of the bar graph from the VWO app showing conversion rate of each variation:
Why Variation 1 Outperformed the Other Two Versions of the Product Page?
As an online store selling natural and organic items, the pressure on Manna to prove their credibility is much more than any other web-shop selling cosmetics and skincare. This is because in addition to having great product pages and a seamless web experience they also have to constantly re-iterate the fact that they sell authentic natural products made from chemical-free substances. Having certifications, right on product pages, thus helped them boost the clicks on “add to cart” button. This is why even version 2 recorded a whopping increase of 490% in click rate.
Variation 1 emerged as a winner because of showing the certifications and ease of payment option together. This addressed the two most common concerns faced by customers — whether the product is authentic and if their preferred mode of payment is available as an option.
Have you done any tests on your product pages? I’d love to know what you learnt about your visitors from them in the comments section below.
One of the things I absolutely love about split testing is its inherent ability to humble opinions of even the most seasoned testers. Sometimes even the most well researched hypotheses fail.
This is the strongest reason why companies must test everything from offer copy to page design, instead of relying on gut instinct or personal preferences. In this post, I’ll share some case studies that either saw huge opinion disparity among the WhichTestWon community, or whose results absolutely shocked our editorial team and Testing Awards’ judges.
Social Proof Is Not the End-All-Be-All
Can you guess which of these two versions of an in-line form generated more opt-ins for a well-known Web design blog?
As of today, 71% of the WhichTestWon community picked the version on the left – the one with social proof copy – as the winner. When I present this test to live audiences at conferences, usually 90-95% of attendees pick the social proof version.
However, they are all wrong. The variation at the left — without the subscriber count — saw 122% more newsletter opt-ins than its social proof counterpart. In a world where we are jaded by the user counts of Facebook, Twitter, etc…it seems that 14k visitors wasn’t compelling enough to get prospects to act.
Normally we see companies just add social proof without testing it because virtually every blog post and ‘social media expert’ has told them it will help conversions. Thankfully the team at this site tested this before implementation, or they would have missed out on a lot of opt-ins – more than double the opt-ins in fact.
Don’t get me wrong, there is a ton of value in social proof. For one, it is great at reducing a visitor’s anxiety and can give your brand a sense of authority. The trick is finding out where it is best to publish your social stats and what exactly is worth sharing. Should you post newsletter subscribers, Facebook likes, awards, or all of the above? Simply put: test it out. Never add something blindly; you don’t know how it will impact the bottom line.
Icons May Be Trending, but they Might Hurt Conversions
Icons have been making a major comeback in web design. Overall, icons have been useful, especially when they are used as a replacement for bullets in stylized lists. The team at Build.com wanted to find out whether icons would be a useful navigational tool…the results surprised them.
Here are the two versions of the header they tested, one with icons and one without:
The challenger variation included icons that represented different category pages on the site. The team believed that an increased focus on navigation with their most visited categories would increase interactions and sales. However, the version without the icons saw 21% more product purchases.
Why? We suspect that although the icons provided a sleek navigation pane, overall they likely added more clutter that confused the visitor.
Security Seal on a Lead Gen Form Test
The highest point of friction on any lead generation page is the form itself. You need to identify the optimal number of form fields, choose an intuitive design, and add visible privacy policies and/or security icons to reduce anxiety.
These are well-known best practices that all lead gen marketers understand… and that’s probably why 74% of the WhichTestWon community guessed the wrong winner for this A/B test.
The variation without the TRUSTe logo next to the button got 12.6% more completed forms. Yes, the ‘submit’ button did shrink to accommodate the TRUSTe logo; but, we strongly suspect the primary cause for this lift has to do with the logo itself.
Trust seals can be essential to your conversion rate; the real trick is knowing where and when to place them.
In this particular circumstance, the TRUSTe logo was the wrong security seal at the wrong time. Visitors are used to seeing this seal, and others like it, directly in a shopping cart; not on a top funnel lead generation form. It’s quite likely that many of them suspected a payment transaction when they saw the trust seal here.
Remember, context is the key!
If Your Conversion Rates are Falling, Put on a Happy Face?
CRO specialists and designers love using faces! I get it; faces are the first thing the eye identifies when it looks at a web page. Numerous eye tracking studies support this claim.
However, sometimes a human face can be too much of a distraction. So, you need to test when you add faces to your page to make sure they aren’t competing with your important headlines and calls to actions (CTAs).
Here’s an example:
The version without the image won 24% more form completions. This wasn’t a perfectly clean test. There were some slight alterations in copy but nothing too dramatic. To tell you the truth, I’m more of a fan of the story behind this test than the actual unexpected results. However, it’s not the first or the last test we’ve seen where removing a face increased conversions.
What’s perhaps most amazing is that at the time of this split test HubSpot was about to make it a mandatory practice to include a person’s image on each of their landing pages. On some level this makes sense, they had found that some of their pages performed better with people’s pictures. However, what’s true for some pages may not always be true for all pages. Luckily, this test casted a seed of doubt and the company changed their design mandate.
Before you create any new landing page or design policies, please test beforehand…you have no idea just how many conversions you could leave on the table.
Video Icons – Product-Centric or Person-Centric?
Here is another case that tests whether using a face is appropriate.
The version of this Autodesk product page that used faces got 50% fewer video clicks. Nothing else on this page changed except for the video preview image. I am not anti-faces on websites; I simply want you to test before you implement!
Needless to say, the testing team was surprised by the results. So they ran a user survey to try to figure it out. The responses showed that Autodesk’s prospective buyers were more interested in seeing how the product worked over individuals talking about the product.
This just comes down to a case of knowing your audience and that best practices are not one-size-fits-all!
Leaders in the testing field have all been stumped by the unexpected results before, and will be stumped again. The trick is to understand what to do after your test goes counter to your hypothesis or flat-lines.
Your next steps may include evaluating a litany of things such as your hypothesis, technology, source traffic, device…the list goes on. You need to learn if the test itself was flawed – or if your understanding of what your visitors really want from the page was flawed. Either way, you’ve learned something valuable.
Remember testing is an evolving process, future iterations are borne from our successes and our failures.
Keep testing my friends! There are so many variables to consider while running a test, it is no wonder that we often see lifts or losses where we least expect it.
Imagine there’s a cupcake fair in your community. You have your heart set on Red Velvet and there are two counters selling the same. Counter A has a animated bunch of customers digging into the cake and has a banner saying “last 20 pieces left” while Counter B adorns a deathly, almost funereal look.
Which one would you go for? If you are thinking Counter A, you are not alone.
A study was conducted in 1975 where researchers wanted to know how people would value cookies in two identical glass jars. One jar had 10 cookies while the other contained just two. Though the cookies and jars were identical, participants valued the ones in the near-empty jar more highly.
And that’s the scarcity principle at play. It essentially means that people tend to place higher value on an object that is scarce and a lower value on one that is available in abundance. No wonder, marketing guru Robert Cialdini cites ‘Scarcity’ as one of the six golden persuasion principles in his book “Influence”. When combined with Urgency, which is essentially the other side of the same coin, the two make for a potent weapon for increasing eCommerce sales.
Here are some examples of how different eCommerce websites creatively use this persuasion principle to increase conversions.
1) Stock scarcity
Displaying your stock meter on the eCommerce product page is always a good conversion practice. Not only does it ensure there are no last-minute heartbreaks for the customer, it also speeds up the buying process. A user might be convinced to make a purchase, but he/she might not be always willing to buy right away. They might want to compare the prices on other websites, look for discount coupons or simply forget about the product — thanks to the myriad distractions of the web.
Look how ModMomFurniture flashes a message that ‘only 3 items left’ of a particular product.
Here, Boticca almost urges visitors to complete the purchase right away. The use of an active verb like ‘Act’ is used to drive immediate action. Here is a list of more words that drive urgency and sales.
2) Size scarcity
You head to a shop to buy a denim and figure out that the last piece in your size has just gone out of stock. Old story? Well, if it could happen in the real stores all the time, why couldn’t it happen online? Intimating the buyers when a particular size goes out of stock is killing two birds with one arrow. Not only is it a huge favor for the prospects, the information also works as a positive reinforcement of the product.
See how Jabong represents the unavailable size in grey.
Zappos goes a step further and even shows how many items are left for a particular size and color combination.
3) Time-bound purchase for next day shipping
If you are already offering next-day delivery, then ‘time-bound purchase’ won’t cost you any additional resource. You just need to inform the users how many hours/minutes do they have to complete the purchase so that their order qualifies for next day delivery.
When you ask the visitors to make the purchase in a specific amount of time, you not only make them more proactive towards the purchase, it also eliminates any kind of confusion at their end as to when they will receive the order.
Zappos has a permanent banner on its homepage saying that you need to order before 1pm to qualify for next-day shipping.
Amazon shows the exact number of hours/minutes within which the purchase needs to be completed to qualify for next-day shipping.
4) Make them see other buyers
Two women fighting over the same item of clothing in a fashion store is not just a devilish mind’s fictitious construct. While popular Hollywood movies might have taken these fights too far, the fact remains that people are much more inclined to buy something when other people desire them too.
So while they are considering whether or not to buy those shiny leather shoes and they suddenly realize that another person is mulling the same purchase, it replicates the physical store scenario where two people grab the same item at the same time.
When you look at a property in Booking.com, it shows you exactly how many people are checking out the same property at that exact moment.
Hotels.com even gives you information about how many people are viewing properties in a particular city. A modal box pops opens and shows you your virtual competition/companions.
5) Limited-period discounts
The fear of missing out (FOMO) is a legitimate one. It is the anticipated regret of not being able to seize an opportunity. A limited-time discount works exactly at that level. It makes the offer look so tempting and fleeting that one is compelled to seize the opportunity.
OverStock dramatically uses a running countdown to accurately show how long the sale will last.
MakeMyTrip.com shows an alert when the last few discounted airline tickets are left in stock. See how they use color psychology here to instill urgency. The use of the color ‘Red’ is not a mere coincidence. The color is associated with energy, increased heart rate and is often used in clearance sales.
5) Limited-time discount on abandoned cart items
The average online cart abandonment rate is 67%. While a majority of these carts are not redeemable, re-marketing efforts can salvage some of these lost sales. Offering limited-time discounts on abandoned cart items is a great way to use the urgency principle to re-market products. However, the risk here is of overdoing it. You do this too often and you will see your ‘e-mail open rate’ nosedive.
See how Miracas offers a 5% discount for just the next two days.
6) Shopping cart item sold out
Seeing some items disappearing from the cart might just be the reality check users need to wake up and salvage the rest of the cart items. See how Snapdeal shows a ‘Sold Out’ message next to one of the books in the following example.
7) Limited-time free delivery
Well, this works the same way as limited-time discounts. The desire to avail free delivery could possibly offset the visitors’ tendency to procrastinate the purchase.
8) Special discount hours
A two-day or a weekend sale has its own charm but a special discount hour can be used to galvanize excitement around that specific hour. Zivame sent out this mailer to subscribers to build up craze for its hour-long sale.
9) Last chance emails
E-mails letting subscribers know about last day of sale is another great way to get their attention. See how Myntra makes use of Orange color to drive action here. According to color psychology, just like Red, Orange also has an aggressive feel and creates a sense of urgency to do something.
Before you set out to use any of these tactic mentioned above, there are three golden rules you should keep in mind.
1) Don’t expect ‘scarcity’ to create demand
‘Scarcity’ or ‘Urgency’ work best as motivators to quicken the buying process but the customer will have to be already convinced to make the purchase decision. While they are great procrastination killers, don’t expect them to generate demand. If you go back to the initial example, it’s only once you have decided to buy a cupcake that you will value the cupcakes in Counter A more. You will have to truly convince the user before you flash the scarcity card.
2) False urgency can backfire
Urgency is a subtle art. Yes, I know that sounded slightly oxymoronic but don’t go about faking urgency or your customers will get a whiff of it. Be honest. Don’t try to get rid of the stock that won’t move by flashing messages like ‘hurry, only last two pieces left’. First they will catch your bluff, then you will become a laughing stock and then they will leave you. Here’s a real life example of how that happened.
3) Don’t overdo it
Even if you are using the scarcity principle in all earnestness, don’t go about overdoing it. You don’t need to employ all the practices with multiple countdowns breathing down customer’s neck. You will come across as pushy and shady and untrustworthy and manipulative. Moderation is the key, though testing will give you better insights as to what will work for you.