Tag Archives: product


Promo Code Box on your Shopping Cart Page could be Bleeding Dollars. A/B Test it.

The Company

Bionic Gloves is an online store that designs and sells a range of gloves, such as golf gloves, fitness gloves, and more. Their focus is to provide customers with gloves that have fine grip, comfort, and durability.

To increase sales from their eCommerce shop, they decided to optimize their website. The task was given to Portland-based marketing & conversion optimization agency, Sq1.

The Test

Sq1 performed many tests on the Bionic Gloves website. In this case study I’ll be taking you through an interesting test that was performed on one of the most important pages of any eCommerce website, the shopping cart page. In fact, one study by Surepayroll estimated that each year eCommerce websites lose a whopping $18 billion because of shopping cart abandonment.

To test their hypothesis that removing the ‘special offer’ and ‘gift card’ code boxes from the shopping cart page would result in more sales and less cart abandonment, they set up an A/B test in VWO.

This is how the original shopping cart page looked like:

Bionic AB - Control

The Result

The test was run on close to 1400 visitors for a duration of 48 days. This is how the variation page (without the code fields) looked like:

Bionic AB - Variation

The primary goal that they were tracking was the revenue made. The variation won and increased the total revenue by 24.7%, and revenue per visitor by 17.1%.

Why the Variation Won?

In the words of David from Sq1, “Anytime you leave the door open for a user to leave the conversion funnel – even if it seems like they’d come right back – you risk losing sales. By showing the Promo Code field on the cart, users were enticed to leave the site in search of a promo code. At that point, the conversion process is interrupted and you are more likely to lose potential customers. As such, hiding it was a very logical test.

A shopping freak myself, I wouldn’t lie that I, too, have gone looking for coupon codes a number of times in the middle of my purchasing process. This, as David pointed out, has a number of risks:

  • The sight of the coupon box triggers visitors to look for one on Google and other places. I did a quick Google search of “Bionic Glove”, and look what I found in the auto-complete searches:
  • eCommerce websites also risk losing money to affiliates and websites offering deals, coupons, etc.
  • Many a times, visitors end up finding a better deal on another web store.

To avert this, I have seen many websites now show all available coupon codes right on the product page and also on the cart page. Not only does this help them reduce cart abandonment, but also helps them increase their average order value as many shoppers go ahead and buy more stuff to cross the threshold at which coupons can be applied.

See how Myntra, a fashion ecommerce website based out of India, does this beautifully:


Let’s Talk

Tell me what you think about this case study in the comments section below. I am also available for intellectual discussions on CRO and A/B Testing which can fit in less than 140 characters on Twitter @taruna2309. See ya!

8 Checkout Optimization Lessons Based on 5+ years of Testing

The post Promo Code Box on your Shopping Cart Page could be Bleeding Dollars. A/B Test it. appeared first on VWO Blog.

Read this article: 

Promo Code Box on your Shopping Cart Page could be Bleeding Dollars. A/B Test it.


Build Your Own Product Hunt With Telescope And Meteor

Over in startup land, one of the big stories of 2014 was, without a doubt, the success of Product Hunt1. It’s is a community where people post, vote on and comment on new products they’ve discovered or launched. Whether you’re looking for the next big thing to invest in or just want to find a better weather app, Product Hunt has got you covered.

Coincidentally, in addition to being a fan of the website, I also have a pretty personal connection to the company. I’ve been online friends with Product Hunt’s designer Jonno Riekwel2 for years, and I was part of founder Ryan Hoover3’s previous project, Startup Edition4.

Product Hunt (View large version6)

All this to explain that Product Hunt has been on my radar for a while now. In fact, what many people don’t know is that Ryan initially considered7 basing Product Hunt on Telescope8, an open-source app that I had just started working on! Sadly, Telescope was still too immature at the time, and Ryan wisely decided to build his own solution.

But I’m here to tell you that things have changed. As we know, Product Hunt has gone on to become Silicon Valley’s latest darling. But Telescope has also kept quietly evolving, and it’s now a full-featured solution for people to build their own communities. So, today, let me show you how you too can set up your own personal Product Hunt-style website.

Introducing Meteor

Product Hunt wasn’t the only success story of 2014. Another one was certainly Meteor9, the company behind the open-source JavaScript framework of the same name. Meteor completely changes the way we write apps by getting rid of the age-old client-server divide. Unlike other JavaScript frameworks such as AngularJS and Ember.js, Meteor runs not only in the browser, but also on the server. So, with Meteor, not only is your whole app written in JavaScript, but you can even share the same code and call the same functions in both environments!

The Meteor JavaScript framework (View large version11)

I decided to pick Meteor to build Telescope, and in fact I liked the framework so much that I ended up writing a book about it12. But that’s a story for another time. Don’t worry: Meteor may be cutting-edge, but it’s also very approachable. Even if you’re not a JavaScript expert, we’ll have you up and running in no time (about 45 minutes, to be exact)!

Installing Meteor

If you’re on Mac OS or Linux, installing Meteor is as simple as opening a Terminal window and typing:

curl https://install.meteor.com/ | sh

Though Meteor doesn’t quite support Windows yet, a preview release13 is available, and an official version should be coming very soon.

Installing Telescope

Next, you’ll need to clone Telescope’s code base from GitHub14. Just type:

git clone https://github.com/TelescopeJS/Telescope.git

If you don’t have Git, you can download the command-line utility15 or use an app, such as the free GitHub for Mac16.

Running Telescope

Almost there! First, drill into the newly created telescope directory with this:

cd Telescope

Then, run your app with this:


Your app might take a minute or two to initialize the first time. But once it’s done, your brand new Telescope app should be up and running! Head to http://localhost:3000 to check it out.

Your new Telescope app17
Your new Telescope app (View large version18)

If at any point you need to stop your Meteor app (maybe it’s crashed or you want to get the command prompt back), you can do so by opening the Terminal tab where your app is running and hitting Control + C.

Configuring Telescope

Let’s personalize the app a bit. The first thing to do is create a user account by clicking the “Sign Up” link in the top navigation menu. Because this will be the first user account you’ve ever created, it will automatically be assigned administrative rights, which will let you access Telescope’s admin area.

Telescope's settings panel19
Telescope’s settings panel (View large version20)

Once your account is created, head to the “Settings” page via the “Admin” menu. There, you can set various options, such as your website’s title, description and color scheme.

For example, try setting the header’s color to #1282A2 (or any other color!), and then click the “Submit” button at the bottom of the form.

Customizing colors21
Customizing colors (View large version22)

Activating “Product Hunt Mode”

Telescope was originally based on the popular hacker hangout Hacker News, which is why the default view is just a uninterrupted list of posts. But with the success of Product Hunt, more and more people started asking for an option to use a day-by-day view, which I like to call “Product Hunt mode.”

To enable it, just go to the “Settings” page, scroll down to the “Default View” option, and set it to “Daily.” Save the settings and refresh the home page. You should now see a much nicer layout!

Product Hunt mode23
Product Hunt mode (View large version24)


Let’s show the world the result of our hard work. Deploying an app is usually a painful process, but not today. It turns out that Meteor provides its own free hosting service for quickly deploying small apps and prototypes.

Open a new Terminal window, browse to your app’s directory, and then type the following:

meteor deploy my-app

Replace my-app with the name of your app.

If this is your first time deploying on Meteor’s service, you’ll be prompted to create a developer account25.

Once the deployment process is finished, your app will be live at http://my-app.meteor.com!

Note that the settings of a Telescope app are stored in its database, which is not transferred as part of the deployment process. So, you’ll need to re-enter any settings that you’ve previously set.

Customizing Your App With Packages

So far, we’ve changed the default view and tweaked a few colors. We can take things much further than that. Let’s start by adding a few extra CSS customizations.

To do so, we’ll create our own package. At this point, we could just go ahead and modify Telescope’s code, but that could lead to Git conflicts when we try to update the app down the road.

Instead, we’ll use Meteor’s package system to make sure that any customizations we implement stay independent of the main code base and will not be overwritten even when we pull in future updates.

By the way, the code for this tutorial is also available on GitHub26.

Inside your app, locate the /packages directory, and create a new telescope-custom directory inside it (either with the mkdir command or by going to File → New Folder). From now on, we’ll put the files we create inside this directory.

Each Meteor package needs a file named package.js, which contains the package’s manifest, a simple set of instructions that tell Meteor what the package actually contains.

So, create the package.js file inside telescope-custom, and type the following:

  summary: 'Telescope custom package',
  version: '1.0.0',
  name: 'telescope-custom'

Package.onUse(function (api)   
  api.addFiles('custom.css', 'client');

The Package.describe block provides some basic information about the package. But the interesting part is the Package.onUse block, which tells Meteor which files to load in our package. In this case, we’ll say we want to add a custom.css file.

Remember I said Meteor could handle code on both the client and server? This is why we have to specify where we want to add our files — in this case, on the client.

All that’s left is to actually create this custom.css file!

Customizing The CSS

Let’s open the custom.css file we just created and add Product Hunt-style yellow hover states to each of our posts:

  background: #fcf5e2;

We’re almost done. We still need to tell Meteor that we want to use this new package. We can do so with the following command:

meteor add telescope-custom

Once you’ve added the package, your browser tab should automatically refresh, and you’ll see that our new yellow hover state is working!

Hover effect27
Hover effect (View large version28)

But wait, we didn’t even tell Meteor where to import that new style sheet from. How can that even work?!

This is one of Meteor’s cool time-saving features. It’s smart enough to notice when you add a new CSS file to your code base. It will automatically concatenate it, minify it and load it in your app.

Customizing Templates

A nice touch in Product Hunt’s UI is how it includes banners between days. It’s a great way to link out to other content and break up the monotony of listed links, while maintaining a clean layout and not disrupting the experience. Let’s see how to do the same thing with Telescope.

Telescope already has templates before and after each day, but the default templates are empty. So, our mission will be to override the afterDay template with one that contains actual content.

Add a banner.html file to your package, and add a simple banner:

<template name="banner">
  <div class="custom-banner"><a href="http://twitter.com/telescpe">Follow us on Twitter!</a></div>

Next, we’ll tell Telescope to override the default afterDay template with this one. To do so, we create a new banner.js JavaScript file at the root of our package:

templates["afterDay"] = "banner";

Before calling the afterDay template, Telescope will check in the templates array to see whether we have provided a replacement. And since we have, it will use the template code from banner.html instead!

We’re not quite done yet, though! Just as before, we need to add our new files to the package’s manifest. And this time, we’ll also need to add package dependencies to it.

We’ll add the templating package, which lets us use templates, as well as the telescope-base package, which exposes the global templates object that we’re calling in custom.js. So, let’s go back to package.js and modify its contents:

  summary: 'Telescope custom package',
  version: '1.0.0',
  name: 'telescope-custom'

Package.onUse(function (api) 


  api.addFiles('custom.css', 'client');
  api.addFiles('banner.html', 'client');
  api.addFiles('banner.js', 'client');

Finally, let’s add a little CSS for good measure. Go back to custom.css and add this:

  background: #c4e6ef;
  padding: 20px;
  text-align: center;

Twitter banner29
Twitter banner (View large version30)

Adding Some Logic

Things are looking good, but you’ll notice that our banner is repeated after every single day.

This might get a little repetitive. Let’s see how to show our banner after only the third day.

To do this, we’ll need a bit of JavaScript magic. First, we’ll use Meteor’s templating logic (known as Spacebars) to add an if statement to the template:

<template name="banner">
#if isThirdDay}
  <div class="custom-banner"><a href="http://twitter.com/telescpe">Follow us on Twitter!</a></div>

We haven’t yet defined what isThirdDay is supposed to be. We’ll do that through a template helper.

Template helpers are simply bits of JavaScript code you can assign to variables and use in your templates. Let’s open banner.js and add the following code to the file:

  isThirdDay: function () 
    return this.index == 2;

Because our afterDay template is called from a loop, we can access the current day’s index using this.index. And because that index is zero-based, the third day will be the one whose index is 2, which in turn gives us the contents of our helper.

Here you go! Now our banner is appearing after only the third day and no other.

Twitter banner, not repeated31
Twitter banner, not repeated (View large version32)

Other Features

We’re running out of time, but this is by no means the extent of Telescope’s features. Out of the box, Telescope also makes the following possible:

  • Add a banner to sign people up to your email list.
  • Generate and send a newsletter of the best posts on your website.
  • Add thumbnail previews to each new posted link.
  • Automatically import posts from an RSS feed.
  • Require new posts to be approved by an admin.
  • And much more!

If you want to go further, a good place to start would be Telescope’s documentation33. You can also find a nice introduction to Meteor itself on the official website34.


As Product Hunt’s success has shown, there’s a big demand for websites that help us deal with information overload by streamlining and centralizing content. Telescope is a fast, modern platform on which to build your own community, social news app or link-sharing website. And, as you’ve just seen, extending it is very easy. I encourage you to give it a try. And if you end up building the next Product Hunt with Telescope, I hope you’ll remember who first told you about it!

Thanks to Loren Sands-Ramshaw, Phil Pickering, Nigel Anderson, Austin Stoltzfus and others for reviewing a draft of this article.

(vf, il, al)


  1. 1 http://www.producthunt.com/
  2. 2 http://www.producthunt.com/Jonnotie
  3. 3 http://www.producthunt.com/rrhoover
  4. 4 https://twitter.com/startupedition
  5. 5 http://www.smashingmagazine.com/wp-content/uploads/2015/02/01-product-hunt-opt.jpg
  6. 6 http://www.smashingmagazine.com/wp-content/uploads/2015/02/01-product-hunt-opt.jpg
  7. 7 http://ryanhoover.me/post/69599262875/the-wisdom-of-the-20-minute-startup
  8. 8 http://telesc.pe
  9. 9 http://meteor.com
  10. 10 http://www.smashingmagazine.com/wp-content/uploads/2015/02/02-meteor-homepage-opt.jpg
  11. 11 http://www.smashingmagazine.com/wp-content/uploads/2015/02/02-meteor-homepage-opt.jpg
  12. 12 http://discovermeteor.com
  13. 13 https://github.com/meteor/meteor/wiki/Preview-of-Meteor-on-Windows
  14. 14 https://github.com/TelescopeJS/Telescope
  15. 15 http://git-scm.com/downloads
  16. 16 https://mac.github.com/
  17. 17 http://www.smashingmagazine.com/wp-content/uploads/2015/02/03-telescope-fresh-install-opt.jpg
  18. 18 http://www.smashingmagazine.com/wp-content/uploads/2015/02/03-telescope-fresh-install-opt.jpg
  19. 19 http://www.smashingmagazine.com/wp-content/uploads/2015/02/04-telescope-settings-opt.jpg
  20. 20 http://www.smashingmagazine.com/wp-content/uploads/2015/02/04-telescope-settings-opt.jpg
  21. 21 http://www.smashingmagazine.com/wp-content/uploads/2015/02/05-telescope-color-opt.jpg
  22. 22 http://www.smashingmagazine.com/wp-content/uploads/2015/02/05-telescope-color-opt.jpg
  23. 23 http://www.smashingmagazine.com/wp-content/uploads/2015/02/06-product-hunt-mode-opt.jpg
  24. 24 http://www.smashingmagazine.com/wp-content/uploads/2015/02/06-product-hunt-mode-opt.jpg
  25. 25 https://www.meteor.com/services/developer-accounts
  26. 26 https://github.com/TelescopeJS/Custom-Package
  27. 27 http://www.smashingmagazine.com/wp-content/uploads/2015/02/07-telescope-hover-opt.jpg
  28. 28 http://www.smashingmagazine.com/wp-content/uploads/2015/02/07-telescope-hover-opt.jpg
  29. 29 http://www.smashingmagazine.com/wp-content/uploads/2015/02/08-banner-repeated-opt.jpg
  30. 30 http://www.smashingmagazine.com/wp-content/uploads/2015/02/08-banner-repeated-opt.jpg
  31. 31 http://www.smashingmagazine.com/wp-content/uploads/2015/02/09-banner-not-repeated-opt.jpg
  32. 32 http://www.smashingmagazine.com/wp-content/uploads/2015/02/09-banner-not-repeated-opt.jpg
  33. 33 http://telesc.pe/docs
  34. 34 https://www.meteor.com/install

The post Build Your Own Product Hunt With Telescope And Meteor appeared first on Smashing Magazine.

View original article – 

Build Your Own Product Hunt With Telescope And Meteor


The Basics Of Test Automation For Apps, Games And The Mobile Web

Mobile application ecosystems — let’s count Android and iOS here — are unbelievably dynamic, but they also suffer from both software and hardware fragmentation. This is especially true for Android, but fragmentation also exists in the iOS ecosystem1, as experienced with the rollout of iOS 8. As the latest version of iOS was released, many existing apps were made clumsy on updated devices.

Even the new iPhone 6 and iPhone 6 Plus have had not-so-typical issues2 for Apple devices. In addition, a significant proportion of users with older devices have very few options: essentially, buy new hardware (i.e. a new device) to get everything working well.

In the Android world, things are different. As OEMs launch new devices, software updates and customizations for their devices, application and game developers get serious headaches3 trying to keep their latest products up to snuff and fully compatible with all possible device variants. Making a certain app or game work only on high-end devices is out of the question, though. Why would a developer want to miss out on a significant chunk of potential users?

Automation enables simultaneous testing on hundreds of real devices4
Automation enables simultaneous testing on hundreds of real devices. (Image credit: Testdroid6854403733282517135) (View large version6)

Professional automated testing software is a solution to a common problem: how to produce high-quality, robust and reliable software with the ever-growing complexity of technology and under massive competitive pressure. Automated software testing is a cost-effective solution to this problem. Not to mention, it provides three business benefits:

  • increased testing efficiency,
  • increased testing effectiveness,
  • faster time to market.

This article walks through a sample use case for test automation and provides a downloadable example to get you started. Also, we’ll focus on different aspects of mobile test automation and explain how this relatively new yet popular topic can help mobile app and game developers to build better, more robust products for consumers. With the advanced example later in the article, we’ll show how image recognition can be used to test mobile games; specifically, we’ll run Appium’s test automation7 framework against Supercell’s Clash of Clan8 game to illustrate how image recognition can be built into the test-automation process.

Test Automation Is Perfect For Mobile App Development

Developing mobile applications is very different from developing PC software or even embedded software. Mobile development is meant to be agile9, and a lot of great tools and practices have been developed for that agility. However, doing something manually — such as testing an app — is never agile, which is why test automation has shown tremendous growth among app and game developers, speeding up their doings and yielding robust and better results.

To achieve compatibility between users, devices and the market, including test automation as a part of the agile development process10 is typical. Fortunately, a lot of tools are available, and test automation is a perfect fit for this process. For example, let’s say your typical development sprint is two weeks. You have daily standups and a lot of scrum activities, and you own internal policies that gear development to the final product. Test automation offers a significant value-add by enabling testing to be done in parallel — for example, as nightly test sessions. By the next morning, the tests will have been finalized and the results of the latest regression will be ready for review. Fixing an issue earlier will save a lot of time and get developers to finalize the product sooner; most importantly, it cumulates to better quality11, with fewer bugs.

Value that test automation brings to agile process.12
The value that test automation brings to the agile process. (Image credit: Testdroid6854403733282517135) (View large version14)

Test automation offers the possibility to test mobile apps instantly and effectively. Once tests have been automated, they can be executed quickly and repeatedly, again and again. In almost all cases, this is the most cost-effective method for regression testing15 of software products that have a long maintenance life. In fact, test automation of any mobile app is the best way to increase the effectiveness, efficiency and coverage of the testing process. The true benefit of automation comes not only from the repeatability of tests, but also from the ability to execute tests that probably could not even be performed manually.

Things To Consider When Automating Mobile App Testing

Let’s look at how test automation (as opposed to manual testing) can improve the development process, add value and speed up development.

Costs and Assets

Regardless of whether you go for manual or automated testing, you’ll need the following assets and resources (all of which cost money): time, people, infrastructure, tools and training. Depending on the size of the project and the application, test automation will quite obviously provide a good return on investment. For example, once test cases have been created, automated tests can be run over and over again at no additional cost, and these can be more rapidly completed than any manual tests. Automated software testing can reduce the time required to run repetitive tests from weeks to hours. This is a significant time-saving that translates directly into cost-saving.

Continous integration together with mobile app testing16
Continous integration together with mobile app testing. (Image credit: Testdroid6854403733282517135) (View large version18)

Integrated App Development And Testing Cycle

Increasing efficiency and productivity with automation actually starts with adopting a new mindset. Software tests have to be repeated often during all development cycles to ensure the best possible quality of an application. Every time source code is modified19, software tests should be repeated. For each release, your software should be tested on all supported variants of operating systems and all configurations of hardware. Manually repeating these tests would be costly and time-consuming. For example, running comprehensive tests manually on all variants of Android and on actual devices would take a lot of time.

Tools and Technology: Test Automation Frameworks

To get the most out of your efforts and maximize testing coverage, select the most robust and most cross-platform20 method. That automated methods can be used both to validate requirements and to reduce costs through automated test-case generation is well known. However, the full automatization of large software entities also comes with a cost that many companies haven’t been ready to pay for. Historically, one of the reasons has been a concern over the lack of adequate integration with well-established development life cycles.

Test Coverage and Reusability: Open Standards Mean No Vendor Lock-In

Automated testing can increase the depth and scope of tests and significantly improve software quality. Lengthy and thorough tests — often not doable with manual testing — can be run automatically. Ideally, test cases should have full access to an application and test all aspects of it21 — memory contents, data tables, file contents and internal program states — to determine whether the product behaves as expected. Automated tests can easily execute thousands of complex test cases during every test run, providing coverage that is simply not possible with manual testing. Developers, freed from repetitive manual tests, will have more time to create new automated test cases and build more compelling features (and then more test cases).

Time to market for mobile web22
(View large version23)

Improve Effectiveness and Finalize Sooner

In a nutshell, professional automated testing software is a solution to a common problem: how to produce high-quality, robust and reliable software with the ever-growing complexity of technology and under massive competitive pressure. Automated testing improves business results in three ways: greater testing efficiency, greater testing effectiveness and a shorter time to market.

Different Ways to Automate Mobile Testing

In general, there are three ways to automate the testing of mobile apps.

Handwritten Test Scripts

Typically, this is the best choice when you know what you’re doing and when you have programming-capable people doing the test scripts. Plenty of options are available for test automation frameworks, tools and integration — both commercial and open-source offerings. Having your engineers write all of the test scripts will take up some time and tie up resources, but it will get you exactly what you want: well-structured, thorough scripts that test precisely the aspects of your app that you want to test.

Manual versus automated testing for mobile apps24
Manual versus automated testing for mobile apps. (Image credit: Testdroid6854403733282517135) (View large version26)

Record-Playback Approach

This approach is less error-prone because nothing needs to be written in code, but it is typically more limited in functionality. Tests can be quickly recorded and then played back again and again against different OS versions and device configurations. These tests focus on user interactions and user-driven activities. Some things might fall beyond a test’s scope (such as integration with other technologies and compatibility with other software).

Automatic Test Exercisers

Automatic test exercisers provide a great way to smoke-test applications. No specific tests are needed; rather, the focus is on testing user-interface logic (such as opening menus, clicking buttons, swiping and multi-gesture actions). Automatic test exercisers yield the least exact results but provide quick feedback on each iteration of an app.

Focus Areas In Testing For Mobile Apps And Games

Different ways to automate mobile app testing27
Different ways to automate mobile app testing. (Image credit: Testdroid6854403733282517135) (View large version29)

User Interface and Functionality

A user interface and its overall functionality will directly affect how successful your app or game30 will be. These two things, which encompass visual appeal and gameplay, are the most important things to get right — and you must ensure that device fragmentation doesn’t break any of these. Various things in the UI need to be tested:

  • UI layouts and elements
    Games especially are typically targeted at a high number of different screen resolutions and screen types. Regression testing should be done each and every time the UI’s layout changes to ensure that the game works.
  • Menu structure and functions
    Testing menu structures, functionality and behavior can be automated with instrumentation and the help of different test-automation frameworks.
  • Screen orientation
    Surprisingly, so many apps and games out there get this wrong. If a screen’s orientation changes during an interaction, for example, what happens? What is supposed to happen? Does the app or game work well in both landscape and portrait modes?
  • Screen resolution
    A lot of screen resolutions exist, especially on Android, and auto-scaling will usually help developers. However, test your game across these resolutions to ensure that the graphics do not stretch.

Graphics Performance

Performance needs to be consistent across all device variants31 among your users. Because of this, test on as many real devices as possible. To determine how well your app or game responds to various levels of usage, including performance and battery usage, consider creating tests that last for hours. To determine whether your game runs effectively under a heavy load for a long time, run load (or stress) tests. These performance tests will measure, for example, how responsive your game is on real devices.

Performance testing of CPU load and memory consumption32
Performance testing of CPU load and memory consumption. (Image credit: Testdroid6854403733282517135) (View large version34)

Usability and User Experience (i.e. Good Entertainment)

Testing usability, navigation flow and user experience simply cannot be done on a desktop with a mouse and keyboard. So, forget emulators and use only real devices35. And to test how usable and entertaining your app is, consider these two important things:

  • User interaction and responsiveness
    Testing performance is critical because this will make make or break the user experience. Performance lag, for example, is easy to expose with real devices.
  • Background events
    Interruptions, battery consumption and the effect of battery chargers on overall performance and usage all have a significant impact on the user experience — and entertainment value.

Multi-User Features

Nowadays, multi-user support is common in both apps and games. Testing multi-player capabilities is important and is naturally more challenging, requiring real users to measure performance. A typical case is a game communicating with the back-end server. In this case, connectivity is essential, to synchronize the back end with devices that need to get information about the gameplay. You should test a ton of different scenarios, many of which could severely affect the game’s experience, resulting in negative feedback and the game being uninstalled by users.

Social Integration

Testing social integration in mobile apps36
Testing social integration in mobile apps. (Image credit: Testdroid6854403733282517135) (View large version38)

Integration with social networks is another important factor. Being able to share something across an ecosystem, with friends or just with oneself, is essential in many apps. Test this thoroughly with real Android and iOS devices, with different OS versions and different device configurations, to assess functionality and ease of use.

Security and Liabilities

Nearly all developers use some open-source components in their apps. This practice is widely accepted and highly recommended because it offloads the development of code for non-core functionality. However, identifying vulnerabilities and licensing restrictions with third-party code is often neglected by developers.

Breakdown: Android Test Automation Frameworks

Comparison of test automation frameworks39
Comparison of test automation frameworks. (Image credit: Testdroid6854403733282517135) (View large version41)

Robotium42 is an Android test automation framework that fully supports native and hybrid applications. Robotium makes it easy to write powerful and robust automatic black-box UI tests for Android applications. With the support of Robotium, test case developers can write function, system and user acceptance test scenarios, spanning multiple Android activities.

UIautomator5143, by Google, provides an efficient way to test UIs. It creates automated functional test cases that can be executed against apps on real Android devices and emulators. It includes a viewer, which is a GUI tool to scan and analyze the UI components of an Android app.

Espresso44, by Google, is a pretty new test automation framework that got open-sourced just last year, making it available for developers and testers to hammer out their UIs. Espresso has an API that is small, predictable, easy to learn and built on top of the Android instrumentation framework45. You can quickly write concise and reliable Android UI tests with it.

Calabash46 is a cross-platform test automation framework for Android and iOS native and hybrid applications. Calabash’s easy-to-understand syntax enables even non-technical people to create and execute automated acceptance tests for apps on both of these mobile platforms.

And then there’s Appium. OK, let’s get into this one.

Appium: Executing Tests On Real Devices On A Cloud Service

In a nutshell, Appium47 is a mobile test automation framework (and tool) for native, hybrid and mobile-web apps for iOS and Android. It uses JSONWireProtocol48 internally to interact with iOS and Android apps using Selenium’s WebDriver49.

In its architecture, Appium is an HTTP server written in Node.js that creates and handles multiple WebDriver sessions. Appium starts tests on the device and listens for commands from the main Appium server. It is almost the same as the Selenium server that gets HTTP requests from Selenium client libraries.

In fact, Appium is a pretty good choice for both apps and games because, in many cases, apps and games tend to be identical (or at least very similar) on both platforms, Android and iOS — and so the same test script can be applied to both. Another significant benefit of Appium is that users can write tests using their favorite development tools, environment and programming language, such as Java, Objective-C, JavaScript, PHP, Ruby, Python or C#, among many others.

Appium enables users to execute tests on mobile devices regardless of OS. This is possible because the Appium framework is basically a wrapper that translates Selenium’s WebDriver commands to UIAutomation50 (iOS), UIautomator5143 (Android, API level 17 or higher) or Selendroid52 (Android, API level 16 or lower) commands, depending on the device’s type.

For Android, this is how Appium compares to other test automation frameworks:

Family tree of Android Test automation frameworks53
Family tree of Android test automation frameworks. (Image credit: Testdroid6854403733282517135) (View large version55)

Android test suites are based on JUnit56. In addition, Android provides an architecture and fully integrated testing capabilities with its standard tools57, which help developers to test at every level, from unit to framework. Android instrumentation58 is a set of control methods in the Android system. These methods control an Android component independently of its normal life cycle.

One of the best things about Appium is that, despite sounding architecturally complex, it actually isn’t — at all. For developers, it provides support for various programming languages, freedom from having to select tools, compatibility across the most important platforms (Android and iOS), freedom from having to install and configure devices to test and more.

If you are familiar with Selenium, then you’ve got Appium covered. An Appium test is pretty much the same as a Selenium test: They use the same WebDriver, and DesiredCapabilities59 is used the same way. Configuring an application to run on Appium has a lot of similarities to Selenium — for example, those DesiredCapabilities. We’ll configure a sample test later in this article.

Also, Appium includes a component called the inspector. This inspector enables a host of functionality — for example, showing all of those UI elements in the application and enabling basic recording and playback.

However, you may not need the inspector because everything can be done in the code. The example later provides all of the scripts and instructions step by step, without using it.

More information about Appium can be found in “Getting Started With Appium60.”

Setting Up The Environment

To get started, first download our Appium example61. The example is available in Python, Java and Ruby. Then, depending on which programming language you will be using, select the appropriate client library62. Appium’s documentation63 provides all instructions, help and additional material.

Mac OS X and Linux

Ensure that Python 2.7.x or later is installed (it’s preinstalled on Mac OS X):

$ python --version

Install Python if it’s not already (Linux only):

$ sudo apt-get install python2.7

Check whether Python’s “pip” module is already installed:

$ pip --version

If you’re on Mac OS X, install pip if it’s not already:

$ curl https://raw.githubusercontent.com/pypa/pip/master/contrib/get-pip.py > get-pip.py
$ sudo python get-pip.py
$ rm get-pip.py

If you’re on Linux, install pip if it’s not already:

$ sudo apt-get install python-pip

Install the Selenium module for Python:

$ sudo pip install selenium

Verify that Selenium is installed:

$ pip freeze | grep -i selenium


Ensure that Python 2.7.x or later is installed:

$ python --version

If it’s not installed, download and run the setup from Python’s download center64. To add Python environment variables, go to “System properties” → “Advanced System Settings” → “Environment Variables” → “System Variables” → “Edit Path,” and then insert C:Python27;C:Python27Scripts at the end (assuming you have installed Python in the default location). Make sure to restart the command prompt to bring new environment variables into effect.

Check whether Python’s pip module is already installed:

$ pip --version

Install pip if it’s not already:

$ curl https://raw.github.com/pypa/pip/master/contrib/get-pip.py > get-pip.py
$ python get-pip.py
$ del get-pip.py

Install Python’s Selenium module:

$ pip install selenium

Running The First Tests

Download the Appium sample test script and sample app65. The sample package includes the APK for Android, the IPA for iOS and one sample test script. In addition, there are three samples:

  • appium_example_ios.py
  • appium_example_android.py (for Android API level 17 and above)
  • appium_example_selendroid.py (for Android API level 16 and below)

Step 1: Create Account and Upload the APK

Create an account.66 Note that this service provides a freemium option, and you don’t need a plan to complete these examples using real devices on a cloud service. If you want to access hundreds of device models, then different plans are available.

Before proceeding with running the test script, you’ll need to upload the APK or IPA to Testdroid’s cloud service via HTTP POST. Let’s try that using cURL.

For Mac OS X and Linux:

$ curl -s --user username@example.com:password -F myAppFile=@"/absolute/path/to/app/SampleApp-iOS.ipa" "http://appium.testdroid.com/upload"

For Windows:

$ cd C:pathtodirectorycontainingtheapplication
$ curl -s --user username@example.com:password -F myAppFile=@"SampleApp-iOS.ipa" "http://appium.testdroid.com/upload"

Upon successful upload, you will get a JSON response with status:0 and an unique identifier for the uploaded app:

   “status”: 0,
   “sessionId”: “abcdefg-1234-5678-abcd-111122223333”,
      “message”: “Uploads Successful”,
      “uploadCount”: 1,
      “rejectCount”: 0,
      “expiresIn”: 1800,
         “myAppFile“: “abcdefg-1234-5678-abcd-111122223333/SampleApp-iOS.ipa“
      “rejects”: {}

Step 2: Set Credentials and Other Parameters

Open the test script, appium_sample_ios.py, in any text editor. Set screenshotDir to the path where you want those screenshots to be saved on your machine. Set your credentials on testdroid_username and testdroid_password in DesiredCapabilities. Set the myAppFile identifier against testdroid_app in DesiredCapabilities. It should look like this:

desired_capabilities_cloud = 
   ‘testdroid_username’:    testdroid_username,
   ‘testdroid_password:     testdroid_password,
   ‘testdroid_project:      ‘Appium iOS Project 1’,
   ‘testdroid_target:       ‘iOS’,
   ‘testdroid_description:  ‘Appium project description’,
   ‘testdroid_testrun:      ‘Test Run 1’,
   ‘testdroid_device:       testdroid_device,
   ‘testdroid_app:          ‘sample/SampleApp-iOS.ipa’,
   ‘testdroid_platformName: ’iOS’,
   ‘testdroid_deviceName:   ‘iPhone device’,
   ‘testdroid_bundleId:     ‘com.bitbar.testdroid.SampleApp-iOS’

Step 3: Run the Test Script

Execute the following command:

$ python sample-appium_ios.py

The console’s output should look like this:

Console output after test script67
Console’s output after test script. (Image credit: Testdroid6854403733282517135) (View large version69)

Step 4: Get Results From Cloud

The screenshots will be available locally on your machine in the directory that you specified in step 2. Log into your cloud service and navigate to the project’s name, as defined in the _project attribute of DesiredCapabilities, to access the following log files:

  • Appium’s server log,
  • Logcat and instruments log.

Advanced Example: Using Appium For Mobile Game Testing With Image Recognition

In this example, We’re using Supercell’s popular mobile game Clash of Clans. It’s a fantastic game and I’ll bet many of you have played it, so you should be pretty familiar with its look and gameplay. This example does basic clicks through the game’s tutorial.

## Example script that tests Clash of Clans tutorial first steps
## Works on different resolutions, both iOS and Android

import unittest

from time import sleep
from AppiumCloudTest import AppiumCloudTest, log
from selenium.common.exceptions import WebDriverException

class ClashOfClansTest(AppiumCloudTest):
   def setUp(self):
      # AppiumCloudTest takes settings from environment variables
      super(ClashOfClansTest, self).setUp()

   def test_tutorial(self):
      driver = self.get_driver() # Initialize Appium connection to device

      sleep(10) # Wait for game to load

      # Use this to get detected screen hierarchy
      # print self.driver.page_source

      # Dismiss the in-app purchases dialog if it shows
      okayButton = None
      if self.isAndroid():
            okayButton = driver.find_element_by_id('button3')
         except WebDriverException:
            log("There was no in-app purchases dialog")
      else: # iOS
         self.driver.implicitly_wait(5) # Wait only 5 seconds to find it
            okayButton = driver.find_element_by_accessibility_id('Okay')
            # No need to sleep here
         except WebDriverException:
            log("There was no in-app purchases dialog")

      # Cancel iOS Game Center login
      if self.isIOS():
         #print self.driver.page_source
            cancelButton = driver.find_element_by_accessibility_id('Cancel')
            log("Cancelling iOS Game Center login…")
         except WebDriverException:
            log("The Game Center login was not displayed")


      # Check that a goldmine is on screen
      rect = self.find_image("queryimages/tutorial_goldmine.png",
         "There should be a goldmine on screen at beginning of tutorial")
      log('Gold mine found at %s %s! Continuing…' % (rect[0], rect[1]))

      # Dismiss the bubbles
      sleep(2) # second blabla
      sleep(2) # Goblin appears

      # Go to shop
      # NOTE: tap_image does also assert, fails test if target not recognized

      # Buy cannon

      # Place the cannon
      self.tap_image("queryimages/place_the_cannon.png", width_modifier=0.75)
      # Use gem to finish right away
      # Bring it on!
      self.tap_image("queryimages/bring_it_on.png", height_modifier=0.75)

      # To be continued…

if __name__ == '__main__':

Let’s look some of the stages in this test script. The test_tutorial contains the following steps:

  1. It first figures out whether the test executes on Android (self.isAndroid()) or iOS. As you can see, it looks for UI content differently. On Android, it tries to find a button by using an element’s ID (element_name) and on iOS by using an accessibility ID with a description (“Okay”). The same check happens for iOS’ Game Center login.
  2. Screenshots are taken at various steps and stored in files entered as a parameter in a function call.
  3. A check is done to see whether “goldmine” appears on screen by comparing two PNG files using a self.find_image call. If these pictures match (i.e. “goldmine” appears on screen), then the script continues executing the game’s tutorial.
  4. The tutorial proceeds with the following steps:
    1. Go to shop.
    2. Buy cannon.
    3. Place the cannon.

    The information about all three of these items is stored in those PNG files: shopbutton.png, cannon.png and place_the_cannon.png.

  5. Finally, the tutorial finishes and the battle starts. After the battle, the application is closed.

The video below shows how the test executes at each step. For the video, I’ve used three devices: one iOS (iPhone 4S) and two Android phones (Samsung Galaxy S3 Mini and HTC One X). You can also watch the video on YouTube70.

How Is Image Recognition Used Here?

The example shown uses image recognition (i.e. template matching71) to identify which features — basically, pixels and graphic content — are shown on screen and to compare the two pictures to each other. The algorithm built to recognize images was used on real devices and on two different platforms (Android and iOS), and it used a single test script for both platforms. This sort of image comparison is even very handy for recognizing UI elements and graphics that are resized and/or rotated.

Let’s say the template image has some distinctive features, such as text that can be easily abstracted from the background content. In this case, feature-based recognition can be used. In our example, if the “Button” text has been resized or rotated (or otherwise transformed), we can quickly and easily identify this and take further action.

The following functions explain the approach of comparing images:

def find_image(self, queryimage_file, screenshot_match=None):
   log('Trying to find %s in current screen' % queryimage_file)

   fd,screenshot_file = tempfile.mkstemp()
      # print screenshot_file
      head, tail = os.path.split(screenshot_file)
      self.screenshot(tail, path=head)
      head,tail = os.path.split(queryimage_file)
      screenshot_match='%s/%s-match.png' %
         (self.screenshot_dir, tail.split('.')[0])
      return ImageFinder.findImages(queryimage_file,
         "%s.png" % screenshot_file,
def screenshot(self, name, path=None):
   if path:
      full_path='%s/%s.png' % (path,name)
      full_path='%s/%s.png' % (self.screenshot_dir, name)

   except WebDriverException: # for iOS, sometimes times out, so retry!
      #log("Failed taking screenshot %s - retrying" % full_path)

   width,height = get_image_size(full_path)
   if(height > width):
      if self.platform_name == 'Android':
         #log("Rotating screenshot 270 degrees")
         os.system('sips -r 270 %s >/dev/null 2&>1' % full_path)
         #log("Rotating screenshot 90 degrees")
         os.system('sips -r 90 %s >/dev/null 2&>1' % full_path)
def tap_image(self, query_image,
   width_modifier=0.5, height_modifier=0.5, retries=2):

   retries_left = retries
   rect = None
   while retries_left > 0 and not rect:
      rect = self.find_image(query_image)
      retries_left = retries_left - 1
   image_name = os.path.split(query_image)[1]
   self.assertIsNotNone(rect, "Image %s is on screen" % image_name)
   x = rect[0]+rect[2]*width_modifier
   y = rect[1]+rect[3]*height_modifier
   log('%s button found at %s %s (%sx%s), tapping at %s %s' %
      (image_name, rect[0], rect[1], rect[2], rect[3], x, y))

def tap(self, x, y):
   log('Tapping at %s,%s' % (x,y))
   driver = self.get_driver()
   if self.isAndroid():
      if self.isSelendroid():
         touch_actions = TouchActions(driver)
         touch_actions.tap_and_hold(x, y).release(x,y).perform()
         action = TouchAction(driver)
         action.press(x=x, y=y).release().perform()
      else: # iOS
         action = TouchAction(driver)
         # TODO: Temporary hack
         # On iPhone 4S = res. 480 x 320 for taps but screenshots are 960 x 640
         action.press(x=int(x/2), y=int(y/2)).release().perform()


Testing is crucial to ensuring success in the highly competitive landscape of mobile apps and games72. But even poorly planned testing can take up 20 to 50% of your total development effort, in which case it would also account for the single biggest cost in your budget. To ensure that testing is extremely efficient, covering the breadth of today’s mobile ecosystems and device models, the best option is an online cloud-based service.

If you only start thinking about testing a few days before the app hits the market, it’s too late. You’ll need to test a wealth of elements, data and functionality from day one. Here are some things to consider in making testing a part of your development process:

  • Plan carefully: Automate generic processes as much as possible.
    When you’re building a mobile app, a well thought out strategy is critical, a great user experience and design are paramount, and solid development and testing are fundamental. Many aspects of testing can be automated, and this automation will increase the depth and scope of your testing and significantly improve the app’s quality. Ideally, test cases should have full access to the application and test all aspects of it — memory contents, data tables, file contents and internal program states — to determine whether the product performs as expected.
  • Your app will change during development: The same goes for testing.
    Many things about your app will change as you’re creating it: the user interface, graphics, functionality, language support, the privacy policy, the use of external resources and much more. Even if 10% of your code changes73 or is added to the app, you’ll still need to test 100% of the features. Manual testing can’t keep up with this, so your best option is to build all of your test cases for new features. Then, when a feature is added, all features will be automatically tested. Building your tests to be maintainable over the various development phases of your app is essential.
  • Choose a testing technology and provider you can grow with.
    If you already have an app on the market and are looking to create a similar one, then select a technology74 and vendor that meet your needs. For example, building your test cases according to a certain method or framework will enable you to reuse those test cases for your new application — at least to some extent. So, choose a technology and vendor that is able to handle your needs as your product scales up and as your testing has to cover new geographical areas and even support new platforms (for example, going from Android to iOS).
  • Test automation is available 24/7.
    Automation will reduce the time it takes to test new features and even the app itself by running 24/7.
  • Use a cloud-based platform for truly global reach.
    With an online cloud-based service, you’ll get instant access to hundreds of real Android devices75. Especially with Android, having access to devices that are used in volume holds significant value. Running automated tests on these devices is easy and fast and provides all of the information you’ll need, preprocessed, summarized and in full detail.

What are your thoughts on test automation for mobile? Please let us know in the comments section!

(da, ml, al)


  1. 1 http://www.zdnet.com/ios-8-usage-figures-showing-signs-of-stress-fragmentation-after-initial-uptick-7000034389/
  2. 2 http://www.gottabemobile.com/2014/10/22/iphone-6-plus-problems-fixes/
  3. 3 http://www.smashingmagazine.com/2014/10/02/what-every-app-developer-should-know-about-android/
  4. 4 http://www.smashingmagazine.com/wp-content/uploads/2014/12/01-device-rack-opt.jpg
  5. 5 http://www.testdroid.com/
  6. 6 http://www.smashingmagazine.com/wp-content/uploads/2014/12/01-device-rack-opt.jpg
  7. 7 http://appium.io/
  8. 8 http://www.supercell.net/games/view/clash-of-clans
  9. 9 http://www.gartner.com/newsroom/id/2823619
  10. 10 http://testdroid.com/testdroid/5876/test-early-test-often-testing-as-part-of-your-app-development
  11. 11 http://autotestcentral.com/how-test-automation-can-help-to-achieve-5-stars/285
  12. 12 http://www.smashingmagazine.com/wp-content/uploads/2014/12/02-value-diagram-opt.jpg
  13. 13 http://www.testdroid.com/
  14. 14 http://www.smashingmagazine.com/wp-content/uploads/2014/12/02-value-diagram-opt.jpg
  15. 15 http://en.wikipedia.org/wiki/Regression_testing
  16. 16 http://www.smashingmagazine.com/wp-content/uploads/2014/12/03-testing-in-ci-process-opt.png
  17. 17 http://www.testdroid.com/
  18. 18 http://www.smashingmagazine.com/wp-content/uploads/2014/12/03-testing-in-ci-process-opt.png
  19. 19 http://testdroid.com/testdroid/5851/increase-efficiency-and-productivity-with-test-automation
  20. 20 http://en.wikipedia.org/wiki/Cross-platform
  21. 21 http://agiledata.org/essays/tdd.html
  22. 22 http://www.smashingmagazine.com/wp-content/uploads/2014/12/04-time-to-market-opt.jpg
  23. 23 http://www.smashingmagazine.com/wp-content/uploads/2014/12/04-time-to-market-opt.jpg
  24. 24 http://www.smashingmagazine.com/wp-content/uploads/2014/12/05-different-ways-to-automate-opt.jpg
  25. 25 http://www.testdroid.com/
  26. 26 http://www.smashingmagazine.com/wp-content/uploads/2014/12/05-different-ways-to-automate-opt.jpg
  27. 27 http://www.smashingmagazine.com/wp-content/uploads/2014/12/06-different-ways-to-test-opt.jpg
  28. 28 http://www.testdroid.com/
  29. 29 http://www.smashingmagazine.com/wp-content/uploads/2014/12/06-different-ways-to-test-opt.jpg
  30. 30 http://www.appannie.com/indexes/all-stores/rank/overall/
  31. 31 http://www.androidcentral.com/devices
  32. 32 http://www.smashingmagazine.com/wp-content/uploads/2014/12/07-performance-testing-opt.png
  33. 33 http://www.testdroid.com/
  34. 34 http://www.smashingmagazine.com/wp-content/uploads/2014/12/07-performance-testing-opt.png
  35. 35 http://testdroid.com/testdroid/5901/rely-only-on-real-emulators-vs-devices
  36. 36 http://www.smashingmagazine.com/wp-content/uploads/2014/12/08-social-integrations-opt.jpg
  37. 37 http://www.testdroid.com/
  38. 38 http://www.smashingmagazine.com/wp-content/uploads/2014/12/08-social-integrations-opt.jpg
  39. 39 http://www.smashingmagazine.com/wp-content/uploads/2014/12/09-frameworks-opt.png
  40. 40 http://www.testdroid.com/
  41. 41 http://www.smashingmagazine.com/wp-content/uploads/2014/12/09-frameworks-opt.png
  42. 42 https://code.google.com/p/robotium/
  43. 43 http://developer.android.com/tools/help/uiautomator/index.html
  44. 44 https://code.google.com/p/android-test-kit/wiki/Espresso
  45. 45 http://developer.android.com/tools/testing/testing_android.html#Instrumentation
  46. 46 http://calaba.sh/
  47. 47 http://appium.io/
  48. 48 https://code.google.com/p/selenium/wiki/JsonWireProtocol
  49. 49 http://docs.seleniumhq.org/projects/webdriver/
  50. 50 https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/UsingtheAutomationInstrument/UsingtheAutomationInstrument.html
  51. 51 http://developer.android.com/tools/help/uiautomator/index.html
  52. 52 http://selendroid.io/
  53. 53 http://www.smashingmagazine.com/wp-content/uploads/2014/12/10-framework-families-opt.png
  54. 54 http://www.testdroid.com/
  55. 55 http://www.smashingmagazine.com/wp-content/uploads/2014/12/10-framework-families-opt.png
  56. 56 http://developer.android.com/tools/testing/testing_android.html#JUnit
  57. 57 http://developer.android.com/tools/testing/testing_android.html
  58. 58 http://developer.android.com/tools/testing/testing_android.html#Instrumentation
  59. 59 https://code.google.com/p/selenium/wiki/DesiredCapabilities
  60. 60 http://appium.io/slate/en/tutorial/ios.html?ruby#getting-started-with-appium
  61. 61 https://github.com/bitbar/testdroid-samples/tree/master/appium/sample-scripts
  62. 62 http://appium.io/downloads.html
  63. 63 http://appium.io/introduction.html
  64. 64 https://www.python.org/downloads/
  65. 65 https://github.com/bitbar/testdroid-samples/tree/master/appium/sample-scripts/python
  66. 66 http://testdroid.com/sm-appium
  67. 67 http://www.smashingmagazine.com/wp-content/uploads/2014/12/11-console-output-opt.jpg
  68. 68 http://www.testdroid.com/
  69. 69 http://www.smashingmagazine.com/wp-content/uploads/2014/12/11-console-output-opt.jpg
  70. 70 https://www.youtube.com/watch?v=Q5U3HwpD1Ec
  71. 71 http://en.wikipedia.org/wiki/Template_matching
  72. 72 http://www.appannie.com/indexes/all-stores/rank/overall/
  73. 73 http://testdroid.com/testdroid/6186/strive-for-hermetic-but-never-compromise-integrity-of-app
  74. 74 http://www.smashingmagazine.com/2014/01/10/four-ways-to-build-a-mobile-app-part2-native-android/
  75. 75 http://insights.wired.com/profiles/blogs/the-best-advice-for-app-developers-skip-emulators

The post The Basics Of Test Automation For Apps, Games And The Mobile Web appeared first on Smashing Magazine.

View post:  

The Basics Of Test Automation For Apps, Games And The Mobile Web


Make your VWO Graphs more Meaningful with Annotations

Ever found yourself puzzled with a sudden spike or drop in test results? Did it finally take you hours of brainstorming and discussions to figure out what caused the change? Now you can avoid all that hassle by simply adding notes to your graphs.

VWO allows you to annotate your graph reports with relevant notes to make your analysis more meaningful. Think of these as sticky notes on your graph reports. You can use these annotations to add comments, mark important events and even add relevant links to your graphs.

annotations walkthrough

Let’s say you are the optimization expert with a large software product company. To get more users for your product, you are running a campaign on your ‘sign-up’ page with a goal to improve conversions. Meanwhile, your marketing team decides to launch a well-targeted newsletter campaign that directs visitors to the sign-up page. The awesome campaign results in a sudden spike in conversions on your sign-up page, right on the date when the newsletter was sent.

A few months down the line, when you analyze the results of your campaign, it would be hard for you to identify what caused the sudden spike in conversions. This is where annotations come to your rescue. Annotations allow you to add context to your results by including notes for other activities/events happening during the test duration. The moment you see the comment with newsletter campaign in your graph, you know what to attribute the spike to.

The annotations are visible to all users in your account so that everyone can have similar context for interpreting results. It also comes with the added advantage of viewing who added the annotation so that any further discussions can flow seamlessly.

Check out this article in our knowledge base to get a step by step view of how to use annotations for your account.

And There’s More…
1. Now you can make changes to the iframe tag on your website. The iframe editor allows you to edit iframe URL, change style and placement, and so on. Simply load the webpage containing the iframe content into the campaign builder and you are ready to go!


2. The select parent option expands to allow you to select the largest parent from any element on the page. If you are making changes to an element on your page, use this option to go up to the largest parent of the element.

select parent

Let us know about your experience with this month’s updates in the comments section. And we would love to hear if you have anything to share with us about VWO!

The post Make your VWO Graphs more Meaningful with Annotations appeared first on VWO Blog.


Make your VWO Graphs more Meaningful with Annotations


15 Essential Elements of an Effective eCommerce Product Page

Product pages are the doorways that lead to conversions and revenue for an eCommerce store. A good product page plays a pivotal role in taking a customer from just browsing to actually adding the product to the cart. Most eCommerce websites, therefore, invest a significant portion of their time and money in making their product pages more attractive and persuasive.

But then, how do you do this?

Essentially, it comes down to creating a great User Experience for your customer. An effective User Experience (UX) requires relevant and useful information designed in a manner which takes the customer seamlessly through the buying process. In this blog post, we are going to talk about 15 product page elements that add to the UX for your product pages and makes them high-converting. You will be familiar with most, if not all, of these. My intention here is to create a short checklist for every element that you can use to optimize that element. There’s also a surprise for you at the end of this post, so hold on!

1) Product Name

  •  Product Name should be descriptive. This gives more clarity to the customer as well as boosts SEO for that page. For example, ’24 Carat Gold Ring’ is much more descriptive and SEO-friendly than ‘Gold Ring’
  • The name should be unique. This sounds like common sense but you’ll be surprised by how often eCommerce websites give similar names to their products which makes finding them through search really difficult. Needless to say, this also negatively affects the SEO for the products.

2) Call to Action (CTA) Button

  • This is the most important element on any product page. The CTA affects the decision – making of the customer through its size, color and text. Thus, marketers should pay significant attention to optimizing all these aspects.
  • Different colors denote different things across different cultures. When deciding on color, keep in mind two things. Firstly, whether that color triggers the emotion in your target audience that you are hoping for. And secondly, how does it contrast with the color scheme of the rest of the page. Ideally, you want the CTA to stand out so that it grabs customer attention.
  • When deciding on the text, keep in mind that certain words mean different things in different countries. For example, see how Amazon changes the CTA text on its US and UK websites to adhere to the local flavor.

Amazon US Page
Amazon US page with CTA text as 'Add to Cart'

Amazon UK Page
Amazon UK page with CTA text as 'Add to Basket'

3) Product Price

  • Understandably, this is the least controllable of the page elements. As a marketer, your main focus should be the appearance of the price tag.
  • The price should be placed close to the Buy button. In fact, the product name, price and CTA should form a kind of visual hierarchy so that the act of buying flows seamlessly in the mind of the customer.

4) Product Image

  • Photos have a huge impact on the usability and overall UX of the website as well as increasing conversions and sales. For product page images, you should be mindful of download times. You need to keep in mind that not everyone has a super-fast
    internet connection, and that high load time can negatively affect the bottom line.
  • Use multiple images clicked from different angles to give the overall look of the product.
  • Use inspirational product images (product image used in a setting; for example, a customer wearing the dress that is displayed on the product page). These images add social proof as well as make the product more desirable.

5) Product Description

  • Keep product descriptions short and make sure to include important keywords so that the page ranks well in search engine rankings

6) Quantity Option

  • Quantity option takes a very small amount of screen real estate but can lead to a bigger sale. For example, if I want to buy 5 copies of a book, I shouldn’t be forced to go to the page and click 5 times on the ‘Buy’ button. Not having this element means you are potentially leaving a lot of money on the table.
  • It should be placed near the CTA.

7) Product Reviews & Testimonials

  • According to an iPerceptions study, 61% of customers read online reviews before making a purchase decision, while 63% of customers are more likely to make a purchase from a site which has user reviews.
  • The problem with having reviews on your product pages is that so many pages just have so few reviews. This can result in negative social proof. To combat this, follow the eBay model of buyer/seller feedback. That means, as soon as someone purchases a product, send them an email asking for feedback about the product. Many customers feel that such feedback request is part of the buying process and they are more likely to fill in the feedback form. This will ensure that your review section is filled.
  • Ask people to rate reviews. This ensures that the most helpful reviews rise to the top.

8) ‘Add to Wish List’

  • This option is particularly helpful in case of indecisive customers who are evaluating your product. Also, while browsing, some customers may stumble upon a product which they like but are not yet ready to buy. This option gives them the flexibility to ‘bookmark’ the product to which they can return later.

9) Cost Savings

  • Many times, if a product has been discounted, the product page shows the original price along with the discounted price. This is a smart trick used by marketers to cash in on the loss aversion tendency of people.
  • Show both the percentage saving as well as the actual saving made on the product. Different customers are induced by different messages.

10) Cross-selling & Up-selling Options

  • As a business owner, you want the user to purchase add-ons, related products and accessories of the products they buy. One way to achieve this is to provide good options for up-selling and cross-selling on the product pages.
  • Good suggestions for similar and related products not only improve the browsing experience but also aid in product exploration.

11) Social Media Integration

  • Social media buttons provides the friends/followers with social proof, which may result in increased purchases of the same item.
  • It helps to spread brand awareness.
  • It may also increase traffic on your website which will help in increase in overall sales, quite apart from the particular product which has been shared.
  • Keep in mind, however, that if the number of likes and share are too few in number, it may result in negative social proof.

12) Delivery & Returns Information

  • Nothing is more irritating to a customer than selecting a product, making a purchase and reaching the checkout section only to discover the addition of unexpectedly high delivery charges and hidden taxes.
  • Posting the total cost on the product page eliminates the surprise of a high shipping cost and also lets the customer factor in the total cost before adding the product to the cart.

13) Live Chat Widget

  •  According to an Econsultancy reportLive chat has the highest satisfaction levels for any customer service channel, with 73%, compared with 61% for email and 44% for phone.
  • Often times people have questions that may not have been answered in the product description or in the customer reviews. If you force them to guess, or leave them wondering, they are going to leave and find the answer to their question elsewhere.

14) Product Videos

  • Visuals work much better than text when it comes to conveying a message. Many eCommerce sites are using videos in different ways to improve customer understanding
    of their products as well as make their content more engaging and intimate. SixPackAbsExercises.com, a VWO customer, A/B tested videos on the sales page which resulted in increased conversions by 46%.
  • Videos also help the customers to gain a more intimate understanding of product look as well as functionality.
  • If you decide to have videos on your website, you may also look at having transcripts for the
    videos. These not only ensure that viewers with hearing disability can access your video content, it also improves the SEO for that video.

15) Breadcrumb Navigation

Flipkart's Breadcrumb Navigation

  • Breadcrumb navigation helps the user to understand the product hierarchy as well as navigate to other areas of interest. They are also known to reduce bounce rates.

That’s it! One last piece of advice: keep testing. One thing that we at VWO have learned over the years is never to trust experts. The best way to improve product page performance is to keep testing out new ideas and concepts with A/B testing and keep optimizing your product pages.

Now it’s time for the surprise we told you about. We’ve designed a beautiful eCommerce product page template for you which employs many of the elements we just talked about. Scroll down to see it.


We have designed 4 more templates specially for you. Click on the button below to get a free eBook on ’5 eCommerce Product Page Templates to Boost your Sales and Conversions”.

The post 15 Essential Elements of an Effective eCommerce Product Page appeared first on VWO Blog.

Read this article:  15 Essential Elements of an Effective eCommerce Product Page


Adding Certificates & Payment Options on Product Pages Increased Clicks on Add to Cart

The Company

Manna is a Hungary-based online store that sells chemical-free, organic and handmade personal-care products like soaps, body butters, essential oils, etc. Their products are available for purchase in Hungary, Germany, Austria and Serbia.

The Hypothesis

Being a Hungarian brand in Germany, they have to deal with a lot of trust issues. To overcome the same they decided to test various certifications and payment seals on the website. The hypothesis was that adding these certificates and payment icons near the “add to cart” button on the product page will increase clicks on the button as well as sales and revenue.

This is how the original product page looked like:

Original product page - Manna A/B test

The Test

The team at Manna decided to a/b test 2 variations of the product page against the original. The first version had a big banner showing the payment options and 3 certificates below it. This is how it looked:

Variation 1 - Manna A/B test

The second version had 2 rows of various certificates. This is how it looked:

Variation 2 - Manna A/B test

The test was run on more than 3000 visitors split equally between the 3 variations.

The Result

Variation 1 won and 11.26% of the visitors, who viewed this version, clicked “add to cart” button, in comparison to 1.48% in original (and 8.73% in variation 2).

Here’s a screenshot of the bar graph from the VWO app showing conversion rate of each variation:

Result graph - Manna A/B test

Why Variation 1 Outperformed the Other Two Versions of the Product Page?

As an online store selling natural and organic items, the pressure on Manna to prove their credibility is much more than any other web-shop selling cosmetics and skincare. This is because in addition to having great product pages and a seamless web experience they also have to constantly re-iterate the fact that they sell authentic natural products made from chemical-free substances. Having certifications, right on product pages, thus helped them boost the clicks on “add to cart” button. This is why even version 2 recorded a whopping increase of 490% in click rate.

Variation 1 emerged as a winner because of showing the certifications and ease of payment option together. This addressed the two most common concerns faced by customers — whether the product is authentic and if their preferred mode of payment is available as an option.

Let’s Talk!

Have you done any tests on your product pages? I’d love to know what you learnt about your visitors from them in the comments section below.

Also, since you are here, why not sign-up for our webinar by Siddharth Deswal on “7 Inspiring Tests to Create High-Converting eCommerce Product Pages”. I promise you lots of useful insights in just 45 minutes on Oct, 29.

The post Adding Certificates & Payment Options on Product Pages Increased Clicks on Add to Cart appeared first on VWO Blog.

Originally from:   Adding Certificates & Payment Options on Product Pages Increased Clicks on Add to Cart


Your Intuition is Wrong – A/B Testing Results that Surprised the Experts

One of the things I absolutely love about split testing is its inherent ability to humble opinions of even the most seasoned testers. Sometimes even the most well researched hypotheses fail.

This is the strongest reason why companies must test everything from offer copy to page design, instead of relying on gut instinct or personal preferences. In this post, I’ll share some case studies that either saw huge opinion disparity among the WhichTestWon community, or whose results absolutely shocked our editorial team and Testing Awards’ judges.

Social Proof Is Not the End-All-Be-All

Can you guess which of these two versions of an in-line form generated more opt-ins for a well-known Web design blog?

With social proofWithout social proof

As of today, 71% of the WhichTestWon community picked the version on the left – the one with social proof copy – as the winner. When I present this test to live audiences at conferences, usually 90-95% of attendees pick the social proof version.

However, they are all wrong. The variation at the left — without the subscriber count — saw 122% more newsletter opt-ins than its social proof counterpart. In a world where we are jaded by the user counts of Facebook, Twitter, etc…it seems that 14k visitors wasn’t compelling enough to get prospects to act.

Normally we see companies just add social proof without testing it because virtually every blog post and ‘social media expert’ has told them it will help conversions. Thankfully the team at this site tested this before implementation, or they would have missed out on a lot of opt-ins – more than double the opt-ins in fact.

Don’t get me wrong, there is a ton of value in social proof. For one, it is great at reducing a visitor’s anxiety and can give your brand a sense of authority. The trick is finding out where it is best to publish your social stats and what exactly is worth sharing. Should you post newsletter subscribers, Facebook likes, awards, or all of the above? Simply put: test it out. Never add something blindly; you don’t know how it will impact the bottom line.

Still not convinced? A VWO customer saw a similar result recently on its product page. Read the whole story here.

Icons May Be Trending, but they Might Hurt Conversions

Icons have been making a major comeback in web design. Overall, icons have been useful, especially when they are used as a replacement for bullets in stylized lists. The team at Build.com wanted to find out whether icons would be a useful navigational tool…the results surprised them.

Here are the two versions of the header they tested, one with icons and one without:

Without icons

With icons

The challenger variation included icons that represented different category pages on the site. The team believed that an increased focus on navigation with their most visited categories would increase interactions and sales. However, the version without the icons saw 21% more product purchases.

Why? We suspect that although the icons provided a sleek navigation pane, overall they likely added more clutter that confused the visitor.

Security Seal on a Lead Gen Form Test

The highest point of friction on any lead generation page is the form itself. You need to identify the optimal number of form fields, choose an intuitive design, and add visible privacy policies and/or security icons to reduce anxiety.

These are well-known best practices that all lead gen marketers understand… and that’s probably why 74% of the WhichTestWon community guessed the wrong winner for this A/B test.

Without trust sealWith trust seal

The variation without the TRUSTe logo next to the button got 12.6% more completed forms. Yes, the ‘submit’ button did shrink to accommodate the TRUSTe logo; but, we strongly suspect the primary cause for this lift has to do with the logo itself.

Trust seals can be essential to your conversion rate; the real trick is knowing where and when to place them.

In this particular circumstance, the TRUSTe logo was the wrong security seal at the wrong time. Visitors are used to seeing this seal, and others like it, directly in a shopping cart; not on a top funnel lead generation form. It’s quite likely that many of them suspected a payment transaction when they saw the trust seal here.

Instead of using a security seal, the team could have tested providing assurance by adding a simple fine-print text link to the privacy policy.

Remember, context is the key!

If Your Conversion Rates are Falling, Put on a Happy Face?

CRO specialists and designers love using faces! I get it; faces are the first thing the eye identifies when it looks at a web page. Numerous eye tracking studies support this claim.

However, sometimes a human face can be too much of a distraction. So, you need to test when you add faces to your page to make sure they aren’t competing with your important headlines and calls to actions (CTAs).

Here’s an example:

Without human face

With human face

The version without the image won 24% more form completions. This wasn’t a perfectly clean test. There were some slight alterations in copy but nothing too dramatic. To tell you the truth, I’m more of a fan of the story behind this test than the actual unexpected results. However, it’s not the first or the last test we’ve seen where removing a face increased conversions.

By the way: I am happy that the team used the photo of an actual employee rather than a stock photo. Models and stock photos tend to get even worse conversions than “real” people.

What’s perhaps most amazing is that at the time of this split test HubSpot was about to make it a mandatory practice to include a person’s image on each of their landing pages. On some level this makes sense, they had found that some of their pages performed better with people’s pictures. However, what’s true for some pages may not always be true for all pages. Luckily, this test casted a seed of doubt and the company changed their design mandate.

Before you create any new landing page or design policies, please test beforehand…you have no idea just how many conversions you could leave on the table.

Video Icons – Product-Centric or Person-Centric?

Here is another case that tests whether using a face is appropriate.

Person-centric video thumbnails

Product-centric video thumbnails

The version of this Autodesk product page that used faces got 50% fewer video clicks. Nothing else on this page changed except for the video preview image. I am not anti-faces on websites; I simply want you to test before you implement!

Needless to say, the testing team was surprised by the results. So they ran a user survey to try to figure it out. The responses showed that Autodesk’s prospective buyers were more interested in seeing how the product worked over individuals talking about the product.

This just comes down to a case of knowing your audience and that best practices are not one-size-fits-all!

In Summary

Leaders in the testing field have all been stumped by the unexpected results before, and will be stumped again. The trick is to understand what to do after your test goes counter to your hypothesis or flat-lines.

Your next steps may include evaluating a litany of things such as your hypothesis, technology, source traffic, device…the list goes on. You need to learn if the test itself was flawed – or if your understanding of what your visitors really want from the page was flawed. Either way, you’ve learned something valuable.

Remember testing is an evolving process, future iterations are borne from our successes and our failures.

Keep testing my friends! There are so many variables to consider while running a test, it is no wonder that we often see lifts or losses where we least expect it.

The post Your Intuition is Wrong – A/B Testing Results that Surprised the Experts appeared first on Visual Website Optimizer Blog.


Your Intuition is Wrong – A/B Testing Results that Surprised the Experts


How to use Urgency and Scarcity Principles to Increase eCommerce Sales

Imagine there’s a cupcake fair in your community. You have your heart set on Red Velvet and there are two counters selling the same. Counter A has a animated bunch of customers digging into the cake and has a banner saying “last 20 pieces left” while Counter B adorns a deathly, almost funereal look.

Which one would you go for? If you are thinking Counter A, you are not alone.

A study was conducted in 1975 where researchers wanted to know how people would value cookies in two identical glass jars. One jar had 10 cookies while the other contained just two. Though the cookies and jars were identical, participants valued the ones in the near-empty jar more highly.

And that’s the scarcity principle at play. It essentially means that people tend to place higher value on an object that is scarce and a lower value on one that is available in abundance. No wonder, marketing guru Robert Cialdini cites ‘Scarcity’ as one of the six golden persuasion principles in his book “Influence”. When combined with Urgency, which is essentially the other side of the same coin, the two make for a potent weapon for increasing eCommerce sales.

Here are some examples of how different eCommerce websites creatively use this persuasion principle to increase conversions.

1) Stock scarcity

Displaying your stock meter on the eCommerce product page is always a good conversion practice. Not only does it ensure there are no last-minute heartbreaks for the customer, it also speeds up the buying process. A user might be convinced to make a purchase, but he/she might not be always willing to buy right away. They might want to compare the prices on other websites, look for discount coupons or simply forget about the product — thanks to the myriad distractions of the web.

Look how ModMomFurniture flashes a message that ‘only 3 items left’ of a particular product.

Stock meter

Here, Boticca almost urges visitors to complete the purchase right away. The use of an active verb like ‘Act’ is used to drive immediate action. Here is a list of more words that drive urgency and sales.

Scarcity principle

2) Size scarcity

You head to a shop to buy a denim and figure out that the last piece in your size has just gone out of stock. Old story? Well, if it could happen in the real stores all the time, why couldn’t it happen online? Intimating the buyers when a particular size goes out of stock is killing two birds with one arrow. Not only is it a huge favor for the prospects, the information also works as a positive reinforcement of the product.

See how Jabong represents the unavailable size in grey.

Size scarcity

Zappos goes a step further and even shows how many items are left for a particular size and color combination.


3) Time-bound purchase for next day shipping

If you are already offering next-day delivery, then ‘time-bound purchase’ won’t cost you any additional resource. You just need to inform the users how many hours/minutes do they have to complete the purchase so that their order qualifies for next day delivery.

When you ask the visitors to make the purchase in a specific amount of time, you not only make them more proactive towards the purchase, it also eliminates any kind of confusion at their end as to when they will receive the order.

Zappos has a permanent banner on its homepage saying that you need to order before 1pm to qualify for next-day shipping.

Next day shipping incentive

Amazon shows the exact number of hours/minutes within which the purchase needs to be completed to qualify for next-day shipping.


4) Make them see other buyers

Two women fighting over the same item of clothing in a fashion store is not just a devilish mind’s fictitious construct. While popular Hollywood movies might have taken these fights too far, the fact remains that people are much more inclined to buy something when other people desire them too.

So while they are considering whether or not to buy those shiny leather shoes and they suddenly realize that another person is mulling the same purchase, it replicates the physical store scenario where two people grab the same item at the same time.

When you look at a property in Booking.com, it shows you exactly how many people are checking out the same property at that exact moment.


Hotels.com even gives you information about how many people are viewing properties in a particular city. A modal box pops opens and shows you your virtual competition/companions.

Last booked

5) Limited-period discounts

The fear of missing out (FOMO) is a legitimate one. It is the anticipated regret of not being able to seize an opportunity. A limited-time discount works exactly at that level. It makes the offer look so tempting and fleeting that one is compelled to seize the opportunity.

OverStock dramatically uses a running countdown to accurately show how long the sale will last.


MakeMyTrip.com shows an alert when the last few discounted airline tickets are left in stock. See how they use color psychology here to instill urgency. The use of the color ‘Red’ is not a mere coincidence. The color is associated with energy, increased heart rate and is often used in clearance sales.

Last few tickets

5) Limited-time discount on abandoned cart items

The average online cart abandonment rate is 67%. While a majority of these carts are not redeemable, re-marketing efforts can salvage some of these lost sales. Offering limited-time discounts on abandoned cart items is a great way to use the urgency principle to re-market products. However, the risk here is of overdoing it. You do this too often and you will see your ‘e-mail open rate’ nosedive.

See how Miracas offers a 5% discount for just the next two days.

Remarketing mail

6) Shopping cart item sold out

Seeing some items disappearing from the cart might just be the reality check users need to wake up and salvage the rest of the cart items. See how Snapdeal shows a ‘Sold Out’ message next to one of the books in the following example.


7) Limited-time free delivery

Well, this works the same way as limited-time discounts. The desire to avail free delivery could possibly offset the visitors’ tendency to procrastinate the purchase.


8) Special discount hours

A two-day or a weekend sale has its own charm but a special discount hour can be used to galvanize excitement around that specific hour. Zivame sent out this mailer to subscribers to build up craze for its hour-long sale.


9) Last chance emails

E-mails letting subscribers know about last day of sale is another great way to get their attention. See how Myntra makes use of Orange color to drive action here. According to color psychology, just like Red, Orange also has an aggressive feel and creates a sense of urgency to do something.


Before you set out to use any of these tactic mentioned above, there are three golden rules you should keep in mind.

1) Don’t expect ‘scarcity’ to create demand

‘Scarcity’ or ‘Urgency’ work best as motivators to quicken the buying process but the customer will have to be already convinced to make the purchase decision. While they are great procrastination killers, don’t expect them to generate demand. If you go back to the initial example, it’s only once you have decided to buy a cupcake that you will value the cupcakes in Counter A more. You will have to truly convince the user before you flash the scarcity card.

2) False urgency can backfire

Urgency is a subtle art. Yes, I know that sounded slightly oxymoronic but don’t go about faking urgency or your customers will get a whiff of it. Be honest. Don’t try to get rid of the stock that won’t move by flashing messages like ‘hurry, only last two pieces left’. First they will catch your bluff, then you will become a laughing stock and then they will leave you. Here’s a real life example of how that happened.

3) Don’t overdo it

Even if you are using the scarcity principle in all earnestness, don’t go about overdoing it. You don’t need to employ all the practices with multiple countdowns breathing down customer’s neck. You will come across as pushy and shady and untrustworthy and manipulative. Moderation is the key, though testing will give you better insights as to what will work for you.

The post How to use Urgency and Scarcity Principles to Increase eCommerce Sales appeared first on Visual Website Optimizer Blog.

This article – 

How to use Urgency and Scarcity Principles to Increase eCommerce Sales