Can Consumers Tell the Difference Between Phones Anymore?

A Smartphone Blind Taste Test with public radio’s Marketplace

Google’s Pixel 3a presents a challenge to the rise in smartphone pricing, which can now easily exceed $1,000. The new $399 Pixel 3a has the same extraordinarily well-reviewed camera as the $799 Pixel 3 and many of the same features, but for half the cost. That got Scott Tong from APM’s Marketplace wondering if regular consumers can tell the difference between different priced phones anymore, so we arranged an experiment.

We invited five people (including Scott) to come to my office for a focus group. None of the participants are gadget enthusiasts. I set up eight smartphones across the price spectrum ($300 - $1,000), covered the logos with electrical tape, and loaded test accounts on them. We then asked each participant to test each phone for 5 – 10 minutes; take pictures indoors, outdoors, and in the dark guest room next to my office; scroll through Facebook, Google Photos, YouTube, and Google Maps (which I moved to the home screen); rate it across four categories (design, display, camera, app/performance); guess how much it cost (from $300 - $1,000); and tell us their key impressions. This was a focus group, not a survey – doing regression analysis on this data will not yield scientifically valid results, but we knew that asking our group to provide a numerical rating would force them to think about these categories, and having some data would help me to identify trends.

The Marketplace segment is here (jump to 21:45)

The results were fascinating, and not necessarily what you might expect.

Some conclusions:

Even our group of regular consumers could easily pick out the pair of $1,000 phones from Huawei (P30 Pro) and Samsung (Galaxy S10+), and they liked them more than any of the others. At the highest level, you do get what you pay for.
While I was setting up the phones, the OnePlus 7 Pro felt viscerally faster and smoother than most of the other phones. Our panelists noticed this, too. While it wasn’t anyone’s favorite, the OnePlus 7 Pro came in second place on the averaged numerical ratings.
Despite picking the most expensive pair of phones as their favorites, our panel did not guess the prices correctly. At all. On average, they underpriced the $1000 phones by hundreds of dollars. They also thought that the Motorola G7 and Google Pixel 3a cost $100 and $200 more than they do. While this might make Google’s marketing team happy, I would be hesitant to get too excited: as mentioned, our panel was not great at guessing pricing, and the Pixel 3a was one of the least favorite phones in the lineup. Numerically, it came in second to last, and complaints focused on the relatively small display – which may be why the camera scored below average – and cheap-feeling plastic materials.
The iPhone XR may be the best-selling phone in the world, but it came in dead last in our test. With the Apple brand hidden, Apple’s own apps moved to another screen, and key features – such as FaceID* – disabled, the $750 phone did not stand out. Participants thought it was a budget Android phone. In our blind taste test, abstract concepts like security and expectations around software updates were excluded, and this hurt Google’s Pixel 3a as well. In the real world, phones are not purchased without taking brand, ecosystem, and privacy considerations into account, nor should they be. If you want an Apple Watch, you need an iPhone to go with it. Still, this test does show that Apple cannot rely on hardware alone for continued success – rivals have definitely caught up in that respect.
I included three phones from Huawei and its Honor brand, and all three were well received. U.S. consumers are missing out on access to these phones, and if Huawei cannot extricate itself from the U.S. government’s entity list, so will the rest of the world.
Finally, while nobody confused Lenovo’s $300 Moto G7 for a $1,000 flagship, it ranked in the middle on the ratings, and all participants thought that it cost at least $100 more than it actually does.

So, there you have it. Not all phones are alike, and consumers are not sure exactly how much they should cost. If you pay $1,000 for one, you will certainly get your money’s worth, but if you spend a lot less, not only will you have more money in your pocket, you’ll feel like you got a bargain.

Thank you to American Public Media’s Marketplace for this collaboration, and to my local NPR station for airing it.

Techsponential clients are encouraged to reach out to discuss the results in detail and for customized recommendations for marketing, positioning, pricing, and design.

* Biometric security was disabled on all the phones to facilitate testing