Schiller calls ‘shenanigans’ on Samsung’s Note 3 benchmark boost

Schiller calls ‘shenanigans’ on Samsung’s Note 3 benchmark boostApple marketing chief Phil Schiller doesn’t generally have a great deal to say on twitter, though he has been known to poke fun at rivals in the past, including a cheeky March 2013 tweet about Android security.

Phil’s back with a one-word tweet this time, and a handy link to an article claiming that Samsung is up to its old benchmarking tricks, this time with the spanking new Galaxy Note 3.

Back in July, Anandtech claimed that Samsung allowed the Galaxy S4 CPU and GPU to run wild in particular benchmarking applications, something Sammo later denied.

This week, it’s ArsTechnica suggesting similar antics on the Samsung Galaxy Note 3, and Phil Schiller wants his 100,000+ followers to know it, with a tweet simply reading: “shenanigans”.

The Ars holes (joking) write: “Normally, while the Note 3 is idling, three of the four cores shut off to conserve power; the remaining core drops down to a low-power 300MHz mode.

“However, if you load up just about any popular CPU benchmarking app, the Note 3 CPU locks into 2.3GHz mode, the fastest speed possible, and none of the cores ever shut off.”

Presumably Samsung will respond soon with the obligatory denial, but in any case, you have to wonder why the hell they would bother. If you base your smartphone buying habits on benchmarks, or consider them anything more than a nice-to-know, you’ve got a problem, buddy.

And heck, I’d wager that 99.9% of the smartphone buying public doesn't have a clue what a benchmark is, never mind caring about the results.

Add a comment
15 comments

barrybarryk  Oct. 2, 2013 at 16:06

You mean the CPU goes into a power saving mode when it's not in use and steps up when it's being stressed? I'm shocked, no wait, even my i5 does that nevermind virtually all ARM SoCs

AhmadCentral  Oct. 2, 2013 at 16:22

So the same thing the iPhone does during benchmarks.... and the same thing every phone does.

Sure Samsung have written a profile specifically for benchmarking apps but it's still a representation of the devices power when under heavy load.

corgi74  Oct. 2, 2013 at 20:42

corgi74  Oct. 2, 2013 at 20:49

According to the original article if you run the benchmark as is it gives one set of figures but if you rename the benchmark exe and run it again you get a lower score.
They said Samsung has set the Note to recognise the exe names and run faster when it sees them in use.

barrybarryk  Oct. 2, 2013 at 21:35

Well not really "faster" it just disables the power saving, it doesn't actually 'overclock' the phone. It's still running at its marketspeed. If anything the differences between the two show a flaw in the mobile benchmarking methodology more than anything else

corgi74  Oct. 2, 2013 at 21:44

Or they are exploiting the over emphasis on specs and speed with Android phones to make their product more appealing to those obsessed by such things.
I'm glad Google have not done it on their products. IMHO the Nexus devices show how Android should be done.

barrybarryk  Oct. 2, 2013 at 21:50

I like the Nexus line myself as well but if you're going to benchmark any device, it should be done without any power saving gubbins turned on. Otherwise, it completely defeats the purpose of the exercise. The benchmarking 'Apps' themselves should be taking care of that to ensure the ramp up from idle to full isn't affecting the end score.

corgi74  Oct. 2, 2013 at 22:03

So you're advocating false benchmarks that emphasise speed over real world performance comparisons?

barrybarryk  Oct. 2, 2013 at 22:12

Benchmarks aren't real world performance indicators, they never have been for any device. They're arbitrary numbers to mark the maximum performance across multiple vendors and products for a comparison, they have absolutely no link to real world performance.

If you're benchmarking an SoC with power saving gubbins on, you're getting an indicator of the max score ramping from idle (some times) to full. Meaning the scores are only comparable to the first few seconds of running a demanding 'app' and only relevant if the other phones were in the same state to begin with and running the same power saving software.

Turn them off and it's a direct comparison of what the hardware is capable of, you're benchmarking the phones performance not its SoC steps.

AhmadCentral  Oct. 2, 2013 at 22:15

So you're advocating false benchmarks that emphasise speed over real world performance comparisons?

It's showing how powerful the device is. That's what a benchmark is.

corgi74  Oct. 2, 2013 at 22:15

The important thing is that its a fair fight. The LG G2 is pretty much the same as the Note 3 internally but it benchmarks a lot lower which is what made Anandtech do the report.
If its no holds barred, unrestricted massaged benchmarks then it becomes a meaningless arms race to be the first to post a certain speed.

corgi74  Oct. 2, 2013 at 22:19

It's not showing true power. If you had 2 cars that weighed the same and had the same engine in you would expect similar performance.
What if one car has then been sent to have the engine remapped? Mechanically it is no different and the mechanics have only increased what was already there to show the maximum potential power but is it then a fair race?

drunken_prozac  Oct. 3, 2013 at 08:49

It's not showing true power. If you had 2 cars that weighed the same and had the same engine in you would expect similar performance.
What if one car has then been sent to have the engine remapped? Mechanically it is no different and the mechanics have only increased what was already there to show the maximum potential power but is it then a fair race?


But the engine isn't being 'remapped'...
What's happening is one car is using 50% throttle and the other has his(or her) foot to the floor when having a race... wonder who will win???

A benchmark is for testing the top speed a device can do, if you have a really powerful device which can do 120fps but the benchmark only wants 60fps then the cpu will never be maxed out so the result will be a lower performance score, what sammy are doing is showing exactly how powerful the device is rather than just letting it work at an average speed.

corgi74  Oct. 3, 2013 at 09:16



But the engine isn't being 'remapped'...
What's happening is one car is using 50% throttle and the other has his(or her) foot to the floor when having a race... wonder who will win???

A benchmark is for testing the top speed a device can do, if you have a really powerful device which can do 120fps but the benchmark only wants 60fps then the cpu will never be maxed out so the result will be a lower performance score, what sammy are doing is showing exactly how powerful the device is rather than just letting it work at an average speed.


It's nothing like one having 50% throttle. It's like everybody agreeing to all go at 50% throttle and then one flooring it. The whole point of a benchmark is to compare similar things against the same standard. If they were both running at 50% of the potential speed it would be a fair race as the faster one would still be faster and have longer bar on the chart.

A benchmark is not for testing top performance. It is for comparing similar things.
bench·mark
[bench-mahrk] Show IPA

noun
1.
a standard of excellence, achievement, etc., against which similar things must be measured or judged: The new hotel is a benchmark in opulence and comfort.

2.
any standard or reference by which others can be measured or judged: The current price for crude oil may become the benchmark.

3.
Computers. an established point of reference against which computers or programs can be measured in tests comparing their performance, reliability, etc.

Samsung (and others) have employed people to create routines to recognise when certain benchmarking software is being used and to increase performance accordingly with the sole purpose of producing a higher score on that software than their contempory devices by other manufacturers. Cheating is cheating.
What is interesting is that the Western/US companies haven't been involved.

Stelph  Oct. 3, 2013 at 09:40

Personally I think it should be benchmark on real world use.

On one side having a benchmark as the absolute top speed that the chip can do is a good thing, however in that case almost all smartphones would be the same speed, what is more interesting is how quick the phone is in the context of what the average user will experience (i.e. bringing in all the background processes that the manufacturer has added as part of their "skin" and/or any battery saving coding that they have implemented)

Essentially it needs to be clear if a manufacturer/test is testing for overall speed or real world use, TBH it would be good to see both as I think it would be quite telling about how much bloat some manufacturers are loading android with...

Email:

You don't need an account to comment. Just enter your email address. We'll keep it private.

Comment: