# How reliable are DxO Mark scores?



## sovietdoc (Mar 26, 2012)

While following the never-endless debate of "Canon vs Nikon", I've stumbled upon a number of people who criticized DxO Mark for being wrong.  I don't usually care about scores produced by any benchmarking suites as they usually don't reflect real life performance, but because a lot of people actually base their opinion on the numbers produced by DxO Labs, I had to look into it deeper.  

Apparently, their Lens benchmarking tool shows 70-200 f/2.8 mark I at 21 points and mark II at 18.  Now, everyone who shot with the Mark I and then went to Mark II will tell you that mark II is better in every single way, yet DxO places it lower by a significant margin.  

DxO shows Mark II at a lot worse resolution, worse distortion, worse vignetting and better CA compared to Mark I.  This is rubbish as many many people have cropped RAWs from either lens and found that mark II was better than mark I in every one of those cases.


This is just one example of obvious error in DxO Mark tests, there are others as well.

So the question is: Why do people think DxO's scores are so significant that they use DxO scores to justify their purchases or in an argument of what's better?  I would seriously doubt any number produced by DxO Labs based on what I've seen.


----------



## analog.universe (Mar 26, 2012)

I agree, browsing through DxO I've seen lots of results that are either obviously wrong, like the example you point out, or don't match up with other review sites.  When I buy a lens I read every test I can get, and the entire internet usually agrees on the numbers, except DxO.  They so frequently contradict ever other test that I've stopped reading them at all.  I never know if I'll get something bogus...


----------



## Trever1t (Mar 26, 2012)

Funny, all the Canon users were touting the benchmarks WHEN Canon tested higher?


----------



## o hey tyler (Mar 26, 2012)

Trever1t said:


> Funny, all the Canon users were touting the benchmarks WHEN Canon tested higher?



He's referencing a difference in scores of the* same Canon lens *(MKI and MKII). This isn't a Canon v. Nikon thing... It wasn't even brought up.


----------



## Trever1t (Mar 26, 2012)

oops.....mybad! Sorry, I totally misread!


----------



## KmH (Mar 26, 2012)

DXO Mark uses industry standard testing methods.

DxOMark - Test result reliability


----------



## Derrel (Mar 26, 2012)

o hey tyler said:


> Trever1t said:
> 
> 
> > Funny, all the Canon users were touting the benchmarks WHEN Canon tested higher?
> ...



It's implied...Canon has been gettig its ASS kicked by Nikon products since 2009...and now, Canon users are questioning the validity of the DxO Mark scores....BUT, when Canon was scoring higher, they were touting the DxO Mark scores all the fricking time, all over the web...

The better question might be, "How reliable is Canon's lens quality control???"

Here is what Axel, at DxO Mark had to say about the test results: *Canon EF 70-200mm f/2.8L IS II USM vs its predecessor :*If we compare these 2 generations of L series telephoto lenses on a Canon 5D Mark II, we can see the following:

These 2 lenses are very similar in term of transmission, distortion and vignetting
The version II gives slightly less chromatic aberration
The Canon EF 70-200mm f/2.8L IS II USM offers slightly less resolution with 51 lp/mm compared to the excellent 61 lp/mm of the Canon EF 70-200mm f/2.8L IS USM
The slightly less good resolution result of the Canon EF 70-200mm f/2.8L IS II USM comes from less homogenous behavior across the field.
.."after checking with all our experts in the lab, there isn&#8217;t really a mistake.

Indeed, the review  was a bit harsh with the new version of the Canon EF 70-200mm f/2.8L IS II USM and we modified it: DxOMark - Canon EF 70-200mm f/2.8L IS II USM measurements and review.
But, overall the *Mark 1 has a slightly higher and more homogeneous resolution*. So, *it scores better on a full frame camera*, like the Canon 5D Mark II used in the review.
But, as we also added to the review,* if you make the same comparison on a APS-C, it&#8217;s the Canon EF 70-200mm f/2.8L IS II USM that comes out slightly ahead*. This may be the reason why the results are in other reviews made on other cameras.


So, what it means is that both lenses are really really close, and if you have a full frame it is better go for the older version and if you have an APS-C the newer version is a better choice.


Axel from DxO Labs"

So...they stand by their results on the 5D-II; the older lens performs better across a wider range of settings. ALSO, the 70-200 Mark II is alleged to be non-parfocal, meaning its focus point shifts as the lens is zoomed....that is a VERY,VERY bad quality to have in a lens like a 70-200/2.8 zoom...

The differences in lens scores are not particularly significant on the 5D-II they tested it on...I would love to see the OP verify his assertions that there are "other errors" in DxO Mark's tests....if so, let's SEE some URL's and some actual proof.


----------



## analog.universe (Mar 26, 2012)

KmH said:


> DXO Mark uses industry standard testing methods.
> 
> DxOMark - Test result reliability



Nothing there seems to explain why so many of their results are either self contradictory, or contradict the results of other respected testers.

Does it not seem odd that the Samyang 14 and 35mm, as well as the Canon 85mm 1.8, all report higher resolution figures than any macro lens they tested?  They claim the Tamron 18-270mm f/3.5-6.3 has a higher resolving power than the Canon 300mm f/2.8 IS II (a lens which broke records on most testing sites).  You could go on and on with the absurdity...  It doesn't matter how objective they claim to be, their figures are in disagreement with reality.


----------



## DorkSterr (Mar 26, 2012)

Yup those guys over at DxO know what their doing.


----------



## Frances69 (Mar 26, 2012)

Nothing there seems to explain why so many of their results are either  self contradictory, or contradict the results of other respected  testers.


----------



## chuasam (Mar 26, 2012)

Reliable, accurate and absolutely pointless in real world applications.


----------



## jaomul (Mar 27, 2012)

chuasam said:


> Reliable, accurate and absolutely pointless in real world applications.



I believe this to be true but- I have to point out that i use Canon so i may be a bit defensive because they make such bad cameras and i wasted my money on 2 of them. It was pointed out on a post above by a very informed source that Nikon are kicking canons ass since 2009. I dont doubt this, the facts say so. This leads me to believe that the nikon cameras before 2009 were even more crap than the crap canon cameras on the market now. I should go get a sight test because i have seen loads great images from older Nikon cameras from before that. How is this. i am confused now


----------



## Crollo (Mar 27, 2012)

about as reliable as the american government


----------



## sovietdoc (Mar 31, 2012)

I think this says it all: Customers worldwide use DxO Analyzer


----------



## 2WheelPhoto (Mar 31, 2012)

they are hiring too

DxO - Careers


----------



## Mrgiggls (Mar 31, 2012)

sovietdoc said:


> I think this says it all: Customers worldwide use DxO Analyzer




I must be confused...it looks like you're using marketing material from the company who makes DxO in a discussion about whether it should be considered accurate and reliable.:scratch:


----------



## sovietdoc (Apr 1, 2012)

Mrgiggls said:


> sovietdoc said:
> 
> 
> > I think this says it all: Customers worldwide use DxO Analyzer
> ...



If you look at that page, DxO says "major players in the imaging industry" and Canon isn't there.  Obviously Canon isn't a major player in the imaging industry.  This may explain why their cameras and lenses score lower across the board in DxO tests.


----------



## Mrgiggls (Apr 1, 2012)

sovietdoc said:


> If you look at that page, DxO says "major players in the imaging industry" and Canon isn't there.  Obviously Canon isn't a major player in the imaging industry.  This may explain why their cameras and lenses score lower across the board in DxO tests.




Ah...now I get it lol.....my bad... I are a little slow


----------



## KmH (Apr 1, 2012)

sovietdoc said:


> I think this says it all: Customers worldwide use DxO Analyzer
> 
> If you look at that page, DxO says "major players in the imaging industry" and Canon isn't there.  Obviously Canon isn't a major player in the imaging industry.  This may explain why their cameras and lenses score lower across the board in DxO tests.


Oh Please! 


> Here is _*just a sample*_ of our current customers:


----------



## Village Idiot (Apr 2, 2012)

Trever1t said:


> Funny, all the Canon users were touting the benchmarks WHEN Canon tested higher?



Wow. This guy's a more serious Nikon fanboy than some others on here. Every thread I've read with your post in it are praising Nikon for what you perceive as infallibility or bashing Canon because they're Satan's camera.


----------



## Nikon_Josh (Apr 2, 2012)

Village Idiot said:


> Trever1t said:
> 
> 
> > Funny, all the Canon users were touting the benchmarks WHEN Canon tested higher?
> ...



Your a famed 'Canon Shill' on the forum as Derrel once mentioned.. I remember you actually making a post to show off about how Canon HDSLRs were being used in Hollywood. Takes a fanboy to know one!


----------



## sovietdoc (Apr 2, 2012)

KmH said:


> sovietdoc said:
> 
> 
> > I think this says it all: Customers worldwide use DxO Analyzer
> ...



So what you're saying is they put HTC, CHIMEI and CCI in that list and didn't put Canon because they were just showing a sample.  It's like making iPad displays and not saying that Apple is one of your customers but listing all those little Chinese ripoffs instead.


----------



## Dao (Apr 2, 2012)

I wonder if it will end like like the PC industry benchmarking in the past.   The manufacturers create hardware that optimizer just for the test.  So a video card score very high in the test does not mean it perform as great in real life example.


----------



## Village Idiot (Apr 2, 2012)

Nikon_Josh said:


> Village Idiot said:
> 
> 
> > Trever1t said:
> ...



Looks like we have another one. Let me guess, the 5D MKII is a novelty piece of equipment at best and could never be used in professional video production?


----------



## o hey tyler (Apr 2, 2012)

Village Idiot said:


> Let me guess, the 5D MKII is a novelty piece of equipment at best and could never be used in professional video production?



You are correct sir. The 5D II is a piece of trash. Especially for video production. No one ever uses that garbage. 

http://www.usa.canon.com/cusa/about_canon?pageKeyCode=pressreldetail&docId=0901e0248045fd9f

[URL="http://www.petapixel.com/2010/04/09/house-season-finale-filmed-entirely-with-canon-5d-mark-ii/"]http://www.petapixel.com/2010/04/09/house-season-finale-filmed-entirely-with-canon-5d-mark-ii/

[URL="http://www.petapixel.com/2010/09/29/canon-5d-mark-ii-used-for-iron-man-2/"]http://www.petapixel.com/2010/09/29/canon-5d-mark-ii-used-for-iron-man-2/

[URL]http://www.techradar.com/news/photography-video-capture/cameras/oscar-nominated-doc-shot-with-canon-5d-mark-ii-1057377[/URL][/URL][/URL]


----------



## KmH (Apr 2, 2012)

You can bet pro video done with a 5D MKII uses a 5D MKII that has at least a couple thousad $$$$'s worth of added hardware, like Redrock Micro

and a good video tripod, like a Manfrotto 501HDV,546BK Video Kit with 501HDV Head and 546B Tripod (Black)

We won't get into the lighting and sound gear.


----------



## analog.universe (Apr 2, 2012)

KmH said:


> You can bet pro video done with a 5D MKII uses a 5D MKII that has at least a couple thousad $$$$'s worth of added hardware, like Redrock Micro
> 
> and a good video tripod, like a Manfrotto 501HDV,546BK Video Kit with 501HDV Head and 546B Tripod (Black)
> 
> We won't get into the lighting and sound gear.



But you add a couple thousand dollars of hardware to your still camera to get the images you want as well....


----------



## Trever1t (Apr 2, 2012)

Village Idiot said:


> Trever1t said:
> 
> 
> > Funny, all the Canon users were touting the benchmarks WHEN Canon tested higher?
> ...




What? Better check yourself and your facts. I never say one brand is better than another nor do I think that would or could be a true statement. Rather, I only point out that when Canon did rank higher scores that Canon users quoted them often. 

I'm not anybody's fanboy....chit, I took off all the Harley emblems off my bike. Not into branding, you got the wrong guy. I just call it like i see it...and as I see you you were way out of line.


----------

