# Interpretation of DxOMark results



## enzodm (May 12, 2011)

While I'm glad to have an independent and quantitative evaluation of sensors and lenses like DxOMark, I'm somewhat puzzled with some of their data. As an example, please look here: DxOMark - Compare sensors
(comparison among 1000D, 50D, 60D).
Sooner or later I wish to upgrade from my 1000D (XS) to 60D or even 50D to save some money. Put apart the better body, 50D seems only slightly better than 1000D from the sensor point of view and worst if looking at ISO performance (perhaps related to different pixel density). 
If I look at details, it seems that the 1000D is better than both 50D and 60D at 100ISO, and then more or less the same. If I look at noise, it seems like they are more or less the same, unless the fact 50D and 60D allow for greater ISO, but with quality much lower than 1000D 1600ISO. I'm not sure how these differences translate in real picture quality - it seems to me to have seen better low light pictures coming from 50D than what I would expect from those data.

So, please give me directions on how interpret DxOMark results for a more informed choice, thanks.


----------



## Overread (May 12, 2011)

The problem with DxOMark results is that they never publish the methods or criteria they use for testing, just the results. This leads to a lot of results they publish being somewhat suspect by the general population as they suggest things that - in the field and in real world experiences - are either different or more marginal than the scores from DxO would suggest.

Personally I would take it with a pinch of salt and read regular reviews and opinions to get a more balanced viewpoint.


----------



## analog.universe (May 12, 2011)

^    +1

If you want to compare image quality from 2 different cameras, look at the actual images they've taken, not some mathematical deconstruction that attempts to explain them.  I've had good experience with Digital Camera Resource Page | Unbiased Digital Camera News and Reviews.  They take essentially the same shots of the same subjects for every camera they review, so comparisons are pretty straightforward.

As you mentioned though, due to the size of the pixel on the sensor, all other things being equal (which they pretty much never are), higher pixel count = higher noise.


----------



## subscuck (May 12, 2011)

^ Hmmm, didn't we just go thru this last night on another thread? DXO Mark is a waste of reading time.


----------



## Overread (May 12, 2011)

DxO is however not all bad as their processing software that they make is very popular - esp with users of wide angle lenses looking to reduce distortion from the lens.


----------



## enzodm (May 12, 2011)

Overread said:


> The problem with DxOMark results is that they never publish the methods or criteria they use for testing, just the results.


 
ok, this is really a problem. No methods, no sensible interpretation.
I always looked at reviews (Analog: for me, DPreviews gives good comparisons), but  numbers are  tempting, at least for my education.


----------



## subscuck (May 12, 2011)

I'm not sure I've seen it here, but on dpreview, which seems to have a lot more DXO Mark discussion, I've clicked on links people have provided to DXO Mark showing consumer and prosumer lenses with better numbers than L or Gold Ring glass. As Overread said, without knowing how these results were obtained, it's just not possible to interpret the numbers in any meaningful way. And vis a vis the thread from last night, raw numbers are meaningless, final output is what counts.


----------



## KmH (May 12, 2011)

DxOMark - Why trust DxOMark?



> Why trust DxOMark?
> 
> When looking for digital camera image quality evaluation data and benchmarks, you want to be sure that the provided measurements have been openly and objectively performed by a completely independent testing operation that uses state-of-the-art equipment and follows industry-standard methods.
> *DxOMark means objective, independent, comprehensive image quality data*
> ...


----------



## enzodm (May 12, 2011)

KmH said:


> [... (read more about How we test).
> *DxOMark measurement methods and protocols are open and compliant with international and industry standards*...


 
ok, thanks: so methods (at least the basis) are public, and I will read them with care. 
But I have to come back to the original question: at this point how do I interpret the comparison between 1000D and 50D? Are they really so close that, _from the sensor point of view_, it is not worth to "upgrade" from one the the other?


----------



## KmH (May 12, 2011)

Maybe!

You have to consider the other camera features and functions too.
Things like the fact the 1000D doesn't have Spot metering mode, while the 50D does.

How about the frame burst rate? How about the auto focus system? how do those compare.

DXO Mark does not compare bells and whistles. DXO Mark only compares the basic Raw image metrics.


----------



## Garbz (May 12, 2011)

subscuck said:


> I've clicked on links people have provided to DXO Mark showing consumer and prosumer lenses with better numbers than L or Gold Ring glass.



While in general the thought is counter intuitive don't automagically assume that a colourful ring at the end of the lens means it necessarily outperforms another lens without the coloured ring. I can think of several Gold Ring lenses which are outperformed in sharpness and CA by my kit lens. In general though my kit lens doesn't have an f/2.8 aperture though, which is what I paid for with my gold ring lens. There are some truly horridly performing gold ring lenses, as well as some truly exceptional "cheap" ones.

But I have no comment on DxO's results on the matter.


----------



## Derrel (May 12, 2011)

I've seen loads of Canon users who try and discredit DxOMark results because so many recent Canon models perform so poorly against the new Nikon and Pentax cameras that have sensors made by Sony in them...they cannot imagine that their higher MP-count Canons cameras could score worse than lower MP count Nikons and Pentaxes...in three metrics: Dynamic Range, High ISO, and Color Depth...it's almost epidemic, the constant attempt to discredit DxOMark results among Canon fanatics. Last year, we had numerous Canon nuts (extremists who have for the most part, left this site) constantly trying to discredit an entire company's worth of people and testing equipment as biased...because Canon products were being out-scored by competing cameras. The list of canards these guys pout forth about DxOMark was a Fear Uncertainty and Doubt campaign worth of Microsoft in the 1990's.

One thing to keep in mind right now, in mid 2011: SONY has developed a new generation of sensors that are now being paired with a new generation of noise-reduction and sensor read-out electronics, and the "problems" that Canon has had with its 17.8 megapixel sensor in its 7D and 60D bodies has been circumvented by Sony with its new slightly larger in area, 16.2 MP sensor. Right now, in May of 2011, the SONY-made sensors used in the Nikon D5100, D7000, and Pentax K5 are of an entirely new and more-advanced generation than those Canon is currently putting into its mid-range cameras...the ACTUAL, real-world results mirror DxOMark scoring. After the testing dPreview conducted they wrote this about the Pentax K5: "One of the best dslrs in terms of High ISO image quality." and "Exceptionally low shadow noise in RAW files at low ISOs." The little Nikon D5100 has image quality that matches the D7000; these three cameras have the best sensor,noise reduction, and read-out electronics in the consumer segment.

The one thing that many people are unaware of right now, is that the newest Sony sensors have managed to achieve extremely low noise levels, so much that GROSS under-exposures, like the exposure for ISO 51,200 can be used with the camera set to ISO 200, and the image successfully "rescued" using some of the more-advanced RAW converters on the market. I have seen some demo frames on the web, exposed with ISO values of over 50,000 (fifty thousand ISO!!) but with the camera actually being set to ISO 200--and the image these new Sony sensors makes is so low in noise, that EXTREME Positive Exposure Compensation can be applied to the RAW file data and an actually decent, usable image can be pulled out. Combine this ability of the new sensors Sony, Nikon, and Pentax are using, and then add in the new read-out electronics hardware, and the image processing SOFTWARE that the camera makers build, and we've come to an entirely new frontier in sensor development, as well as in camera-building. Because DxOMark examines RAW image quality, there is a huge advantage for the newer Sony-sensored cameras, like the Nikon D3x versus the Sony A900 and A 850. All three cameras use the SAME sensel, the same light-sensitive portion of the sensor, but the Nikon D3x has vastly,vastly superior read-out electronics and vastly better image processing capabilities than the Sony A900 and A850. Same thing is happening now with the very-newest cameras, the Nikon D5100, D7000,and Pentax K5---the same newer, better, lower-noise-in-RAW sensor technology that Sony developed in its 24.6 megapixel full frame sensor two years ago, has now been migrated to the new 1.5x 16.2 sensor...meanwhile the "other" camera makers are a generation behind, and their cameras are getting clobbered when measured against newer technology cameras.

SONY has advanced sensor technology to a new level. Canon has not yet caught up


----------



## subscuck (May 12, 2011)

Garbz said:


> subscuck said:
> 
> 
> > I've clicked on links people have provided to DXO Mark showing consumer and prosumer lenses with better numbers than L or Gold Ring glass.
> ...


 
I'm with you 100%. I love my 85 1.8 and see no reason to ever upgrade. It's IQ is outstanding. My point of contention is on DXOMark's lens rankings. For instance, the Canon 50 1.8, on the 1DsIII, at f2.8, outscores both the Canon 1.4 and 1.2L. Really?


----------



## Derrel (May 12, 2011)

enzodm said:


> While I'm glad to have an independent and quantitative evaluation of sensors and lenses like DxOMark, I'm somewhat puzzled with some of their data. As an example, please look here: DxOMark - Compare sensors
> (comparison among 1000D, 50D, 60D).
> Sooner or later I wish to upgrade from my 1000D (XS) to 60D or even 50D to save some money. Put apart the better body, 50D seems only slightly better than 1000D from the sensor point of view and worst if looking at ISO performance (perhaps related to different pixel density).
> If I look at details, it seems that the 1000D is better than both 50D and 60D at 100ISO, and then more or less the same. If I look at noise, it seems like they are more or less the same, unless the fact 50D and 60D allow for greater ISO, but with quality much lower than 1000D 1600ISO. I'm not sure how these differences translate in real picture quality - it seems to me to have seen better low light pictures coming from 50D than what I would expect from those data.
> ...



Keep this in mind" the 50D launched in the fall of 2008, and its sensor and electronics were probably developed 18 months earlier. The 50D is now a camera based on, basically, 5 year old technology and capability. The 50D was a very controversial camera among Canon users, and caused a HUGE disagreement among the Canon community about cramming too many pixels onto too small of a sensor area; its High ISO score of 696 is bettered by the EOS 60D which scored 813, and which was launched a full two YEARS later, with a sensor that has 17 MP. The Rebel XS is also a mid-2008 camera launch. So, you're comparing two 2008 model Canons with a 2010 model. In terms of Dynamic Range and Color Depth, the two newer Canon cameras are almost neck and neck. Why? Because they are still made using "old" sensor technology.

If we eliminate the 10 MP Rebel, and instead drop in the new Pentax K5, we see HUGE improvements in overall sensor score, color depth, Dynamic Range, and Low-Light ISO performance...from a camera that premiered September 20th, 2010, just three weeks before the Canon 60D. As evidence in support of my claims in Post #12 above, I include this three-camera comparison of the Pentax K5, Canon 50D,and Canon 60D. DxOMark - Compare sensors


----------



## enzodm (May 13, 2011)

KmH said:


> Maybe!
> 
> You have to consider the other camera features and functions too.
> Things like the fact the 1000D doesn't have Spot metering mode, while the 50D does.
> How about the frame burst rate? How about the auto focus system? how do those compare.


 
*Keith*: in fact I would upgrade for having some of those bells&twistles, including better viewfinder, but possibly having more sensitivity when needed. I have the possibility to buy a used 50D for about half the price of a new 60D... which would be my natural choice. I could resist without upgrading for some time, and wait for cheaper prices of 60D.

*Derrel*: I know your thoughts about Canon (including your blog post on Canon and Nikon) but I do not think here is matter of religion. As I told, I'm naturally inclined in trusting quantitative evaluations more than samples, but within reason.
Remaining in Canon will cost me less than changing (by the way, here K5 is 200&#8364; more than 60D). However, in case of change I would have no problems in choosing Pentax - one reason I did not start with it (Km) is that was difficult to find one place selling it. When I bought my camera, I had chances to try d3000, Alpha 230/330 and a couple of Olympus. At that time, restricted to entry level, my choice was not bad even considering DXOMark data (although I discovered it later).


----------



## usayit (May 13, 2011)

analog.universe said:


> If you want to compare image quality from 2 different cameras, look at the actual images they've taken, not some mathematical deconstruction that attempts to explain them.



Agreed..  I found this resource very helpful:

Imaging Resource "Comparometer"  Digital Camera Image Comparison Page

They have a good list of samples.... in the same lighting /studio for easier comparison with your own eyes.


----------



## Derrel (May 13, 2011)

Damn all that science at DxOMark!!! It's so scientific, so consistently applied, and so impartially evaluated! The entire web site at DxOMark must be a conspiracy to punish one camera brand at the expense of others. My camera of choice fares poorly when tested....boo hoo hoo....same old chit...


----------



## analog.universe (May 13, 2011)

I think there can be such a thing as "too much science" when it comes to creative equipment.  Assuming the experiments are sound, and the data collected is accurate, it still only describes very specific functions of the camera in very fundamental terms.  Of course there's value in this data, but I don't think it should be too large a portion of the decision making process.  The camera with the most accurate sensor will not necessarily be the one that allows every photographer to get the best pictures.  It will allow some photographers to get the best pictures, either because they interface well with the gear, or because their relationship with the gear isn't important to them.  I think for most people however, interfacing well with the gear, and being able to use it intuitively... even _preferring_ subjectively the results of a less accurate capture, is what gets them the best pictures.  An audience looking at your photos is not going to say "I wish that sensor could've captured 12 stops instead of 11.3, ruined the shot", they say "Sweet landscape".  But only if the photographer can forget about their camera and actually worry about the landscape!


----------



## Derrel (May 13, 2011)

Yeah, too much accurate information about equipment performance/capabilities/characteristics is a bad thing. It's better not to know how your equipment really measures up. Keeps cognitive dissonance at a minimum to not know how badly your gear is, compared with what's also available on the market.


----------



## analog.universe (May 13, 2011)

Hey... I said there's value in the information... it is worth knowing.  But it is not a measure of whether or not a camera is the right one for you...   it is one factor among many to consider, and in my experience, not the most important one.  Of course you should research technical information, but not at the expense of hands-on experimenting and viewing actual images produced.


----------



## enzodm (May 13, 2011)

So, it is not only in the Beginner forum that it is difficult to keep threads on track... 

I think we are all convinced that, in taking one picture, the photographer has the most impact on the result. And also that a camera is made of many components and features, with different weights in personal choices.
Said that, DXOMark helps me in understanding about some of them -those more quantifiable. I could also check some example to understand the impact of those differences on real images. By the way, comparing K5 with 60D on a couple of the mentioned resources somewhat confirms DXOMark (at least on high ISO). 
What is missing is the impact these measures may have on my own pictures - since I'm a bad photographer, I suppose I could not gain much from 1ev more, although better ISO corresponds to something I'm able to exploit, possibly better low light pictures. But if 50D is not much better than my current camera, I'll wait. I'll go with 60D in the future (or 70D with a Sony sensor  ). 

Derrel: are you still shooting 5D?


----------



## Garbz (May 13, 2011)

subscuck said:


> My point of contention is on DXOMark's lens rankings. For instance, the Canon 50 1.8, on the 1DsIII, at f2.8, outscores both the Canon 1.4 and 1.2L. Really?


 
Well do they in practice too? e.g. The 85mm f/1.4 is an orgasmic lens in terms of the image quality it produces, but it's sharpness is actually quite poor in the measurements. What people perceive as image quality is not necessarily reproducible on an MTF chart.


----------

