The notorious 15% tolerance in manufacture and testing!
We are experiencing a boom in testing digital and photographic equipment and in the wake of these activities, we see also a large amount of space and energy devoted to discussing the relative and absolute advantages of film-based and solid-state imagery.
Some time ago I tried to discuss this theme with the Kodak engineers in Rochester. They have in-depth knowledge with both media and should be well placed to offer valuable insights. Their answer was as simple as sobering: we do not conduct comparative tests, as both media are inherently quite different and it is very difficult to make meaningful comments. Their view is quite valid: you choose the medium that suits your style of photography and that gives you the results you wish or your clients demand.
The futile comparison between film-based and solid-state imagery
This is a most sensible standpoint and it should be promoted more often. The person who loves darkroom work, has a knack of working with film and enjoys the look and feel of a well made baryta or RC print, will continue to use the chemical processing techniques, because he sees the virtues and joys of this technique.
We all know that for the best definition of detail and the finest tonal range, we need the large format camera and negatives of large sized sheet film that are contact printed. Still we all, with the exception of a few aficionados, use 35mm film, not because it delivers the best quality, but it delivers what we need and want. And no one will start the discussion what film size is best: this is a boring topic with known conclusions. The choices are made on different arguments, like convenience, speed, cost and suitable image quality.
In the thirties of the past century, 35mm won the contest with roll film and there was a massive swing to the new format and its equipment. Now in the first decade of the 21st century we see a similar tendency: the tiny digital sensor arrays, supported by powerful mathematics and programs, are winning the contest with film in general and with 35mm and roll film in particular. You can see the large format guys smiling and thinking: I told you so!
The practical advantages of solid-state imagery and its large array of options during digital post processing are so evident and useful, that any discussion about the absolute differences in performance is irrelevant. If you follow the discussions and comparisons in magazines, on websites and discussion groups, there is not one new insight or statement that will alter the trend or gives fresh insights or adds something of value to the discussion. In fact with the introduction of the Canon D30 and its offspring, the chapter of the film-based SLR is fast coming to an end, however impressive a Nikon F6, a Canon EOS-1V and others are as electro-opto-mechanical instruments.
Manufacturing tolerances and tolerances in testing
If we wish to embark on absolute and relative testing and comparing of results and equipment, it is wise to reflect on the following issues.
A few weeks ago I was visiting the small company called Image Engineering, located in Cologne, Germany. They have a test lab, loaded with computers and all kinds of test charts and equipment, and provide all test results for some major German magazines, like ColorFoto in the photographic branch and C't in the computer market. Evaluating their test procedures on a regular basis, they have come to the conclusion, that most test charts that are commonly used and that rely after all on visual inspection and comparison have an inherent fault margin of 15%. Observers, comparing the same results differ by 15% in their judgment! And even more problematic, the test charts themselves vary by 15% in bringing identical and reproducible results.
These conclusions are not new: most studies about visual inspection and judgments have concluded that the margin of error is quite big between persons and between results.
Image Engineering also concluded that the autofocus mechanism of most cameras was so unreliable that they had to resort to a manual focus with a statistical analysis of small focus movements to find the optimum sharpness plane. Special testcharts have been designed and constructed (Siemens stars wit a sinusoidal gradient) to be able to evade the sharpening mechanism in the digital cameras that kill any meaningful comparison.
All this sophisticated equipment and statistically controlled computer analysis can not guarantee that the results are relevant for two reasons: the camera is still manually aligned in front of the test board and here we see a remarkable phenomenon: the results for decentring of a lens are impossibly high: they find on average a decentring between 8% and 15%. This is measured as the difference between the performances measured at the edges of the image. But it is almost impossible by visual inspection to align a camera absolutely plane parallel in front of a test plane. This observation is not intended as critique: I think the people at Image Engineering do an excellent job. This story is only told here to give an indication of how easily it is to make errors in testing and how large the effort has to be to control the inevitable tolerances in test equipment and procedures.
The second problem is the manufacturing process: however good a company and is quality control, there has to be some allowance for manufacturing tolerances. A margin of plus/minus 5% is really excellent in consumer electronic products and cameras.
Like it or not, but two identical cameras and lenses can differ by a margin of 10% and sometimes higher. This implies that a judgment that two different products perform equally well may be wrong in a statistically significant number of cases. And the observation that two products differ in their performance may be erroneous too. In most cases we do not note these aspects, because the margin in errors in the test procedures and test methodology is larger (15%) than the differences in manufacture (10%). Often the manufacturing tolerances may be smaller, but the errors in the test methodology are nonetheless always present, even if we are not aware of this.
The upshot
Given the large amount of factors that can influence the results, the margin of tolerance in the test equipment, the margin of tolerance in the manufacture of products, we have to admit that comparative results and single judgments may be off by 15% or even more. Without a clear understanding of the conditions that can influence the result, the conclusions may be at best indicative and often are non-informative. This is an unpleasant conclusion, that limits the value of the numerous comparisons and results that are being published daily in all kinds of media. The best a reader can do is to be aware of a fair margin of error in the conclusions presented and interpret the results with this tolerance band in mind.
This is the reason why I restrict my testing of lenses and films to the aspects that can be measured quite confidently and in lab-situations where the influences can be controlled. I also use several sets of independently acquired data that can be used to find any anomalies and can be used to check the results of every set of data. This is also the reason why I am very reluctant to indulge myself in the art and science of testing digital equipment, where the chain of possible influences is much larger and the visual assessment of results is very prone to errors in judgment, given the wide tolerance band in which one is operating.
