Friday 27 June 2014

24-Bit vs. 16-Bit Audio Test - Part II: RESULTS & CONCLUSIONS


See Part I: PROCEDURE for details around the test samples used and how this study was conducted.

In this installment, let's have a look at the results from the 24-bit vs. 16-bit listening test among respondents.

First I need to remind everyone that the test procedure was not easy. As demonstrated in Part I, the sonic difference between the original 24-bit track and the 16-bit dithered version is down below -90dB. This makes the test much more difficult than the previous high bit-rate MP3 test from last year... Whether you were able to detect the 24-bit version or not, I applaud your efforts and input.

As I noted previously, there were 140 total respondents and looking at the transfer statistics from my FTP server, I know the test was downloaded at least 350 times. Response rate just based on my FTP server transfer was therefore about 40% of all who downloaded. The actual response rate would likely be significantly lower since there were other download sites.

Results

I. Demographics:


First let us consider the characteristics of the respondents taking this blind test. Being that this is an internet test, involves downloading 200MB worth of high-resolution audio data in FLAC, and given the target audiophile forums where the test was advertised, it is reasonable to conclude that many if not most are tech savvy audiophiles rather than the "average" music listener.

Not surprisingly, the vast majority (98%) were men which is expected (just have a look around audio clubs, audio shows, etc.) - thanks to the 2 ladies that responded!:
The age distribution likewise isn't a surprise. Audiophiles tend to be a bit older overall, and the average age if we estimate using the median age in each range comes out to about 44 years old. The distribution looks like this:
Nice to see some teenagers and early 20 year olds with the majority in the 41-50 age category. If one were a computer audio manufacturer, the 40-50 age group would be the one to target for maximal effect in 2014.

The survey also asked if some of the respondents belonged to specific categories such as musicians and those with audio engineering experience. This could be useful  in the sub-analysis to see if there were more "golden ears" in these groups:
 
By self report, there were >20% musicians and audio "engineers". Of course these 2 groups were not exclusive and 17/31 musicians also identified themselves as doing audio recording/mixing/editing.

As for the hardware utilized by the respondents, here is the general layout of the type of gear being used to evaluate:
In terms of operating systems, of the 3 main OSs - Windows, Mac, Linux - it's clear that Windows predominated. 129 respondents used one of these 3 OS's and Windows was 60% of that followed by Mac at 23% and Linux 17%. Among streaming devices the Squeezebox was tops. Most respondents used an external USB/Firewire DAC to conduct the evaluation; not surprising that in the computer audio world, SPDIF interfaces are no longer as common and a few used the HDMI interface (surround receiver devices).

There was an even split in respondents using speakers (bookshelf + tower) of 74 and headphones 72 (a few used both).

Here's how the audio system "cost structure" looked (US$):
Weighted average using the median price in each category yields a system price of around $8160 on account of the number of expensive 5-figure systems reported (22% had systems >$10,000). The median audio system price is in the $1000-$3000 range. This is very reasonable and again speaks to the demographic who would download and try a test like this. Objective >16-bit resolution is easily achieved in a $1000-3000 system as demonstrated with even relatively inexpensive DACs measured here over the last year and by having a look at the Stereophile objective results.

Many respondents went into detail describing their systems in the survey.  The first 25 responses included full Meridian active speakers, Sennheiser HD800 headphones with upgraded cabling, custom amplifiers, tube amplifiers, custom ESS9023 DAC, NAD amp, Lyngdorf TDAI 2170 digital amps into Intonation Terzian speakers, Overdrive SE USB DAC, Parasound Halo JC-1 monoblocks, custom ribbon speakers, Cambridge Azure 840E, Focal 1028BE speakers, Sonus Faber Cremona Auditor M speakers, Sony MDR-7509HD headphones, Grado SR325 headphones, Audiolab M-DAC, Chord Hugo DAC, AKG Q701 headphones, Squeezebox Transporter, PS Audio 4.6 preamp, Pass Aleph 5 amplifier, Devialet 170 integrated DAC/amp, Martin Logan Montis speakers, Geek Out 720. Clearly, many respondents used very high quality equipment for this test.

As a reflection of the technological savvy of the respondents, many utilized ABX testing such as the Foobar ABX tool:
20% utilized listening tools to evaluate (ignore that 3rd bar above since it's just a reflection of how many left a description, 29/140 used an ABX tool). Other than Foobar ABX, Mac ABXTester was common, and others described their own script.

II. Were the 24-bit audio files distinguishable from the same files dithered down to 16-bits (and fed into the DAC in the 24-bit container) by the respondents as a whole?

In total, the final result looked like this:




As you can see, in aggregate there is no evidence to show that the 140 respondents were able to identify the 24-bit sample. In fact it was an exact 50/50 for the Vivaldi and Goldberg! As for the Bozza sample, more respondents actually thought the dithered 16-bit version was the "better" sounding 24-bit file (statistically non-significant however, p-value 0.28).

Looking at the individual responses, there were a total of 20 respondents who correctly identified the B-A-A selection of 24-bit samples, and 21 selected the opposite A-B-B. This too is in line with expectations that 17.5 would pick each of these patterns based on chance alone.

III. How certain were the respondents that they answered correctly (ie. able to identify the 24-bit sample)?

24-32% of respondents felt they were unable to hear a difference (1 star = "Guessing"). If we consider that those who chose "2 Stars = more than a guess" also represent a very low level of certainty, then we can see that 45-52% of respondents really had quite low confidence that they were able to tell the difference.

Fewer respondents were "certain" about the solo piano piece (Goldberg), and in general more seemed confident about the Bozza piece. This could be listening fatigue if one were to progress through Bozza-Vivaldi-Goldberg in sequence to account for this result.

IV. Were the respondents who felt more certain about their answer more likely able to identify the 24-bit audio?

Let us have a look at the results reported by those who rated their confidence level as 4 or 5 ("very confident" to "certain" - 25-30% of all the responses):

"Correct" responses being the ones who were successful in identifying the 24-bit sample. As can be seen, there is no evidence to suggest that even in those respondents with a strong sense of confidence were able to identify the 24-bit sample (as sounding better). In fact, for the Goldberg sample, only 44% of those who were quite "certain" selected the 24-bit version correctly.

V. Were the subgroups (musicians, sound engineers, hardware reviewers) able to identify the 24-bit audio better?

Due to the fact that respondents admitting to "guessing" tended to answer with A-A-A and this would severely impact a small sample size, I decided to not count the "guesses" in these smaller subgroups and see if there was any pattern of higher accuracy compared to all respondents.

Musicians:


As a subgroup (total of 31 respondents), the self identified respondents with a "good amount" of musical background did not do well. In fact, this group of respondents consistently scored worse than the combined result. Curiously, the musician group seemed to select the 16-bit dithered Vivaldi as the "better" sounding version (p-value 0.047).

Sound "Engineers" (those with experience recording, mixing, editing):


As a group the "engineers" faired better than the musicians in terms of accurately identifying the 24-bit tracks. This subgroup surpassed the accuracy of the combined respondents marginally. Again, the number of individuals was small (34). There was an overlap between the "musician" and "engineer" group with 17 individuals identifying themselves as both.

Hardware Reviewers:

This was an optional survey item that could be interesting to look at since audiophiles who provide hardware review opinions can have significant influence on sentiment and purchasing decisions.


With only 8 respondents, it would be difficult to draw any firm conclusion other than there is no evidence to suggest this subgroup was any more able to identify the 24-bit from dithered 16-bit audio.

VI. Were those with more expensive hardware able to identify the 24-bit audio better?

In total, there were 44 (31.4%) respondents using $6000+ equipment to perform this test, let us see if they were more accurate than the group average in identifying the 24-bit sample:


As you can see, the ~30% of respondents utilizing equipment costing >$6000 were not able to accurately identify the 24-bit audio track any better than the group average. The Vivaldi track was exactly at 50% accuracy.

VII. Did Headphone Use Improve Accuracy?

72 respondents used headphones in their evaluation. Since headphones can be potentially more accurate (no room acoustics, better noise isolation) at a lower overall cost, it would be interesting to see if accuracy in determining which was the 24-bit sample was any better.

As you can see, headphone use did not result in any appreciable improvement.

VIII. Did age have any effect on the accuracy?

There were 44 respondents 51+ in age. As a group, this is how they did compared to the overall result:


No evidence again of any significant change in accuracy in identifying the 24-bit audio.

Conclusions:


In a naturalistic survey of 140 respondents using high quality musical samples sourced from high-resolution 24/96 digital audio collected over 2 months, there was no evidence that 24-bit audio could be appreciably differentiated from the same music dithered down to 16-bits using a basic algorithm (Adobe Audition 3, flat triangular dither, 0.5 bits).

This survey was targeted to audiophile enthusiasts who in general reported using equipment beyond typical consumer electronics. The majority (77%) were using audio systems reported in excess of US$1,000 and 22% were listening with systems in excess of $10,000. Furthermore, 20% used an ABX utility in the evaluation process suggesting good effort in trying to discern sonic differences. There were no surprises in terms of demographics with the vast majority being males, with an age distribution centred around 41-50 years old.

Subgroup analysis of "musicians" and those who work with the technical aspects of recording, editing and mixing ("engineers") did not demonstrate evidence of special abilities at discerning the 24-bit audio. The "engineers" group did perform slightly better overall. The small group of individuals who identified themselves as writing hardware reviews did not show an increase in accuracy.

About 50% of respondents admitted that they had low confidence in their ability to discern differences. Conversely, 25-30% (depending on which musical sample) of respondents reported a strong sense of "certainty" that they were correct in identifying the 24-bit sample. Nonetheless, analysis was not able to demonstrate improved accuracy despite claims of increased subjective confidence by the respondents.

Furthermore, analysis of those utilizing more expensive audio systems ($6,000+) did not show any evidence of the respondents being able to identify the 24-bit audio. Those using headphones likewise did not show any stronger preference for the higher bit-depth sample. No difference was noted in the "older" (51+ years) age group data (not surprising if there is no discernible difference even with potential age-related hearing acuity changes).

Limitations of the study includes the fact that this was an open test distributed via the Internet in an uncontrolled fashion. This allowed the opportunity for test subjects to analyze the audio files objectively rather than through pure listening. However, this is also the mechanism of delivery for high-resolution downloads and the test participants would likely be using the same equipment to listen. The benefit of course is that the results may reflect realistic feedback from potential consumers (if not the target audience) of high-resolution audio. Respondents were able to listen in their own home using their own equipment rather than an artificially controlled environment. The fact that there was no time limit (other than a 2 month window to gather survey submissions) should have been a less stressful experience for the testers.

140 participants is not a particularly large number of data points but it was adequate to demonstrate an even 50/50 split in preference across the 3 musical samples; a level of consistency which adds to the idea that listeners were unable to differentiate 24-bit audio from the dithered 16-bit counterpart. Replication of the results is of course advised.

As expressed previously in "High-Resolution Expectations" (See "Good Enough Room?" section), there is no good rationale for a dynamic range of greater than 16-bit digital audio in the home environment. The results of this survey appear to support the notion that high bit-depth music (24-bits) does not provide audible benefits despite the fact that objectively measurable DACs capable of >16-bit resolution are readily available at very reasonable cost these days.

If 24-bit audio imparts no audible benefit when listening to music compared to the same data dithered down to 16-bits, how certain can the audiophile consumer be that higher sampling rates (eg. 88/96/176/192kHz) would make much of any audible difference? This perhaps should be the target for another blind test. Methodologically, it would be extremely difficult to maintain the blind testing condition over the internet since it would be trivial to run the audio files through a spectrum analyser with no easy mechanism to conceal the bandwidth limitation of lower sampling rates (eg. 22kHz frequency headroom for 44kHz sampling). The reader is encouraged therefore to explore the effect of higher sample rates for him/herself.

One final comment in closing. Notice that the Goldberg track was soft and had a peak amplitude of -10.35dB as demonstrated by the DR Meter (see PROCEDURE post). This means that the full potential dynamic range was not being utilize and for the 16-bit dithered sample, the dynamic range can be encapsulated in <15-bits. Even with this limitation, there was no evidence that respondents were significantly able to identify a difference in aggregate or within subgroups.
 

-------------

As usual, I encourage others to do their own testing. Feel free to drop a link especially if there are other controlled, preferably blind tests showing a significant audible difference between 24-bit and 16-bit audio.

I will put up a Part III over the next week as well documenting the subjective comments made by respondents and final observations... Stay tuned.


54 comments:

  1. Great stuff. It would be interesting if you did a follow-up survey of these participants to see how many, if any, have changed their opinions concerning what matters concerning bit depth and audibility. For example, choices like "I never thought bit depth made a difference and this confirms it for me", "I think bit depth makes a significant audible difference regardless of this or any other test", etc

    ReplyDelete
    Replies
    1. Hi Dfs. No, I did not do any follow-up. However, I will have a Part III where I'll post the subjective impressions from the respondents and whether the respondents thought it was harder / easier. I think some of that could be quite insightful...

      Delete
  2. Very interesting results, and many thanks for your efforts. Clearly, my gear is fine, but I am going to have to get my ears upgraded!
    I appreciate your efforts to create a truly blind test, but, speaking for myself, I wasn't interested in "cheating" so much as what experience passed from source file to brain and finding out whether I personally could discern a difference. I am definitely going to reconsider my quest for 24/96 recordings, but I shall also continue my avoidance of MP3 compression
    I shall continue to watch this space with great interest...

    ReplyDelete
  3. Playing the role of 'subjectivists advocate' the conclusion: 'there was no evidence that 24-bit audio could be appreciably differentiated from the same music dithered down to 16-bits using a basic algorithm' may not be shared by the 15% who gave the correct answers.

    I am convinced that blind tests with statistically enough samples will most likely yield a (close to) 50% score with the general public it may still be possible that at least some of the 20 people (of the 140 participants) have 'Californium-252 ears' and MAY indeed have heard differences where the rest (me included) won't have.

    We will never know IF the people that gave the correct answers truly HEARD the differences or just guessed or analysed the files correctly.
    This could only be done in a controlled environment with those who answered correctly with a statistic more relevant blind test.

    What may be interesting could be the demographic background of those that gave the correct answers. how confident were they, what equipment and experience did they have etc.

    I fear the hardcore subjectivists will still be crying they can 'CLEARLY hear differences between 16 and 24 bits' and will shout that the 15% who heard correctly were dismissed.

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
    2. Thanks Solderdude. I'll have a little more to say about this is Part III. I'll show some comments like the ones who felt this test was "easy" and correlate the comments with some scores.

      Good idea with the 20; I'll see about putting something together...

      Going off camping with the wife and kids for the weekend :-)

      Delete
    3. I was sure of my selections and they were all true. I am surprised you cannot hear the difference. I'm not even good with music or anything and have $30 speakers but the 24 bit versions were just clearer, richer, almost louder. the 16 bit ones were flatter, maybe tinnier. I first heard this difference when listening to compressed vs uncompressed output from my camcorder, so maybe I knew what I was listening for.

      Delete
    4. Sorry, Solderdude Frans, but having 20 people in the sample that could reliably identify the 24 bit signal would have altered the results significantly. That would leave 120 who chose the 24 bit by chance, but the 20 that could always identify the deeper bit depth. So you would end up with 80/140 that identified the 24 bit samples, or 57%. That would be a statistically significant result at 95% confidence. So the data here does not support you hypothesis.

      Delete
  4. This comment has been removed by the author.

    ReplyDelete
  5. As one of the 20 who answered all three questions correctly I can state the following: Only on the piano piece was I reasonably certain of my answer but I wasn't simply guessing on the others - I was just less sure of myself. Whilst I don't generally go out of my way to buy hi-res albums (with a few notable exceptions), for me this test has confirmed that the differences ARE there to be heard, but they are perhaps not worth worrying about most of the time. I won't generally buy hi-res unless I have reason to believe the album has been very well recorded in the first place. Hi-res on anything but a top-notch production seems like a waste of time and money to me. I'd rather the recording engineers spent more time on other aspects of production that have a far more profound effect on sound, such as dynamic range, clipping (especially with loud vocals) and overall tonal balance. That said, all other things being equal (including price) I'd still prefer hi-res if the recording justifies it.

    For anyone interested, my system is nothing special:

    Logitech Squeezebox Touch > Arcam rDAC > Arcam AVR280 > Usher V602.

    My credentials are simply being a music lover, hi-fi enthusiast and 'Philips Golden Ears' Gold level achiever. Oh, and having an OCD level of attention to detail, which is a curse as much as a blessing!

    ReplyDelete
    Replies
    1. Appreciate the comment from "one of the 20", Stephen.

      As with all tests looking at group effects, I certainly do not discount some people having very excellent audio acuity! The point of course is that statistically within the 140 respondents, it seems to be uncommon enough that I believe most audiophiles / audio enthusiasts should accept that it doesn't likely make a difference even if the audiophile you're chatting with seems to have strong confidence that he/she can perceive a difference! Testimony of strong confidence in one's own perceptions of sonic difference this minute does not appear to predict accuracy. This point should be even more obvious given the quality of the test tracks used which I believe is of much better pedigree than probably 90+% of what is being sold as "high-resolution" these days!

      Personally, if I were to participate in (my own) survey, I would only score 2/3 and would not grade my confidence as high. This is enough for me personally to not worry much about hi-res music also assuming it's the same mastering.

      Well said about the importance of better audio engineering; never mind the tendency for us music consumers to obsess about whether one digital "format" (DSD vs. PCM, 44 vs. 96 vs. 192) is "better".

      I appreciate your openness to also describe the system you used both here and in the survey itself.

      Delete
    2. This comment has been removed by the author.

      Delete
    3. You touch upon another issue here - that of variations in mastering between CD and hi-res versions. One example is the (stunning) debut album by London Grammar. On my deluxe CD the tracks segue into one another, albeit slightly. On the hi-res download there's a distinct silence between each song. I imagine for technical reasons this will apply to all downloadable formats of the album. Anyway, I much prefer the latter approach and this alone makes me favour the high resolution version regardless of any sonic differences. Mind you, I'm far more bothered by the overuse of compression and occasionally distorted vocals (in louder sections) that can be heard on ALL versions of the album. They detract somewhat from an otherwise good production.

      You know, there are times when I'm thankful for my cheap car stereo which forces me to simply listen to the music and forget about all this technical stuff! Much as I love hi-fi and technology sometimes it can get between you and the music rather than bringing you closer to it.

      Delete
    4. Apologies for the double post. The first one produced an error on submission and was presumed lost!

      Delete
    5. probability theory suggests that if 16bit and 24bit tracks cannot be distinguished by any human then there is 50:50 chance of getting any track correct.

      So there will still be about 17 subjects in this test that get 3/3 by chance alone.

      Anyone that thinks they have golden ears need to take a cold shower and sober up.

      The study design could be strengthened by adding 8bit and 6bit tracks so hopefully some non-random results are obtained.

      Delete
    6. Yes, of course the study could be strengthened.

      Remember guys, this is an open test on the Internet for anyone with the time and effort to participate. Already I was concerned that I had 3 test samples for people to listen to knowing (at least speaking personally!) that this was going to be difficult!

      Delete
    7. Keep in mind that with 3 tracks, you have a one in 8 chance of guessing all three correctly. With 140 total respondents, 1 in 8 comes to 17.5, of which 20 is within standard expected error bounds.

      Not discernible difference, but simply statistics! This is entirely the expected result if we assume that the results are entirely chance. If you were one of the people who got all the answers right, you need to understand that the chance was actually quite high, and based on the results it has nothing to do with your ears, no matter how much you want to believe it does. One data point does not a sample make.

      I see Nada came to the same conclusion, but it's worth saying again :)

      An anecdote that's fun and gets the point across:

      'As to the influence and genius of great generals — there is a story that Enrico Fermi once asked Gen. Leslie Groves how many generals might be called “great.” Groves said about three out of every 100. Fermi asked how a general qualified for the adjective, and Groves replied that any general who had won five major battles in a row might safely be called great. This was in the middle of World Wat II. Well, then, said Fermi, considering that the opposing forces in most theaters of operation are roughly equal, the odds are one of two that a general will win a battle, one of four that he will win two battles in a row, one of eight for three, one of sixteen for four, one of thirty-two for five. “So you are right, general, about three out of every 100. Mathematical probability, not genius.”'

      Delete
    8. You give me 100 other pieces, and I will guess it correctly everytime. It is that clear. Why are you denying what people are hearing?

      Delete
  6. It's worth noting that in the Philips Golden Ear challenge roughly 13% of participants reach the top level. It would suggest that subtle (but perceptible) differences in sound can only be appreciated by a small percentage of people. Taking that into account, it perhaps makes the argument for hi-res even less credible. However, even after a cold shower and strong coffee my ears still tell me piano piece A sounds superior to piano piece B. :)

    ReplyDelete
  7. I passed the Golden ears challenge reasonably quick and didn't feel I really needed 'golden ears' to do so.
    It is a fun, enlightening and entertaining test non the less.
    Even though my hearing is officially 'golden ears certified' I can't hear a difference between 16 and 24 bit but it may take some education for me to learn WHAT to listen for. It may also be possible I will never hear it.

    I am not dismissing some people can actually hear things I can't but as mentioned before the amount of test files is too small.
    IMO one needs for instance 10 unknown files which consists of a few (random numbered) 16 bit files and the rest 24 bit files.
    To ensure file sizes don't give the 2 files away at first glance it would be best to use WAV.
    The 10 test files can be packed in an uncompressed ZIP as a single folder.

    Yes, it will be easy to cheat and the test would ONLY be to satisfy your own curiosity/ease of mind and thus cannot be used as 'proof' unless it is taken under controlled conditions.

    I always hate to see these discussions end in arguments back and forth without any progress made in researching this, especially on the more subjective based forums, where it always ends in harsh words and allegations.

    ReplyDelete
  8. Hi Archimago.

    Great job!

    Glad to see the results. Just as I expected.

    Just for the record, I also achieved Golden Ears "title" but I don't see that as some special achievement, and I am sure that 13% of Goldies is not a realistic number. Pretty sure that many contestants give up the test due to inadequate equipment, fatigue, or simply loss of interest.

    So I have "40years old Golden Ears", +15k$ DIY system in acoustically treated room, and even so I was not sure about my answers. As it turns out I got right the first two samples, and maybe guessed the third. I don't even know which answer did I choose for the solo piano piece so I don't know if I am one of the "blessed 20" (for first two answers I tagged the files). Archimago help - parallel 211SET from CRO ;-)

    First two answers were just a bit more then a guess, seemed like B&A samples had better dynamics and nothing else (dynamics is what I was looking for, so autosuggestion is very possible), and for the Goldberg piece both samples sounded exactly the same in all aspects.

    Anyway, in Golden Ears challenge the differences were much greater, few dB and very audible. In this case I am not sure the difference is audible at all. Unfortunately did not have enough time to repeat the test many times (holidays + system refresh :), and did not want to do the test using my headphones, so I will ask my wife to retag the files, and take the test again.

    I'll try to convince my "subjectivist blogger" friend to do the test together at his place on a very respectable setup. If he fails the test, It Is NOT Audible :-) But since he is a "subjectivist true believer" I clearly doubt that he will accept the challenge...as usual. Anyway friendship is much more important than audibility of HiRes formats :-)

    ReplyDelete
    Replies
    1. Hey Marin, I believe I see ya...

      Hard luck dude, sorry to say you were not "blessed" :-(. Time to turn in that "Golden Ear" certificate and pray harder! :-)

      I see B-A-B. First 2 you said "more than a guess", Goldberg you said "guess". Better luck next time!

      Glad to see all the Croatian input!

      On a serious note, you of course did better than the majority of respondents. And of course this is *much harder* than the Philips Golden Ear Challenge...

      Delete
    2. Yep! that's me.

      Blessing or curse, that is the question :-)

      You said it right: "hard LUCK".I think that luck had a lot to do with it.

      It would be nice if some of the "golden 20" especially "five stars", writes a few words (Stephen Ward thnx for your comment) about the procedure of listening and maybe point out few details about the differences between samples.

      Maybe I have "Golden Ears" but don't know how to use them.


      PS
      Bravo for the "sampling rates" blind test idea. Keep your faith in people ;-)

      Delete
  9. A big thanks to Archimago, the participants and those who submitted thoughtful comments. My limited experience with hi-res has not been very rewarding. I prefer a well-recorded CD to vast majority ordinary of hi-res recordings. A couple of Linn SACDs have been happy exceptions.

    My system includes Musical Fidelity V-Link (USB to SPDIF conversion), Sony (optical disk source), Marantz receiver (dac and amplification) and PSB and Dynaudio speakers.

    ReplyDelete
  10. I have a CD album collection numbering well over a thousand. In my entire music collection I have just three 24 bit recordings and only one of them is appreciably different to its CD counterpart (Kate Bush - 50 Words For Snow). Even after scoring three out of three in this test, I'm still not likely to start snapping up hi-res versions of new releases because I don't consider the difference in sound (assuming the mastering is the same) to justify the extra cost and disk space. I believe the overall test results are valid and significant. I'm just not sure I would conclude that no human can detect a difference between 16 bit and 24 bit under any circumstances.

    Going back to that Kate Bush album, the 24 bit version is sonically better than the CD and if you check out the DR levels it's easy to see that it's markedly different. Now the question is, is the hi-res version better because those extra bits allowed for a superior mastering or did they just do a poor job of the CD by comparison? Either way it's a very clear example of an album where the 24 bit version is superior and I'm very glad it's available to buy. If the 'premium quality' perception and expectation of 24 bit gives recording engineers a reason to make more of an effort then that alone justifies the existence of hi-res formats for me.

    With each new generation of computer graphics cards we usually have to wait a while before games exist that take advantage of the extra performance. Is 24 bit similar in that respect? In a few years time will sound engineers do a better job of getting the most from the format? Hey, perhaps in a few thousand years ALL humans will have evolved the hearing to appreciate their efforts! That's meant to be tongue in cheek by the way. I have very large ears but they're most certainly not golden! :)

    ReplyDelete
    Replies
    1. Interesting comment on the Kate Bush album. If you downsample that album to 16/44, the DR will *not* change significantly. The 24-bit version must therefore be a better mastering job than what they did on CD.

      I came across a few of these examples in my "upsampled" SACD post awhile back. They used a better, more dynamic version on the "high-res" version. Also, multichannel DVD-A and SACDs showed this tendency for the multichannel version to retain better dynamic range.

      So here's the thing... I think 24-bit "premium" versions could be worth it if the mastering is better. However, we should be careful about attributing it to the fact that it was in a 24-bit container!

      Delete
    2. I totally agree with your final paragraph. Thanks by the way for taking the time and trouble to create this test. It's been enlightening and even fun! It takes me back to my twenties when I was making A/B comparisons with DIY turntable modifications and component level upgrades. I'm wiser now of course but they are fond memories nonetheless!

      Delete
  11. I didn't submit the test, but I bombed it (1/3). Granting all the facts about how little we SHOULD be able to detect the difference, both from human anatomy and limitations of recordings, I wonder if it matters that most (all?) currently-in-production DACs don't really use multibit output. Our UD-501, for instance, does multibit for the top 6 bits and streams the rest as delta-sigma, if I understand the term correctly. Would a DAC with a bank of 24 itty-bitty output resistors reveal a difference more reliably for PCM?

    ReplyDelete
  12. i better keep the 24byte digital file than 16byte ... if i cannot hear the different is ok .. but from the the spec tech the different is true .....

    ReplyDelete
  13. Not me. I can save two 16-bit albums in the same space than one 24-bit does. Lucky you to have tons of hard disks.

    ReplyDelete
  14. As predicted. And I keep saying it: What makes the biggest audible difference between digital audio formats, and even digital vs analog comparisons, is the person at the controls in the mixing/mastering studios.

    There are plenty of horror stories out there, such as the 24bit Nirvana 'Nevermind' downloads, which were tested to have between 3-4dB less dynamic range than the original 1991 CD release. Had a DIRECT to 24/96 transfer of the original master tapes been done, then there would have been a case for considering them.

    ReplyDelete
  15. would 16 vs 24 make a difference when pressing vinyl?

    ReplyDelete
    Replies
    1. Yes - said the family dog!

      For the rest of us? Not really. Again, choices made in production(dynamic compression, EQ, etc.) matter far more. Besides, the difference between a 16 & 24bit vinyl master would be eaten up in the surface noise.

      Delete
  16. Top! Thank you for this test and all the effort! I like this kind of real knowledge - in opposite to the meanings and opinions and feelings, concerning technology.
    I am a cameraman and would love to see this test translated to video resolutions (480i-720P-1080i-1080P-4K)
    I am convinced the electronic industry will get a shock how stupid all this high-resolutioning-excess is.

    ReplyDelete
    Replies
    1. I've been saying this to the walls: Two things more important than resolution - #1 a good, well-produced source material, and #2 a correctly adjusted consumer display(see calibration).

      Factory settings are terrible for TVs! :)

      Delete
  17. Nearly 10 years ago, I was involved in the development of a high-end, HD audio capable direct digital amplifier (Zetex DDFA); of course we had a SACD to do comparison against a CD with the same material - but we weren't able to hear a difference. The expensive measurement equipment said that our chip did meet our target, i.e. it could play HD audio. Of course, it also did the best it could with CD Audio, i.e. fully reconstruct the waveform and play it through the HD-Audio pipe.

    The CD spec really is more than good enough, with proper noise-shaping dither, you can even achieve 20 bits of resolution for the most sensitive region between 2 and 4kHz (at the expense of less resolution at higher frequencies above 16kHz, where the the ear of young people is already much less sensitive and many people are completely deaf); this is already beyond the spec of the ear.

    ReplyDelete
  18. Thanks for this very interesting study. One things which everyone can plainly hear is the difference between a recording which is well engineered and one which isn't. This is where the improvements are really needed.

    ReplyDelete
  19. Thanks for sharing, nice post!

    - Máy đưa võng tự động hay may dua vong cho em bé là sản phẩm tinh túy của công nghệ, vong tu dong là món quà tuyệt vời cho những ông bố bà mẹ bận rộn. Nó đang ngày càng trở thành 1 phần của các gia đình Việt nhất là những nhà có trẻ nhỏ hoặc người lớn tuổi.
    - Hiện nay trên thị trường máy đưa võng em bé có nhiều sản phẩm với nhiều kiểu dáng, màu sắc và giá máy đưa võng khác nhau làm cho người tiêu dùng phải phân vân khi lựa chọn sản phẩm. Bất cứ mặt hàng nào cũng vậy, luôn có những “con sâu làm rầu nồi canh” với hàng giả, hàng nhái kém chất lượng với may dua vong tu dong gia re hoặc siêu mắc. Bạn cần chọn lựa kĩ càng máy chính hãng để được đảm bảo chất lượng, an toàn!
    - Với nhãn hiệu uy tín, sản phẩm máy đưa võng chất lượng và giá bán phải chăng, máy đưa võng tự động An Thái Sơn tự hào là địa chỉ bán may dua vong tu dong tốt nhất, uy tín nhất cho bé.

    Bạn xem thêm bí quyết và chia sẽ kinh nghiệm làm đẹp:

    Những thực phẩm tốt cho tại http://thucphamtotcho.blogspot.com/
    Những thực phẩm tốt cho da tại http://thucphamtotchoda.blogspot.com/
    Chăm sóc da mặt ở http://chamsocdamato.blogspot.com/
    Cách chăm sóc da mặt bằng http://cachchamsocdamatbang.blogspot.com/
    Dạy trẻ sơ sinh tại http://daytresosinh.blogspot.com/
    Bí quyết giảm cân hay tại http://biquyetgiamcanhay.blogspot.com/

    Chúc các bạn ngày mới vui vẻ!

    ReplyDelete
  20. Thanks for sharing your thoughts with us.. they are really interesting.. I would like to read more from you.
    Sound Testing

    ReplyDelete
  21. Thanks for the great study, another weapon in my arsenal against "24 bit tinfoil hat" musicians.

    But, seriously... "biological gender"? Please...

    ReplyDelete
  22. Good stuff. Although I am surprised it isn't more common knowledge that 24bit and 16bit audio are exactly the same down to about 90dB. It's no wonder no-one can hear a difference as there literally is none until you get down to extremely low levels.

    People should be more inquisitive and less trusting of their own bias and preconceptions. A simple null test in a DAW would answer this question. And it does.

    ReplyDelete
  23. 4 years down the road I found this post and the results were not that surprising. Quoting: "Curiously, the musician group seemed to select the 16-bit dithered Vivaldi as the "better" sounding version (p-value 0.047)." I participated in an ABX test back in 2006. One of the samples was solo piano. One sample had been passed through 20 TL072 op-amps and the other had passed through one. I could determine which was which 82% of the time through 20 plus tests. The funny thing was I preferred the "degraded" one over the "pristine" one. Something about the decay of individual notes sounded better compared to the "better" sample. I believe that the slightly higher noise floor of the "degraded" recording acted as dither, smoothing out the tails of the notes. In the lossy compression vs. uncompressed portion of the test once the bitrate passed 192kbs I could no longer hear the difference in the samples of music. As a live sound engineer I believe one place where bitdepth/samplerate does make a huge difference is when multiple ad/da conversions happen in the chain. It is not uncommon for a live signal to snake its' way through a path like this: Microphone to preamp, converted to digital, mixed in a digital mixer where it is eq'd, compressed, delayed, and combined w/ other channels, converted to analog and sent to a loudspeaker DSP where it is converted back in to digital and then eq'd, frequency divided and compressed/limited then converted back to analog and sent to the amps. If that chain is 24/96 then losses are minor. That trip still sounds worse than keeping the signal in the digital domain after it leaves the microphone until the power amp. Think of re-ripping MP3s as a comparison.

    ReplyDelete
    Replies
    1. Hi Bitmap,

      Thanks for the note! Yeah, at the end of the day, I think the rational audiophile realizes that although it's "best" to maintain best practices to prevent as much loss as possible, the reality is that small changes here and there should be nothing to freak out about and in some instances, some folks might even have a preference to what would be the mildly degraded version.

      Fascinating test on the 20x TL072 opamp! Know of any testing with 20x NE5532 opamps which I believe is a "classic", inexpensive and very common in audio devices?

      Delete
  24. The NE5532 is a classic. I have never seen a 20x test but I am sure I have heard plenty of signal chains with 20+ 5532s in them! They may be inexpensive today but in the 1980s they were a couple bucks a pop! 4558s and TL07x series were the inexpensive alternates. The TL072/5532 comparison is really an apples/oranges thing. The TL072 is a hi impedance FET device that is (comparatively) noisy. It has some other quirks that make designing circuits for it challenging but I always considered it a good choice for filter circuits (EQs, x-overs). The 5532 is a low impedance, wide bandwidth and low noise device. If a circuit needed gain or drive then it was the best choice. I actually replaced a blown op-amp (4556) w/ a 5532 last week. Similar specs, the 4556 was never really on my radar. This was in an old Yamaha mixer I picked up on Craigslist for $30. Interesting note is that the mic preamps in this console are discrete class A circuits. The op-amps are only used as unity gain buffers, for the balanced line outputs and, amazingly, the headphone cue amp. The thing sounds amazing and it is built like a tank compared to most of the modern stuff. My son will use it as a front end for recording. He likes the vintage gear feeding his modern 24-bit/96-khz multi track digital recorder. Best of both worlds from his perspective. I tend to agree.

    ReplyDelete
  25. The gadget is mainly used for the purpose of combining, routing, and changing the level, tone or dynamics of audio signals. Best Audio Mixers

    ReplyDelete
  26. This comment has been removed by the author.

    ReplyDelete
  27. This doesn't mean that a dealer can just turn around and sell the item for their cost. Dealers spend significant amounts of time and money to sell an item, and as such mark up the price of an item article

    ReplyDelete
  28. I am against any kind of blind test for at least 2 fundamental reasons.
    the first, it is scientifically proven that the memory of a sound sensation disappears after a few seconds unless we compare 2 sounds with macroscopic differences.
    so all blind tests are a question of skill. but we don't want to establish who is better at it ..
    then we have all the mechanisms that the. our brain puts in place when we listen to how, expectation, excitement, etc. etc. that will overwhelm your judgment.to be sure you hear a difference you must listen several times, without haste, on different days with different pieces of music, when the same judgment is repeated for many times you are reasonably sure that you have heard a small difference.and yet ... even if you've heard a difference, do you have to compare and remember which one is better?
    how many friends have listened with haste and compared have changed an audio device and then go back and resume the previous one.
    to judge it takes calm and time, otherwise you deceive yourself.

    ReplyDelete
  29. Why Blind test is a vrong method for audio,
    A good scientist explanation,
    http://www.anstendig.org/ABTesting.html

    ReplyDelete
  30. Extremely helpful!! thank you very much. Now I can be confident by just buying a 16bit/44.1khz flac rather than spending extra money on Hi-Res music. Personally, I think buying expensive headphones/earphones and AMPs are much more cost-effective than buying 24bit/192khz music. Unlike high quality music, expensive electronics definitely show the difference.

    ReplyDelete
  31. grazie, nice test and nice readings! our ears are such an amazing device

    ReplyDelete
  32. Stephen Dawson here. I've got to congratulate you on the rigour of this test. (I hope I'm not saying that merely because the test confirms my priors.) I'm glad you've been able to gather an audience sufficient to provide you with an N of 140.

    ReplyDelete
  33. I have also 18 bit music, and thuis has a beter dept and forard, left and richt planning.

    ReplyDelete