It’s time we blow up PC benchmarking

Welcome to The Full Nerd newsletter—your weekly dose of hardware talk from the enthusiasts at PCWorld. Missed the burning topics on our YouTube show or the latest news from across the web? You’re in the right place.

Want this newsletter to come directly to your inbox? Sign up on our website!

Bigger bar better. My late colleague Gordon Mah Ung coined this catchphrase to summarize the general attitude toward benchmarks. When each successive generation comes out, most people look to see if the numbers went up—and by how much.

You already know the outcome. The bigger the jump in number, the happier the conversations. (Or at least, one “side” in the debate is much happier.) When the numbers appear to only crawl forward, everyone reacts in a more subdued way.

But is this truly natural behavior—or is it learned? I’d say it’s both.

I bet many of you remember when benchmark data felt like a win. In the late 1990s, when Gordon first devoted himself to the benchmarking grind, having such data simplified what felt complex to measure. (Ex: Framerates could tell you what to expect from new hardware when gaming—and helped us calculate how long to stretch the life of our existing setups.) More importantly, the community could replicate the tests and thus verify. If it all checked out, then reviewers could be viewed as trustworthy.

Hardware has increased in complexity, though—as you’d expect over the course of almost 30 years. It jumped particularly dramatically in the last ten, with the focus on chiplet designs, more silicon layers, and additional processors to help with specialized tasks. These advances make the hardware great to use, but difficult to evaluate.

Willis Lai/Foundry

I think we’re overdue for a wide conversation around benchmarks. Balancing consistency, repeatability, and simplicity has been a core tenet of testing. In our chat this week with guest Matt Bach, who oversees benchmarking for respected workstation vendor Puget Systems, these themes span our entire discussion. But I think this balance is fraying at its edges.

It’s time to blow things up. The decades of expectations around simple numbers to encapsulate complex situations is doing us no favors. I think reviewers and consumers alike need to rethink what consistency truly encompasses these days.

Because from where I sit, variability is the key factor in the quality of our experience on PC these days. Your 1 percent lows or microstutters? Their frequency and severity impact the fluidity of your gameplay more so than raw framerates. You could argue for similar impact with how cores and threads boost or how efficiently instructions pass between CPU chiplets. Like with medical research, a harder and more thorough look at variables and their effects would address a wider range of experiences, and in some ways, more accurately identify and address subtle nuances that have big impact.

To be clear, we shouldn’t do away with benchmarks performed with consistent, repeatable factors. But the data that comes from purposefully examining situations where all factors can’t be controlled? Trends and even patterns lurk within that seeming chaos, too.

I have hope that the internet can adapt.

In this episode of The Full Nerd

In this episode of The Full Nerd, Adam Patrick Murray, Alaina Yee, Will Smith, and special guest Matt Bach, Labs Supervisor and PugetBench PM of Puget Systems talk about hardware, benchmarking, and the reliability of modern parts. The most important detail we cover: How to properly pronounce “Puget.” (It’s “pew-jet.”)

We also dig into the importance of PC reliability overall, first from the perspective of consistent performance, and then of failure. In fact, the idea of consistency comes up often in our discussion—so much so that it may just have influenced my focus for the newsletter this week.

Matt digs deep into his behind-the-scenes reveal of how the Puget Systems team benchmarks—not just how they come to deciding how to craft the tests, but also their philosophy and approach. Our almost two hour talk all but flew by!

Willis Lai / Foundry

Missed our live show? Subscribe now to The Full Nerd Network YouTube channel, and activate notifications. We also answer viewer questions in real-time!

Don’t miss out on our NEW podcast series too—our first episodes of Dual Boot Diaries and The Full Nerd: Extra Edition just launched!

And if you need more hardware talk during the rest of the week, come join our Discord community—it’s full of cool, laid-back nerds.

This week’s eclectic nerd news

A major change to Intel’s ownership, the new Commodore 64 smashing sales records, an actually cool use of AI, and yet another appearance of Doom in an unexpected place—the most interesting news to me currently spreads off in every direction. (And there’s so much!)

I love it.

Aaron Christophel / YouTube

• The U.S. will now own almost 10 percent of Intel: The government is currently promising to be a “passive” investor. Meanwhile, Intel has warned that its ability to secure non-U.S. business could be impacted by this involvement.

• Ready to rumble? Rumors suggest that AMD’s upcoming RDNA5 architecture may trade blows with Nvidia at the top of the stack. (No one tell Brad I’m sharing speculation.)

• Battlefield devs wish secure boot wasn’t necessary: But it’s going to be a requirement anyway. Blame this outcome on our fellow humans who won’t stop cheating.

• Huh, I don’t hate this use of AI: I actually love the idea of hobbyist AI models helping people access and experience history more easily. It’s possibly a more accessible format for digital museums. Key to this anecdote is the use of high quality data—if only that were a universal approach.

• Autofill is a double-edged sword: I like convenience, but not at the expense of security—and this week’s news about 11 password managers vulnerable to clickjacking attacks proves out my wariness.

• Is an intervention needed? Gen X and Millennials love throwing money at nostalgia. It’s not a new phenomenon—my Boomer relatives’ repeat purchases of 1950-1970s music CD sets prove that out—but let’s be real. We don’t have the same amount of real estate to store continual reproductions of PC and gaming hardware from our youth. At the same time, please take my money.

• Slap some duct tape on that problem: More than once, Will and I have bemoaned the wretched state of identifying USB cables and ports. This tester helps mitigate some of the issues, if you want to get deep in the weeds. Gordon would have loved it, but also that meme of slapping tape over a crack in a water tank is applicable here.

• Speaking of duct tape solutions: Amid all the reports of melting power cables, ASRock has released a cable with overheating protection. I think we’d all rather not have to worry about fires.

• RIP to my youth: The death of TypePad is just adding to all the signs that significant time has passed since I was young. Kind of hilarious that LiveJournal still is shambling about while its more elegant competitor is being laid to rest.

• What can’t Doom run on? I firmly believe the answer is “nothing.”

• I’m not ready to say good-bye yet: AMD’s Wraith Prism cooler is one of the most underappreciated stock coolers, in my opinion. (I guess this opens the opportunity for Hall of Fame nomination, though. Hmm.)

• A Redditor made a Lego 3D printer, and I’m a fan: It’s slow, cute, and perfect just as it is. Just like me.

• Happy birthday, Linux: It’s your year, or so I hear. Look, we even have a whole new podcast series dedicated to you!

Catch you all next week—I’ll make a bold prediction that I’ll still be as unprepared then as I am now for autumn’s imminent arrival. How is September already on our doorstep?

Alaina

This newsletter is dedicated to the memory of Gordon Mah Ung, founder and host of The Full Nerd, and executive editor of hardware at PCWorld.

Author: Alaina Yee, Senior Editor, PCWorld