Better. Faster. Cheaper?
When a skeptic encounters a claim, the examination begins. The biggest story of the past few weeks (months?) has been the impact of a new AI model that challenges prior assumptions.
So, how does the skeptic make a valid comparison? Next, how does this apply to the seemingly non-stop claims of better, faster, and cheaper ways to bring AI into everything, everywhere imaginable?
There is also a saying that “if you have to ask, they can tell you anything”. As vendors are constantly innovating and bringing those innovations to market, the benchmark for better or improved is arguably best kept open and transparent.
ML Commons
MLPerf and Moore's Law
The Goals of ML Commons
During AI Field Day 6, David Kanter of ML Commons shared perspectives on the goals of MLPerf for getting more results into GitHub for transparency and traceability. To Wardly this out a bit, the goal is to be associated with the leading direction and sensibly retiring benchmarks that are no longer represented as performance, efficiency, and qualitative (risk, responsibility) — more pioneer within the edge of a settler than attention to the laggard claims of best.
Learn more… https://github.com/mlcommons
MLPerf Storage Divisions and Categories
MLPerf Storage Submitters
Architectures and Innovations
Also, during AI Fields Day 6, Curtis Anderson explained the challenges within storage systems that are slow to change (architecturally) with the rapid innovation timelines of AI. As such, past storage-specific benchmarks for storage are not suitable for the nature of the AI workloads that use storage in vastly different ways than a traditional Enterprise workload — and contrary to what you might have heard — storage is not “dead”.
Learn more… https://github.com/mlcommons/storage
Industry Pace and Rate of Claims
As a side note, more and more claims will be made as models come to market. It will be interesting to see how benchmarks evolve in response.
ML Commons Techmeme
Elsewhere
To see more coverage of ML Commons by the AI Field Day 6 delegates, please visit https://techfieldday.com/event/aifd6/
About the Author
Jay Cuthrell
Chief Product Officer, NexusTek
Jay Cuthrell is a seasoned technology executive with extensive experience in driving innovation in IT, hybrid cloud, and multicloud solutions. As Chief Product Officer at NexusTek, he leads efforts in product strategy and marketing, building on a career that includes key leadership roles at IBM, Dell Technologies, and Faction, where he advanced AI/ML, platform engineering, and enterprise data services.
