The DatacenterDude Dispatch is a subscriber-only bi-weekly newsletter. Don’t miss an issue. Subscribe below.
For about fifteen years now, the pitch for object storage has been basically the same every time.
Cheap.
Scalable.
Durable.
Put your cold data here and stop paying for expensive flash to babysit files nobody’s touching. It was the digital equivalent of a climate-controlled storage unit. You fill it up, lock the door, and mostly don’t think about it again until auditors ask where the records went.
“Write once, read never” was not a marketing slogan. It was the honest description of how most organizations were using the technology. And for what it was being asked to do, it worked fine.
Then AI showed up and started asking very different questions of that storage unit.
I was talking about this with Duncan Moore, VP of Object Storage at NetApp, last week. We covered a lot of ground on my show, NetApp ONAIR. But one thing that was mentioned almost in passing has stuck with me. Clement Delangue, the CEO of Hugging Face, posted about OCR’ing 27,000 academic papers into Markdown. Open 5B model. 16 parallel GPU jobs on L40S instances. A single mounted object bucket. Total cost: $850. Total time: 29 hours.
My response to this was: “Hell, twenty years ago, that job would have taken a small army of data entry clerks the better part of a year.”
The number is impressive. But that’s not the part that matters. The bucket in that story wasn’t a cold archive. It was a production input. The data went in to be consumed, processed, transformed, and acted on at machine speed. It was a raw material, not a waiting room. That is a fundamentally different relationship with object storage. And most of the industry hasn’t caught up to what it means yet.
Object storage didn’t suddenly become important because of AI. It became visible because of AI.
The properties that make it work for AI workloads…
- flat namespace
- rich metadata
- S3 API compatibility
- horizontal scale
- protocol agnosticism
…those were always there. We just spent a decade pointing object storage at the wrong problem and then got surprised when it showed up at the right one.
Market data backs this up. Unstructured data is growing at 55-65% annually. Nearly 30% of cloud object deployments are already sitting at 10 petabytes or larger. These are not cold archives. These are active data estates that applications and, increasingly, agents are crawling constantly. The shift isn’t subtle anymore.
As one industry report put it bluntly…
“…object storage isn’t an alternative to file storage for AI.
It is the AI data store.”
I think that’s right. And I’d even take it one step further… if your object platform wasn’t designed for the access patterns that AI workloads impose, you’re going to feel that gap very soon. The AI workloads most infrastructure teams have been designing for… RAG pipelines, batch training jobs, periodic analytics runs… those are comprehensible. They have schedules. Defined inputs. Defined outputs. You can build for them.
Agents are different. And they’re coming very, very fast.
An autonomous agent doesn’t come to your data estate on a schedule. It comes when it has a question. It might reach into a bucket your team hasn’t touched in three years because something in a user’s conversation made that data suddenly relevant. It will read metadata you didn’t know you had. It will follow references you didn’t intend to expose. It will build context from data that was never designed to be context. That’s not a bug. That’s the whole point of agents in the first place.
But it means your object storage layer needs to be something it was probably never designed to be. Always on. Deeply indexed. Metadata-rich. Capable of serving low-latency, concurrent, random reads across a distributed data estate without falling over. Most object platforms were optimized for one thing… high throughput at ingest and low cost at rest. Agents impose the opposite access pattern. And that gap between what the platform was built for and what it’s about to be asked to do is where the bodies are going to start piling up.
So yea, in a nutshell, the hardware and software decisions you made five years ago are about to matter in ways nobody predicted.
I work at NetApp. Bias disclosed. Take that for what it is.
That said, I’ve been watching StorageGRID for a long time, and I think it’s one of the most underrated platforms in the industry. Not because it’s flashy, but because of what it was architecturally designed to do from day one.
StorageGRID was built around a premise that most object platforms treat as an afterthought… not all data is the same, data changes over time, and governing data at scale requires policy, not people. Its Information Lifecycle Management (ILM) engine isn’t a feature bolted-on after the fact. It’s the architectural core of the whole thing. You define rules. Where the data lives. How many copies. What protection scheme. When it transitions. Where it goes as it ages. The system executes those rules automatically at petabyte scale while your team does other things. That’s ILM. And that capability, which was built for the compliance and governance era, turns out to be exactly what an AI data pipeline needs, because agents operating across a multi-petabyte data estate need the data to be organized, consistently protected, and accessible. Not sometimes. Always.
StorageGRID 12.0 shipped in September 2025 and made a pretty clear statement about where this platform is headed. The headline number is a caching layer delivering up to 20x the throughput of previous appliances for AI training and HPC workloads. That’s real. But the feature I keep coming back to is branch buckets. Duncan described it as “Git for data” on the show, and honestly that’s a really good analogy. You can take an instant, space-efficient clone of a bucket containing billions of objects and petabytes of data, hand it to a data science team or an agent workflow, let them operate independently, and reconcile back later. You can also branch as of a specific point in time. Not just now. As of 3am two days ago.
By the way, for anyone running AI pipelines in a regulated industry, data reproducibility and versioning aren’t nice features. They’re load-bearing requirements.
Branch buckets treats them that way.
The sheer scale specs of StorageGRID are worthy of their own callout…
600 billion objects per cluster.
220 nodes.
16 geographically distributed sites under a single S3-compatible namespace.
Zero RPO in synchronous multi-site configurations.
15-nines durability through layered erasure coding.
S3 Object Lock with no proprietary APIs for WORM compliance.
Oh, and you can run the same bits on hardware appliances, VMware VMs, bare-metal Linux, or Docker containers.
This isn’t a cold archive platform that got retrofitted for AI. It’s a data governance platform that was always built for the moment when the data mattered at scale. I’d be doing you all a disservice if I skipped that part.
StorageGRID is not cheap. It is not simple. ILM is powerful because it’s expressive, and expressive systems can fail in expressive ways. A poorly designed lifecycle policy applied to an existing petabyte-scale dataset can kick off massive, unexpected data movement that eats bandwidth and blindsides your team. The trick, however, is to design your policies before you start ingesting data. Test in non-production. Budget for expertise upfront, either in-house or professional services.
The 12.0 caching layer shipped six months ago. It’s production-ready. As always, be sure to validate your specific workload sizing with NetApp before you put it in a critical pipeline architecture. And if you’re a small shop that just needs S3-compatible object storage without the lifecycle complexity, MinIO exists and it’s dramatically cheaper. Know the actual size of your problem before you size the solution. Or as I always say, start at the end and work backwards. Reverse engineer your way to the proper solution.
Where This Is All Heading
AWS announced Files on S3 (blog). ONTAP has had native S3 on top of file for a while. Every major storage vendor is converging on the same conclusion… that the persistent store of record for enterprise data is object, and everything else, file, block, cache, is a performance and service layer sitting above it or adjacent to it.
That should in no way be viewed as a threat to the architecture NetApp has been pushing for years now… It’s a validation of it.
File at the center, high-performance AFF in the middle, StorageGRID as the outer ring where the petabytes live.
Agents are going to navigate that entire stack. The data will be hot and cold and everywhere in between simultaneously, depending on who’s asking and when. Your object platform needs to be ready for the question that shows up at 3am. Not just the one you scheduled for 3pm.
I talked through all of this with Duncan last week on NetApp ONAIR. If you want to go deeper on the hardware, ILM mechanics, branch buckets, and yes, I promise this is real, the NetApp campus beekeeping program and the honey-finished lager that came out of it, the full episode is below.
The infrastructure instincts you’ve built over two or three decades still apply here. Now, you’re just applying them to a data estate that’s about to become a lot more alive than it used to be.
/Nick
If this was worth your time, the newsletter goes further. The DatacenterDude Dispatch is bi-weekly, subscriber-only, post that dives deeper than my typical blog posts on specific topics. Hit the subscribe button and you won’t miss the next one.
Frequently Asked Questions
What is making object storage relevant for AI workloads in 2026?
AI workloads, particularly autonomous agents, access data constantly, randomly, and without schedules. That access pattern rewards exactly what object storage was already built for: horizontal scale, metadata richness, and S3-compatible APIs. AI didn’t reinvent object storage. It revealed what it was always capable of.
What is the difference between how AI agents and traditional workloads use object storage?
Traditional workloads are predictable. They have schedules, defined inputs, and defined outputs. Agents are none of those things. An agent comes to your data estate when it has a question, not when you tell it to. That requires storage that is always on, deeply indexed, and capable of serving concurrent random reads at scale.
What is NetApp StorageGRID and why does it matter for AI?
StorageGRID is NetApp’s enterprise object storage platform, built around policy-driven data lifecycle management at petabyte scale. Its ILM engine automatically governs where data lives, how it’s protected, and when it transitions, without human intervention. That governance capability, built originally for compliance, turns out to be exactly what AI data pipelines need.
What are branch buckets in StorageGRID 12.0?
Branch buckets let you instantly clone a bucket, including billions of objects, hand it to a data science team or agent workflow, and let them operate independently before reconciling back. Think of it as Git for data. You can also branch as of a specific point in time, not just the current state.
Is StorageGRID the right object storage platform for every organization?
No. StorageGRID is enterprise-grade and priced accordingly. If you need simple S3-compatible storage without lifecycle complexity, cheaper options exist. StorageGRID earns its cost at petabyte scale, in regulated industries, and anywhere data governance and reproducibility are load-bearing requirements.
Discover more from DatacenterDude
Subscribe to get the latest posts sent to your email.