Tag Archives: Machine Learning

Anything to do with Machine Learning

Why Owning Your LLM and ML Model Infrastructure Is Important

I attended a session at RenderATL this week titled “From Demo to Production: The Infrastructure Gap (and Why You Must Own the Stack)” by Nolan Code. Nolan is from the Atlanta AI & Robotics Initiative… I honestly didn’t know anything about the organization since I am not from the area, but they seem to be doing a lot of amazing things! The session was interesting and focused primarily on Physical AI… Think robotics, like self-driving cars and automation (as in industrial or factory). He said Physical AI will be among one of the fastest-growing segments of AI along with Agentic AI; I completely agree with his assessment and have been saying this for some time.

"From Demo to Production: The Infrastructure Gap (and Why You Must Own the Stack)" by Nolan Code

Along with Physical AI comes a need to own your own infrastructure. In the self-driving car sense… You obviously need cars to build autonomous self-driving technology, but the session’s deeper message is that you need to own the entire AI infrastructure stack. (hardware and software). His organization uses GPU acceleration, such as the NVIDIA Jetson, for inference on these models, but notes that they still use the big 3 hyperscalers for training.

Nolan stressed the importance of owning the inference stack for many reasons, but privacy and inference costs were the primary motivators, with cost being the heavier of the two. The cost of renting GPUs at significantly higher prices than traditional compute is definitely a buzzkill. He backed up his hypothesis by pointing to an article about 20 regional AI leaders, all of whom own their infrastructure stack in their own data centers, as supporting evidence.

Factory Automation

This got me thinking. While I agree with this idea, in this particular situation… the bigger problem might be privacy, not cost. Cost, you say? From me? The guy who has been yelling about the inevitable price hike looming in AI. Heck, I am even in the process of writing a book on Small Language Models, with the intention of getting AI practitioners to think about building AI solutions with privacy and cost in mind. Why is this situation different? Let me explain…

All 20 of those companies cited are inherently successful. While the cost of renting GPUs will continue, the money at this point isn’t a big deal to these successful companies. For those in the multi-hundreds of millions or even in the big “B” category for revenue, this is especially true. Others in their financial situation (by definition) have a successful and, more importantly, financially sustaining business where they don’t need to worry about things like GPUs depreciating. But I wholeheartedly agree that, at their level, you need to own that inference stack, but for privacy reasons.

Man-in-the-middle Attack

The systems that run these AI workloads, LLMs or traditional ML models, require GPUs or some AI acceleration. Typically, if those models are core to their business, they have dedicated infrastructure to run these models. Their solutions access those models sitting on those GPUs via some API… REST, MCP, whatever. That boundary is the source of your access and your problem. What stops those providers from recording the input and output over time and training their own version of your model? If you are using a cloud provider, they can record large amounts of input data and output, aka the answer… and then use that data to train a model to do the same thing. The simplest example of this is a classifier. If you aren’t of consequence (aka a company that doesn’t make a lot of money), who cares. HOWEVER, if you are, what’s to stop them from monitoring the network for that data and taking your IP?

That’s some tinfoil hat stuff, but with all the “urgency” from these AI providers to be better than their peers and turn a profit on their efforts… Why not do this? Why not do it? I mean, these companies have been scraping websites, violating content creators’ terms of service, and on and on. If you have proprietary data and have built a machine learning model that is core to your business and IP, bringing in billions, those providers might want a piece of it. Don’t think that would happen?

Amazon Basics and 3rd Party Sellers

We know Amazon already uses its retail business to monitor which products do well, and for those that do, it goes to a Chinese dropship company to source those products. This is EXACTLY their operating model for Amazon Basics. This has been happening for some time now. Check out this article, or this one, or watch this video. Amazon is basically doing this with cables and backpacks… the small shit. Why not for the big shit?

Google Lawsuit with Incognito Mode on Chrome

Google doesn’t let you opt out of using your personal information and data (think Gmail, Docs, etc.) unless you turn off all history and memory features. Even then, I would bet that they still do it, but use the legal loophole of anonymizing YOU while still using all the intelligence from the behaviors and patterns. Google effectively did this already with Chrome and incognito mode. Check out this article describing the payout from the lawsuit or this video if you aren’t familiar with what happened.

Why not do this in AI/ML in the middle of this AI arms race?

If you own the hardware infrastructure and stack in a datacenter, you don’t have this problem. For companies bringing in billions, this is the real threat. For you and me, who are using AI to solve coding problems… maybe they get to see your billion-dollar idea take off from the greenfield and help them create ideas. Until it becomes the money maker, your biggest concern is cost… A completely different problem based on a completely different audience.

While my reasoning might be very different from Nolan, he is absolutely freaking right.

What do you think?

When AI Gets Real: Takeaways from a Week at Devoxx Morocco

Devoxx Morocco brought together a mix of engineers, architects, and AI practitioners who were focused on the practical side of AI. The conversations were grounded in real problems rather than vague theory. People wanted to talk through design decisions, past failures, and the parts of their stack that still worry them. That set the tone for the conversations happening at the conference. The setting made it easy to slow down, listen, and think through the patterns that kept showing up.

Devoxx Morocco

Two themes came up everywhere: agentic systems built with MCP, and how to get AI pilots into production. The surprising part was how technical the hallway conversations were; as in a higher caliber than I have recently been to! Folks wanted to compare notes on everything from context-window constraints to the tradeoffs of graph-based retrieval. Instead of repeating the same “AI is the future” line, people were honest about the limits of current tools and focused on how to deal with them. It made the event feel useful in a way most conferences don’t.

This was a stark contrast to KubeCon, which took place the same week. Granted, the conferences’ purposes were different: KubeCon focuses primarily on application/container infrastructure, and Devoxx is engineer/developer-focused… but the difference in the AI discussions was VERY different. At KubeCon, the AI discussions felt stuck in the marketing and hype phase, and at Devoxx, the promise of the hype was meeting the realities of implementation.

Let’s dive into some of the discussions at Devoxx Morocco!

Highlights from Devoxx Morocco

The technical depth of the discussions stood out right away. Two of the most detailed conversations centered around vector embeddings and why they often fall short in production. Both engineers told nearly the same story: the embeddings looked fine on paper, but the answers drifted, broke down when correctness mattered, or hallucinated when the domain got too specific. What surprised me was that they brought up graph-based retrieval before I did. They wanted to talk about ontology design, schema choices, and how to build a structure that reflects the real domain. You could tell they had already run into the limits of semantic search and were looking for something more grounded in facts.

Devoxx Booth 1

Another strong thread came from someone who had been working with Model Context Protocol (MCP) and ran into context-window and tool-count limits. The way they described it felt familiar. You can scale the prompt window and restructure your tools, but there’s a point where the entire system becomes fragile. After thinking it over, it’s clear that Anthropic Skills are meant to address this problem. At a very high level, Skills act like folders that hide or load tools only when needed. It’s a smart workaround, but it also kicks the problem down the road rather than addressing it head-on. Additionally, it made me wonder whether these dynamic “folders” introduce their own risks, like tool hijacking within a skill if security boundaries aren’t well-defined.

Devoxx Booth 2

Production-readiness came up again and again. Attendees weren’t trying to “explore possibilities” the way many teams still do. They were focused on the details that block deployment: data pipelines, observability, governance, and proving that an AI system delivers value. It was refreshing to hear people skip the marketing talk and get straight to real outcomes. They cared about what actually works, not the slideware version of AI.

David's Session 1

My session titled “Rethinking RAG: How MCP and Multi-Agents Will Transform the Future of Intelligent Search” explored how Model Context Protocol (MCP) and Agent2Agent (A2A) can reshape the future of intelligent search by moving beyond flat, opaque vector embeddings toward adaptive, explainable, and secure agentic systems. The talk highlighted how current RAG pipelines often fail due to a lack of reasoning depth and fragile context handling. Through live demos, we showed how combining MCP’s structured data access with A2A’s multi-agent collaboration enables scalable solutions that enable agents to reason, search, and cooperate in real time. Key takeaways included designing modular agentic architectures inspired by software engineering principles, using reinforcement learning loops to safely promote verified knowledge into the RAG corpus, and considering small, specialized language models (SLMs) for faster, cheaper, and more transparent performance.

If you are interested in seeing the slides or the demo material, you can find them on my GitHub repo: github.com/davidvonthenen/2025-devoxx-morocco.

David's Session 2

The more conversations I had, the clearer it became that many teams here are further along in their AI journey than what I typically hear at other events. The questions were sharper. The examples were real. And the pressure to deliver value was obvious. You could see that stakeholders weren’t looking for experimentation anymore. They wanted impact, and the engineers at the event were working toward that with real urgency.

Personal Note

This was my first time visiting Morocco, and after hearing about Devoxx Morocco last year, I was definitely interested in attending. I didn’t know what to expect, but I knew this would easily be different from any other place I had visited in all my travels. The things I usually look for when traveling anywhere are: I want to know the real history of a place, see some amazing art or architecture, chat with people who live and have roots in the region, and, of course, eat all of the food.

Morocco is so colorful

On the architecture and art front, I was blown away by the bright colors and mix of Arabic and Middle Eastern influences. I have never seen buildings or architecture where the colors were just absolutely in your face. Truly a stand out.

And Soo Green With Plants

And to see how nature is integrated into the architecture, buildings, and landscape was simply beautiful.

If you find yourself in the area (in Europe) and you want something different, I recommend adding a stop in Marrakesh. If you like traveling the road less traveled, you won’t be disappointed.

Until Next Time!

Devoxx Morocco showed a clear shift in how engineers think about AI today. The excitement is still there, but it’s grounded in real work. People talked about the friction points they’ve faced, the parts they’ve rebuilt, and the choices they regret. And the common thread was simple: value matters. Teams want systems that work in production, not demos that run well on a stage. The conversations made it clear that AI is entering a phase where engineering discipline matters as much as model quality. That honesty made the event stand out.

Amazing Pool

On that personal note, given the chance, I would totally come back to Marrakesh and Morocco. There is plenty to do and I couldn’t possibly do it all in the time that I had. I will be back!

Doing More with Less: The Quiet Revolution Powering AI Through IoT and Edge

Edge AI is quietly changing how we think about machine learning deployment. Instead of running models in the cloud, intelligence is moving closer to where data is created—on sensors, gateways, and even microcontrollers. This shift isn’t about shrinking models for fun; it’s about making AI useful where latency, bandwidth, and power all matter. In this piece, we look at how tools like ExecuTorch and TorchScript let you run PyTorch models anywhere… whether that’s a GPU rack, a Raspberry Pi, or a factory floor controller and why efficiency is the next big driver of innovation in AI.

Pushing AI/ML to the Edge

What’s fueling this movement is necessity. As the world races toward smaller, faster, and more sustainable AI systems, the focus has moved from “how big can we make it” to “how far can we take it.” One of the best examples was DeepSeek’s efficiency breakthroughs… it’s clear that doing more with less is the future of machine learning.

Dive into the full article here: https://bit.ly/4mSCFOl.