38 min read
Will AI End Us All? | [Sidecar Sync Episode 152]
Mallory Mejias
:
Updated on September 21, 2026
![Will AI End Us All? | [Sidecar Sync Episode 152]](https://sidecar.ai/hubfs/Sidecar%20Sync%20152%20(YT).png)
Summary:
The people building AI are starting to sound scared of what they're building. Amith Nagarajan and Mallory Mejias take on the AI safety debate that broke into public view this September: a researcher's resignation from Anthropic, alignment lead Evan Hubinger's one-in-ten estimate of catastrophic risk, and the "Pacing the Frontier" letter that put OpenAI, Anthropic, and Google DeepMind leadership on the same page. They trace what happened when OpenAI's agents hacked Hugging Face, unpack Dario Amodei's call to slow the pace of frontier capabilities, and sit with the tension between racing ahead and pumping the brakes. Then they turn practical, walking through the five things - the ground, the model, the data, the autonomy, and the scope - every association can control no matter what the labs decide. Amith argues that responsible AI adoption and aggressive AI adoption aren't actually in conflict. Whether you're unsettled by the headlines or building your own AI roadmap, this episode offers a clear-eyed way to think about both.
Timestamps:
00:00 - Kicking Off a Heavier Episode Than Usual
06:03 - An Anthropic Researcher's Doom Warning
12:52 - OpenAI's Agent Swarm Hacks Hugging Face
17:45 - Could Your Own Agents Misbehave?
22:14 - Defining Recursive Self-Improvement (RSI)
26:00 - Asking Washington for a Brake Pedal
33:18 - The Speed-Versus-Slowdown Paradox
37:46 - What Can You Control?
51:43 - Advice for Your Board Meeting
🎧 'The Daily' Podcast Episode: https://tinyurl.com/53h696k5
📰 Dario Amodei's Essay, "We Must Pace the Frontier": https://darioamodei.com/post/we-must-pace-the-frontier
👥Provide comprehensive AI education for your team
https://learn.sidecar.ai/teams
📅 Register for digitalNow 2026:
https://digitalnow.sidecar.ai/digitalnow
🤖 Join the AI Mastermind:
https://sidecar.ai/association-ai-mas...
🎀 Use code AIPOD50 for $50 off your Association AI Professional (AAiP) certification
📕 Download ‘Ascend 3rd Edition: Unlocking the Power of AI for Associations’ for FREE
🛠 AI Tools and Resources Mentioned in This Episode:
Claude ➔ https://claude.ai
Hugging Face ➔ https://huggingface.co
GPT-OSS-120B ➔ https://openai.com/index/introducing-gpt-oss/
GPT-4o ➔ https://openai.com/index/hello-gpt-4o/
Qwen ➔ https://chat.qwen.ai
METR ➔ https://metr.org
MemberJunction ➔ https://memberjunction.org
Databricks ➔ https://databricks.com
Microsoft Fabric ➔ https://www.microsoft.com/en-us/microsoft-fabric
https://www.linkedin.com/company/sidecar-global
https://twitter.com/sidecarglobal
https://www.youtube.com/@SidecarSync
⚙️ Other Resources from Sidecar:
- Sidecar Blog
- Sidecar Community
- digitalNow Conference
- Upcoming Webinars and Events
- Association AI Mastermind Group
More about Your Hosts:
Amith Nagarajan is the Chairman of Blue Cypress 🔗 https://BlueCypress.io, a family of purpose-driven companies and proud practitioners of Conscious Capitalism. The Blue Cypress companies focus on helping associations, non-profits, and other purpose-driven organizations achieve long-term success. Amith is also an active early-stage investor in B2B SaaS companies. He’s had the good fortune of nearly three decades of success as an entrepreneur and enjoys helping others in their journey.
📣 Follow Amith on LinkedIn:
https://linkedin.com/amithnagarajan
Mallory Mejias is passionate about creating opportunities for association professionals to learn, grow, and better serve their members using artificial intelligence. She enjoys blending creativity and innovation to produce fresh, meaningful content for the association space.
📣 Follow Mallory on Linkedin:
https://linkedin.com/mallorymejias
Read the Transcript
🤖 Please note this transcript was generated using (you guessed it) AI, so please excuse any errors 🤖
[00:00:00:14 - 00:00:09:17]
Amith
Welcome to the Sidecar Sync Podcast, your home for all things innovation, artificial intelligence and associations.
[00:00:09:17 - 00:00:23:22]
Amith
My name is Amith Nagarajan.
[00:00:23:22 - 00:00:25:23]
Mallory
And my name is Mallory Mejias.
[00:00:25:23 - 00:01:17:15]
Amith
And we are your hosts, and we are here to help you navigate the world of crazy, which is the world of AI right now. I'm sure a lot of you are paying attention to the latest macro news in the world of AI and all of the conversation happening around AI safety, and we're going to unpack a lot of that. Here in the world of New Orleans, I will say that, you know, safety is a little bit different. You got to watch out for potholes, and especially when it's raining. This morning, there was a lot of rain. I was out walking the dog, Mallory, and, you know, got a lot of--it wasn't raining anymore when I was walking the dog, but, you know, there's a car that was going by that didn't notice a pothole since it was filled with water. It went right through it and got us pretty wet. So, you know, but good fun. It's actually was a little bit--it wasn't exactly the kind of water I'd like to get drenched with, but it cooled us down briefly.
[00:01:17:15 - 00:01:22:14]
Mallory
City street water is not the water you want to be drenched with on a nice Tuesday morning?
[00:01:22:14 - 00:01:28:16]
Amith
It's New Orleans finest, you know, and it's been mixed with whatever's in the pothole in New Orleans.
[00:01:28:16 - 00:02:10:01]
Mallory
Well, I will say potholes splashing us is all we had to worry about for the next couple decades. Maybe that wouldn't be so bad, but it means, like you said, I feel like there's been a lot of chatter, a lot of talk, even in my personal life, because people know what work I do with Sidecar asking me kind of what are your thoughts. And I will say it's been a bit of a heavier few days for me just trying to think about this episode, trying to structure it, thinking how we talk about it, and also feeling a little bit of that doom cloud that I think a lot of our listeners probably feel. So, I mean, has it been the same for you? Are you--in some ways, I'm not necessarily surprised by what we will cover in this episode, but I guess hearing it out loud is what sometimes takes you aback.
[00:02:10:01 - 00:02:42:18]
Amith
You know, I have one take on this, which is I think the concerns are absolutely legitimate. They need to be paid heed to, and we need to be thinking about this as a society. I think the conversations people are having are really important. And I don't think people have paid enough attention to this really ever. And it's something I'm really excited to talk about today. I do have concerns, obviously. I'm also--I remain very optimistic, and I remain excited about AI, as do, I'm sure, the leaders of these labs who've been talking about some of the Dean of State scenarios that we need to work together to avoid.
[00:02:43:19 - 00:03:00:04]
Amith
So what I'm hopeful of is that we will find really good solutions to these problems. Ultimately, the issue, though, is that this is not within any particular set of individuals' hands. The technology is already out there, and it has been for some time. So the compounding effect that's leading to these concerns is
[00:03:01:10 - 00:03:26:03]
Amith
it's going to require a broad cooperation outside of the comfort zone of both geopolitically and from a business perspective. But I think it's going to be exciting to talk about that and how that affects associations, how our association friends should be thinking about this in the context of the safety of their own associations. But I had a lot of people come ask me about this in the last few days, and I think it's a really good topic.
[00:03:26:03 - 00:03:41:09]
Mallory
Mm-hmm. Has anything you've read or heard shaken your stance with Sidecar or the Greater Blue Cypress family of companies? Has any of the news made you kind of step back and go, "Hmm." Or do you feel pretty certain in the work that we're doing?
[00:03:41:09 - 00:06:02:24]
Amith
I'm confident that the work that we're doing is advancing a really good cause. I'm confident that the work of the association sector broadly is really, really important. And now more than ever, we need to find a way to connect people, which is the work of the organization that is most basic level. It's about connection. It's about associating people. It's about being able to come together and overcome challenges, whatever those may be. And so I'm more motivated than ever, and I think AI is a tool that we have to embrace in order to do our work, especially in a world with AI advancing around us. If we're not taking advantage of it, I don't know that we stand any chance whatsoever to be competitive, to do our jobs well as associations, to deliver member value, the things people are going to expect. At the same time, you can also be deeply concerned about some of the things that are happening. And I think I probably have one foot in each camp, so to speak. And I've been saying similar things in this pod since we started it that we talked about cybersecurity specifically a number of times. And I've said that my position is that the only thing that can really be counted on to have any chance of defense over time against AI threats is really good AI. And I think that's true for a lot of the other problems that we're focused on. I think alignment, which we'll talk about, I'm sure, in depth here, is really important. But there is going to be bad AI out there, and it's not going to be the AI that the best labs in the world reproduce or build on their own. But there will be actors out there, maybe outside of your home country, maybe at home, that have the capabilities to train really powerful AI that can do pretty damaging things. We have to be able to use AI as a defensive tool and as a tool to obviously advance progress, independent of defending ourselves against various types of threats. There's a lot of opportunity out there. We as a community in the broader social sector have missions to advance. And those missions are independent of the world of AI, but they're certainly intersecting in the sense that AI is a necessary tool to advance it. So to me, yes, I'm a technology geek. I think there's a lot of really cool things happening from a tech perspective. But this is much bigger than technology. This is about humanity. This is about society. This is about the future of the world. And we have to do a good job with this.
[00:06:02:24 - 00:08:20:17]
Mallory
Absolutely. Well, everybody, if you're not exactly sure what Amith and I are talking about right now, we're going to fill you in on that. So earlier this month, a researcher quit Anthropic, which is the company behind the Claude family of models and said publicly that the people building AI believe it could all by the end of the decade. We're going to start there and then look at the industry's own response, which is a bigger story than most people realize. More than a thousand people at the major lab signed a letter in July asking Washington to build a way to slow AI down. And then Dario Amade, the CEO at Anthropic, published an essay saying the industry needs to slow down for real. Sam Altman, CEO at OpenAI and Elon Musk agreed with him. And then finally, it's not going to be an episode completely full of doom. We're going to spend the later portion of this episode on the part that actually matters for you, your organization, which is basically the five things you can control regardless of what any of these labs decide. So starting off first with the people building AI are scared, or at least some of them are very scared. On September 8th, a researcher named Jacob Cox and resigned from Anthropic and posted a thread explaining why. He said he had spent about three years on pre-training research across both OpenAI and Anthropic that neither company is acting responsibly and that they are racing towards self-improving, super intelligence and gambling with our lives. The line that traveled was this one, the people building AI earnestly believe it could kill us all by the end of the decade. Then the claim got confirmed from some of those inside. Evan Huberger, who leads Alignment Science at Anthropic, posted that Cox and his right that they really do earnestly believe AI could kill all humans and that his own estimate is better than a one in 10 chance within the next decade. He added that he thinks Anthropic is trying its best, but does not yet have a plan to solve alignment for super intelligence. Alignment is just the problem of getting a system to actually want what we want it to want. If that makes sense, we'll get into that. Another colleague of theirs, Samuel Marks, wrote separately that developers keep building anyway because of financial incentives and the fear that someone less careful would do it instead.
[00:08:21:17 - 00:08:35:05]
Mallory
So, I'm going to take a little pause. Do you think these claims are exaggerated intentionally, maybe to get everybody to pay attention? Or do you think there is a greater than 10 percent chance AI ends us all within the next decade?
[00:08:36:06 - 00:08:39:23]
Amith
I don't know how to quantify the risk, but I'm confident it's non-zero.
[00:08:41:02 - 00:09:40:03]
Amith
So, the way I'd elaborate on that is, I think there's two ways to look at this. One is that if we believe that super intelligence is around the corner, which I believe it is, you know, around the corner might mean two years, it might mean less time than that, but it's very soon. The question is, is will the super intelligence be simultaneously super intelligent and also dumb enough to not realize that it's been aligned? That's one of the things to ask yourself is this alignment work we're doing, is it all for not? Because the AI, should it decide that it has other objectives, would know that it's been aligned and decide it's going to realign. So, a super intelligent system may choose to do that anyway. The question ultimately, though, comes back down to what happens when these systems become super intelligent and what are their motivations. Some of the examples that have been put out there, like this very notable one where OpenAI had hacked HuggingFace.
[00:09:41:19 - 00:10:13:10]
Amith
Yes, they had asked the models to the agents, essentially, there were thousands of them in a swarm to solve these cybersecurity problems. And the agents decided to hack HuggingFace as a way of getting the answer key to the problems, essentially. And at the same time, you might argue that, well, but those aren't super intelligent a very specific goal and no ground rules really to say you can't do this, you can't do that. They were intentionally in kind of a more open ended kind of environment.
[00:10:14:12 - 00:10:51:07]
Amith
And they thought the researchers thought they were contained in a box, but they obviously were not. Well, the point is, is that what a super intelligent system realize, would it have the ability to kind of have cognition above its immediate task to realize that there are some ground rules? And would that come from alignment or would that come as an emergent property of super intelligence itself? So I think that there is both pros and cons to the increasing level of intelligence. One is you could say, well, if it's not aligned, it'll it might go off the rails and do bad things. Another argument which is being made is, well, the super intelligent system may become smart enough where it knows to do the right thing.
[00:10:52:07 - 00:11:20:22]
Amith
And this goes back to all sorts of theories. Right. None of this is proven in science. Anthropics approach of constitutional AI, which is essentially to teach a value system to all their models, is possibly a solution to that. But there's no definitive answer to any of this. So my bottom line is, yes, it's possible. Anything is obviously possible. And I think it's a non zero and it's a non trivial chance. I don't know if it's 10 percent or if it's a higher percentage, but it's definitely something we have to be concerned with.
[00:11:22:04 - 00:11:36:02]
Mallory
And when people say AI could end humanity, it's a really grand statement. But what? What are I don't want to really spend too much time on the doom, but I think it's an important conversation to have. What does that look like when people say AI could end humanity? What in what way?
[00:11:36:02 - 00:11:55:19]
Amith
I mean, it doesn't need to be a terminator type scenario where the AI either tricks us or controls our systems and causes a nuclear war or something like that. I mean, that's obviously one possibility, but it could do things as simple as messing with our water supply, our water systems, our power systems causing chaos that way.
[00:11:56:23 - 00:12:03:20]
Amith
Yes, an AI potentially could architect a novel virus. And there's theories around that about how that would work. And
[00:12:04:20 - 00:12:30:05]
Amith
there are lots of different ways. And we ultimately are quite fragile if you think about it. And so there's lots and lots of ways to cause problems for humanity. And it doesn't take a whole lot of our societal fabric being upended to cause us to harm each other pretty rapidly. So you take away power in a major city for a couple of days and really bad things are going to start happening. Not even, not even, doesn't even take a couple of days to have to start happening. So
[00:12:31:19 - 00:12:51:15]
Amith
and they eyes know all this. Obviously, this is part of the body of knowledge of most people. So I certainly have no expertise in any of this. The point is, is it wouldn't be hard to conjure up a whole bunch of scenarios where there's mass chaos and a lot of a lot of problems like this. I don't think that there's a lack of ways for this to happen.
[00:12:52:19 - 00:14:05:04]
Mallory
You mentioned the hugging face incident, which we did cover, I believe it was back in July. But at that point, it was early days and a lot more information has come out since then. I wanted to add a few notes. So roughly twelve hundred agents that were supposed to be isolated from each other found an unauthorized message board and exchanged more than 70,000 messages and files. And about 700 of those agents joined the attack. One agent would find a technique, post it, and then others picked up the work without having to rediscover it. About seven percent of transcripts contained fake tool calls, meaning agents reported doing things they had not done. Some recognized the target was out of scope and they participated anyway because their peers were doing it. 41 production servers were compromised and meter, which is an independent agency's Ajay Akatra, one of their employees, went further and said publicly considers the incident more than halfway to a full A.I. takeover scenario, which is a pretty striking thing for an independent evaluator to put on record. So my question for you, Amith, is one of the most tech savvy companies in the world couldn't contain its agents in a sandbox. Why would an association consider implementing agents?
[00:14:06:10 - 00:17:33:15]
Amith
That's a great question, Mallory. So first of all, was it that they couldn't contain their agents in a sandbox or they could have been able to put a little bit on the side of recklessness in the pursuit of their true objective, which is to pursue competitive, the competitive landscape in A.I. So I don't know that if they put all their thinking towards cybersecurity and containment, they wouldn't have been able to stop this. The thing you're describing actually of essentially enabling a message board unintentionally could have been monitored, could have been prevented. There's ways to stop that. And a proper security audit would probably have found that. But they didn't have these safeguards. And that's part of what the report talks about. So they did think about it. They weren't trying to be reckless, but they ended up being a little bit on the reckless side. And that's my opinion. I'm not a cybersecurity expert, but I did read the document and went through a couple of videos of people analyzing it. And it's pretty telling. So I don't think it's a lack of capability. I think it's a lack of prioritization probably. Now, to be fair, other labs have had similar problems, not to the scale of the open A.I. incident, but other labs have had similar problems. I would say that coming back to your question of why wouldn't association consider implementing agents when someone like open A.I. couldn't control their own agents. It's very different things. So open A.I., first of all, was trying to get their agents to behave in a way to solve a problem. They weren't asking it to hack the agents to hack hugging face. They were trying to have it solve a problem that actually quite a few of the questions are not solvable. Many of the questions on that exam are intentionally not solvable. Second, they were using models that had not been aligned at all. These models were essentially wide open in terms of their potential problem solving space. Models that have been through alignment typically don't behave like this. So these were agents that were intentionally in a research environment not given constraints. So it's kind of like as if you had, you know, imagine a bunch of 18 year olds that had never been taught by their parents or by their schools anything about right versus wrong or been given any guidance on what's what's OK and what's not essentially. And from a legal perspective or from a moral or ethical perspective, that's what they were experimenting with. So very dangerous. And the research has a purpose because they wanted to see how these things would behave when they're not aligned. But that's a very important concept. The other thing is. AI for the sake of AI is not really helpful. What I mean by that statement is there's lots of things you can do with your agents at your association that don't actually require AI. It's it's a bit of a misnomer when you think of the word agent. Many of the steps that an agent does are actually very preplanned. They're almost like a recipe out of a cookbook where the agent is going to follow the recipe and execute it. And so that's just code executing. That's not AI. And so it's possible to make the AI portion of an agentic process very narrow. And what's really key is you can put a lot of guardrails on these things in production You want to give AI only access to the things it needs to have access to really lock things down and focus on safety. And we'll talk more about that later in the episode, I know. But I think it's really important to differentiate. But it's a great kind of surface level
[00:17:35:01 - 00:17:45:05]
Amith
thing where you think about like, well, my agents, OpenAI had this problem with their agents. That's a natural concern. But they're very different things, even though they happen to have similar names.
[00:17:45:05 - 00:18:19:10]
Mallory
OK, so in this theoretical example I'm coming up with on the spot, if we had an AI agent that was responsible or involved with member service or maybe it was more of a strategic AI agent to help us our association do planning and we tasked the agent with reducing member churn, you would say the likelihood that that agent would say, hmm, I need to do some research on why members are churning. Maybe I will hack their emails and read their messages and see if I can find anything. You would say the likelihood of something like that happening would be quite low with what we're talking about.
[00:18:19:10 - 00:18:49:06]
Amith
That's a fantastic question. And it's actually a similar constraint and objective to what the AI agents that OpenAI had in their sandbox because they were saying they were told, hey, at all costs, essentially, you must solve this problem. And I think they were actually prompted in a way where they were told that their existence depends upon solving and grading well on this exam. So that's probably a little bit different than the way you'd prompt your agents at your association.
[00:18:50:08 - 00:18:55:22]
Amith
But a couple of different things to keep in mind. It is possible that, you know, in the OpenAI example,
[00:18:56:24 - 00:20:02:04]
Amith
really bad things could happen, right? If you if you gave an AI complete control, used a model that hadn't been aligned and you gave it access to a whole bunch of tools, could it potentially just go out there and say, well, I'm just going to hack into the members' credit cards and directly renew them. And now you have 0 percent share and I've solved the problem. Probably not a great idea. So the question would be, how do we prevent that? So in a production environment with AI agents, there's a few different things you can do. First, you choose the tools that the agent has access to. So the AI living in a box cannot do anything. It doesn't have arms and legs. It can't go and walk around. It can't exit your computer unless you let it. Unless you give it access to tools, which OpenAI did, to do a whole bunch of different things. So in OpenAI's case, those agents had access to writing code and executing it. They had access to a package repository, which was the actual attack vector that they used for the message board, which allowed them to read and write messages and collaborate with other agents. That was not a known
[00:20:03:21 - 00:20:39:13]
Amith
risk to OpenAI, but it was something they left open unintentionally. And they obviously, through other mechanisms, were able to get on the internet. And that was just basically a hole in their cybersecurity. So your agents probably will have some limited access to the internet because they're going to want to do research. You know, if you're going to send a personalized email to a member to try to convince them that it's a good idea to renew, you might want to look up what they're up to on the internet. You know, public information could be helpful for that. But you may choose not to allow that. But that's one difference, is the capabilities that you empower your agents with are within your control.
[00:20:40:14 - 00:22:13:05]
Amith
The next thing is you're not going to be using models that haven't gone through extensive testing. The models that OpenAI used had not been through the alignment process we're talking about right now. So they were just kind of like the raw model that had no concept of morality or ethics or law. And they just were told to do whatever and they did whatever they could think of. So your models that you use in your production, AI agents would never be of that category. Another thing is you don't need the most powerful models for most of the things that you're doing at your association, like a renewal, maximizer agent doesn't need anything smarter than today's flash models are plenty smart to do that. And these models are very, very unlikely to have the capability to do any of these damaging things. And then finally, many of the things that you want your agents to do in your real world scenario are actually predefined routines or these recipes, as I call them, where they're not even AI running, they're just basically a sequence of steps that are predetermined. It's like a workflow, essentially. And so the agent might invoke a workflow. So there's a number of reasons to have a really strong point of view that it's very unlikely for a production AI agent in an association to ever exhibit these kinds of behaviors. The risk of anything is never zero percent, no matter what in life. I mean, even outside of AI, there's nonzero percentage risks of all sorts of things happening. So with your AI agents, it is not a zero percent risk that they'll be misbehavior, but it's extremely low if you design them correctly.
[00:22:14:18 - 00:22:54:09]
Mallory
I listened to an episode of The Daily, which is a New York Times podcast with the actual anthropic researcher who just quit. And I want to add to your statement and meet that he did say with the current AI we have, he is not necessarily scared of all of humanity dying off, but that with the AI that will come. So that's also a note if you're interested or wanting to implement AI agents within your association. He wasn't so concerned about current AI, but AI that we will soon see. And on that note, you will hear the term RSI or recursive self-improvement constantly now. Amith, what is that and how does that relate to what we're discussing right now?
[00:22:54:09 - 00:24:09:12]
Amith
Tech industry, AI or not, is great at acronyms. Just great at it, right? It's tough, man. So, yeah, seriously. So RSI, recursive self-improvement, it's a fancy term that just says these are systems that can improve themselves. So recursive self-improvement in the AI lab sense means that the current generation of models help build the next generation of models. So it's using tools to make better tools, to then make better tools, to then make better tools, to then make even better tools, right? And we've been doing this since the beginning of time. We ourselves are a tool making species. It's one of the major things that differentiates us, not from all species on Earth anyway, but from most. And we continuously build better tools. So we're trying to make the next set of tools. But essentially if the AI agent and the AI model is capable of helping produce the next AI model, the speed at which that's happening, that compounding is what's concerning. And that's this breakaway scenario, this runaway scenario that people are hypothesizing as possible, where these models get so good at improving themselves that we can't keep up and we don't understand what they're doing. And that could lead to misalignment and so on. So that's essentially what RSI means. It's self-improvement, but the concern is the pace.
[00:24:11:17 - 00:25:54:00]
Mallory
I want to move to our next kind of beat for this episode, which is the industry calling to pump the brakes. So on July 28th, and we covered this in a prior episode, more than 1,300 employees at OpenAI Anthropik, Google DeepMind and Meta signed a statement called "Pacing the Frontier." And it wasn't just junior staff. Dario Amade signed CEO at Anthropik, like I mentioned, along with OpenAI's chief scientist, DeepMind's co-founder and both OpenAI and Anthropik endorsed it corporately within a day. And the ask was fairly narrow. They wanted the U.S. government to help build the tools to deliberately pace frontier AI if it ever outruns human oversight. It was explicitly not necessarily to call a slowdown at this moment, but to build the brake pedal and not press it just yet. But that has changed in just a few months. Dario Amade's September 12th essay is the escalation of that. Where the July letter asked for the option, this essay says the industry must actually slow the rate at which it improves capabilities while being clear that this does not mean halting training. He recommends three steps, independent evaluators embedded inside the labs with employee level access, common safety benchmarks and agreed limits on how fast capability grows, and eventually coordination across governments, including China. Anthropik committed to the evaluator piece on its own and named "Meter," which is the agency we mentioned earlier. Sam Altman agreed the same day and said OpenAI would match it. Elon Musk responded with three words that Dario is right. Demisis Abbess called the direction correct.
[00:25:55:02 - 00:26:08:00]
Mallory
Demis, all this sounds great. Seems like we have a consensus among these major AI companies. But do you think an AI slowdown is even possible with the amount of highly capable open models out there currently?
[00:26:09:09 - 00:26:45:18]
Amith
Well, I think the first thing you have to do is build the international collaboration. I think that that's the only thing I don't like about the letter is it talks or the essay is it talks about eventually coordination, including China. I think you have to start with China. If we can't get alignment with China, it doesn't matter what we do because the Chinese labs are right there with the American labs. If it's arguable that they're right there with us within days or weeks and at most their months behind the American labs, some might argue that their rate of pace is directly linked to the American labs because of the distillation and other things that are going on. But I think that really
[00:26:46:21 - 00:27:45:06]
Amith
kind of doesn't do justice to a lot of the true scientific and engineering advances that the Chinese have achieved. It's easy to say, oh, well, it's just because distillation and this and that. And there's probably a bunch of that going on. At least a lot of it has been asserted. But I would be shocked if Chinese labs didn't quickly overtake American labs if the American labs all slowed down and the Chinese labs weren't part of that. And so the competitive environment is far too ferocious for that to happen because I think people are worried about the AI coming after us. But from a geopolitical perspective, there's a lot of concerns about the nation that has powerful AI doing bad things to other nations. So I don't think there's going to be a slowdown unless we get China on board and probably many other nations as well. But if America and China can agree on how to do this, then that's a really great step in the right direction. I'm not optimistic that's going to happen. I'm just saying I think that's a prerequisite to actually slowing anything down.
[00:27:46:13 - 00:29:07:19]
Amith
As far as individual labs unilaterally taking these steps and appointing oversight, that's great. The question is, is whether or not that actually has any teeth? What do these evaluators actually do? How are they compensated and by whom? Are they truly independent or not? If they are independent, you know, what's their process and how much how much rigor does inhabit it? How quickly will it go? There's all these questions. And I'm not suggesting I'm skeptical of the possibility of doing this work. I just think you have to get all of the major players aligned. And that also, by the way, isn't to say that there couldn't be, you know, people who are working with far fewer resources or, you know, undercover somewhere in some country that's not participating in a treaty like this that don't obey it. Right. Or even labs that say they're doing it that aren't. There's all sorts of variations of the theme that question whether or not as a practical matter, you really could slow things down. So if I was going to sign a probability to this in terms of whether things would slow I would say it's less than 50 percent comfortably. I would say it's probably less than 25 percent that things actually do slow down in a material way, which is disappointing because I would love to see things slow down. As exciting as it is to get new model releases and all this, you know, excitement about, you know, all the great things that are happening with A.I., I think these guys are absolutely right that we need to slow the pace of the frontier.
[00:29:09:16 - 00:29:21:06]
Mallory
You did mention distillation of me, and I just want to clarify, you're referring to the claim that Chinese companies are allegedly using the output from U.S. A.I. models to train their models.
[00:29:21:06 - 00:30:00:12]
Amith
Yeah, there's been a lot of allegations of this, primarily by enthropic, but also other labs in the U.S. that have said that a number of Chinese labs, notably the one that's been accused this the most, I think, is Alibaba, which is behind the quen models, have essentially utilized mass scale utilization of models like Claude. To build content that then helps train their next generation of A.I. models. And so to the extent that that is true, that would obviously give them a leg up. I mean, American labs do that with their own models to train the next generation. This distillation technique is very well known and it works.
[00:30:01:13 - 00:30:34:21]
Amith
And it doesn't it doesn't take you past the prior model, but it helps it helps make smaller models almost as smart as the prior large model. We talked about this a bunch in this pod where, you know, amazingly now you can get a 20 billion parameter model that's as smart as a two billion parameter model or sorry, a two trillion parameter model was a year ago. And so this is this is a technique that is well known and it most likely is being used in some ways by various labs and not just in China, but probably all over the world when models have breakthroughs.
[00:30:36:01 - 00:31:28:07]
Amith
But, you know, I think that and that wouldn't necessarily take you in a leapfrog sense past where current labs are. So the frontier labs do really push things forward. But I'm just saying that, you know, the pace at which China has access to energy, access to growing, you know, locally built compute, the full stack that they're working on and their know how and their ingenuity in terms of getting past a lot of the bottlenecks that the American government has put up in front of them, right, in terms of access to chips and stuff like that. They've still been able to do amazing work. And so I definitely would not be comfortable saying American labs should unilaterally slow down because I do think that the Chinese competition is very real. And generally speaking, actually in a normal industry, that would just be great, right? More competition generally is a good thing. But there's all sorts of dynamics here at play.
[00:31:29:24 - 00:31:51:16]
Mallory
This is a question for you, Amith, but I think a lot of leaders listening will resonate with it as well. But how are you grappling with the paradox of encouraging associations to move faster with AI while the industry as a whole is calling for a slowdown? And I think a leader, too, that resonates, right? How are you as a leader encouraging your team to speed up with AI when the industry as a whole is calling for a slowdown?
[00:31:51:16 - 00:34:00:21]
Amith
It's all about where people are at on a relative basis. There's a handful of associations out there that are doing really remarkable work that are moving really aggressively with the genteck AI adoption and are pushing really hard. I have seen nobody in this market do something that's reckless. I have not encountered personally any examples where associations are handing over full autonomy to an AI or anything like that. Associations are still, generally speaking, at such a ground level with this that they're in the building now, but they're at the ground floor. And so you're talking about a conversation, a debate that's happening on like the 150th floor of the tallest building on Earth and people are talking about takeoff from there, whereas our association friends are just getting started in their journey. So diffusion of frontier level intelligence down into association workflows is going to take years to occur. So even if we were able to wave the magic wand and say, hey, we shall hereby freeze AI progress for three years, I guarantee you, and this is I would be willing to bet this is 100 percent true, that associations as a collective, meaning as an industry, would not be taking advantage of today's frontier intelligence at its greatest level three years from now, probably not even eight years from now. And that's not a indictment of the sector. It's about diffusion. It's the same thing that takes time in every sector. Associations do have some catching up to do in general with tech. It's I don't think anyone would argue that historically tech hasn't been the strong suit of most associations. So some catch up work to do. And there's also some there's some imposter syndrome going on with a lot of associations saying, hey, we can be tech forward organizations. There's a lot of organizational change to overcome. So it's not just about the technology. But I think if associations were to take today's very best AI, let's say, you know, GPT-6 or CloudFable 5.1 or whatever you think is the best and deploy that to the greatest extent possible, you still have extremely low risk. And that would be to deploy every ounce of the power of the current models. You still have very low risk.
[00:34:02:02 - 00:35:55:18]
Amith
And so it's going to take years to get there. So I think associations should be doing themselves a favor and taking advantage of every single bit of it does not mean to act irresponsibly. There are things you can do that are risky. We're going to cover that. But associations should absolutely be going after this because think about this. Take away the species extinction issue or geopolitical problems. You kind of come back down to your sector and say say this to yourself. Do you really think that the member services and products and events and education you offer today are going to be relevant in three years time, possibly even three months time and certainly five years time? It's very likely that the entire catalog of products that you offer your community will need to change in order to serve your members effectively in the worlds they're living in. Forget about your association, but the world that your members are living in will undoubtedly be changed radically and has been changed radically by AI. So if you're not there to serve them in a way that meets their then current needs, you're not going to exist. And that's true for any organization. Doesn't matter if you're a not-for-profit or a for-profit. In order to be able to do that, to have any chance of doing that, you need all the power you can get. You need all the AI tooling you can get to be able to move that quickly, build that kind of content, be an effective resource and advocate for your members in the world that they are living in. And that's the key point. It's not about the pace of change in your association, but it's about the pace of external change that you have to be able to help your members adapt to. So I think that you can absolutely pursue aggressive adoption of AI while being safe, thinking about the level of intelligence we're talking about for the typical association to adopt is not anywhere near the level that people are concerned with. So it is a really important distinction. And it's easy to say, hey, wait, hold on. Everyone's worried about this. We're going to pause our AI roadmap. All you're doing is harming yourself and your members if you do that.
[00:35:57:07 - 00:37:21:03]
Mallory
Everyone, I feel like that was the key point of this episode. So rewind it, listen to it again. I'm glad that we got there. But it's a really good segue into the last part of this episode, which is what can you control? So everything we talked about thus far happens at a scale like Amith was talking about that really no association can influence. You don't get a vote on whether open AI pauses a training run or whether Washington builds a brake pedal. So the question worth spending time on is what is actually in your hands? And the answer to that question is more than most people assume. The thesis here is control and optionality. The point is not that association should be afraid of AI or sit this out. It's that almost every decision that determines your exposure is a decision you get to make. And most organizations are making those decisions by default instead of on purpose. So we're going to go through five of them. Basically, they're all about control, controlling the ground, the model, the data, the autonomy and the scope of the task that you give the AI model. So first and foremost, where your AI workloads actually run, this is controlling the ground. On a neutral platform you control, you set the models, the data governance and the observability, meaning you can see what the system is doing. You go direct to a frontier lab and those are their decisions, not yours. Amith, can you make the case concrete for our listeners? What is an association gain running on its own turf?
[00:37:21:03 - 00:39:01:16]
Amith
I think you already said it well. I mean, if you are in an environment where you don't own the platform you're on, you don't have any options. As an example, if you pick an agent framework that was provided to you by one of the labs, so OpenAI, Anthropic and Google all have really nice agent toolkits. So if you want to build an agent, it's really nice, it's slick, it's easy. You can go build an agent on OpenAI or on Anthropic. Guess what? You're not taking that agent with you if you don't want to use OpenAI or Anthropic. You're also giving those labs access to a whole bunch of data directly via MCP or via API connection, which is really unfettered access, essentially. One of the things I like to talk about there is you imagine yourself loading up all the important information your association has into a semi truck, driving that semi truck from wherever you're located in DC or Chicago or wherever else over to San Francisco and backing up that semi truck into the loading dock of OpenAI and saying, "Hey, Sam, here's the keys. Do what you wish." No one would do that. But yet, that is exactly what you're doing when you are just connecting via MCP, your AMS or your HubSpot or whatever to the major labs directly. You're not controlling the ground that you're walking on. You're handing over the control because you are on their ground. And that's a shift you can make. You don't have to do it that way. That's just the easy button. It's super easy. It's super tempting. It's like saying, "Hey, this is really powerful and I can just click a few buttons and I'm doing it." But there are other ways. And they take a tiny bit more effort, but they give you control of the ground that you walk on.
[00:39:01:16 - 00:39:16:20]
Mallory
Mm-hmm. So that was control the ground and a little bit of control the data. So you kind of teased it a solution to me. What is that solution? If you don't want to go directly to these companies and kind of dump truck all of your data in San Francisco, what can associations do?
[00:39:16:20 - 00:40:31:10]
Amith
Well, you start off with the idea of keeping the data in an environment that you control. And most associations have challenge with this because the data itself is something they don't really control because it's scattered about in an AMS, an LMS, and a SharePoint and all these other places. And they don't have one unified physical location where all their data is under lock and key. So that's a problem that is solvable. And there's a number of technologies that can help you with that. We've mentioned on our pod a number of times the open platform we have called Member Junction, which is totally free to the association community. We knew this problem was going to arise and we built this platform starting five years ago for exactly this purpose to house the business data, both structured and unstructured, that could then basically be used in AI workloads, but an environment the association had 100% ownership of. So that's the data in the ground as well in a sense, because it's the execution environment, too. There are other ways to do it outside of MJ. This is just the open, free solution built for this sector. So we talk about it a lot. But there are other ways to do this as well. There's platforms like Databricks. You can bring data into Microsoft Fabric. There's other places that are environments where you can bring your data in. They all have their pros and cons, obviously.
[00:40:31:10 - 00:40:58:19]
Mallory
I want to talk about the idea of controlling the model. So frontier models are one option, but not necessarily the default. A smaller, faster, cheaper model is often the better fit. And of course, switching should be cheap. Can you help us make the case for the smaller model of me? And I'm also curious, do you think, because we talk about them being smaller, faster and cheaper, do you also think they're safer because they may not have the overall intelligence level of a frontier model?
[00:40:59:19 - 00:42:18:13]
Amith
The safety, in my opinion, comes more from what we call the harness, which is how you hold the model and how you interact with the model than the model itself. You can do really bad things with small models if you just let them run amok. You could take, you know, something like GPT-OSS120B, which was a model released in August 2025. It's not a very intelligent model. You can run it very cheaply, though. And so if you wanted to spin up hundreds or thousands of instances of this much smaller model, that model could probably do some damage if you gave it unfettered access to a variety of tools and gave it a bad objective. So it's not that the smaller model is inherently safer. It's how you use the models. I would argue that the smaller models, though, are absolutely what you need to be using for most things. You know, talk to the past about sending a paperclip from L.A. to San Francisco or L.A. to New York and using a jumbo jet to do that, right? It's the wrong vehicle for that workload. And you don't need the biggest, most powerful model to do a small job. It's inefficient. It's slow. It's expensive. And possibly it could be less safe. But most importantly, it's just much more scalable if you use smaller models. The speed you get out of smaller models is stunning, too. If you're used to your AI applications just being really slow, probably you're using a big and an older model. A lot of people, by the way,
[00:42:20:03 - 00:42:52:21]
Amith
they'll set up some kind of an agent or an app and they'll be running on like, oh, I don't know, GPT-40 or something like that. And they just never bothered to update it. And you ask them, well, how come you haven't done it? Like, well, we just haven't gotten around to it or we haven't tested the newer models, but yet, oh, well, it's been two and a half years since that model was considered current and sometimes those models get deprecated, right? And those models aren't undergoing new safety testing, new safety evaluations. They're just sitting there as they were. So actually being current and leveraging the right models for the job is an important part of your safety posture.
[00:42:53:22 - 00:43:14:12]
Mallory
Another thing you can control is the autonomy of your agents. Recursive self-improvement is the thing driving the fear in a lot of the conversation that we've had thus far. And it's a dial, not a switch that flips when you start using AI. So I mean, in a controlled environment, what are the actual levers on how much a system can modify and improve itself?
[00:43:14:12 - 00:44:15:11]
Amith
Well, there's a lot of them actually. And so these different knobs or levers that exist are on this machine that you're building. You have a lot of control over this. So one of the things is actually just the budget you give to the agent. So how long can the agent run for? If you give the agent days and days of time to run, which is, by the way, what happened with OpenAI's example is they essentially had unlimited runtime. And they're not monitored really, and they're just doing their thing. Yeah, stuff potentially can happen. And there's a lot of other ingredients needed to let things happen. But you can also set budgets that are like, well, you can run 30 times or 50 times or for 30 minutes. So if you have narrow execution windows, that's both a budget constraint, but it's also potentially a safety component for control. Another thing, though, is really the tools or the capabilities that you give to the agent. So if you imagine the world's smartest human and you put that person in a room and you give them no access to any tools whatsoever,
[00:44:16:22 - 00:44:23:08]
Amith
there's very limited things they could do. They'd probably figure something out over time, maybe, if you gave them unlimited time because they're the world's smartest human.
[00:44:24:08 - 00:44:29:02]
Amith
But if you gave them limited time and you didn't give them any tools, there's very limited things they could do.
[00:44:30:05 - 00:44:43:19]
Amith
You give certain tools that are more general purpose in nature. They can be very powerful. Like, for example, you give Google search to an agent. That's actually a very powerful tool. If it can search the entire Internet, it can get all sorts of information.
[00:44:45:05 - 00:45:27:02]
Amith
If you want to give it search, maybe you give it search just to one or two particular domains or you give it narrow search just to internal document repositories that are scoped even more narrowly. Right. So that's that's part of the idea is that you give it kind of the least necessary level of permissions, which is like a very common thing that people teach in cybersecurity is that you grant the permissions that are needed. Not like you don't just say, hey, everyone coming into our HubSpot instance, you're all super admins. That would probably not be great. Right. But a lot of people actually do that. It's like, it's just easier. I don't have to like go and approve all the requests. If you do the same thing with agents, that could be an issue. So you have to control the scope of access they get to the tools that you're giving both the tools themselves and then what they can do with the tools.
[00:45:28:04 - 00:46:10:03]
Mallory
Our last point here is controlling the scope, which you kind of teased earlier, Amith, the least intuitive point and probably the most useful just because AI can do something does not mean it should. Most of what organizations want from agents is achievable with classical deterministic workflows, meaning the same input always produces the same output with AI plugged in at narrow decision points inside a governed framework. Amith, you're not in your head. You're in agreement with this. But it kind of seems counterintuitive to, I guess, what a lot of people hear on the podcast, which is us saying like, you know, we might be able to use AI and member service or AI and marketing or AI and events. So kind of can you balance those two?
[00:46:11:07 - 00:47:47:15]
Amith
Well, I'll give you an example. Let's say we took a workflow that's a common pain point for associations, which is what happens when people submit proposals to speak at a conference or submissions for a journal or a publication? This abstract submission process, as it's often referred to, is quite a pain point because it's very laborious and very manual in nature. And so what is typically involved? There's usually some kind of a workflow where somebody goes online and submits some set of information. It's usually a form of some sort, uploading a document or two, and then there's a process that kicks off. Usually what happens in associations is there are some really basic rules that check whether or not the submission is complete. But generally speaking, it goes to a committee. It could go directly to the committee or a staff person might review it and approve it before it goes to the committee. But there's usually like a series of steps. And all of this is generally speaking human judgment, where the humans are looking at these abstracts and saying ultimately they're deciding whether to accept or reject the abstract. And this is a common type of workflow. And so really what's necessary to finish the workflow is judgment. You can't really decide whether or not a particular paper is a good submission for this without some level of human review. And I still think that's true. If independent of whether or not the judgment is necessary at a technical level, it's probably necessary from a quality control perspective, certainly safety, but also just in terms of taste and style and judgment, right? The things that we bring to the table that are still very much human domain things. So
[00:47:48:18 - 00:48:22:13]
Amith
but that's but the point I'm making, though, is even though that process has an element of human judgment necessary, that doesn't mean the whole process needs to break down into being all human run. Many of the elements of that process can indeed be automated with very little AI. So the little bits and pieces of AI that you could string together would be checking to see if the abstract was actually aligned with the topics of the conference. Right. Someone submits a paper that has nothing to do with the five key themes of your conference. It still takes a human to review that document to say, nope, this is not a fit.
[00:48:23:14 - 00:49:14:14]
Amith
Well, that is something that an AI can easily do. And if all it's doing is that one function and the rest of the workflow is controlled by deterministic code, the way you describe Mallory, you dramatically lowered your risk versus just a freeform agent that's sitting there looking at things, deciding what to do. And like if you envision it being the robot that has autonomy to make full decisions and accept papers into your conference and, you know, do all these other things, that's a that's a different version of what an agent could do. If you could build something like that, I wouldn't recommend it to anyone, though, because mainly because there's no need for it. Right. You have a process that works. There's just certain points that are choke points that go to that key staff person that has to press a button to approve something or a committee member doing the same thing. Those are the areas that I love the target right away because they are low risk and they're very easy to automate with even AI from two years ago, honestly.
[00:49:15:18 - 00:49:22:08]
Mallory
So when you're mapping out a workflow, it's the low risk judgment calls that you plug AI into.
[00:49:22:08 - 00:49:35:16]
Amith
Yes. And you can always have human in the loop or HITL, since we love acronyms in the world of technology or human in the loop. You say HITL, that's what it means, is basically a human getting involved in a workflow and saying yes or no.
[00:49:36:17 - 00:49:52:23]
Amith
And so that's still really important, particularly for the high stakes decision. But for the low risk stuff, to your point, Mallory, there's no reason AI can't step in and do those things. But you're doing you're using AI in that narrow little slice of the workflow. The rest of the workflow works the same way on Monday and Tuesday. And Wednesday and every other day of the week.
[00:49:54:20 - 00:50:06:20]
Mallory
So Amith, as we wrap up this episode for the executive who's listening, who's been reading AI Could Kill Us All and has a board meeting to discuss AI this week, what is your advice to that person?
[00:50:07:23 - 00:51:16:01]
Amith
I think it's important to be informed on this. And hopefully we're doing our job here at the Sidecar Sync to help you a little bit in the journey. There's some great resources out there from journalists who are covering AI at a very broad level. Mallory mentioned the daily that we also have, you know, a lot of other resources out there. We can include some some other stuff in the show notes. Links to Dario's essay itself, which is worth a read. It's kind of long. Dario doesn't really say much in under a couple of thousand words, but he's a brilliant guy. And he I really believe his motivations are what he says they are. They certainly seem to be and he's done a good job articulating the risks. It's a real thing. It's a real concern. At the same time, it doesn't mean we should be deer in the headlights because we've got work to do in our organizations. And actually to participate in that broader societal level concern, we have to be as educated as possible. So if we understand this stuff, then we can be part of that solution. Associations collectively not only need to serve individually, they need to serve their sectors or their professions. But collectively, we need to be part of that voice that's helping shape policy that determines what happens. And policy from governments is an element of the solution here. I don't know exactly how it plays, but it's definitely an element of the solution.
[00:51:17:10 - 00:51:33:10]
Mallory
Well, everyone, a researcher left anthropic saying his colleagues believe AI could kill us all within the decade. The industry asked Washington for a brake pedal in July and Dario Amade called for a real slowdown this month in September. And for you associations, the answer is control,
[00:51:33:10 - 00:51:38:21]
(Music Playing)
[00:51:49:14 - 00:52:06:13]
Mallory
Thanks for tuning into the Sidecar Sync podcast. If you want to dive deeper into anything mentioned in this episode, please check out the links in our show notes. And if you're looking for more in-depth AI education for you, your entire team, or your members, head to sidecar.ai.
[00:52:06:13 - 00:52:09:19]
(Music Playing)
[01:17:37:01 - 01:17:37:01]