Evaluating AI

Effectiveness

Do what you love and let AI do the rest.

Matt Chmiel

Toby Daniels

Co-Founder, ON_Discourse

Editor’s Note: This is a dispatch Toby wrote from his recent trip to Web Summit in Lisbon.

“When you deploy AI in your business, you have two fundamental paths: you can either enhance the things people love doing or eliminate the things they hate doing. This distinction might sound basic, but it’s crucial.”

This is what Nicholas Durkin, the CTO of Harness, an AI-delivery platform said during a roundtable discussion that Dell and NVIDIA hosted during Web Summit in Lisbon.

Take AI code generation, for example. Developers generally love writing code—it’s their craft and their passion. But when AI tools like code generators were introduced, there was pushback. In fact, recent DORA metrics showed that developers using AI code generation tools were less efficient than they were before adopting them. Why? Because these tools inadvertently disrupted the part of their job they enjoy most—writing code.

Durkin went on to say “It’s like telling a chef, we’ll handle the cooking for you,” but leaving them with all the prep and cleaning instead. Chefs thrive on the act of cooking; they don’t want to lose that joy. Conversely, if you use AI to handle the worst parts of the job—like prep work or cleanup—you empower the chef to focus on what they love. This approach doesn’t just maintain their passion; it enhances their ability to excel.

Discover more discourse directly in your inbox.

Sign up for the ON_Discourse Newsletter.

SUBSCRIBE

It’s like telling a chef, we’ll handle the cooking for you, but leaving them with all the prep and cleaning instead.

I spent time with Nick after the roundtable and went much deeper into this topic with him, he went on to outline a model that he uses with his clients. He considers these three metrics when evaluating AI effectiveness:

  1. Efficiency – Does it make processes faster?
  2. Reliability – Is the output consistent and of high quality?
  3. User Experience – Does it make people feel good about their work?

But the most critical factor is alignment with people’s passions. If your AI diminishes the best parts of someone’s job, you’ll face resistance. If it tackles the worst parts, people will embrace it. Focus on “love” and “hate.” Build AI for things people love to do but can’t due to limitations, or for things they can do but don’t want to because it’s boring or repetitive. That’s where AI can make the biggest impact.

We run events every week. If you want to participate, inquire about membership here. If you want to keep up with the perspectives that we hear, you can subscribe to our weekly newsletter.

Twelve Provocations

About 2025

We do provocations, not predictions.

Matt Chmiel

Matt Chmiel

Head of Discourse

Editor’s Note: On the last Friday before Thanksgiving, we assembled members and guests for our second annual End of Year provocations call.

Predictions are boring. None of us knows the future and therefore no one cares what you think is going to happen. Your prophecy is as valuable as mine because you're as stupid about the future as me.

Another problem with predictions is that they are designed to benefit the prophet. The end result is either credit or performative humility; either way you win.

Provocations are different. There is no right or wrong, just boring or stimulating. As a result, they can take many more forms: a meaningful question, a firm statement, or ambiguous feeling.

For the second year in a row, ON_Discourse asked members to prepare a provocation - not a prediction - about the year ahead. In 60 minutes, we tackled the following 12 provocations. We hope at least one of these items stirs up a reaction in you, because this call was full of them.

Discover more discourse directly in your inbox.

Sign up for the ON_Discourse Newsletter.

SUBSCRIBE

1.
Content is dead. Experience is king.

No one reads. Social posts vanish into the algorithmic void. AI has not democratized quality. It has democratized quantity. Too many creators are stuck in the old paradigm of making more, but more isn't better. Better is better, and the real magic I believe, exists in crafting immersive, adaptive experiences that matter.

This provocation did not go unchecked. It was immediately hit with pushback. "Participation in media requires a seismic change in habits, and most people are too passive for that shift."

2.
Why don’t brands do drugs?

There are no mainstream brands that talk about recreational drugs in any kind of a interesting or fun way. There are all these kind of legal psychedelic things coming up, but they're all medical, and they're really, really boring.

At best you see tangential associations with recreational drug use - see: Snoop Dog at the Olympics - but no brands seem willing to commit to this direct messaging, yet.

Someone noted: "Will Coca-Cola return to its roots?"

3.
AI won’t take jobs; it will take tasks.

AI won't eliminate jobs. That's what most people are afraid of. Still. It will just redefine them, and it's already starting to redefine them by taking over mundane tasks.

This is an optimistic expression that was not shared by some. This reaction says it all: "AI has already started taking jobs. I am seeing it. The first hits are in marketing functions, especially here in the Bay Area, where these big tech companies need to hire more AI tech talent, which is extremely expensive. They're doing that by cutting back on their marketing teams, quite simply, laying off a whole bunch of marketers and having the rest of the folks that are left in those marketing departments use their own AI tools."

4.
Streaming platforms are the new Facebook.

Streaming platforms are doing to Hollywood what Facebook did to publishers. What's happening to the studios is what happened when Facebook did with their famous pivot to video, forcing all these publishers to change their models and bend to the whims of Facebook's massive algorithm. I think the difference here is that there's more than one streaming platform.

Nobody pushed back on this one, so I won't invent an argument. I'll just build on the provocation by emphasizing the cascading effect of this system. The commodification of content is diminishing the operational model in Hollywood.

5.
I want the AI bubble to burst.

Right now, AI equals LLMs and Gen AI, but we will not revolutionize the world by generating words and fake videos. AI is much more than that.

As always, I can't tell you who said this, but I can tell you that this person is a founder in AI technology and the emphasis of looking beyond the LLM version of AI got a lot of people excited by this provocation.

6.
Sam Altman is the new Elizabeth Holmes.

The house that Sam Altman built is actually going to create a lot of mini Elizabeth Holmes; we're entering this unregulated environment where there's just going to be more fraud because general AI literacy is still low.

People in the room wanted to edit this take; the underlying theme landed in the room, but the reference was wrong. "I think FTX and SBF in particular are a better parallel because there is a technology at the foundation has real utility but no regulation yet."

7.
Google will win the search wars.

It's very easy for them to flip the switch from being a search dominant metaphor to a chat based one with ads, in the way that perplexity is trying to get towards. I think Google is going to keep winning, and it's very hard for anyone to compete with them.

This provocation had some pushback, but the most effective reaction came from another participant who agreed. This particular member run an AI service provider and has decades of experience in ML, saying: "Google has the best training data by a billion miles with every single click that everybody's done for every Google search and how long they spent on a page and where they went and what they saw when they were there. That's why their current version of search, with AI summaries, has less hallucinations. It's very rare to get hallucination in their AI search summaries. I think they will continue to crush search just because the data."

8.
AI will break the Gartner Hype Cycles.

I am sick and tired of hearing about people talking about Gartner's hype cycle, and I think that, like hype cycles aren't real. Things do not happen linearly, and I think we need a better framework to talk about what's happening right now.

Most people in the group saw some value in the hype cycle. Others gave Gartner credit with turning a banal observation about technical adoption into an evergreen marketing funnel.

9.
The Trump bump will not return for news companies.

The spectacle that inflated the value of some of the news platforms in the first term is being met with a total exhaustion of interest in news. And so now they have to come up with a way to sort of engage audiences on things outside of that.

News is entertainment and people are switching the channel. What are newsrooms to do in this environment? The bro podcast network came up; so did Jon Stewart and The Daily Show. Nobody knows what happens next.

Are the tired of the content or the outrage? The notion of fatigue kept coming up. Someone summed it up like this: "Outrage fatigue is temporary, but trust, once broken, is permanent."

10.
AI will kill the resume.

Every resume looks perfect, looks the same, every candidate feels the same kind of thing.

Everyone is having a hard time finding talent and reporting back from friends who are having a hard time finding work. The outdated notion of CVs feels like they are starting to break conventions.

11.
LLMs are the NFTs of AI.

I think that the LLM bubble bursting. I think people are realizing that LLMs can take you some of the way there or but then they hit a wall. But the AI bubble is not bursting. People are applying different AI techniques to make AI work properly, so you can use it at scale and trust it.


We have heard this notion repeated in several events. AI is bigger than LLMs and maybe 2025 will be the year we move beyond that aspect of it.

12.
We're all cowards.

I think we're afraid to disagree. I count myself in this by the way. We're afraid to call things out and put ourselves out there.

As one can imagine; people were not ready to embrace this one. A few reactions to it: "I thought that provocation sucked."

We run events every week. If you want to participate, inquire about membership here. If you want to keep up with the perspectives that we hear, you can subscribe to our weekly newsletter.

Group Chat Recap
11 • 08 • 24

Your eCommerce site isn't ready for an agent

eCommerce needs to focus on the plumbing first

Matt Chmiel

Matt Chmiel

Head of Discourse

Editor’s Note: We invited an AI expert to talk about the future of eCommerce in the agentic era.

Are AI agents coming to eCommerce platforms? Not if you believe the takeaways from this Group Chat. It features a bonafide AI expert with academic credentials and multiple successful ventures (including a new one that we cannot mention now, that is serving eCommerce sites).

This session paired AI optimists with eCommerce realists. It was a good one. Here are some takeaways.

Discover more discourse directly in your inbox.

Sign up for the ON_Discourse Newsletter.

SUBSCRIBE

Personalization is a mirage

I think the promise of personalization that we have today on our platforms is not there. There's a lot of personalization that's happening on these sites, but it's not real, and it's certainly not a value to most of us. It's why there's so many rails that are run across a PDP or a PLP because they're like shit, maybe they’ll click on this one for no reason.

The future is better search results, not agents

The conversation was practical; before we let agentics magically interpret our mood and behavior with perfect recommendations, we should maybe focus on search results that properly understand customer intent.

One on hand this feels obvious, on the other hand, as one of our guests admitted, this is not always the case.

So getting customers to feel after getting better search results: ‘Oh yeah, that heard me.’ Like, that's amazing. We have a lot of those insights during gifting moments, whether it's holiday, Mother's Day, Father's Day, Valentine's Day, finding. But all we do right now in gift seasons, the lowest common denominator…

eCommerce tech stack is outdated

The most pervasive issue, and blocking a lot of eCommerce companies from being able to take advantage of this technology is their tech is stuck in 2010. So far what they've done now is build wrappers around this so they've put lipstick on it to make it kind of look pretty but it doesn't function.

We run a Group Chat every week. If you want to participate, inquire about membership here. If you want to keep up with the perspectives that we hear, you can subscribe to our weekly newsletter.

Group Chat Recap
11 • 15 • 24

AI and the Law

Finding the line between government and governance

Matt Chmiel

Matt Chmiel

Head of Discourse

Editor’s Note: We invited a legal expert on AI and copyright law to talk about the ethical and legal landscape for an AI-powered internet. We can’t tell you who was there but we can tell you what was said.

AI is the ultimate blackbox. Does that mean entrepreneurs are shielded from liability? We hosted a Group Chat that dug into this question. Here are 3 takeaways:

Discover more discourse directly in your inbox.

Sign up for the ON_Discourse Newsletter.

SUBSCRIBE

Section 230 and AI

Liability remains murky when it comes to AI-generated content. The legal shield of Section 230, which protects platforms from user-generated content liability, extends ambiguously into the AI realm.

Questions of liability scatter across the tech stack. The platform that hosts the agent; the brand that sponsors it; and the software developer that launched it all play a role in this equation.

If you are creating an agent that goes out and makes false claims about a product, the developer could claim immunity under Section 230… but this is not a certainty.

Good Governance

You have to understand what your AI does and then you have to rigorously test it against every bad-case scenario. And then you have to keep records of what you're doing.

Red teaming—stress testing AI systems to expose vulnerabilities—is emerging as a critical practice for ethical and functional deployment.

One of the interesting takeaways we heard was a distinction between communication and reporting tools.

You have to make the decision on whether you are a communication tool or a reporting tool. Then we had to make a decision about how our AI responds to extreme scenarios like suicide threats or violence.

Colorado and the States

The federal government is probably not going to set any standards for AI. The real action is happening in the states.

Emerging regulations like the Colorado AI Act reflect a trend toward greater oversight of AI's societal impacts.

If you don't know about it now, you should read up on the Colorado AI Act.

If you don't know what data is training your model by February of 26 and you know you're going to be deploying your tool in Colorado or to consumers or businesses in Colorado, you're going to be out of compliance. So you know you really need to be thinking about that.

We run a Group Chat every week. If you want to participate, inquire about membership here. If you want to keep up with the perspectives that we hear, you can subscribe to our weekly newsletter.

Breakfast Recap
10 • 10 • 24

Inevitable AI

The unavoidable future of generative content?

Editor’s Note: 18 Industry leaders in NYC joined us for breakfast to talk about the future of content. The word inevitability was uttered many times, generating a lot of pushback.

We hosted our breakfast in the back room at Gemma Italian Trattoria at the Bowery Hotel in New York City.

The discourse centered on this prediction about AI:

The future of content will be generative, ephemeral, and prompted by your needs, wants, and desires.

This future will come in 3 distinct phases
(we are currently in phase 1)

Phase 1

AI tools for professionals to produce more content more efficiently.

Phase 2

AI tools for non-professionals to produce professional-looking content (similar to blogs and print)

Phase 3

Brands generating consumer content based on targeted user data.

This theory does not predict a total takeover of the content supply chain; cinema will still exist; live sports will still exist; but the majority of content consumption will be generative.

Why it matters

As AI moves from assisting creators to generating personalized content in real time, we’re on the brink of a media transformation that could either revolutionize how we consume or isolate us further into echo chambers.

Matt Chmiel

Matt Chmiel

Head of Discourse

Discover more discourse directly in your inbox.

Sign up for the ON_Discourse Newsletter.

SUBSCRIBE

The table had several key reactions:

The Dystopian Risk

Some attendees warned this will create tribal bubbles, reinforcing personal biases and isolating communities. One participant pointed to political radicalization, noting AI’s role in amplifying divisive content.

"We’ll end up in a civil war before we reach this inevitability."

The Loss of Choice

Concerns were raised about algorithms controlling our consumption:

“We’re just TikTok zombies at this point.”

The fear is that AI will strip away free will, reducing us to passive consumers.

Hope for Personal Content

Others embraced the potential for wildly creative, personalized experiences, like one attendee’s dream of blending Darth Vader with Rocky in a custom AI-generated adventure.

Interactive, participatory content could reshape entertainment into a collaborative experience—where viewers not only watch but star in their own creations.

Too Much?

What happens to shared experiences? If everyone gets a different ending to the same movie, are we losing cultural touchstones? Some worried this could fragment society further, erasing the moments that bring us together.

We run closed-door events every week. If you want to participate, inquire about membership here. If you want to keep up with the perspectives that we hear, you can subscribe to our weekly newsletter.

Group Chat Recap
10 • 03 • 24

Escaping Data Jail

AI is very quickly changing the SaaS business model

Matt Chmiel

Matt Chmiel

Head of Discourse

Editor’s Note: How much are you paying your SaaS vendor? Do you think AI - whatever that means to you - can change that cost burden and redefine your relationship? This recap is for you.

We invited 2 SaaS founders into a Zoom call with 3 other executives. I can’t tell you who was there, but you can preview some of the discourse.

The takeaways on this list might drive the next conversation you have with your SaaS vendor.

Discover more discourse directly in your inbox.

Sign up for the ON_Discourse Newsletter.

SUBSCRIBE

AI Won’t Replace SaaS

SaaS is not going anywhere, but the way it has traditionally operated is about to change dramatically. One of the founders put it this way, “AI stitches together smaller, bespoke solutions,” allowing businesses to create customized workflows that fit their needs. The days of relying on one-size-fits-all platforms are over. Those on the cutting edge are using AI to personalize and streamline their SaaS stacks. If you’re not exploring how AI can take your tools to the next level, you’re already falling behind.

SaaS companies are under more financial strain than ever before.

The Golden Age of High Margins Is Fading

The pressure on SaaS margins is mounting. As interest rates rise and customer demands shift, the once-lucrative model of massive upfront spending with guaranteed long-term revenue is fading. One CEO cautioned, “SaaS companies are under more financial strain than ever before.” The key takeaway? Profitability now hinges on constant innovation and real-time value creation. SaaS clients are already eyeing exists from long-term contracts. Why is that?

AI Can Break Out of Data Jail

AI is demolishing the barriers that once made switching SaaS platforms difficult. “Data jail” is becoming a relic, with AI-driven solutions making it easier to move from one system to another. This shift has empowered businesses to pivot quickly and ditch legacy systems without the massive headaches of the past. Is your SaaS provider evolving fast enough? If not, expect to see churn rates rise as switching costs plummet.

You Still Need SaaS

The buzz around AI-generated rapid prototyping is real, but don’t be fooled. While AI makes it easier than ever to build flashy demos, turning these prototypes into robust, scalable solutions is still a major challenge. As one respected executive observed, “Prototypes are about 2% of the work.” The real value lies in operational excellence, something that can’t be replaced by a few lines of code. The smartest leaders are focusing on building sustainable, scalable systems with nimble SaaS solutions.

Do You Want to be Klarna Right Now?

They generated cutting out all SaaS contracts, but our SaaS founders were not convinced this is the right long term move.

“Klarna said we just tore out their CRM, etc. I'm really looking forward to the stories from that. I don't know Klarna at all - and no offense to anybody there - but man, that sounds like a really aggressive move. And I have a feeling that we're going to find some people leaving saying some interesting things about operations.”

Usage-Based Pricing is Coming

The era of flat monthly fees is fading, and usage-based pricing is quickly becoming the new norm, particularly in AI-driven SaaS. One of our guests put it this way, “This is going to create friction for finance teams” SaaS leaders need to prepare for a future where billing is based on usage, which means rethinking everything from budget forecasts to financial planning. If you're not ready, this shift could catch you off guard.

SaaS Agents?

The future of SaaS isn’t just about automation—it’s about intelligent, autonomous agents that can seamlessly integrate into your workflows. "SaaS is moving towards agentic architectures” but for how long?. This trend is reshaping how businesses think about software. These agents, often paired with conversational UIs, can handle complex tasks, but there’s a catch: without structured data, their effectiveness plummets. The next wave of SaaS innovation will hinge on balancing flexibility with the structured environments needed to make these agents work reliably. Those who master this dynamic will lead the pack; those who don’t will struggle to keep up.

We run a Group Chat every week. If you want to participate, inquire about membership here. If you want to keep up with the perspectives that we hear, you can subscribe to our weekly newsletter.

Group Chat Recap
09 • 27 • 24

'A Bleeping Nightmare'

Top-down transformation nightmares and 2 other takeaways about integrating AI into enterprises from members and guests

Editor’s Note: It was the last Friday in September and so we assembled all our members, experts and guests for our end of the month virtual event. These events are shaped by the group dynamic; we start with a plan, but the group dictates the discourse.

0:00 / 0:00

We came into this session with a plan: provoke Peter Pawlick, head of experience design at Proto. Pawlick published a 5 part series on our platform about AI simulations and his perspective deserved attention. If you haven’t read it yet, here it is in a nutshell: design thinking is dying and many brands and agencies don’t know it yet.

Design thinking is the engine that drives digital transformation. It is an endless iterative and agile process that is designed to ‘move fast and break shit.’ It is more than a method, it is a culture. And that culture is about to be replaced.

Synthetic data and other generative AI systems make it possible to preview ideas before they are designed or built. This means that an enterprise that wants to chase a north star vision can stress test 10,000 pathways to get there before deploying any capital on design or development.

In other words, move fast and break synthetic shit so the first public launch is a guaranteed success (or so the theory goes).

As I was saying, we came into the session with, dare I say, a good plan. But like all good plans, it fizzled on first contact; the group took over the discourse. I love when that happens.

The group was full of agency leaders, enterprise representatives, and founders in AI startups. In other words, all sides of the digital transformation spectrum were covered.

Here are 2 other takeaways from the group - anonymized in the way we always do:

Matt Chmiel

Matt Chmiel

Head of Discourse

Discover more discourse directly in your inbox.

Sign up for the ON_Discourse Newsletter.

SUBSCRIBE

1. The AI Firewall

We heard perspective from members who have been struggling to sell AI builds and services into corporate clients.

The only groups willing to consider these services are the marketing and customer service departments. The use-case is pretty clear: the content and services can live freely outside of the company firewall; mistakes in this instance are a minor issue rather than a compliance nightmare.

The rest of the organization is dragging its feet, for good reason. As we’ve heard in other events, AI is great at rapid prototyping but not running a product or ensuring compliance. As one insider put it, AI might work for "marketing procurement, but AI within the rest of the organization is going to be pretty slow.”

Top-down AI transformation is a fucking nightmare.

2. Realistic AI

Now let’s drill deeper: what is the conversation like for the few corporations that are willing to integrate AI deeper into the org-chart? We heard a few perspectives that might influence the way you collaborate with partners.

“AI is not a magic button” and “LLMs are not always the answer to the problem.” The real problem is that expectations for how this technology can help is not fully understood. This creates a communication issue that blows up if not addressed the right way.

One of our members put it bluntly: “Top-down AI transformation is a fucking nightmare.” In other words, employees reject it and compliance pushes back. One blocker comes in after another.

Real change is coming from the bottom-up, where employees (or agencies) are hacking together solutions, often without waiting for permission. One group is speeding up brand assets and media buying. Another group uncovered a way to track and deal with disinformation in social media using AI tools. This is opening up new service-offerings that were never before considered.

This perspective is important because it represents an agency offering for recalcitrant enterprises: do not look for the silver bullet; instead, build tools that solve specific problems. Keep adding new tools, solving new problems, eventually developing a suite of services.

We run a Group Chat every week. If you want to participate, inquire about membership here. If you want to keep up with the perspectives that we hear, you can subscribe to our weekly newsletter.

Group Chat Recap
09•19•24

'Holy Bleeping Bleep'

What it feels like to build a functioning CRM in 3 hours, and 4 other takeaways about AI and team management.

Editor’s Note: This Group Chat was a prolific blend of personalities and perspectives spanning a variety of unrelated industries. We provoked this group with practical questions about AI and management. Their respective answers fed into a good discourse. Here’s a public-facing recap)

0:00 / 0:00

People are always asking what happens in our events. We can’t tell you. That’s part of our deal; we have closed-door sessions with real discourse. Our members get to share raw thoughts, compare notes, and develop new connections. And you get this sanitized recap.

This group had 4 members:

  • 2 Founders (1 AI startup and 1 large public company & venture development firm)
  • 1 Chief Digital Officer in prestigious cultural commerce
  • 1 Chief Operating Officer of an agency

They generated 5 takeaways.

Matt Chmiel

Matt Chmiel

Head of Discourse

Discover more discourse directly in your inbox.

Sign up for the ON_Discourse Newsletter.

SUBSCRIBE

1. Middle Management is Ready for an AI Take Over

In private they don’t mince words: “We want to replace the manager, just like Uber replaced the dispatcher. AI can do a better job at managing people than humans. Managers are the most dysfunctional part of an organization, and people leave jobs because of bad managers, not bad companies.” The role of the manager as we know it is obsolete.

Another participant, tired of slow corporate change, took drastic measures within their company, reshuffling the hierarchy to move faster: “We took our chief creative officer, made them Chief AI Officer, and launched initiatives to trim the fat.” The focus here was cutting out inefficiency and embracing AI as a force for operational speed and innovation.

Managers are the most dysfunctional part of an organization, and people leave jobs because of bad managers, not bad companies.

2. AI Prototyping: Faster Than Bureaucracy Can Keep Up

With greater efficiency comes an unexpected issue. As one exec put it: “We spun up an AI-driven CRM prototype in just three hours, and the reaction was ‘holy fucking shit.’” But the challenge surfaced when this quick success bumped into a familiar corporate problem: maintenance. As this leader explained, “Now, I don’t know what to do. If everyone starts using it, it becomes a product, and then I need to support it.”

The prototype worked so well but it was just a proof of concept. Implementing this thing is where it gets sticky again.

This dilemma—building too fast to scale—captures the double-edged sword of today’s AI landscape. It’s faster to build something tangible than to even discuss building it, but as they pointed out, “The challenge comes when you have to support and scale what was meant to be a prototype.”

3. Mediocrity Won’t Survive

Another issue: people. They put it bluntly: mediocrity will not survive in this new AI-driven world. “Being mediocre is just going to be very difficult.” The pace of AI innovation is outstripping human adaptability. The faster the models retrain, the faster the gaps between skill levels grow. In this environment, anyone slow to adapt will be left behind.

The Chief AI Officer that was mentioned earlier is in charge of hiring new resources. All of them have to demonstrate real opinions, experience, and output from various AI platforms if they want to get hired.

4. Donkeycorn Ventures

Forget unicorns. While some chase billion-dollar valuations, a new concept emerged during the discussion—the “donkey-corn.” This represents small, high-performing companies with $2 million in revenue, built by tiny teams grinding hard in niche markets. The future belongs to companies focused on efficiency and scalability without bloating. Someone referred to the Sam Altman question: “Who’s going to be the first billion-dollar brand owned by one person?” That question does not imply a single business. In this executive's mind, it is a network of donkeycorns.

5. AI-First Companies

One consistent theme throughout the chat was the consensus that any new company starting today is an AI-first company. “You have two types of people: those who embrace change and those who fight it. Only one of them wins.” Whether it’s AI managing teams or personalizing brand experiences, the clear takeaway was this: the future is not waiting for anyone. The business leaders in this chat agreed—AI isn’t just a tool; it’s a revolution in how work will be done.

We run a Group Chat every week. If you want to participate, inquire about membership here. If you want to keep up with the perspectives that we hear, you can subscribe to our weekly newsletter.

SAFE TO FAIL

The Future is a Simulation

Part 5

How

to Get

Started

Peter Pawlick

Principal, Experience, Head of Experience Design, Proto

Peter Pawlick

Principal, Experience, Head of Experience Design, Proto

Editor’s note: At this point, you have spanned the history, and future of simulation and been given the tools to use for this process. The final installment of this series is a metaphorical baton pass to get you started on your simulation.

The future of simulation is ready to go mainstream, expanding far beyond capital-intensive industries, and into the digital realm. If you are leading a digital transformation agency or running an in-house transformation, the next step is to figure out how to leverage this powerful technology in the right way.

The previous article laid out a so-called 4V Framework you can use to shape your simulation. Now you need a determine the value of this process for your own needs. Consider the MASTERY framework as a diagnostic tool to identify how simulation might be relevant to your own company or team:

SAFE TO FAIL

The Future is a Simulation

PART 1

An Ancient Strategy

PART 2

The AI Spring

PART 3

Virtual Prototyping

PART 4

Focus Areas and Tools

PART 5

How to Get Started

Mine

Uncover actionable insights from complex data and market analysis, identifying emerging trends and opportunities. What if you could predict market shifts or new consumer behaviors before they become apparent to your competitors? How might this foresight influence your strategic investments and guide your innovation roadmap?

Architect

Build and test virtual models of potential organizational changes or new business ventures. What if you could visualize the impact of a proposed change across your entire organization before implementation? How would this ability to test different scenarios inform your decisions and mitigate risks associated with transformative changes?

Strategize

Leverage strategic simulations to evaluate and refine various business strategies to understand their potential impacts thoroughly. What if you could simulate the outcomes of different strategic choices and their effects on your business? How could this capability enhance the precision of your strategic planning and accelerate effective decision-making at the executive level?

Train

Enhance your team's capabilities through targeted simulation-based training in a controlled, risk-free environment. What if your leadership and most critical teams could practice navigating complex business scenarios without any real-world risks? How might this training prepare them better for strategic pivots and innovation challenges?

Engage

Align organizational efforts and test the impact of new policies or operational changes before they are implemented. What if you could preview the results of a new organizational policy or a shift in workflow? How would this visibility help in aligning departments and optimizing overall decision-making processes?

Refine

Continually optimize your operations by simulating various process configurations to identify the most efficient models. What if you could streamline operations and improve productivity continuously? How would this commitment to ongoing refinement support sustained growth and operational excellence?

Yield

Maximize the effectiveness of your strategic decisions and operational processes by testing and enhancing outcomes before full-scale implementation. What if you could fine-tune your strategies and operations to ensure they deliver maximum value and impact and focus resources on the strongest areas of return? How would optimizing these yields transform your organization's performance and competitive positioning?

As simulation becomes more accessible, it addresses the critical barrier of risk, enabling businesses to model and test scenarios virtually. This reduces the need for costly physical prototypes and mitigates risks associated with new innovation initiatives. Consequently, organizations can move more confidently and quickly from concept to execution, exploring innovative solutions, iterating rapidly, and bringing advancements to market faster.

The democratization of simulation technology, propelled by AI, promises to reverse the "innovator's dilemma," enabling large corporations to take on a level of risk formerly reserved for startups. These organizations can leverage extensive data for precise simulations, accelerating innovation processes. However, this raises questions about the broader impact. While simulation capabilities might give established companies a competitive edge, relying on vast data suggests those with access to such resources could disproportionately benefit, leading to control consolidation and dominance by larger entities.

Nevertheless, simulation holds potential for broader inclusion by allowing organizations to predict outcomes and evaluate ideas more objectively. This could lead to a more inclusive approach to innovation, where decisions are based on outcomes rather than assumptions.

Accelerating innovation without allowing time for adaptation could cause social and economic disruptions. Balancing rapid technological advancement with strategies to help societies adjust is crucial. As simulation becomes more widespread, fostering thoughtful integration over speed is vital to ensuring these advancements benefit a broader population and bring people along (both employees and customers).

The democratization of simulation offers unprecedented opportunities for precise, cost-effective decision-making. Yet, it also creates a divide between those who can harness its power and those who cannot. Organizations that adapt to this new era are poised for breakthroughs, while others may struggle and face increased risks. Preparing for this shift is crucial; businesses must invest quickly in tools, training, and strategies to leverage simulation effectively and navigate the evolving landscape.

Discover more discourse directly in your inbox.

Sign up for the ON_Discourse Newsletter.

SUBSCRIBE

PREVIOUS

PART 4

Focus Areas and Tools

START OVER

PART 1

An Ancient Strategy

Discover more discourse directly in your inbox.

Sign up for the ON_Discourse Newsletter.

SUBSCRIBE

SAFE TO FAIL

The Future is a Simulation

Part 4

Focus

Areas

and Tools

Peter Pawlick

Principal, Experience, Head of Experience Design, Proto

Peter Pawlick

Principal, Experience, Head of Experience Design, Proto

Editor’s note: You've spent the past three installments understanding where simulation technology comes from and why it matters. The fourth update to the series breaks down the available tools and steps that are available today that bring simulation to your innovation project.

In corporate innovation, a range of conditions and variables — including organizational structure, company culture, market dynamics, customer needs, regulatory environments, and technological trends — play crucial roles. Each variable can significantly impact the success and direction of innovation efforts.

At Proto, our Experience Design group uses simulation to increase our clients’ velocity and success rate in corporate venturing. Our approach is organized around four focus areas:

SAFE TO FAIL

The Future is a Simulation

PART 1

An Ancient Strategy

PART 2

The AI Spring

PART 3

Virtual Prototyping

PART 4

Focus Areas and Tools

PART 5

How to Get Started

1. Visualization

We help clients find the best way to represent hypotheses, context, and scenarios to facilitate understanding and decision-making.

2. Virtualization

We help clients model systems and conditions to test and explore future possibilities in a controlled, safe way.

3. Validation

We help clients lower their organizational barriers to experimentation by making it faster, easier, and less risky to learn by doing.

4. Variation

We help clients test assumptions across a range of conditions to explore how different variables influence outcomes.

In today's complex business environment, senior executives and C-level officers should promote the use of simulation technologies.

These tools help navigate strategic challenges effectively and ensure that all organizational levels understand and pursue common goals.

Discover more discourse directly in your inbox.

Sign up for the ON_Discourse Newsletter.

SUBSCRIBE

1. Strategic Clarity and Alignment

Visualization simplifies complex data, ensuring everyone in the organization understands the strategic goals and their roles in achieving them. This is particularly important for addressing complicated business issues or entering new markets.

Claude Sonnet’s ability to produce simulated 3D physics using WebGL allows designers to create and test detailed digital prototypes early in development. This reduces the need for costly physical prototypes and extensive user testing, facilitating rapid iteration and real-time feedback. Similarly, automation powered by GPT-4 generates Figma designs based on product requirement documents (PRDs). In both cases, AI-enabled visualization bridges the imagination gap and accelerates the transition from concept to tangible reality, making it faster and easier to test and iterate on ideas quickly and cost-effectively. 

Future advancements will integrate real-time adjustments based on predictive analytics and user feedback, further shortening development cycles and allowing for comprehensive virtual prototyping environments. This transforms the design process by bridging the gap between imagination and empirical validation.

2. Risk Mitigation and Cost Efficiency

Virtualization allows for the testing of strategies in a simulated environment, reducing the financial risks of new initiatives. This is crucial in industries where errors are costly, as it allows potential problems to be identified and addressed before they occur.

The Decision Lab leverages behavioral science and AI to de-risk innovation by predicting consumer behavior and refining products to better meet market demands. This helps businesses develop new revenue models aligned with market needs, reducing risks associated with new launches. In 2023, IBM introduced Watsonx, a suite of AI tools designed to modernize business processes and improve data-driven decision-making, further reducing risks in innovation planning by analyzing data and simulating business scenarios. This virtual approach to risk management allows strategies to be tested in a controlled environment, identifying potential problems early and significantly reducing financial risks. 

As AI evolves, tools will incorporate advanced machine learning techniques like reinforcement learning and federated learning, leveraging real-time data streams for deeper insights and accurate predictions while maintaining data privacy. This enhances risk mitigation by allowing businesses to address issues proactively and make more informed decisions.

3. Data-Driven Decisions

Validation through simulation provides data that supports decision-making processes. This method ensures that new ideas are not only theoretically sound but also empirically viable, allowing decisions to be based on solid evidence rather than assumptions and anecdotes.

Pecan AI uses predictive analytics to analyze customer data, helping businesses forecast demand and tailor their offerings to specific consumer needs. Similarly, Neurons Inc. utilizes neuromarketing tools to simulate and predict consumer responses to various stimuli, enabling companies to refine products and marketing strategies. These tools simulate responses of virtual consumers to various stimuli, providing deep insights into consumer behavior without the need for traditional market research.

Findings from recent studies show that large language models (LLMs) like GPT-4 can predict the outcomes of social science experiments with high accuracy, suggesting their potential for product testing and market analysis. Future advancements will further enhance these capabilities, allowing for hyper-personalization and continuous product adjustments based on real-time data. This approach transforms the innovation process by validating concepts with data from synthetic audiences, reducing product failure risks, and ensuring alignment with market demands.

4. Agility and Market Responsiveness

Variation in simulation tests the impact of changes in one part of the system on the entire business. This capability is essential for quickly adapting to market changes and maintaining competitiveness, as it allows for swift and efficient responses to new challenges and opportunities.

AI platforms like Arena AI enable continuous monitoring and real-time optimization of processes, allowing companies to swiftly adapt to market changes. This agility is achieved through real-time adaptive learning mechanisms that predict market trends and enable rapid scenario testing for dynamic strategy adjustments. Future AI systems will further enhance this capability by integrating more sophisticated real-time learning and prediction algorithms, allowing businesses to dynamically adjust their strategies and maintain a competitive edge. This enhances market responsiveness, ensuring that companies can quickly respond to new challenges and opportunities, maintaining competitiveness in a rapidly changing market landscape.

By anticipating these shifts and advocating for these simulation principles, leaders ensure their teams are equipped to make informed decisions and adapt to changes, positioning the organization for ongoing success and innovation.

PREVIOUS

PART 3

Virtual Prototyping

NEXT

PART 5

How to Get Started

Discover more discourse directly in your inbox.

Sign up for the ON_Discourse Newsletter.

SUBSCRIBE